Failsafe / loss-of-control / disarm not honored on SPRacing H7RF

[blockfeed]

Describe the bug

After about an hour of flight time, the input suddenly locks up and stick movements are not registered properly. Disarm also appears to not be honored when activated (appears related).

To Reproduce

Flash 4.3.0-20220118-1357 (SPRacing H7RF, which appears to be rebased to master/9c3562a).
Fly as usual
Eventually controls feel as if "locked up", but are being registered sporadically/randomly

Expected behavior

Fly normally

Flight controller configuration

board_name SPRACINGH7RF

# resources
resource RX_SPI_CS 1 NONE
resource RX_SPI_EXTI 1 NONE
resource RX_SPI_EXPRESSLRS_RESET 1 NONE
resource RX_SPI_EXPRESSLRS_BUSY 1 NONE

# feature
feature -LED_STRIP

# serial
serial 3 1 115200 57600 0 115200
serial 4 1024 115200 57600 0 115200
serial 7 64 115200 57600 0 115200

# aux
aux 0 0 0 1700 2100 0 0
aux 1 13 3 1700 2100 0 0
aux 2 26 4 900 1700 0 0

# rxrange
rxrange 0 988 2011
rxrange 1 988 2011
rxrange 2 988 2011
rxrange 3 988 2011

# master
set acc_calibration = -11,-25,-1,1
set mag_hardware = NONE
set rssi_channel = 15
set serialrx_provider = CRSF
set dshot_bidir = ON
set use_unsynced_pwm = OFF
set motor_pwm_protocol = DSHOT600
set bat_capacity = 850
set vbat_warning_cell_voltage = 370
set vbat_scale = 111
set ibatv_scale = 920
set small_angle = 180
set osd_units = IMPERIAL
set osd_warn_rssi = ON
set osd_warn_rssi_dbm = ON
set osd_rssi_alarm = 96
set osd_rssi_dbm_alarm = -100
set osd_cap_alarm = 71
set osd_alt_alarm = 0
set osd_vbat_pos = 480
set osd_rssi_pos = 2400
set osd_link_quality_pos = 352
set osd_rssi_dbm_pos = 320
set osd_tim_1_pos = 352
set osd_flymode_pos = 2496
set osd_current_pos = 2464
set osd_mah_drawn_pos = 2336
set osd_craft_name_pos = 73
set osd_gps_speed_pos = 288
set osd_gps_lon_pos = 192
set osd_gps_lat_pos = 160
set osd_gps_sats_pos = 224
set osd_home_dir_pos = 192
set osd_altitude_pos = 18816
set osd_warnings_pos = 14729
set osd_avg_cell_voltage_pos = 2368
set osd_esc_tmp_pos = 288
set osd_core_temp_pos = 408
set osd_stat_max_curr = OFF
set debug_mode = RC_SMOOTHING_RATE

profile 0

profile 1

profile 2

profile 0

rateprofile 0

rateprofile 1

rateprofile 2

rateprofile 3

rateprofile 4

rateprofile 5

rateprofile 0

# resource show all
Currently active IO resource assignments:
(reboot to update)
--------------------
A00: FREE
A01: FREE
A02: FREE
A03: FREE
A04: FREE
A05: FREE
A06: MOTOR 3
A07: MOTOR 4
A08: FREE
A09: FREE
A10: FREE
A11: USB
A12: USB
A13: SWD
A14: SWD
A15: GYRO_CS 1
B00: MOTOR 1
B01: MOTOR 2
B03: SPI_SCK 6
B04: SPI_MISO 6
B05: SPI_MOSI 6
B07: FREE
B08: I2C_SCL 1
B09: I2C_SDA 1
B10: I2C_SCL 2
B11: I2C_SDA 2
B12: PREINIT 1
B13: FREE
B14: SPI_MISO 2
B15: SPI_MOSI 2
C00: FREE
C01: ADC_CURR
C02: ADC_EXT
C03: ADC_BATT
C04: FREE
C05: FREE
C06: FREE
C07: FREE
C08: SDIO_D0
C09: SDIO_D1
C10: SDIO_D2
C11: SDIO_D3
C12: SDIO_CK
C13: SDCARD_DETECT
C14: SYSTEM
C15: PINIO 1
D00: SERIAL_RX 4
D01: SERIAL_TX 4
D02: SDIO_CMD
D03: SPI_SCK 2
D04: FREE
D05: SERIAL_TX 2
D06: SERIAL_RX 2
D07: FREE
D08: SYSTEM
D09: SYSTEM
D10: FREE
D14: FREE
D15: GYRO_EXTI
E00: SERIAL_RX 8
E01: SERIAL_TX 8
E03: FREE
E04: BEEPER
E05: OSD
E06: OSD
E11: OSD
E12: OSD
E13: OSD
E14: OSD
E15: OSD
F00: FREE
F01: FREE
F02: FREE
F03: FREE
F04: FREE
F05: FREE
F06: FREE
F07: FREE
F08: FREE
F09: FREE
F10: FREE
F11: FREE
F12: FREE
F13: FREE
F14: FREE
F15: FREE
G00: FREE
G01: FREE
G02: FREE
G03: FREE
G04: FREE
G05: FREE
G06: FREE
G07: FREE
G08: FREE
G09: FREE
G10: FREE
G11: FREE
G12: FREE
G13: FREE
G14: FREE
G15: FREE
H00: FREE
H01: FREE
H02: FREE
H03: FREE
H04: FREE
H05: FREE
H06: FREE
H07: FREE
H08: FREE
H09: FREE
H10: FREE
H11: FREE
H12: FREE
H13: FREE
H14: FREE
H15: FREE

Currently active Timers:
-----------------------
TIM1: FREE
TIM2: FREE
TIM3: FREE
TIM4: FREE
TIM5: FREE
TIM6: FREE
TIM7: FREE
TIM8:
    CH1 : DSHOT_BITBANG 2
    CH2 : DSHOT_BITBANG 1
TIM12: FREE
TIM13: FREE
TIM14: FREE
TIM15: FREE
TIM16: FREE
TIM17: FREE

Currently active DMA:
--------------------
DMA1 Stream 0: DSHOT_BITBANG 2
DMA1 Stream 1: DSHOT_BITBANG 1
DMA1 Stream 2: FREE
DMA1 Stream 3: FREE
DMA1 Stream 4: FREE
DMA1 Stream 5: FREE
DMA1 Stream 6: FREE
DMA1 Stream 7: FREE
DMA2 Stream 0: FREE
DMA2 Stream 1: FREE
DMA2 Stream 2: ADC 3
DMA2 Stream 3: FREE
DMA2 Stream 4: FREE
DMA2 Stream 5: OSD
DMA2 Stream 6: OSD
DMA2 Stream 7: OSD

Flight controller

SPRacing H7RF, purchased from SPRacing directly

Other components

Matek R24-D ELRS (2.1.0), wired to UART 8
Caddx Vista (latest), wired to UART4
TX16S running EdgeTX 2.6rc3
Thor TX running master (dynamic power, 1:64, 500MHz, 250mw max, 3.75M baud, built the day before 2.1 release)

How are the different components wired up

See above.

Add any other context about the problem that you think might be relevant here

Please keep in mind that despite this board being an SPI ELRS board, I have the built-in receiver disabled and am using a dedicated Matek ELRS receiver (in this configuration). I mentioned the issue to @hydra in the SPRacing discord and he recommended raising this issue with the BF team.

[blockfeed]

The blackbox file from this flight, please skip to 1:30 for the relevant portion.
LOG00124.zip

DVR of crash (if needed):
https://imgur.com/a/3ZBMKP2

[hydra]

Yes, disarm not being registered is very serious.

Here we can see the FC gets the RX link back after the failsafe, the FC registers the Flight Mode Change (ARM OFF), but doesn't disarm.

Someone else posted a similar looking log in the ELRS discord server a while ago, but I don't have the log or details handy.

To clarify, the FC wasn't powered on for over an hour, the batteries were changed and the FC was power cycled right @blockfeed ?

[hydra]

FLIGHT_LOG_EVENT_FLIGHTMODE is being logged, but not FLIGHT_LOG_EVENT_DISARM.

i.e. this isn't being called, in processRcStickPositions:

[hydra]

@SteveCEvans do you think this could be as a result of the scheduler not scheduling the processing of the RX state 'RX_STATE_MODES' in taskUpdateRxMain which calls processRxModes which in turns calls processRcStickPositions?

[pjpei]

I have also had this happen to me, and I have been injured as a result of this. I panic-reflex caught my drone when I disarmed and it didn't land, cutting my finger badly. Although my finger should be fine by next week or so, it sorta put me off fiddling with this in the interim.

It would really be great if we could get this fixed! I was testing new PID tunes and a custom lowpass filter so I also have a blackbox log of this happening. Running ELRS and HGLRCF722 flight controller (STM32F7X2).

[McGiverGim]

Both of you are flying and unsupported firmware file? I see they are from spracing, not directly from betaflight. Maybe the error is in the betaflight code too, but it makes the thing more difficult.

[pjpei]

Note I'm using ELRS (2.0.1) wired to a serial port too, BetaFPV receiver.
(Corrected above)

Mine was a fork from Betaflight, and I modified only the filters code, not any other files. Modified on my side are gyro_init.c, filters.c, filters.h I think is all. I'm testing the same firmware on a SPI FRSKI whoop and it doesn't have any disarm issues. Also note that (according to the original poster) ELRS discord also reported some issues.

EDIT:
This is my change list. I'll gladly share the details I've modified, and I don't think it could affect arm/disarm.

# On branch master
# Your branch is up to date with 'origin/master'.
#
# Changes to be committed:
#       modified:   src/main/cli/settings.c
#       modified:   src/main/common/filter.c
#       modified:   src/main/common/filter.h
#       modified:   src/main/sensors/gyro.h
#       modified:   src/main/sensors/gyro_init.c
#

[haslinghuis]

We are unable to look at this issue now. We first need #11110 which is scheduled for BF 4.4.

[pjpei]

Should I make another ticket for the HGLRCF722 which I'm flying currently with the same issue?

[haslinghuis]

Yes please. This will keep this issue specific to the H7RF. There could be some common underlying (configuration or code) problem.

[hydra]

@McGiverGim @haslinghuis The SPRacingH7RF support does not change any flight code. It uses the same ELRS RX code that is in master and simply has the addition of load/save to memory mapped flash and PixelOSD support. @pjpei is not using an SP Racing FC and has the same issue indicating that the problem is indeed in the same code that both targets use in master and that the changes to support the SPRacingH7RF are indeed unrelated to this issue.

I don't feel we need a new issue for this, but if one is created please reference this issue from it.

[hydra]

@pjpei Are you able to replicate this on an unmodified target for you 'HGLRCF722' board?

Maintainers: This is a very serious safety issue which, IMHO, should be treated as a high-priority bug that deserves investigation and fixing prior to any 4.3 release and not deferred to 4.4. The last thing we want is users injuring themselves and loosing confidence in the software over a known issue that wasn't looked at.

[pjpei]

@hydra As I have injured myself already, I'm sorta waiting for at least some acknowledgement of the bug and some form of bugfix before I fly with it again on my HGLRCF722. If it helps, I have tested with MATEXF411RX with my custom stuff many times with no issue of this sort.

I did file another issue as requested by haslinghuis, #11344 .

[pjpei]

Correction: I'm on ELRS 2.0.1, not 2.1.0. Correcting description above.

[JBKingdon]

FLIGHT_LOG_EVENT_FLIGHTMODE is being logged, but not FLIGHT_LOG_EVENT_DISARM.

i.e. this isn't being called, in processRcStickPositions:

The requirement for 4 packets signaling disarm needs to be relaxed if the rc packet interval exceeds some threshold. The log provided has crsf debug enabled and we can see a collapse of the packet interval at the end of the log, with intermittent packets getting through at around 2Hz. We could either expand the definition of failsafe to include a min packet rate, or we could relax the rcDisarmTicks threshold when the packet interval is large.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs within a week.

[hydra]

This is waiting on scheduler investigation and updates from @SteveCEvans.

What's the latest @SteveCEvans ?

[SteveCEvans]

@hydra Please test with #11380 as that includes the latest failsafe code (which I'm not sure you're using) as part of the ELRS work. Failsafe most certainly works with that. You probably want to be trying this for your H7RF anyway.

[hydra]

I'm guessing that #11459 is also related. For those following I updated the H7RF branch in order to test #11380 but it didn't work. Issue: #11451, Fixes: #11456 and #11457. I'll wait for all these 3 PR's to be merged and then request @blockfeed to re-test. @blockfeed if you can't wait, then grab https://github.com/spracing/betaflight/commits/bf-h7rf-wip-14 and cherry-pick the commits from #11459 and retest and report back here.

[haslinghuis]

Please test with latest development firmware if issue still persists as it should be resolved.

[hydra]

@blockfeed Please test with https://github.com/spracing/betaflight/releases/tag/spracing-20220406-2041 which is rebased on master and includes a critical scheduler bug fix from #11459 and report back.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs within a week.

[github-actions]

Issue closed automatically as inactive.