VTail issue on 2.7.1 with LUX & SPF3 Clone

[HorusDrones]

I have experienced two crashes due to unresponsive flight controllers using the VTail mix. I've spoken to Dominic about as well, however in these instances when we have been flying acro. While doing flip and rolls, the flight controllers have suddenly become unresponsive to input. This issue does not persist after the crash, which makes it even more interesting. Both frames, after a prop swap and a quick check-up have been flying with the same hardware since. So here are the basics of the two frames this occurred with.

VTail #1 - 250mm, 2600kv motors, DYS XM30 V2 w. Multishot, LUX FC set at 8K with the standard VTail mix.
VTail #2 - 180mm, 2950kv motors, LittleBee Pro 20A w. Multishot, SPF3 Clone, set at 4k with the standard VTail mix.

On both occasions, each airframe had been flying fine, exhibiting no issues during maneuvers, flipping and rolling with aplomb. Then suddenly, on a particular flip or roll, it froze while upside-down and then either dove for the ground or free fell, depending on the angle of the pitch at that moment. Myself and my tester both dropped the throttle and actively tried to get it to respond to yaw, pitch or roll inputs, but no input was exhibited by the airframe. In the most recent incident, he was roughly 120 or more feet off the ground, no rssi issues, nothing of note. The airframe was flying great he was very happy with it, so he goes for another high-rate roll and we all watched as it just locked when he got inverted. He dropped the throttle to roughly 30% and quickly, but not frantically, tried every input he could. Given the height and size he had plenty of time to do this, however it resulted in a crash. Just one broken part aside from the props.

The importance of describing the second incident vs the first is that this happened with VTail #2. Number 2 uses the SPF3 Clone uses the MPU on I2C, whereas the LUX using the 6500 has noise which is often blamed for moments like this (and I believe you have addressed), but is clearly not the culprit if it can be replicated on completely different hardware.

We will be pulling our blackbox to see if we can recover any information, however I wanted to bring this to your attention because we suspect it maybe something within the code (why I also brought it up to Dominic), however I don't know or understand how this would happen with a VTail vs a Quad based on my limited understanding of how the mixing factors in to the PID loop.

Warm Regards
Sherif Manganas
Horus Drones

[HorusDrones]

CLI DUMP:

version

BetaFlight/SPRACINGF3 2.7.1 May 16 2016 / 20:47:21 (74cd38a)

dump master

mixer

mixer VTAIL4
mmix reset
smix reset

feature

feature -RX_PPM
feature -VBAT
feature -INFLIGHT_ACC_CAL
feature -RX_SERIAL
feature -MOTOR_STOP
feature -SERVO_TILT
feature -SOFTSERIAL
feature -GPS
feature -FAILSAFE
feature -SONAR
feature -TELEMETRY
feature -CURRENT_METER
feature -3D
feature -RX_PARALLEL_PWM
feature -RX_MSP
feature -RSSI_ADC
feature -LED_STRIP
feature -DISPLAY
feature -ONESHOT125
feature -BLACKBOX
feature -CHANNEL_FORWARDING
feature -TRANSPONDER
feature VBAT
feature RX_SERIAL
feature FAILSAFE
feature ONESHOT125
feature BLACKBOX

beeper

beeper GYRO_CALIBRATED
beeper RX_LOST
beeper RX_LOST_LANDING
beeper DISARMING
beeper ARMING
beeper ARMING_GPS_FIX
beeper BAT_CRIT_LOW
beeper BAT_LOW
beeper GPS_STATUS
beeper RX_SET
beeper ACC_CALIBRATION
beeper ACC_CALIBRATION_FAIL
beeper READY_BEEP
beeper MULTI_BEEPS
beeper DISARM_REPEAT
beeper ARMED
beeper SYSTEM_INIT
beeper ON_USB

map

map TAER1234

serial

serial 0 1 115200 57600 0 115200
serial 1 0 115200 57600 0 115200
serial 2 64 115200 57600 0 115200

led

led 0 15,15:ES:IA:0
led 1 15,8:E:WF:0
led 2 15,7:E:WF:0
led 3 15,0:NE:IA:0
led 4 8,0:N:F:0
led 5 7,0:N:F:0
led 6 0,0:NW:IA:0
led 7 0,7:W:WF:0
led 8 0,8:W:WF:0
led 9 0,15:SW:IA:0
led 10 7,15:S:WF:0
led 11 8,15:S:WF:0
led 12 7,7:U:WF:0
led 13 8,7:U:WF:0
led 14 7,8:D:WF:0
led 15 8,8:D:WF:0
led 16 8,9::R:3
led 17 9,10::R:3
led 18 10,11::R:3
led 19 10,12::R:3
led 20 9,13::R:3
led 21 8,14::R:3
led 22 7,14::R:3
led 23 6,13::R:3
led 24 5,12::R:3
led 25 5,11::R:3
led 26 6,10::R:3
led 27 7,9::R:3
led 28 0,0:::0
led 29 0,0:::0
led 30 0,0:::0
led 31 0,0:::0

color

color 0 0,0,0
color 1 0,255,255
color 2 0,0,255
color 3 30,0,255
color 4 60,0,255
color 5 90,0,255
color 6 120,0,255
color 7 150,0,255
color 8 180,0,255
color 9 210,0,255
color 10 240,0,255
color 11 270,0,255
color 12 300,0,255
color 13 330,0,255
color 14 0,0,0
color 15 0,0,0

aux

aux 0 0 0 1850 2100
aux 1 27 1 1300 2100
aux 2 28 1 1300 1700
aux 3 0 0 900 900
aux 4 0 0 900 900
aux 5 0 0 900 900
aux 6 0 0 900 900
aux 7 0 0 900 900
aux 8 0 0 900 900
aux 9 0 0 900 900
aux 10 0 0 900 900
aux 11 0 0 900 900
aux 12 0 0 900 900
aux 13 0 0 900 900
aux 14 0 0 900 900
aux 15 0 0 900 900
aux 16 0 0 900 900
aux 17 0 0 900 900
aux 18 0 0 900 900
aux 19 0 0 900 900

adjrange

adjrange 0 0 0 900 900 0 0
adjrange 1 0 0 900 900 0 0
adjrange 2 0 0 900 900 0 0
adjrange 3 0 0 900 900 0 0
adjrange 4 0 0 900 900 0 0
adjrange 5 0 0 900 900 0 0
adjrange 6 0 0 900 900 0 0
adjrange 7 0 0 900 900 0 0
adjrange 8 0 0 900 900 0 0
adjrange 9 0 0 900 900 0 0
adjrange 10 0 0 900 900 0 0
adjrange 11 0 0 900 900 0 0
adjrange 12 0 0 900 900 0 0
adjrange 13 0 0 900 900 0 0
adjrange 14 0 0 900 900 0 0

rxrange

rxrange 0 1000 2000
rxrange 1 1000 2000
rxrange 2 1000 2000
rxrange 3 1000 2000

servo

servo 0 1000 2000 1500 90 90 100 -1
servo 1 1000 2000 1500 90 90 100 -1
servo 2 1000 2000 1500 90 90 100 -1
servo 3 1000 2000 1500 90 90 100 -1
servo 4 1000 2000 1500 90 90 100 -1
servo 5 1000 2000 1500 90 90 100 -1
servo 6 1000 2000 1500 90 90 100 -1
servo 7 1000 2000 1500 90 90 100 -1

set mid_rc = 1500
set min_check = 1100
set max_check = 1900
set rssi_channel = 0
set rssi_scale = 30
set rssi_ppm_invert = OFF
set input_filtering_mode = OFF
set rc_smoothing = OFF
set roll_yaw_cam_mix_degrees = 0
set max_aux_channels = 6
set debug_mode = NONE
set min_throttle = 1050
set max_throttle = 1980
set min_command = 1000
set servo_center_pulse = 1500
set 3d_deadband_low = 1406
set 3d_deadband_high = 1514
set 3d_neutral = 1460
set 3d_deadband_throttle = 50
set unsynced_fast_pwm = OFF
set fast_pwm_protocol = MULTISHOT
set motor_pwm_rate = 1600
set servo_pwm_rate = 50
set disarm_kill_switch = ON
set gyro_cal_on_first_arm = OFF
set auto_disarm_delay = 5
set small_angle = 25
set fixedwing_althold_dir = 1
set reboot_character = 82
set gps_provider = NMEA
set gps_sbas_mode = AUTO
set gps_auto_config = ON
set gps_auto_baud = OFF
set gps_wp_radius = 200
set nav_controls_heading = ON
set nav_speed_min = 100
set nav_speed_max = 300
set nav_slew_rate = 30
set serialrx_provider = SBUS
set sbus_inversion = ON
set spektrum_sat_bind = 0
set spektrum_sat_bind_autoreset = 1
set telemetry_switch = OFF
set telemetry_inversion = ON
set frsky_default_lattitude = 0.000
set frsky_default_longitude = 0.000
set frsky_coordinates_format = 0
set frsky_unit = IMPERIAL
set frsky_vfas_precision = 0
set frsky_vfas_cell_voltage = OFF
set hott_alarm_sound_interval = 5
set battery_capacity = 0
set vbat_scale = 110
set vbat_max_cell_voltage = 43
set vbat_min_cell_voltage = 33
set vbat_warning_cell_voltage = 35
set vbat_hysteresis = 1
set vbat_pid_compensation = OFF
set current_meter_scale = 400
set current_meter_offset = 0
set multiwii_current_meter_output = OFF
set current_meter_type = ADC
set align_gyro = DEFAULT
set align_acc = DEFAULT
set align_mag = DEFAULT
set align_board_roll = 0
set align_board_pitch = 0
set align_board_yaw = 90
set max_angle_inclination = 700
set gyro_lpf = OFF
set gyro_sync_denom = 2
set gyro_lowpass = 100
set moron_threshold = 32
set imu_dcm_kp = 2500
set imu_dcm_ki = 0
set alt_hold_deadband = 40
set alt_hold_fast_change = ON
set deadband = 5
set yaw_deadband = 6
set throttle_correction_value = 0
set throttle_correction_angle = 800
set yaw_control_direction = 1
set yaw_motor_direction = 1
set yaw_jump_prevention_limit = 200
set tri_unarmed_servo = ON
set servo_lowpass_freq = 400
set servo_lowpass_enable = OFF
set super_expo_factor = 30
set super_expo_factor_yaw = 30
set super_expo_yaw = OFF
set failsafe_delay = 10
set failsafe_off_delay = 10
set failsafe_throttle = 1400
set failsafe_kill_switch = OFF
set failsafe_throttle_low_delay = 100
set failsafe_procedure = OFF
set rx_min_usec = 885
set rx_max_usec = 2115
set gimbal_mode = NORMAL
set acc_hardware = NONE
set acc_lpf_hz = 10.000
set accxy_deadband = 40
set accz_deadband = 40
set acc_unarmedcal = ON
set acc_trim_pitch = 0
set acc_trim_roll = 0
set baro_tab_size = 21
set baro_noise_lpf = 0.600
set baro_cf_vel = 0.985
set baro_cf_alt = 0.965
set baro_hardware = NONE
set mag_hardware = NONE
set mag_declination = 0
set pid_process_denom = 2
set blackbox_rate_num = 1
set blackbox_rate_denom = 1
set blackbox_device = SPIFLASH
set magzero_x = 0
set magzero_y = 0
set magzero_z = 0

rxfail

rxfail 0 a
rxfail 1 a
rxfail 2 a
rxfail 3 a
rxfail 4 h
rxfail 5 h
rxfail 6 h
rxfail 7 h
rxfail 8 h
rxfail 9 h
rxfail 10 h
rxfail 11 h
rxfail 12 h
rxfail 13 h
rxfail 14 h
rxfail 15 h
rxfail 16 h
rxfail 17 h

profile

profile 0
############################# PROFILE VALUES ####################################

set gps_pos_p = 15
set gps_pos_i = 0
set gps_pos_d = 0
set gps_posr_p = 34
set gps_posr_i = 14
set gps_posr_d = 53
set gps_nav_p = 25
set gps_nav_i = 33
set gps_nav_d = 83
set yaw_p_limit = 500
set dterm_lowpass = 110
set dynamic_pterm = ON
set iterm_always_reset = OFF
set iterm_reset_degrees = 200
set yaw_iterm_reset_degrees = 50
set iterm_reset_offset = 15
set yaw_lowpass = 80
set pid_controller = MWREWRITE
set p_pitch = 44
set i_pitch = 42
set d_pitch = 23
set p_roll = 40
set i_roll = 36
set d_roll = 20
set p_yaw = 88
set i_yaw = 49
set d_yaw = 0
set p_alt = 50
set i_alt = 0
set d_alt = 0
set p_level = 50
set i_level = 50
set d_level = 100
set p_vel = 55
set i_vel = 55
set d_vel = 75
rateprofile 0

rateprofile

rateprofile 0

set rc_rate = 114
set rc_expo = 65
set rc_yaw_expo = 20
set thr_mid = 40
set thr_expo = 0
set roll_rate = 76
set pitch_rate = 78
set yaw_rate = 110
set tpa_rate = 0
set tpa_breakpoint = 1500

[HorusDrones]

blackbox_log_2016-06-27_100139.TXT

Blackbox Log

[blckmn]

What you describe is like a hard fault has occurred. Or a loop that never exits.

[HorusDrones]

How can we diagnose a hard fault? I'm not entirely clear on what would be the trigger.

Speaking with my friend and test pilot today, he intimated he suspects that its an issue of noise. In our latest conversation he explained that the build up of noise, user error, bad PIDs or mounting leads to accumulated noise, preventing the closing of the loop and leading to desync issues. With the LUX board, I can see this as the primary source of the issue. It has two points of potential noise; the FC mount and the V itself, which has a direct line to the central chassis. However, with the SPF3 board, mounting is not an issue and the V is carbon vs. 3D printed engineering grade material.

The method of testing, for him, is to flip and roll throttling in and out. This seems the most likely way to to reproduce the issue. Do this enough times and it seems to be method for achieving this.

For the SPF3 board, I believe we will keep the looptime to 2Khz w. the F396 ESCs. With the LUX, maybe we will limit it to 4Khz.

I'm building a third bird now, with SPF3, DYS BLHeli_S 30A ESCs, new 3D material and Rebel 2600kv motors. I'm going to load this with Betaflight 2.9.0 and the latest BLS firmware to see if we can reproduce the issue. As with the previous one, I'll limit this build to 2Khz.

[martinbudden]

There is a remote possibility that this could be caused by an integer overflow error in the MWREWRITE PID. I don't think it is that likely, but it is worth checking for the purpose of elimination.

Can try retesting with the LUXFLOAT PID to see if you can reproduce the problem?

[borisbstyle]

@blckmn @martinbudden
I dont think thats the case at all. When things only occur during flips and rolls that typically indicates mechanical issues or other electronical issues on the quad. If there was a hardfault it would even occur when not armed

[borisbstyle]

@HorusDrones
Your log doesnt show any signs of problem. 8MB flash I guess which got full

[HorusDrones]

Both birds are mechanically sound however I can't claim to be 100% on the electronic side.

That said, I've been reviewing some of the previous release notes and updating myself on the CLI commands and it occurred to me that despite using the Multishot protocol, the unsynced_fast_pwm is OFF and the motor_pwm_rate = 1600. Do you think that there is something happening here?

I've seen in 2.8 and on, you added the anti_desync_power_step option. Based on my understanding, could this address any issues assuming that the motor_pwm_rate and unsynced_fast_pwm is set correctly?

That said, I've found limited information about what are the proper setting for the motor_pwm rate. With regard to unsynced_fast_pwm, the idea is run it when you're going over 2Khz correct?

[blckmn]

One way in which we can rule out a hard fault is by flashing a LED constantly and at a high rate in the hard fault handler. That way, if it crashes due to a fault (which it shouldn't - but I have seen it before when buffers overflow or similar) then it will sit there with a high speed flash until power is cycled.

Thoughts?

[borisbstyle]

@blckmn
Once you cannot reproduce hardfault when not inflight it will also never occur during flight. FC is technically always flying, except it doesnt produce motor output when disarmed

@HorusDrones
The only thing what gets stressed out during flips and rolls are the ESC's as that is the part where huge currents do occur. From the software perspective the flips and rolls are not different than normal situation.

[borisbstyle]

@blckmn
The best way to troubleshoot this is to get it in the log ;). Hardware fault would make log to stop and other failures will be logged

[blckmn]

true a good BB log will always help diagnose. I just don't like the symptom of completely unresponsive. It smells of a lock up, like a never exiting loop, or hard fault. Of course there's a thousand possibilities - so I'm probably wrong.

[borisbstyle]

I am yet to reproduce that on the current code base :). Simply 100% sure it
is not the case here.

What about rx ;)....power brown out.....
Op 11 jul. 2016 17:34 schreef "J Blackman" notifications@github.com:

true a good BB log will always help diagnose. I just don't like the symptom
of completely unresponsive. It smells of a lock up, like a never exiting
loop, or hard fault. Of course there's a thousand possibilities - so I'm
probably wrong.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#616 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AKQlhL2KVz3GLsWFKb9cspAqkicSyGiqks5qUmKQgaJpZM4I_TAj
.

[HorusDrones]

@borisbstyle I’ll get you new logs ASAP. Any commentary on the motor_pwm_rate / unsynced_fast_pwm question? Or the anti_desync_power_step option?

The thing that is most confusing really, is that the motors have full throttle authority, however do not respond to aileron, elevator or rudder inputs.

If you’d like to test this out on some actual hardware, I’d love to send you a frame for free. Visit http://www.horus-drones.com and let me know if you’d like a Harpy or Merlin in an email, to vision@horus-drones.com, with your address and I will send you one.

[borisbstyle]

@HorusDrones
I would like to accept your offer, but there are so many things I need to build now but dont have time :(

[martinbudden]

@HorusDrones ,
I've been doing too much coding and not enough flying recently. So if the offer applies to other betaflight developers, then I'd like to test out a V-tail. I've also felt that the default values for the V-tail mixer don't really look right and it would be an excuse to do the maths and calculate and flight test some new mixer values.

[borisbstyle]

Yeah @martinbudden deserves it!

[HorusDrones]

@borisbstyle @martinbudden Thank you both, I'd love to take you up on it! Having anyone working closely with the code development is a dream come true for me. @martinbudden Send me an email to vision@horus-drones.com with your address and I'll ship out a frame or two to you asap!

[borisbstyle]

closing this one due to lack of activity. Open new issues if the issue is still there