Keep getting EKF High Bias error

[DonLakeFlyer]

Log is here: https://review.px4.io/plot_app?log=0e28f7f8-65c7-44c2-a8ff-fcdbd2ca2c4b

I was getting this intermittently both inside and outside at boot. I tried recalibrating outside. No difference. Recalibrating inside,. no difference. It's raining now so I can't go outside and I can't seem to make it happen any longer inside.

I'm running a modified master on a new hexacopter+. The modifications are due to running an old pixhawk but I need new mag support. Don't know if that is affecting things since I can run regular master on this vehcle:

diff --git a/ROMFS/px4fmu_common/init.d/rc.sensors b/ROMFS/px4fmu_common/init.d/rc.sensors
index 7cac355f2..041f4afc0 100644
--- a/ROMFS/px4fmu_common/init.d/rc.sensors
+++ b/ROMFS/px4fmu_common/init.d/rc.sensors
@@ -76,6 +76,8 @@ then
# External I2C bus
hmc5883 -C -T -X start
lis3mdl -X start
+ ist8310 -C -b 1 start
+ ist8310 -C -b 2 start

# Internal I2C bus
hmc5883 -C -T -I -R 4 start
diff --git a/cmake/configs/nuttx_px4fmu-v2_default.cmake b/cmake/configs/nuttx_px4fmu-v2_default.cmake
index 369fc5568..9838b5251 100644
--- a/cmake/configs/nuttx_px4fmu-v2_default.cmake
+++ b/cmake/configs/nuttx_px4fmu-v2_default.cmake
@@ -31,6 +31,7 @@ set(config_module_list
drivers/imu/lsm303d
drivers/magnetometer/hmc5883
drivers/magnetometer/lis3mdl
+ drivers/magnetometer/ist8310
#drivers/mb12xx
#drivers/mkblctrl
drivers/imu/mpu6000
@@ -39,7 +40,7 @@ set(config_module_list
#drivers/protocol_splitter
drivers/pwm_input
#drivers/pwm_out_sim
- drivers/px4flow
+ #drivers/px4flow
drivers/px4fmu
drivers/px4io
drivers/rgbled
@@ -50,14 +51,14 @@ set(config_module_list
drivers/vmount

# distance sensors
- drivers/distance_sensor/ll40ls
- drivers/distance_sensor/mb12xx
- drivers/distance_sensor/sf0x
- drivers/distance_sensor/sf1xx
- drivers/distance_sensor/srf02
- drivers/distance_sensor/srf02_i2c
- drivers/distance_sensor/teraranger
- drivers/distance_sensor/tfmini
+ #drivers/distance_sensor/ll40ls
+ #drivers/distance_sensor/mb12xx
+ #drivers/distance_sensor/sf0x
+ #drivers/distance_sensor/sf1xx
+ #drivers/distance_sensor/srf02
+ #drivers/distance_sensor/srf02_i2c
+ #drivers/distance_sensor/teraranger
+ #drivers/distance_sensor/tfmini
#drivers/distance_sensor/ulanding
modules/sensors

[Antiheavy]

I've been seeing this warning recently on a Pixracer sitting my my desk connected over USB (px4 master within the last week). I chalked it up to a lazy sensor calibration since I tried to re-do the sensor calibration awkwardly connected to a USB cable. Or maybe the poor quality GPS signal I was getting indoors was a factor. I'll look to see if I can find a log that had it happening...

[Antiheavy]

I went back to look on the SDcard of the Pixracer I mentioned and I could not find any logs that showed "high bias error". The I remembered that the way I "fixed it" was to completely wipe the SDcard, reset all params to factory defaults, re-install firmware fresh, and then re-calibrate from scratch and re-apply my other params. As fas as I can tell this pixracer has not shown the "high bias error" since doing that... maybe a couple dozen boot cycles and several days of use on the bench. Don't know if that is helpful...

[priseborough]

If I calibrate the mag on my pixracer with usb cable connected, the calibration is invalid and it will not pass arming checks when the cable is disconnected or vice versa due to proximity of usb connector to sensor. I recommend all mag calibration be done with USB disconnected.

In this instance the preflight fail was due to an accelerometer bias warning error. Looking at the data your accelerometer is out of calibration. The vertical axis is measuring 10.25 m/s/s.

[DonLakeFlyer]

If I calibrate the mag on my pixracer with usb cable connected, the calibration is invalid and it will not pass arming checks when the cable is disconnected or vice versa due to proximity of usb connector to sensor.

Which I have done like 10 times. I have only ever calibrated while plugged in to battery.

Looking at the data your accelerometer is out of calibration.

So how do I get it calibrated correctly?

[DonLakeFlyer]

So I flashed this same vehicle with regular master. Disconnected the I2C so the new compasses on my GPS wouldn't show up. Recalibrated everthing. Same result. 10 m/s/s vertical accel:
https://review.px4.io/plot_app?log=dde3ffee-c4f0-417b-8733-7cb743499cc0

[DonLakeFlyer]

So I flashed this same vehicle with regular master. Disconnected the I2C so the new compasses on my GPS wouldn't show up. Recalibrated everthing. Same result. 10 m/s/s vertical accel:

Nope, wrong. Gravity is correct in that case. So something with my setup which is adding newer compass support to older fmu-v2 build (described above) is causing things to go off. Does that make any sense? Is this not a supported configuration?

[DonLakeFlyer]

Let me sanity check the initial problem again. I'll flash my custom build again, recal everything and get a new log just to be safe.

[DonLakeFlyer]

Ok, I tried my changes on v1.7.3 and master all over again. And now the logs look fine in the vertical axis. Daniel had originally mentioned possibly a temperature differential problem. Could that be it? Log with bad vertical accel was taken outside.

[dagar]

@DonLakeFlyer it's a real error.

1st log (ist8310) - https://review.px4.io/plot_app?log=0e28f7f8-65c7-44c2-a8ff-fcdbd2ca2c4b
Learned accel bias.

Commander error threshold (COM_ARM_EKF_AB) is 0.0024

[DonLakeFlyer]

In order to try to verify that this is a temp problem I did this:

• Full calibration indoors
• Left vehicle in cold garage overnight
• Brought in this morning, booted up
• Got EKF Bias error right away
• Left the vehicle powered up to come to room temp

Log here: https://logs.px4.io/plot_app?log=318c288b-b9b6-461d-b09e-9d434db8a5a1

[dagar]

Z accel straight from each sensor.

Temperature from each over the same period.

This is what the EKF learned accel bias looks like. Again, the threshold for the "PREFLIGHT FAIL: EKF HIGH IMU ACCEL BIAS" is only 0.0024 (COM_ARM_EKF_AB).

[priseborough]

The EKF uses units of delta velocity (m/s) instead of acceleration (m/s/s)for its inertial nav.

the EKF runs its internal updates using a 8 msec average frame time (outputs are predicted forward at the full IMU rate of 4 msec). Therefore a bias state of 0.0024 m/s is equivalent to a sensor bias of 0.0024/0.008 = 0.35 m/s/s

[priseborough]

Your board has excessive temperature sensitivity. It happens with a small percentage of boards because the board manufacturer does not screen boards and the sensor manufacturer does not screen individual sensors. We do provide a thermal calibration and compensation function.

[DonLakeFlyer]

Thanks everyone. I'm going to move into the current century and go to a PixRacer instead.

[Antiheavy]

@DonLakeFlyer we've been doing temp calibrations for a while now in our lab and on our engineering test vehicles. I think we are going to start doing it on our production vehicles soon (once we figure out how to do a better curve fit on the Baro). Ping us if you want to try it. The documentation is okay.

[priseborough]

@Antiheavy I think one of the problems with calibrating the Baro could be that we are applying our polynomial compensation in series with the piecewise continuous correction performed in the driver. It would be interesting to do a thermal calibration with the driver correction turned off.

[Antiheavy]

I don't mean to hijack this thread, but here is an example of the (IMHO) bad baro curve fits we are getting using the "offline" scripts. I worry that if the curve fit is extrapolated to colder temperatures it will be vastly diverged from the actual bias.

[DonLakeFlyer]

@Antiheavy The whole temp thing has me a bit concerned with respect to the Phoenix Pro in Kenya. I ran into problems because I didn't deal with Temp correctly in Zimbabwe last time as well with my multi-rotor. My fault I think, but still concerning. So my guess is that it would be best to do the temp cal thing with the Grevy's Zebra Phoenix somehow. Lot's of temp differential in Africa.

[priseborough]

@Antiheavy The compensation is clipped at the minimum and maximum calibration temperature as specified by the parameters - which is still bad for the example you show. We need a warning to the user and/or arming check fail if the sensor temperature is outside the calibration range.

It was this type of sensor behaviour at low temperatures that motivated ProfiCNC to implement an IMU heater function for the 'cube' (which PX4 does not support last time I checked).

[Antiheavy]

I'll probably start another issue to discuss possible improvements to Baro calibration.
edit: see here #9266

[Antiheavy]

On the original issue topic here, I recently started getting occasional high IMU bias erros in SITL too. Is this a new(ish) warning/error which might be why @DonLakeFlyer is seeing this only recently on his hardware?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

Closing as stale.