[PixHawk3] Accel1 startup bias huge changes

[bresch]

On my Drotek PixHawk Pro v1.3 I'm facing a strange behavior.
First I saw sometimes the warning "Accels inconsistent, check cal". After further investigations, I saw that usually, the Z-axis value is usually around 9.8m/s2, but sometimes, the value of the Acc1 is completely wrong (e.g.; 2 or 15). The value of the accelerometer 0 is always correct.

timestamp: 11548976

integral_dt: 4000

error_count: 0

x: -0.0971

y:  0.0563

z:  9.8932

x_integral: -0.0003

y_integral:  0.0002

z_integral:  0.0391

temperature: 42.7938

range_m_s2: 78.4532

scaling:  0.0024

x_raw: 7

y_raw: 40

z_raw: 4074

temperature_raw: 5815

device_id: 3608074

timestamp: 163650940

integral_dt: 4006

error_count: 0

x:  0.0778

y:  0.4095

z:  2.6501

x_integral:  0.0003

y_integral:  0.0014

z_integral:  0.0081

temperature: 59.1025

range_m_s2: 156.9064

scaling:  0.0048

x_raw: 53

y_raw: 29

z_raw: 480

temperature_raw: 8701

device_id: 1442826

After a few consecutive reboots on the pixhawk, I obtained the following sequence:
The behavior was the following: first calibration -> -9.8 -> reboot -> -15 -> 2nd calibration -> -9.8 -> reboot -> -3 -> 3rd calibration -> -9.8 -> reboot -> -15 -> reboot -> *-9.8* -> reboot -> -15 -> reboot -> -15 - reboot 3x -> -9.8 -> reboot -> -15 .........

I tried with two other new PixHawks Pro v3.1 and I can reproduce the error. However, strange thing, I tried on a v1.2 and I wasn't able to obtain the same result after more than 10 consecutive reboots.

Here is a log where the value is correct:
http://review.px4.io/plot_app?log=170ce58a-794e-414d-8607-168ce8c734ef
And here, one with the incorrect bias:
http://review.px4.io/plot_app?log=e0bba09b-1129-4438-8ded-baede72f24cc

The logs come from a device uncalibrated, but I had the same issue with an autopilot fully calibrated (thermal + scale&offset).

[davids5]

@klopezal , @LorenzMeier FYI.
@bresch Here is what I can tell you.

IMPORTANT: Without having found the root cause yet, I am concerned that the chips may have been stressed and that the reset fix below is just getting it started, but in reality it is defective. Please use caution if you test the commit below.

The incorrect values are coming out of the MPU9250 - ACELL Z register. Normal values are -2085.
On my board I see to failure poles. 6.9528 and -14.2173. ACELL Z register is -2954 when it fails at -14.2173.

1. The problem follows the IMU module. I moved a V1.2 IMU to my V1.3 board. It worked fine.

2. Using fmu sensor_reset 50 will fix the sensor, once the problem is evident.

3. The power on the sensor module is not shut off clean at startup. I saw 2.31V. Adding a sensor reset call with 250 Ms delay to board startup (px4fmu_init.c) does create a clean power up sequence (0V to 3.3V) but it did not help the issues when present, nor does a fmu sensor_reset 50 in the sensor.rc solve it.

4. This commit seems to work, It resets the Sensors just after the start of the MPU9250.

5. I compared the registers of the same device while failed and working. There are no obvious differences in any documented registers. In the AM. I will review the code, and see if anything looks to make decisions based on versions of the IC.

[klopezal]

I checked 3V3_SENSOR on a high number of power cycles and did not see any issue :

Rise time is about 60 µs.

The only difference between V1.2 and V1.3 IMUs is the ICM sensor :

• V1.2 has ICM-20608-G
• V1.3 has ICM-20602

We tried the following operations :

• desolder ICM-20602 from V1.3 and replace it by ICM-20608-G -> the problem remains
• desolder ICM-20608-G from V1.2 and replace it by ICM-20602 -> the problem doesn't appear

However :

• desolder MPU9k from V1.3 and replace it by MPU9k from V1.2 -> the problem disappears
• desolder MPU9k from V1.2 and replace it by MPU9k from V1.3 -> the problem appears

It looks like the problem is the sensor itself. @davids5, can you cross-test?
Issuing mpu9250 reset on console does appear to solve the problem when it happens. Some V1.3 boards did not show the issue after 10 reboots.

[LorenzMeier]

@bresch Can you please re-try as soon as possible on HW?

[TSC21]

@bresch any updates on this on the mean time?

[bresch]

With #7554 , I'm unable to reproduce the error I had previously (after 12 reboots at least).

[davids5]

@bresch - thank you. Closing as resolved.