Pixracer magnetometer preflight error

[r0gelion]

Master loaded to Pixracer (px4fmu-v4_default.px4), we were able to calibrate the compass, but when we try to arm the motors QGC returns "mag sensors inconsistent" error:

We noted this happened after commit 0def4ac

This is our setup:

[dagar]

Can you post the log? What's your calibration procedure? Are you inside or outside? Near any buildings or vehicles?

[r0gelion]

We had the SDLOG_MODE set to default (when armed until disarm) but we can't arm due to the preflight check error, so the log was not generated.

Changing the SDLOG_MODE to "from boot until disarm" and to "from boot until shutdown" allowed to create logs, but when we tried to calibrate the compass we got "out of space" error and the calibration failed.

Here you can find the logs:
https://logs.px4.io/plot_app?log=10f23a59-ec89-481f-8c64-c0b8ace51029
https://logs.px4.io/plot_app?log=149544a2-e582-41f4-98b3-9143a4534c63
https://logs.px4.io/plot_app?log=03978cfa-78ab-48b3-8328-22dbf5c5a5e6
https://logs.px4.io/plot_app?log=b09ea596-f321-4773-8e46-4150cc78c00f

We calibrate using QGC, inside the office. We can try again in a better environment, but we didn't have this problem before.

[priseborough]

#8039 was merged prematurely and should be reverted. Until there is sufficient logging to diagnose pre-flight mag inconsistencies and a means to disable faulty mags it is not ready.

[priseborough]

Alternatively the check threshold can be set to something large.

[priseborough]

Can you try setting COM_ARM_MAG to a large value, eg 1.0 and see if it will pass the check?

[r0gelion]

I couldn't find the parameter

These are the parameters that start with "COM_ARM":

[priseborough]

It shows up on QGC with my pixracer when flashed using master

[dagar]

Mag cal is failing because the free memory is dangerously low.

Are you seeing any of these messages in QGC?

Can you set SYS_COMPANION 0, disconnect your px4flow (temporarily), and try again?

[dagar]

Disconnect USB as well.

[r0gelion]

Disconnecting USB cleared the "out of memory" error and allowed us to calibrate compass when SDLOG_MODE is set to "from boot until disarm" or to "from boot until shutdown". After calibration, we got an error of a high offset in an internal mag.

We couldn't arm

Also tried to change COM_ARM_MAG to 1.0 and noted this parameter is only available when SDLOG_MODE is set to default (when armed until disarm). We still got the mag inconsistent error and "NOT ARMING: Preflight checks failded" error.

[r0gelion]

Logs:
https://logs.px4.io/plot_app?log=296ed941-817e-45b6-bba1-e8e3ff03a423
https://logs.px4.io/plot_app?log=26f872ca-11e6-4dd5-b3d0-cce5cb2eadc3

[priseborough]

The TODO items are:

1. Increase the default value of COM_ARM_MAG until there is support for logging and sensor deactivation.
2. Fix the out of memory issue
3. Add an option to log all physical mag units pre-flight. This could be either set by parameter or preferably triggered if the preflight check is failing.
4. Add a method to disable a magnetometer by parameter setting.

I can start on items 1) and 3)

[yashmulgaonkar]

+1 for item 4!
I am working on some micro pixhawks that don't have a mag :)
Thanks!

[taileron]

Got now a new PixRacer (R14) for testing exactly with the same issues. (Master 1.7.2)
Solved it with 2 methods: SYS_COMPANION = disabled - clears the "out of memory" message at the calibration attempt.
CAL_MAG0_EN = disabled or COM_ARM_MAG = 0.5 afterwards an additional compass calibration.
By the way, PixFalcon shows this issues not at all !!

[LorenzMeier]

@taileron Could you please re-enable SYS_COMPANION and use this branch and try again?
#8533

If that doesn't fix it, please lower the log buffer further and let us know which value works.

@priseborough Could you comment on the two other parameters?

[priseborough]

@taileron There is a potential problem created by the close proximity of the usb connector to the magnetometers on the pixracer that means they cannot be reliably calibrated with usb connected because the ferrous material in the plug creates a magnetic field distortion that affects each magnetometer differently.

Please calibrate and do all testing with USB disconnected and if not using an external mag you should not see the 'MAG SENSORS INCONSISTENT' message. If you have an external compass and still get the message, then check alignment of the external sensor.

Was there a reason you disabled MAG0 instead of MAG1? Why did you simultaneously disable a mag and increase the check threshold? If should only be necessary to either one or the other.

The check is showing there is major interference if COM_ARM_MAG needs to increase to 0.5 to get checks to pass. Did you try other values? The parameter represents the max allowed magnitude of the difference between the vector measured by the different sensors.

[priseborough]

The inconsistency level can be determined be checking the sensor_preflight_0.mag_inconsistency_ga message in the log (if pre-arm logging is enabled) or by starting an nsh console and using the listener command to check the sensor_preflight message.

[taileron]

It seems that there is another problem with the usb connection itself. If the connection is done via WIFI and usb only as power supply it works. Mag calib should only be done with the same metallic environment like it is in flight that concerns magnetic canopy locks, cooling fans, usb connectors and all stuff around the sensors. (I corrected a typo in first post)

[taileron]

@LorenzMeier #8533 fixes primarely the memory issue while the QCG connection is done via WIFI (no inconsistence with com_arm_xxx default values anymore) but as soon as QCG is connected via USB most common nsh commands don´t work anymore:
NuttShell (NSH)
nsh> ver all
nsh: ver: command not found

and mostly preflight fail: accels inconsistent" appears as well

still only SYS_COMPANION disable makes a USB connection possible

probably it needs to free up a bit more memory.

[priseborough]

@taileron please repeat the accel calibration and see if the "preflight fail: accels inconsistent" message goes away. It can be caused if a board has a sensor with high thermal sensitivity combined with a temperature that is sufficiently different to when the board was calibrated.

The test limit is set by the COM_ARM_IMU_ACC parameter at 0.7 m/s/s which is not overly conservative, so if that check is failing, then you do need to recalibrate.

[taileron]

@LorenzMeier set LOGGER_BUF 1 still doesn´t offer the usb connection without sys_companion disable but the inconsistence messages of all sensors are gone
@priseborough in my Convergence the PixRacer got a cooling fan cause it generates significantly more heat than expected but in my opinion this controller does a quite good compensation cause the betaflight fc´s acc located above the PixRacer in the plane gets much more detuned by temperature change

[priseborough]

@taileron The pixracer does not do temperature compensation unless you follow the thermal calibration procedure provided by PX4.

There is a lot of variation between different serial number boards of the same model in terms of temperature sensitivity. At the price point you are using, the board manufacturer cannot afford to screen sensors for temperature sensitivity.

[taileron]

@priseborough thanks, learned more about sensors, the old R12 board seems to have the better selected ones ... how was this done in V 1.6.5 flew it after few calibrations several months successful in very different temperatures. Now (R14 / PX4 1.7.2) "sensors status" shows if the board is very cold in the first 60 seconds a higher third value of acc 0 is there a parameter to delay the sensor check or repeat it after boot to have more time for e.g. closing a magnetic locked canopy ?

[taileron]

is PX4 able to read a temperature from one of the sensors anywhere on the board?
which (nsh) command can show it?

[LorenzMeier]

If you have a recent board with enough flash this would do it:

[taileron]

... works great but the baro seems to show a more reliable temperature (listener sensor_baro) tested with an infrared thermometer pointed to the baro

[priseborough]

For effect on sensor calibration it is the internal temperature, not the surface temperature as measured by an IR thermometer that is relevant. The internal temp will be hotter due to thermal resistance of the case and the difference depends the sensor type and also its mode of operation.

[taileron]

... Baro is only 7° (26° absolute) above it´s surface Mag 16° (35°) above surface (19°) directly measured few sec after start and with an additional 20mm cooling fan it remains stable after a few minutes.
Pixracer metal case and without fan the mag sensor rises very fast above 55°

[priseborough]

Individual sensors can be disabled via the CAL_..._EN parameters, the check level is appropriate for safe operation and @taileron was able to resolve the problem by calibrating in the flight configuration.

Sensor comparison data can be logged by setting SDLOG_MODE = 1 and setting bit position 6 in SDLOG_PROFILE to 1

This issue can be closed, however I will raise a new one because our documentation for the setup procedure is not providing the following information:

1. Magnetometer calibration should be done with usb disconnected and the vehicle in a flight configuration.
2. Raw comparison data for all sensors can be logged by setting SDLOG_MODE = 1 and SDLOG_PROFILE = 64
3. Sensors with poor performance can be disabled using the CAL_..._EN parameter