New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pixracer magnetometer preflight error #8098

Closed
r0gelion opened this Issue Oct 9, 2017 · 29 comments

Comments

Projects
None yet
6 participants
@r0gelion
Copy link
Contributor

r0gelion commented Oct 9, 2017

Master loaded to Pixracer (px4fmu-v4_default.px4), we were able to calibrate the compass, but when we try to arm the motors QGC returns "mag sensors inconsistent" error:

v4 preflight check fail

We noted this happened after commit 0def4ac

This is our setup:
img_20171009_095317

@dagar

This comment has been minimized.

Copy link
Member

dagar commented Oct 9, 2017

Can you post the log? What's your calibration procedure? Are you inside or outside? Near any buildings or vehicles?

@r0gelion

This comment has been minimized.

Copy link
Contributor

r0gelion commented Oct 9, 2017

We had the SDLOG_MODE set to default (when armed until disarm) but we can't arm due to the preflight check error, so the log was not generated.

Changing the SDLOG_MODE to "from boot until disarm" and to "from boot until shutdown" allowed to create logs, but when we tried to calibrate the compass we got "out of space" error and the calibration failed.

Here you can find the logs:
https://logs.px4.io/plot_app?log=10f23a59-ec89-481f-8c64-c0b8ace51029
https://logs.px4.io/plot_app?log=149544a2-e582-41f4-98b3-9143a4534c63
https://logs.px4.io/plot_app?log=03978cfa-78ab-48b3-8328-22dbf5c5a5e6
https://logs.px4.io/plot_app?log=b09ea596-f321-4773-8e46-4150cc78c00f

We calibrate using QGC, inside the office. We can try again in a better environment, but we didn't have this problem before.

@priseborough

This comment has been minimized.

Copy link
Contributor

priseborough commented Oct 9, 2017

#8039 was merged prematurely and should be reverted. Until there is sufficient logging to diagnose pre-flight mag inconsistencies and a means to disable faulty mags it is not ready.

@priseborough

This comment has been minimized.

Copy link
Contributor

priseborough commented Oct 9, 2017

Alternatively the check threshold can be set to something large.

@priseborough

This comment has been minimized.

Copy link
Contributor

priseborough commented Oct 9, 2017

Can you try setting COM_ARM_MAG to a large value, eg 1.0 and see if it will pass the check?

@r0gelion

This comment has been minimized.

Copy link
Contributor

r0gelion commented Oct 9, 2017

I couldn't find the parameter

These are the parameters that start with "COM_ARM":
param

@priseborough

This comment has been minimized.

Copy link
Contributor

priseborough commented Oct 9, 2017

It shows up on QGC with my pixracer when flashed using master

@dagar

This comment has been minimized.

Copy link
Member

dagar commented Oct 9, 2017

Mag cal is failing because the free memory is dangerously low.

image

Are you seeing any of these messages in QGC?

image

Can you set SYS_COMPANION 0, disconnect your px4flow (temporarily), and try again?

@dagar

This comment has been minimized.

Copy link
Member

dagar commented Oct 9, 2017

Disconnect USB as well.

@r0gelion

This comment has been minimized.

Copy link
Contributor

r0gelion commented Oct 10, 2017

Disconnecting USB cleared the "out of memory" error and allowed us to calibrate compass when SDLOG_MODE is set to "from boot until disarm" or to "from boot until shutdown". After calibration, we got an error of a high offset in an internal mag.
error1

We couldn't arm
error2

Also tried to change COM_ARM_MAG to 1.0 and noted this parameter is only available when SDLOG_MODE is set to default (when armed until disarm). We still got the mag inconsistent error and "NOT ARMING: Preflight checks failded" error.
error3

@priseborough

This comment has been minimized.

Copy link
Contributor

priseborough commented Oct 10, 2017

The TODO items are:

  1. Increase the default value of COM_ARM_MAG until there is support for logging and sensor deactivation.
  2. Fix the out of memory issue
  3. Add an option to log all physical mag units pre-flight. This could be either set by parameter or preferably triggered if the preflight check is failing.
  4. Add a method to disable a magnetometer by parameter setting.

I can start on items 1) and 3)

@yashm

This comment has been minimized.

Copy link

yashm commented Dec 12, 2017

+1 for item 4!
I am working on some micro pixhawks that don't have a mag :)
Thanks!

@taileron

This comment has been minimized.

Copy link

taileron commented Dec 28, 2017

Got now a new PixRacer (R14) for testing exactly with the same issues. (Master 1.7.2)
Solved it with 2 methods: SYS_COMPANION = disabled - clears the "out of memory" message at the calibration attempt.
CAL_MAG0_EN = disabled or COM_ARM_MAG = 0.5 afterwards an additional compass calibration.
By the way, PixFalcon shows this issues not at all !!

@LorenzMeier

This comment has been minimized.

Copy link
Member

LorenzMeier commented Dec 28, 2017

@taileron Could you please re-enable SYS_COMPANION and use this branch and try again?
#8533

If that doesn't fix it, please lower the log buffer further and let us know which value works.

@priseborough Could you comment on the two other parameters?

@priseborough

This comment has been minimized.

Copy link
Contributor

priseborough commented Dec 28, 2017

@taileron There is a potential problem created by the close proximity of the usb connector to the magnetometers on the pixracer that means they cannot be reliably calibrated with usb connected because the ferrous material in the plug creates a magnetic field distortion that affects each magnetometer differently.

Please calibrate and do all testing with USB disconnected and if not using an external mag you should not see the 'MAG SENSORS INCONSISTENT' message. If you have an external compass and still get the message, then check alignment of the external sensor.

Was there a reason you disabled MAG0 instead of MAG1? Why did you simultaneously disable a mag and increase the check threshold? If should only be necessary to either one or the other.

The check is showing there is major interference if COM_ARM_MAG needs to increase to 0.5 to get checks to pass. Did you try other values? The parameter represents the max allowed magnitude of the difference between the vector measured by the different sensors.

@priseborough

This comment has been minimized.

Copy link
Contributor

priseborough commented Dec 28, 2017

The inconsistency level can be determined be checking the sensor_preflight_0.mag_inconsistency_ga message in the log (if pre-arm logging is enabled) or by starting an nsh console and using the listener command to check the sensor_preflight message.

@taileron

This comment has been minimized.

Copy link

taileron commented Dec 28, 2017

It seems that there is another problem with the usb connection itself. If the connection is done via WIFI and usb only as power supply it works. Mag calib should only be done with the same metallic environment like it is in flight that concerns magnetic canopy locks, cooling fans, usb connectors and all stuff around the sensors. (I corrected a typo in first post)

@taileron

This comment has been minimized.

Copy link

taileron commented Dec 28, 2017

@LorenzMeier #8533 fixes primarely the memory issue while the QCG connection is done via WIFI (no inconsistence with com_arm_xxx default values anymore) but as soon as QCG is connected via USB most common nsh commands don´t work anymore:
NuttShell (NSH)
nsh> ver all
nsh: ver: command not found

and mostly preflight fail: accels inconsistent" appears as well

still only SYS_COMPANION disable makes a USB connection possible

probably it needs to free up a bit more memory.

@priseborough

This comment has been minimized.

Copy link
Contributor

priseborough commented Dec 28, 2017

@taileron please repeat the accel calibration and see if the "preflight fail: accels inconsistent" message goes away. It can be caused if a board has a sensor with high thermal sensitivity combined with a temperature that is sufficiently different to when the board was calibrated.

The test limit is set by the COM_ARM_IMU_ACC parameter at 0.7 m/s/s which is not overly conservative, so if that check is failing, then you do need to recalibrate.

@taileron

This comment has been minimized.

Copy link

taileron commented Dec 28, 2017

@LorenzMeier set LOGGER_BUF 1 still doesn´t offer the usb connection without sys_companion disable but the inconsistence messages of all sensors are gone
@priseborough in my Convergence the PixRacer got a cooling fan cause it generates significantly more heat than expected but in my opinion this controller does a quite good compensation cause the betaflight fc´s acc located above the PixRacer in the plane gets much more detuned by temperature change

@priseborough

This comment has been minimized.

Copy link
Contributor

priseborough commented Dec 29, 2017

@taileron The pixracer does not do temperature compensation unless you follow the thermal calibration procedure provided by PX4.

There is a lot of variation between different serial number boards of the same model in terms of temperature sensitivity. At the price point you are using, the board manufacturer cannot afford to screen sensors for temperature sensitivity.

@taileron

This comment has been minimized.

Copy link

taileron commented Dec 29, 2017

@priseborough thanks, learned more about sensors, the old R12 board seems to have the better selected ones ... how was this done in V 1.6.5 flew it after few calibrations several months successful in very different temperatures. Now (R14 / PX4 1.7.2) "sensors status" shows if the board is very cold in the first 60 seconds a higher third value of acc 0 is there a parameter to delay the sensor check or repeat it after boot to have more time for e.g. closing a magnetic locked canopy ?

@taileron

This comment has been minimized.

Copy link

taileron commented Jan 1, 2018

is PX4 able to read a temperature from one of the sensors anywhere on the board?
which (nsh) command can show it?

@LorenzMeier

This comment has been minimized.

Copy link
Member

LorenzMeier commented Jan 1, 2018

If you have a recent board with enough flash this would do it:

screen shot 2018-01-01 at 15 24 32

@taileron

This comment has been minimized.

Copy link

taileron commented Jan 5, 2018

... works great but the baro seems to show a more reliable temperature (listener sensor_baro) tested with an infrared thermometer pointed to the baro

@priseborough

This comment has been minimized.

Copy link
Contributor

priseborough commented Jan 8, 2018

For effect on sensor calibration it is the internal temperature, not the surface temperature as measured by an IR thermometer that is relevant. The internal temp will be hotter due to thermal resistance of the case and the difference depends the sensor type and also its mode of operation.

@taileron

This comment has been minimized.

Copy link

taileron commented Jan 8, 2018

... Baro is only 7° (26° absolute) above it´s surface Mag 16° (35°) above surface (19°) directly measured few sec after start and with an additional 20mm cooling fan it remains stable after a few minutes.
Pixracer metal case and without fan the mag sensor rises very fast above 55°

@priseborough

This comment has been minimized.

Copy link
Contributor

priseborough commented Jan 8, 2018

Individual sensors can be disabled via the CAL_..._EN parameters, the check level is appropriate for safe operation and @taileron was able to resolve the problem by calibrating in the flight configuration.

Sensor comparison data can be logged by setting SDLOG_MODE = 1 and setting bit position 6 in SDLOG_PROFILE to 1

This issue can be closed, however I will raise a new one because our documentation for the setup procedure is not providing the following information:

  1. Magnetometer calibration should be done with usb disconnected and the vehicle in a flight configuration.
  2. Raw comparison data for all sensors can be logged by setting SDLOG_MODE = 1 and SDLOG_PROFILE = 64
  3. Sensors with poor performance can be disabled using the CAL_..._EN parameter
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment