IO lose connection after a FMU warm reboot

[KeyneZzz]

Recently I met the problem that px4io would fail after a warm reboot sometimes,turned out that the pwm setting would fail to load, but not very often, about once in 300 to 400 reboots.

The problem happened with Pixhawk2.1 and firmware v1.8.2. I assume that the reboot causes some problem, but I can only find the px4io driver sending a reboot-into-bootloader procedure, but not any with normal reboot. I want to know how the normal reboot command was sent and how px4io firmware deal with a normal reboot, so I can find some clue about the problem.

[julianoes]

Interesting, I have heard something similar recently.

I'm not sure what you mean with the reboot in bootloader or not, I don't understand why it would matter for the problem you're seeing.

Do you have any console output for the case when it happens? And how did you make it happen? Did you try it 300 times 😄 ?

[KeyneZzz]

I'm not sure what you mean with the reboot in bootloader or not, I don't understand why it would matter for the problem you're seeing.

https://github.com/PX4/Firmware/blob/master/src/modules/px4iofirmware/registers.c#L648 to L663
I mean this part. The command indicates that it's used when a px4io firmware update is needed. But how it would process when we only asked for a reboot? Or the reboot doesn't care about whether a firmware update is applied, and always process with the same command? During my trials, I reduced the wait time in schedule_reboot(100000); to zero, but it turns out that doesen't affect normal reboots.

Did you try it 300 times 😄 ?

Yes, with mavlink command. It's just an estimated average, sometime happend after around 200 reboots and sometime after around 400 reboots,not sure.

[julianoes]

https://github.com/PX4/Firmware/blob/master/src/modules/px4iofirmware/registers.c#L648
I mean this part. The command indicates that it's used when a px4io firmware update is needed.

Interesting. I'd have to check.

Yes, with mavlink command.

Nice, ok.

[julianoes]

Ok, and how did you verify that px4io was ok? Just trying to reproduce your setup...

[KeyneZzz]

Ok, and how did you verify that px4io was ok? Just trying to reproduce your setup...

When it was not ok, it would generate logs like this

and no px4io in device list:

during the boot, the buzzer would play px4io programming tune （tune 18） and then error tune（tune 20）

[julianoes]

So one thing to note is that the px4io firmware does not reboot when you reboot fmu.

However, the fact that sometimes the driver doesn't correctly reconnect to the IO board is odd, and it's something I have heard from someone else as well.

[KeyneZzz]

So one thing to note is that the px4io firmware does not reboot when you reboot fmu.

However, the fact that sometimes the driver doesn't correctly reconnect to the IO board is odd, and it's something I have heard from someone else as well.

Yep. Now that the problem always happen after warm reboot, I think it necessary to figure out how the reboot process, and there might be some step with uncertainty which can cause the problem.

[KeyneZzz]

I saw px4io led blinking when fmu reboots previously, thought it rebooting, but turned out to be just a signal of losing fum input.

But once the problem happened, the following reboots won't make any change to the px4io led, which just keep constantly on. So the px4io firmware might still work, and the status page keeps unchanged even if the fum is rebooting. Should be something wrong with the reconnection.

[julianoes]

I saw px4io led blinking when fmu reboots previously, thought it rebooting, but turned out to be just a signal of losing fum input.

Correct.

So the px4io firmware might still work, and the status page keeps unchanged even if the fum is rebooting.

Maybe the px4io firmware crashes in that case. Just a guess.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. Thank you for your contributions.

[julianoes]

@KeyneZzz did you get to the bottom of this?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. Thank you for your contributions.

[MikeCharikov]

It looks like we reproduced this issue on 1.9 version and Cube 2.1

~1200 reboots and 8 problems with restart with similar symptoms

Will try to get more info soon.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. Thank you for your contributions.