Plane: Camera feedback causing internal error 0x2000000

[Naterater]

Bug report

Issue details

Recently upgraded from 4.0.6 to 4.1.0Beta (downloaded 2021-07-17). Survey mission now can cause an internal error. It seems to already be reported: https://discuss.ardupilot.org/t/internal-error-0x2000000/70858

Note that this configuration was working properly before. As previously reported, changing RELAY DEFAULT from 0>1 MAY work, and I can test tomorrow. It might be coincidence though.

Version
Plane 4.1.0 Beta

Platform
[ ] All
[ ] AntennaTracker
[ ] Copter
[X] Plane
[ ] Rover
[ ] Submarine

Airframe type
Electric twin

Hardware type
Cube Orange, Sony A7RIV with multiport triggering and hotshoe feedback.

Logs
https://1drv.ms/u/s!AunBK4rE1ZDChJJOj2yiDsBtE049JQ?e=BRa3ur
https://1drv.ms/u/s!AunBK4rE1ZDChJJPa1S6QlIjao-pdQ?e=meH8sl

[tridge]

The key message in the log is this one:
2021-07-18 07:16:41.039 ISR flood on pin 53
that means you do have a hardware problem, as you got more than 10k interrupts on that pin in 100ms. It probably means you are missing a pulldown resistor on the pin, or the resistor needs to have a smaller value.

[Naterater]

Why would this behavior not be a problem prior to this beta firmware test? This system has been flying in this exact same configuration for quite a while without the error.

Furthermore, this isn’t happening on every photo capture, and it’s not capable of generating 100kHz. It’s quite a simple feedback system that’s been working for years.

[tridge]

Why would this behavior not be a problem prior to this beta firmware test? This system has been flying in this exact same configuration for quite a while without the error.

I don't know. It could be that it was close to the edge and the EMI has changed slightly with slight changes in PIDs, but that is just a wild guess.

Furthermore, this isn’t happening on every photo capture, and it’s not capable of generating 100kHz. It’s quite a simple feedback system that’s been working for years.

this is why you'd need a scope trace. I'm pretty sure you did get 100kHz for a short time. That can happen if the shift from a 0 to a 1 doesn't happen really cleanly or if there is a lot of noise from an ESC or some signal wire (eg. i2c).
Do you have a resistor to ground?

[Naterater]

I believe there is a resistor to ground, and the normal voltage is 3.5ish, upon the drop, I do not see much if any noise with the system as flown. The wires are generally isolated from interference from neighboring cables. The motors are 30+cm away, and the I2C cable is running a few cm away and it is shielded & grounded.

This is a zoom in near the zero voltage line in an attempt to highlight the noise or lackthereof.

Should my next test be 4.0.9 to confirm it still works without this error?

[tridge]

Should my next test be 4.0.9 to confirm it still works without this error?

I don't think that will really help. The condition is clearly quite rare as it took 25 mins of flying to happen with the log you sent me.
If you want to check it out properly I suggest you lower the threshold from 10000 per 100ms down to 100 per 100ms, by changing this line:
https://github.com/ArduPilot/ardupilot/blob/master/libraries/AP_HAL_ChibiOS/GPIO.cpp#L458
that will make the problem much more likely to trigger. You can try this with old or new versions of the firmware.
Note that the code that detects this issue has not changed since 4.0.7 when it was introduced.

[Naterater]

Now that you say 4.0.7, I’m pretty certain that the prior firmware was actually 4.0.6. I’m assuming this code was added to avoid issues. Would you be able to message me with a plane build to test?

[Naterater]

Also we have this with every flight now, usually within the first 30 minutes.

[tridge]

I’m assuming this code was added to avoid issues.

yes, hard lockup and the plane falls from the sky as the MCU is spending all it's time processing constant interrupts. That is why we have a quota. If you get more than 10k in 0.1s on a single pin then we shutdown that pin handler. I chose that threshold as it is around the level that it starts to have a significant impact on total CPU load.

Would you be able to message me with a plane build to test?

sure. Here is 4.1.0beta4 with the GPIO threshold set to 1000 interrupts/sec (100 interrupts in 0.1s)

http://uav.tridgell.net/tmp/plane-4.1.0-beta4-GPIO-100-CubeOrange.apj

that should make the issue easier to reproduce

[IamPete1]

@Naterater any update? Were you able to reproduce with Tridge's build?

[IamPete1]

@Naterater any update?