Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Board crashes when 12V is dropped after servo_init #7

Closed
trickeydan opened this issue Jun 9, 2019 · 6 comments · Fixed by #15
Closed

Board crashes when 12V is dropped after servo_init #7

trickeydan opened this issue Jun 9, 2019 · 6 comments · Fixed by #15

Comments

@trickeydan
Copy link
Contributor

This has caused me a lot of issues.

This seems to be due to the i2c being polled from a timer. When the 12V drops, the IO expander will stop responding, causing an error.

This could be resolved by:

a) Disabling the timer when 12V drops.
b) Adding a command to disable the timer.
@PeterJCLaw
Copy link
Member

What is the useful scenario which could happen after 12V power has been lost? I appreciate that there could be a clearer handling of the error state for development (the only scenario where this issue has been reported), however from some perspectives crashing out feels like a reasonable response to the board becoming unusable.

@trickeydan
Copy link
Contributor Author

What is the useful scenario which could happen after 12V power has been lost? I appreciate that there could be a clearer handling of the error state for development (the only scenario where this issue has been reported), however from some perspectives crashing out feels like a reasonable response to the board becoming unusable.

How about running code again without restarting the kit? It crashes the USB bus.

@PeterJCLaw
Copy link
Member

Ah, crashing the whole USB bus isn't good. Even if we decide that the board entering a crashed state after power loss is ok, allowing that to affect other components clearly isn't.

In terms of re-running other code without restarting the kit, if we fix the crashing of the USB bus then this would become a non-issue as the new code would want to reset and re-initialise the peripheral boards anyway, which seems like it would void this issue?

Since the Servo Board draws its 12V power from the user-controllable outputs on the power board, I assume that also means that this state is one which competitors could get themselves into? It's interesting therefore that we haven't ever (to my knowledge) had anyone report issues from this.
This (competitors getting themselves into bad states) seems to me the most compelling reason to have a non-crashy handling of power loss.

What effect would disabling the timer have on the (normal) operation of the board?

@trickeydan
Copy link
Contributor Author

this would become a non-issue as the new code would want to reset and re-initialise the peripheral boards anyway, which seems like it would void this issue?

Doesn't work. See https://pastebin.com/KQWjXRZe for the errors that occur.

Since the Servo Board draws its 12V power from the user-controllable outputs on the power board, I assume that also means that this state is one which competitors could get themselves into? It's interesting therefore that we haven't ever (to my knowledge) had anyone report issues from this.

I'm really surprised to be honest as this issue is so prevalent during our testing of this board before Smallpeice. I guess this may be due to them usually turning off their robot between code runs.

Kier has observed this issue with me also, and we think that it is essential that it is fixed before the summer school this year.

@PeterJCLaw
Copy link
Member

I wonder whether the fact that the current SR stack gets pretty close to a reboot [1] even in the code-reload scenario anyway contributed to the lack of reports of this?

[1] As the overlay being used lives on the USB stick, removing the USB stick ends up killing the whole lot. When you reinsert the USB stick we end up pretty much fully booting again.

kierdavis added a commit to sourcebots/runusb that referenced this issue Aug 5, 2019
See inline comment.
kierdavis added a commit to sourcebots/runusb that referenced this issue Aug 5, 2019
See inline comment.
@trickeydan
Copy link
Contributor Author

trickeydan commented Aug 6, 2019

I now understand the SR stack futher.

As it pretty much does a whole reboot between every run, the usb lines are reinitialised, as in they power on and off. The side effect of this is that it gets around this issue.

For sbot, we have had to add an explicit workaround as our kit does not reboot every time.

The error otherwise occurs on every single run. Please ask me if you would like a demonstration of this.

kierdavis added a commit that referenced this issue Mar 8, 2020
…responsive

Fixes #7.

If 12V power is lost after servo_init has been called, no acknowledgements
will be received to I2C transmissions. In the current implementation
this would result in an infinite loop waiting for an acknowledgement
that never arrives. Typically this occurs during the timer interrupt
handler, causing the microcontroller to become entirely unresponsive.
This manifests itself as the PLOD (Pink LED Of Death).

This commit adds a timeout to all I2C status checks; if the timeout expires,
it is deemed that the I2C slave is dead and the timer interrupt is
disabled (until the next time servo_init is triggered).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants