I2C bus fault observed when reading RF channel temperature

[hartytp]

Booster still seems to restart intermittently maybe once per hour (judging from the fan noise). This is when it's being left alone

[ryan-summers]

If you configure booster to power-on channels by default (e.g. save settings with a channel enabled), it will not enable on reboot if a watchdog occurs because of https://github.com/quartiq/booster/blob/develop/src/rf_channel.rs#L523. This is a potential case to figure out if this is a watchdog or a panic elsewhere in the code.

I'll continue running booster locally to see if I can reproduce these faults, but I'm suspicious that it may be related to more RF channels being installed.

[hartytp]

Thanks.

I have already made all channels power on by default. I can confirm that Booster stopped putting RF out for 10s or so, the lights went blank, the fans spun up and the lights + RF came back

[ryan-summers]

If the RF came back up without any user interaction, that indicates that this is likely a panic as opposed to a watchdog (which is a good thing, as it should be easier to root cause). I'll poke around

[hartytp]

👍 thanks!

[ryan-summers]

Alright, I was able to see a watchdog occur on my bench set up using the debug build when I increased loops to occur 4 times over channels (since I have only 2 installed, looping 4 times should be roughly similar to scanning 8 channels). I'll do some debugging and analysis on Monday to see what's causing this.

[hartytp]

👍 thanks

[hartytp]

@ryan-summers one other data point here...I haven't confirmed if all three Boosters I currently have display this behaviour or not. But, I've noticed that the one we're using has a red LED on one channel that seems to appear after some time (and is cleared by the crash/restart).

I haven't had a chance to look at what causes this error, but will open an issue when I do.

[ryan-summers]

The red LED is indicative of the BLOCKED channel state. This only occurs if the temperature is measured out of range (e.g. over 60 C or below 5 C) or if the overcurrent alert pin is low. It is intended to be cleared by device reset. My guess would be that it's the overcurrent alert tripping.

[hartytp]

My guess would be that it's the overcurrent alert tripping.

How does that work? Is it a software detect of the current going too high? or do we try to check if the LDO goes into foldback.

[ryan-summers]

I believe that's a software limit we program into the supply monitoring IC. I'll have to look over it monday. By all means, open a separate issue for that one as well

…

On Fri, Jan 15, 2021, 18:59 hartytp ***@***.***> wrote: My guess would be that it's the overcurrent alert tripping. How does that work? Is it a software detect of the current going too high? or do we try to check if the LDO goes into foldback. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#128 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACC5O6RQXQ4TVRFTNL5H6WTS2B7ALANCNFSM4WEJ3RBA> .

[hartytp]

@ryan-summers

If you configure booster to power-on channels by default (e.g. save settings with a channel enabled), it will not enable on reboot if a watchdog occurs because of https://github.com/quartiq/booster/blob/develop/src/rf_channel.rs#L523. This is a potential case to figure out if this is a watchdog or a panic elsewhere in the code.

Doesn't this imply that this issue is a panic and not a watchdog issue?

[ryan-summers]

Doesn't this imply that this issue is a panic and not a watchdog issue?

It very well might. I'm adding service information reporting (e.g. stored panic and watchdog detection) in #131 so we can determine if a panic or watchdog is occurring. If it is a panic, it should give us some information as to what exactly is happening.