SCD30 Stops Communicating Sometimes #61
Comments
|
It's seems possible that slowing down the I2C bus may also help! https://learn.adafruit.com/circuitpython-on-raspberrypi-linux/i2c-clock-stretching |
|
Seems like this can be set via |
|
@mschwanzer this appears to be what's happening with your sensor. I'm going to try slowing the clock speed on I2C to see if this stops it. |
|
I changed the clock speed on a few test devices and verified by running the command:
I'll watch those few to see if I can get the condition to repeat. |
|
Unfortunately after changing this setting down to 10000, I've still seen this sensor disconnection issue reproduce. |
|
@eaudiffred it looks like my software fix here did not resolve the issue with your sensor. I'm going to try a second software fix related to the clock stretching of the communication bus that seems more promising! More information : https://github.com/RequestForCoffee/rpi-i2c-timings |
|
Great, thanks! Let me know when to reboot. |
|
Will do @eaudiffred! Note for myself, the CM4 uses bcm2835 (i2c@7e804000) |
|
Ok I tried for a while to use the rpi-i2c-timing utility above, but couldn't get it to function (see RequestForCoffee/rpi-i2c-timings#1). I found this related issue which suggested using parameter force_turbo=1 in the config.txt, so I'm going to try that. |
|
@eaudiffred if you want to give your sensor another reboot, we can see if that helps things :) |
|
I’m also seeing this occasionally on my sensor (and right now actually). We should see what’s happening on the bus lines w/ a scope. I can do that tomorrow. Because I’m wondering if we’re running into a stuck/blocked I2C bus. A quick Google search shows that the i2c driver for the raspberry pi doesn’t implement any kind of i2c recovery routines, which is unfortunate because we’d have to implement our own. Sounds like fun, though :) |
|
Yes! It would be super helpful if you have a scope or a logic analyzer to
figure out exactly what is happening. I think that something like that
might be the problem as the scd30 data sheet mentions that clock stretching
for i2c needs to be supported. I can imagine that lack of clock stretching
could lead to the stuck bus.
…On Thu, Nov 11, 2021 at 9:32 PM Desmond Good ***@***.***> wrote:
I’m also seeing this occasionally on my sensor (and right now actually).
We should see what’s happening on the bus lines w/ a scope. I can do that
tomorrow. Because I’m wondering if we’re running into a stuck/blocked I2C
bus <https://www.pebblebay.com/i2c-lock-up-prevention-and-recovery/>. A
quick Google search shows that the i2c driver for the raspberry pi doesn’t
implement any kind of i2c recovery routines, which is unfortunate because
we’d have to implement our own. Sounds like a fun, though :)
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#61 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AATQ3FR3QPIGNSLEPA7KVGTULSRHTANCNFSM5G26ZKNQ>
.
|
|
Restarted this morning and moved it back outside. Just checked and unfortunately I don't see it online. I'll give it another restart when I get home from work. In the past it's taken 2 or 3 power cycles before I see the dot on the ribbit network map. |
|
Scope captures Looking at the bus while
No clock cycles so it seems like it's not even trying to clock in data. Kind of a misleading error. I disconnected the DPS310 to see if that would resolve anything, but nothing changed. After power cycling SCD-30 by disconnecting/reconnecting the qwiic connect cable:
Bus looks happy and Less critical but ideally Ribbit would have a way to recover from a stuck bus, since it's bound to crop up somehow. A potential robustness improvement, maybe. |
|
Thanks for those scope traces @djgood! So since you disconnected the SCC30, does that make it most likely that the stuck condition is within the BCM chip on the Raspberry Pi? I believe it uses the hardware i2c. I had theorized that adding a power switch to switch power to the SCD30 on and off in case of a stuck bus would resolve the issue. It seems like your testing confirms this correct? Perhaps there is a better way in software, but as you mentioned maybe the buggy bcm stuff prevents that. Sparkfun used to make a nice QWIIC power module but it's out of production it seems. Shouldn't be too hard for us to reproduce if we had to. |
|
No problem! Hm, I was thinking that it was the SCD-30 that was stuck in that clock stretching condition, which blocks the bus and the Raspberry Pi doesn't know how to detect it. So once the SCD-30 releases the bus when it's powered off the Raspberry Pi can continue driving the bus as normal. I think if we could power cycle the SCD30 (or even reset) that would fix the issue but it seems like the easiest alternative to me is to use the software I2C bus, more info here: https://github.com/fivdi/i2c-bus/blob/master/doc/raspberry-pi-software-i2c.md This is a good discussion on I2C clock stretching on the Raspberry Pis: https://raspberrypi.stackexchange.com/questions/127271/does-the-raspberry-pi-i²c-bus-support-clock-stretching |
|
Yeah makes sense. Seems like if the software i2c is the way to go, let's
try it! I believe the scd30 library can be configured to use the new pins.
There isn't a way to use the existing hardware i2c pins as software i2c
pins right? This would be awesome because we can update all the frogs in
the field without a hardware change, but might not be possible.
…On Fri, Nov 12, 2021 at 4:44 PM Desmond Good ***@***.***> wrote:
No problem! Hm, I was thinking that it was the SCD-30 that was stuck in
that clock stretching condition, which blocks the bus and the Raspberry Pi
doesn't know how to detect it. So once the SCD-30 releases the bus when
it's powered off the Raspberry Pi can continue driving the bus as normal.
I think if we could power cycle the SCD30 (or even reset) that would fix
the issue but it seems like the easiest alternative to me is to use the
software I2C bus, more info here:
https://github.com/fivdi/i2c-bus/blob/master/doc/raspberry-pi-software-i2c.md
This is a good discussion on I2C clock stretching on the Raspberry Pis:
https://raspberrypi.stackexchange.com/questions/127271/does-the-raspberry-pi-i²c-bus-support-clock-stretching
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#61 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AATQ3FWLZJBEYA5U5Y3B2RDULWYFPANCNFSM5G26ZKNQ>
.
|
|
Based on this forum post, it seems like it might be possible to use the existing hardware i2c as GPIOS using the software i2c. That would be rad. |
|
Yeah, looks like that's totally doable! Awesome! |
|
I tried testing the software i2c by adding the following, but it doesn't seem to be working. I created a balena forum post here to see if anyone else has any additional tips. |
|
Have you tried specifying different pins? Wondering if the problem is with disabling the hardware i2c or enabling the software i2c. I’ll try getting something working on my device |
|
Yes I tried the same configuration with different pins, but same thing dtparam=i2c_arm=off |
|
Per this forum post discussing the software i2c implementation, it seems like I should be using the settings below instead of my settings above. Will test shortly. |
Enabling Software I2CAll right! I was able to successfully enable the i2c-gpio interface on the same pins (2 and 3) as the hardware i2c on the raspberry pi via the following configuration This allowed to me to see the i2c and verify everything as shown below: Testing to see if it fixes the issue with I2C bus lock-upI then had to make a few changes to the source code to allow the SCD30 Python code to use the I2C bus. I first install the https://github.com/adafruit/Adafruit_Python_Extended_Bus library in order to easily create the I2C object required by the SCD30 sensor constructor. I modified the co2.py file to initialize the I2C bus as shown below: I started up the python script and I was able to connect and read data from the SCD30 and barometer just like before! Unfortunately, this did not resolve the issue and the bus locked up in the same fashion as before after a few hours. I had to reboot the power on the full system to recover. Next StepsI'm not exactly sure what to try next. I'm going to try slowing down the I2C bus a bit more via the |
|
Setting i2c_gpio_delay_us=100 which should correspond to 10kHz I2C speed |
|
Argggg! That's frustrating that it didn't solve the problem. If slowing down the bus doesn't work, another thing we can try as a last resort is to try to bit bang those GPIOs to generate a bunch clock cycles if we detect that the bus is locked up. That would hopefully to get the SCD30 to free it. Not sure how difficult that would be with our current code, maybe we can interact with the software i2c somehow? Also, that's assuming that the SCD30 isn't just completely wedged and would actually respond. |
|
I have great news! After running the test raspberry pi for over 48 hours, the i2c connection seems to be going strong with the software gpio at 10kHz, so I think I'm feeling comfortable to call this done and start rolling it out to the wider fleet. I'm going to clean up the code a bit a do the rollout next. |
|
Deployed to the Ribbit Network fleet:
Software release: c35bf3ccb713 |
As I've deployed more Frog sensors, I've noticed that occasionally some of the SCD30 sensors will stop responding. The condition is resolved after a power cycle, so I don't believe that it's anything physical (loose connector, etc), but I'm open to be proven wrong.
I haven't been able to correlate this to any particular event.
Theories for cause:
Example Error Log
The text was updated successfully, but these errors were encountered: