I2C reads always returning 0xFF (as of 2.7.4+rev1)

[ericbarch]

Although we're still debugging this one on our end, I wanted to create this issue in case anyone else has run into this.

Starting with release 2.7.4+rev1, all of our I2C reads began returning 0xFF. In 2.7.2+rev1 and prior, this behavior is not observed.

We have tested several releases since 2.7.4 (including 2.9.7+rev1) and find the same behavior. Reverting back to 2.7.2+rev1 or earlier always resolves this issue.

Note that scanning devices (via i2cdetect) behaves as expected. Even writing to the I2C device succeeds. We are performing our tests via a Debain container with privileged mode enabled on a Raspberry Pi 3.

[floion]

Hi,
can you write down a testcase for this issue please?

[ericbarch]

Of course 😀

We have been testing against an Atlas Scientific RTD module: https://www.atlas-scientific.com/product_pages/circuits/ezo_rtd.html connected to a Raspberry Pi 3.

Testing under 2.9.7+rev1 (same result w/ 2.7.4+):

balena run --rm -it --privileged resin/armv7hf-debian:stretch /bin/bash

apt update && apt install -f i2c-tools

root@7c4a31740ab4:/# i2cdetect -y 1
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- 68 -- -- -- -- -- -- --
70: -- -- -- -- -- -- -- --

root@7c4a31740ab4:/# i2cget -y 1 104 0 b
0xff

Testing under 2.7.2+rev1:

docker run --rm -it --privileged resin/armv7hf-debian:stretch /bin/bash

apt update && apt install -f i2c-tools

root@3b5b9cf66681:/# i2cdetect -y 1
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- 68 -- -- -- -- -- -- --
70: -- -- -- -- -- -- -- --

root@3b5b9cf66681:/# i2cget -y 1 104 0 b
0x05

Note that this first register (address 0) is the device identification register. It should always return 0x05.

[ericbarch]

After performing some additional testing, it appears as though the new releases only cause issues when communicating with our Atlas Scientific I2C devices. We also tested against a BME280 sensor and did not notice any issues when upgrading to the latest resinOS.

I grabbed a few captures via our logic analyzer of the Atlas Scientific transactions.

Note that the top (BAD) test was performed on resinOS 2.12.3+rev1. The bottom (GOOD) test was performed on resinOS 2.7.2+rev1.

It appears as though the newer resinOS builds are attempting to perform a repeated start. The Atlas Scientific module we are using does not support repeated start.

In fact, if you look closely at the (BAD) analog waveforms (below the digital waveforms), you can see a slight voltage increase in SCL after SDA goes high (indicating that the Pi is attempting to perform a repeated start). Because the change is ~150mV, this would seem to indicate that the slave device is holding the clock low to prevent the repeated start.

Is it possible the underlying I2C drivers have changed to perform repeated starts? And if so, is there any way to prevent this?

[ericbarch]

We've now tracked this issue down to a change in the official RPi kernel. It appears that when the bcm2835 I2C module was made the default (with repeated starts enabled), it broke compatibility with our Atlas Scientific board. This change can be found here:

raspberrypi/linux@fe92705

Will continue to investigate on our own. But closing as this is not a resinOS specific issue.

[agherzan]

Thanks, @ericbarch for investigating this. It was hard on our side as we lack the specific hardware. Update us if we can help in any way.