RP2350-E9 Erratum: can get "stuck" at 2V state even without pulldown

[dhalbert]

I'm debugging a CircuitPython issue in which we see input pins stuck at slightly above 2V, even without all the conditions described in the RP2350-E9 erratum. I reproduced the problem in MicroPython as well.
adafruit/circuitpython#9541
https://github.com/orgs/micropython/discussions/15621#discussioncomment-10446747

The basic conditions are:

• A pin is set up for GPIO input, with no internal pulls enabled.
• Unconnected, the pin initially reads low.
• If the pin is jumpered high temporarily, and then the jumper is removed, the pin reads high, and remains stuck at some voltage above 2V, as measured with a voltmeter.
• Then connecting the pin to ground with a 1M resistor and waiting for any charge to dissipate does not help. The pin remains stuck at the same voltage and reads as high.
• Jumpering the pin directly to ground clears the "stuck" state and the pin reads low.
• The process can be repeated.

For example, in CircuitPython:

Adafruit CircuitPython 9.2.0-alpha.2351-3-g16b6c8877e-dirty on 2024-08-25; Pimoroni Tiny 2350 with rp2350a
>>> import board, digitalio
>>> gp0 = digitalio.DigitalInOut(board.GP0)
>>> gp0.pull = None       # pullups should already be disabled, but make sure
>>> gp0.value  # initial state - pin unconnected
False
>>> gp0.value  # connect to 3.3V with a 1M resistor
True
>>> gp0.value  # Remove resistor: pin reads 2.3V on voltmeter
True
>>> gp0.value  # connect via same resistor to ground: pin still reads 2.3V even after many seconds: no bleed-off
True
>>> gp0.value  # Connect directly to ground with a jumper
False
>>> gp0.value  # Remove jumper
False

I can reproduce the same kind of behavior in MicroPython.

I am at a loss to explain this. It seems like a more general problem than RP2350-E9. I've added some extra code to call gpio_disable_pulls() even before calling gpio_init() on the pin, to make sure we are not getting into the E9 state. Am I misunderstanding something about how to deal with pin setup?

The next step would be to reproduce this in pico-sdk, and I'll try that tomorrow.

[dhalbert]

Here is a very simple pico-sdk reproducer.
Tested with pico-sdk 2.0.0.
arm-none-eabi-gcc is 13.2.rel1.

cmake -DPICO_BOARD=pimoroni_tiny2350 ..

CMakeFiles.txt

cmake_minimum_required(VERSION 3.13...3.27)

# initialize the SDK based on PICO_SDK_PATH
# note: this must happen before project()
include(pico_sdk_import.cmake)

project(my_project)

# initialize the Raspberry Pi Pico SDK
pico_sdk_init()

add_executable(e9
        e9.c
        )

# pull in common dependencies
target_link_libraries(e9 pico_stdlib)

pico_enable_stdio_usb(e9 1)
pico_enable_stdio_uart(e9 0)

# create map/bin/hex file etc.
pico_add_extra_outputs(e9)

e9.c

#include <stdio.h>
#include "pico/stdlib.h"
#include "hardware/gpio.h"

#define PIN 0

int main() {
    stdio_init_all();
    sleep_ms(2000);

    gpio_disable_pulls(PIN);
    gpio_init(PIN);
    gpio_disable_pulls(PIN);  // make sure no pulls

    // More reliable reproduction with high drive strength.
    gpio_set_drive_strength(PIN, GPIO_DRIVE_STRENGTH_12MA);


    int count = 0;
    while (true) {
        printf("%d: pin is %d\n", count, gpio_get(PIN));
        sleep_ms(1000);
        count++;
    }
}

Output:

[pin is unconnected; voltmeter on pin reads 0v]
1: pin is 0
2: pin is 0
...
9: pin is 0
10: pin is 0
[1M resistor jumpered between pin and 3.3V]
11: pin is 1
12: pin is 1
13: pin is 1
14: pin is 1
[1M resistor removed; voltmeter reads 2.3V]
15: pin is 1
16: pin is 1
17: pin is 1
18: pin is 1
[1M resistor jumpered between pin and ground; voltmeter reading still 2.3V and does not change over many seconds]
19: pin is 1
20: pin is 1
21: pin is 1
22: pin is 1
23: pin is 1
...
[wire jumpered between pin and ground, so pin pulled to ground hard; voltmeter goes to 0v]
59: pin is 0
60: pin is 0
61: pin is 0
62: pin is 0
[wire jumper removed]
63: pin is 0
64: pin is 0

[dhalbert]

@Gadgetoid for interest

[waiweng83]

@dhalbert from our testing, an external pull-down resistor lower than ~9K ohm is needed to make it low again.

[DangerousPrototypes]

Can confirm. I want to do a bit more digging, but the pull-down does not seem to be required for latch up.

The latest Bus Pirate firmware has a bug command that can reproduce the E9 bug without the pull-down ever being active.

9K external pull-down probably isn't a solution for us.

[DangerousPrototypes]

I extended my bug replicating command to try four tests:

DIO> bug e9
Replicate bug E9

Test 1:
Pull-down disabled...
Disabling Bus Pirate pull-ups
Making IO0 buffer and GPIO input
Making IO0 buffer an output
GPIO pin should be 0: 0
Making IO0 buffer and GPIO input
Enabling Bus Pirate pull-ups
Making IO0 buffer an output
Disabling Bus Pirate pull-ups
GPIO pin should be 0: 1
GPIO is 1, E9 found

gpio_disable_pulls is run before testing for E9. E9 is cleared, and then found without gpio pin pull-down enabled.

Test 2:
Set pulls disabled...
Disabling Bus Pirate pull-ups
Making IO0 buffer and GPIO input
Making IO0 buffer an output
GPIO pin should be 0: 0
Making IO0 buffer and GPIO input
Enabling Bus Pirate pull-ups
Making IO0 buffer an output
Disabling Bus Pirate pull-ups
GPIO pin should be 0: 1
GPIO is 1, E9 found

gpio_set_pulls(bio2bufiopin[BIO0], false, false); is run. E9 is cleared and then detected without gpio pin pull-down enabled.

Test 3:
Disabling Bus Pirate pull-ups
Making IO0 buffer and GPIO input
Making IO0 buffer an output
GPIO pin should be 0: 0
Making IO0 buffer and GPIO input
Enabling Bus Pirate pull-ups
Making IO0 buffer an output
Disabling Bus Pirate pull-ups
GPIO pin should be 0: 1
GPIO is 1, E9 found
GPIO.IE = false...
GPIO pin should be 0: 0

Replicate E9, then disable GPIO.IE. When GPIO.IE is disabled, E9 is no longer detected.

Test 4:
Strong low test...
Disabling Bus Pirate pull-ups
Making IO0 buffer and GPIO input
Making IO0 buffer an output
GPIO pin should be 0: 0
Making IO0 buffer and GPIO input
Making IO0 buffer an output
Disabling Bus Pirate pull-ups
GPIO pin should be 0: 0

Hold pin strong low, DO NOT expose to a high level voltage (eg don't latch up), then let float. E9 does not occur if the pin is not exposed to a high level voltage (obviously, but just to demonstrate).

Source code for the bug command.

I get the vibe that a software fix is not possible, but has there been anything more from Rpi on this issue?

[DangerousPrototypes]

Went to visit Matt Mets at Blinkinlabs. He soldered 4.7K 0201 resistors on top of our existing 100K pull-down array. This appears to be enough (at least for this chip) to counteract the latch up.

DIO> bug e9
Replicate bug E9

Test 1:
Pull-down disabled...
Disabling Bus Pirate pull-ups
Making IO0 buffer and GPIO input
Making IO0 buffer an output
GPIO pin should be 0: 0
Making IO0 buffer and GPIO input
Enabling Bus Pirate pull-ups
Making IO0 buffer an output
Disabling Bus Pirate pull-ups
GPIO pin should be 0: 0

Test 2:
Set pulls disabled...
Disabling Bus Pirate pull-ups
Making IO0 buffer and GPIO input
Making IO0 buffer an output
GPIO pin should be 0: 0
Making IO0 buffer and GPIO input
Enabling Bus Pirate pull-ups
Making IO0 buffer an output
Disabling Bus Pirate pull-ups
GPIO pin should be 0: 0

[calumapplepie]

I don't think this is a true issue; if a pin has no pull-down, it is in a high impedance state. The reading from it should really be considered undefined. It is likely another view of the same problem as in RP2350-E9; its just that the erratum doesn't mention it, because reading high-impedance inputs is poorly defined anyways.

The software fix is to keep the IE register disabled, and perform the IE on -> read pin -> IE off dance for every read, if high-impedance inputs or internal pull-downs might be involved.

[todbot]

I don't think this is a true issue

Respectfully disagree. High-impedance inputs are common in multi-drop networks, capacitive touch sensing, and low-power button inputs. I currently have several Pico-based projects that cannot use Pico2/RP2350 because of this problem.

[gemarcano]

@todbot beat me to it, I also think this is a valid issue, especially given that at least some (if not all?) the tests in this report include external pull-down resistors and still exhibit the bug. From the first comment, they reproduced the issue while the pin was connected to a 1 Mohm resistor to ground. Others have stated that an external resistors no larger than ~8kohm is needed to make sure the pin is actually pulled down once a high level is removed.

Personally, if this bug (extension to E9?) is confirmed, it's somewhat alarming because most people around me use 10kohm resistors as standard jelly-bean pull-up and pull-down resistors, and it sounds like an external 10kOhm pull-down wouldn't be able to pull down the input pin's level to ground. I guess it's something that can be "fixed" with very loud documentation stating to use external sub 7kohm resistors? But then that might run afoul of use cases like what @todbot mentioned.

[barawn]

to add on to @gemarcano and @todbot , if the pin is reading 2.15V, that means that the Pico is driving current (possibly sinking, too? did anyone try a 1M pullup to 3.3V?) onto the line in that state, when it's an input. That is 100% an issue - it's not just about what the Pico's reading, it's also about what it's doing to the rest of the circuit. ~0.2 mA isn't always a trivial amount of current.

[JamesH65]

We know this is an issue, after all, we published an errata about it. We are still investigating, but so far the errata is complete and accurate - we will rephrase some of it to clarify things, and provide more detail asap.

[DangerousPrototypes]

I don't think this is a true issue

It's a huge issue for us, and we have halted production of rp2350 boards.

so far the errata is complete and accurate - we will rephrase some of it to clarify things, and provide more detail asap

It's accurate in that the small subset of "when pull-down is enabled it latches up" is true. But it is not complete in the much more expansive scope of "any pin will less than >8K pull-down latches up".

I sent Bus Pirates to Raspberry Pi, you can load the most recent firmware and run the bug e9 command to see a repeatable demonstration of the bug.

[NNNILabs]

I don't think this is a true issue

I'd like to add another data point to prove how it might be a problem in certain use cases, if I may.

I'm using PIO to communicate with an MCP3201 ADC via SPI. The ADC does not set the data out pin till the second clock cycle in the transaction, leaving it floating between conversions. Because of the latching issue, PIO interprets the ~2V as two leading 1s in the conversion, which would not happen normally. (Note: in the above image, the ADC is outputting all ones after a single null bit)

Here's another image of the same ADC being read by the RP2040:

Of course, the simple workaround in my case was to simply mask the result using 0xFFF, but other use cases might not have an obvious (and feasible) software or hardware solution.

[kevinjwalters]

The Pi Pico 2 (like other non-W Pico boards) has GPIO24 sensing VBUS using a 5.6k/10k resistor pair. Will this issue have any effect on that?

GPIO24 monitors the existence of VBUS, while R10 and R1 act to pull VBUS down to make sure it is 0V if VBUS is not
present

[robert-rozee]

to add on to @gemarcano and @todbot , if the pin is reading 2.15V, that means that the Pico is driving current (possibly sinking, too? did anyone try a 1M pullup to 3.3V?) onto the line in that state, when it's an input. That is 100% an issue - it's not just about what the Pico's reading, it's also about what it's doing to the rest of the circuit. ~0.2 mA isn't always a trivial amount of current.

has ANYONE checked if there is any change in current consumption of the RP2350 device when a pin enters this 'latch-up' state? while the pin itself may only be able to source 200uA before unlatching, whatever mechanism is holding the pin at 2.1v may well be consuming far more - or not. this is something rather important to confirm either way.

and what happens if ALL 48 of the GPIO pins on a RP2350B (80-pin) device end up in the 'latch-up' state? does the 10mA of unlatching current flow through a single internal node? and again, what is the effect of ALL GPIO pins sitting in 'latch-up' on the devices current consumption? testing should be conducted at normal CPU clock rates, as well as at the minimum and possibly also while in sleep modes.

cheers,
rob :-)

[todbot]

Until more is known, I recommend the "Getting Started with Micropython" site & book linked from https://datasheets.raspberrypi.com/pico/getting-started-with-pico.pdf have a note that certain sections do not work with Pico2.

Specifically, https://projects.raspberrypi.org/en/projects/getting-started-with-the-pico/6
and pages 53-67 of https://hackspace.raspberrypi.com/books/micropython-pico

[mordae]

This looks relevant: https://community.element14.com/products/raspberry-pi/f/forum/55046/rp2350-gpio-pull-down-latching-bug/223658

Driving an input pin with a sine wave (input impedance of 10kΩ) results in the following behavior (measured at pin):

This is with the Input buffer enabled and Pull Up/Down Disabled. As they correctly mention, pin leakage current is obviously wrong and the pins configured as inputs are obviously not always high-impedance.

[dhalbert]

The RP2350 datasheet has been updated and the RP2350-E9 erratum has been extensively revised: https://datasheets.raspberrypi.com/rp2350/rp2350-datasheet.pdf#page=1342 (printed page number is 1341). The erratum title is now "Increased leakage current on Bank 0 GPIO when pad input is enabled".

[dhalbert]

• I have opened pico-sdk support for RP2350-E9 workarounds? pico-sdk#1914 to ask about pico-sdk support.

[lurch]

This has now been thoroughly documented in RP2350-E9 in https://datasheets.raspberrypi.com/rp2350/rp2350-datasheet.pdf and this issue seems to have gone quiet, so I guess this can be closed now?