Confusion among multiple CIRCUITPY drives

[dhalbert]

Originally posted by @garberw in #7409

I have this problem on raspbian 12 bookworm Linux weathers 6.12.28-v8+ #1877 SMP PREEMPT Wed May 14 12:07:03 BST 2025 aarch64 GNU/Linux
lots of weird errors trying to mount 5 picos simultaneously on a Pi4b. I have done the erase_filesystem and installed the latest circuitpython and all my uuids are different and I used "fatlabel" to set different labels on each pico.
it's my special day can someone fix it please. Often the disks don't even show up in blkid although they do in "fdisk -l | grep sd". I also noticed that the raspberry pi program "pcmanfm" shows at most 4 identical copies of CIRCUITPYTHON if you have 5 picos present.

---

@garberw:

so it seems that every time you transfer a file to the pico (a) it ejects the CIRCUITPY drive (b) it starts running from scratch (software reboot) (c) it expects something like pcmanfm to automatically remount it. I have 15 files in my program. So I rsync them to the pico and it gets confused. Also I disabled pcmanfm so when it ejects it does not get remounted. Also some unix process writes to the CIRCUITPY sporadically ejecting the drive. So when not programming (writing to CIRCUITPY) I prefer to leave it mounted read-only. Add to that the fact that it is better to have only one pico attached at a time. As the original poster said "it's the doctor joke if it hurts stop doing it" so instead of leaving all the picos mounted read write and rsyncing them one after another, I rewrote all my scripts. Now they only leave one pico mounted at a time. Note the whole project would not work if the /dev/ttypico? were not working simultaneously. Then I would really be in trouble. Since picos are $15 it's reasonable to expect people to use lots of them at once.

for udev I use

# ACTION=="add",   SUBSYSTEMS=="usb",KERNEL=="ttyACM[0-9]*"                           ,ATTRS{idVendor}=="239a",ATTRS{idProduct}=="8120",ATTRS{serial}=="E66141040383362E",OWNER="garberw",GROUP="garberw",MODE="0600",SYMLINK+="ttypicoi"
# ACTION=="add",   SUBSYSTEMS=="usb",SUBSYSTEM=="block",ENV{ID_FS_USAGE}=="filesystem",ATTRS{idVendor}=="239a",ATTRS{idProduct}=="8120",ATTRS{serial}=="E66141040383362E",SYMLINK+="picoi", RUN+="/usr/bin/logger --tag rpi-pico-mount mounting   pico",RUN+="/usr/bin/mount $devnode /media/picoi"
# ACTION=="remove",SUBSYSTEMS=="usb",SUBSYSTEM=="block",ENV{ID_FS_USAGE}=="filesystem",ATTRS{idVendor}=="239a",ATTRS{idProduct}=="8120",ATTRS{serial}=="E66141040383362E",RUN+="/usr/bin/logger --tag rpi-pico-mount unmounting pico",RUN+="/usr/bin/systemd-umount /media/picoi"
SUBSYSTEM=="block", KERNEL=="sd[a-z]1", ATTRS{idVendor}=="239a", ATTRS{idProduct}=="8120", ATTRS{serial}=="E66141040383362E", OWNER="garberw", GROUP="garberw", MODE="0600", SYMLINK+="picoi"
SUBSYSTEM=="tty",                       ATTRS{idVendor}=="239a", ATTRS{idProduct}=="8120", ATTRS{serial}=="E66141040383362E", OWNER="garberw", GROUP="garberw", MODE="0600", SYMLINK+="ttypicoi"

Note You can't use the ACTION="add" version because it keeps remounting them when not wanted.

for fstab I use
/dev/picoi /media/picoi vfat ro,auto,nofail,noatime,nodiratime,uid=1000,gid=1000,fmask=0022,dmask=0022,sync 0 0
I will post the scripts I use if someone explains how please. Looks like I have to upload them to github first.

---

@eightycc:

so it seems that every time you transfer a file to the pico (a) it ejects the CIRCUITPY drive (b) it starts running from scratch (software reboot) (c) it expects something like pcmanfm to automatically remount it.

You can disable this behavior with:

import supervisor
supervisor.runtime.autoreload = False

---

@garberw:

Yes but sometimes the pico locks up e.g. spewing output to tty and CIRCUITPY (AND the tty maybe ????) may become unaccessible. If this behavior is default (autoreload NOT disabled) I use it as follows. I have a script "zip.sh" which mounts the CIRCUITPY then immediately copies hello_world.py to code.py. Otherwise if you wait too long (not immediately) you can no longer access CIRCUITPY. Then it continues running hello world and is back to normal.

Maybe somehow e.g. using the debug probe or a second raspi you could redirect the tty output so it was not on the same usb connection as the CIRCUITPY. But then it would just crash the usb connection of whatever it was outputting to.

[garberw]

I just did "apt-get update; apt-get upgrade" it was kernel 6.12 and now it has a couple new upgrades (same kernel I think).

All the picos do not even appear in blkid output probably because there are 5 of them. About three of them are there usually, but they come and go.

So you have to physically unplug them or something like that so that only one or two or three of them are there at a time ????

[garberw]

as far as I am concerned for practical purposes it is totally unusable with more than one pico attached at a time. there were so many errors. everything failed. not worth describing.

[garberw]

so you have to wait a few seconds for the disk to appear in blkid then mount. that helps. will test with only four or fewer picos at a time. You probably have to physically unplug the fifth.

[garberw]

there are two awg16 wires going from the raspi 4b server to the pico ground and run pins. There is one pair of wires for each pico.
I use these to do a hard reset. There is a 4.7k pull up on the run pin of each pico. The wires are 10 ft long.
Someone suggested that this wire was losing voltage. The run pin resets the pico when it goes low so normally they have to have
a high voltage (at the end of the wire at the run pin). If there is too much current in the wire it supposedly could sporadically reset the pico.

I don't think that's what's going on but I have some sheilded awg16 wire so I will cut a shorter length and see if that fixes it.
Will get back to you asap. Apologize if this is all a red herring but I doubt it.

I have to have some way to hard reset the pico ... unless the thing worked perfectly in the first place.

[samblenny]

The wires are 10 ft long.

If you have things physically spread out enough that there are multiple 10 foot wires involved, you could potentially have some pretty serious noise on your power and signal lines from sources including RF, inductance, ground loops, etc. If your equipment includes wired connections between devices that are powered from different wall outlets (or otherwise not from one computer's USB), there's a possibility you might have currents running between different ground lines or other spicy analog stuff.

If you need to spread things out that far, you might want to consider a different communication method with better inherent noise rejection than USB. Maybe CAN, 1-wire, Ethernet, WiFi, etc.

I've accidentally done the ground current thing before with a combination of test gear and CircuitPython boards connected to two different computers, and it caused some really weird USB behavior. It's been a while so I'm fuzzy on details, but the symptoms may have included stuff kinda like what you've described here. In my case, the main culprit seemed to be a USB power monitor that I'm guessing probably used a low-side shunt resistor.

[garberw]

I would swear you were pulling my leg. I removed the 10 ft wires and replaced them with 4 ft sheilded wires. The grounding of the sheilding foil was not connected because I am to wiped out to do it. The black and red wires were. I tried it with the surge suppressor for the server and the surge for the picos plugged into different and then the same outlet (well two plugs on the same
gfi bathroom outlet). Never in any of these cases did I get anything in blkid. I unplugged all but one power plug for the picos and
they all started working off the power on the run pin and they all posted in blkid but were in bootsel mode.

So I gave up and completely removed the wires. This is what most people use; just usb for data and nothing on run pin.

Then I consistently get oscillating between 3 and 4 picos showing up in the output of blkid. Which combination of picos changed every 1 or 2 seconds. So I don't see how THIS last bit specifically could have much to do with my wiring.

So next I will observe it with only 1 then 2 ... 4 picos

[garberw]

now it is stable with 5 picos appearing in blkid and /dev/ttypico[ilrtw];
the ones that were oscillating appearing and not showing up were simply boot looping; they crash as soon as software reset;
the only way (other than reflashing .uf2) I know of resetting the boot looping is the zip.sh script which mounts CIRCUITPY then immediately copies hello_world.py to code.py. The immediate copy is the key thing. This helps a lot.

How about a mandatory pause before software reset long enough so you can mount, and wipe out the old code that's locking it up.
watchdog works if locked up for too long but if resets too fast that's what's happening here.

So since my attempt at hard reset over wire was a failure how can you do that? See here
raspi forum
any improvements over that?

[samblenny]

Ouch. That must be pretty frustrating. If that were my project, I would be very suspicious of possible hardware damage to some or all of the RP2040 chips on those boards. It's possible for chips that get exposed to conditions outside their rated electrical limits to begin having intermittent faults rather than a catastrophic failure. That may not be what's happening, but it's likely enough that it would probably be worth your while to try an experiment to confirm that theory or to rule it out.

If I were trying to solve something like this in one of my projects, I would:

1. Quarantine all 5 of the possibly damaged boards in a well labeled box
2. Get a set of brand new Pico boards (and possibly a new known good Pi 4)
3. Hook the known-good Picos up to the Pi 4 using short USB cables with no other wired connections (to avoid possibly damaging the new Picos with problematic wiring)
4. See what happens with USB drives showing up on the Pi 4

If you still get the same strange USB stack behavior from known-good hardware, it's possible there ia a problem with CircuitPython USB (might also be insufficient supply for Pi 4 or USB hub, something about udev or the kernel, etc). On the other hand, if the new picos work fine, then it would be more likely that your original set of boards got damaged by a wiring problem.

If you want help with selecting a distributed power and data setup that will work for what you're trying to do, you might try Adafruit's Discord, maybe the #help-with-projects channel. If you post pictures there of your wiring setup, somebody might be able to help you identify possible problems.

[samblenny]

One other thing... For RP2040 boards, you can trigger the built in bootloader by holding down the BOOTSEL button while plugging in the USB cable. If you do that, it should show up as a usb mass storage volume labeled RPI-RP2, and you can copy whatever UF2 file you want onto that volume.

This learn guide explains how to do that, including how to use the flash_nuke.uf2 rom image if you want to totally erase the flash:
https://learn.adafruit.com/getting-started-with-raspberry-pi-pico-circuitpython/circuitpython

If the boot looping is caused by corrupted data in flash, the flash_nuke UF2 might fix it. If you do the flash nuke thing then flash a CircuitPython UF2, but the boot looping still happens, the board might be damaged.

[samblenny]

Just re-read that Pi forum thread you linked... If I understand correctly, you encountered the problem described in this issue after using multiple Pico boards connected to a Pi 4 with:

1. USB cables that had been modified to cut the +5V wires
2. Pi Picos that were powered through their Vsys pins by an mpm3610 converter in series with a diode, fed from a wall adapter or battery
3. RS485 Hats connected to some or all of the Picos
4. Custom wiring from the Pi 4 to the Picos to enable manual power cycling through the RUN pins

If that's the case, there are many potential issues that I would suspect as maybe more likely than a CircuitPython USB stack bug.

I'm not sure what you're building, but I wonder if you might have a far easier time using ESP32-S3 Feather boards where you'd have WiFi and built in support for battery power. If you're wanting to build a distributed data collection system, Adafruit IO over WiFi is really convenient. It's a pretty easy way to get started with minimal wiring.

[garberw]

That's right. I will be getting rid of the rs485 hat. There was only one. The output of the mpm3610 went to the V+ bar at the top of the perma proto board. There was a possible issue that the sensors were powered directlly from the V+ bar at the top of the perma proto board and the V+ bar was connected to a diode the output of which went to the Vsys to power the pico. So the sensors were at a different voltage level than the pico by the diode voltage drop. I figured the voltage regulator in the pico would make this irrelevant. I will next rewire it so the pico is the same (powered by the mpm3610 after the diode) but the sensors (the rest of the circuit) are powered directly by the 3.3V out of the pico. Tricky resoldering.
thank you for all those interesting ideas. I don't think any hardware is "fried" yet though. My next move will be to try to eliminate any possible ground loops by instead of using a wire directly from the Pi4b to the respective picos, setting up a "controller pico" for each pico which is connected by usb to the Pi4b. Each respective control pico will have a reset wire to the run pin of the "target" pico.

This is the same idea as a debug probe but custom. I could even add a connection for a tty from the target pico to the control pico.

[garberw]

Also there is an adafruit circuitpython "workflow" to connect to the picos in a web browser which I have tried successfully but you have to make the pico read only ... then you can send files to the pico over wifi and even see the tty output. The ESP32-S3 feather is my next choice should I give up on the picos.

[garberw]

circuit diagrams

sorry circuit diagrams are sloppy I was in a hurry and have very shaky hands -- can't do CAD