Flashing a Dongle

Flashing a dongle

You have three paths for putting new firmware on the dongle, all of which use the same esptool underneath:

CLI on a dev box — devices/utils/build_dongle.sh + devices/utils/flash_dongle.py. Same shape as flash_receiver.py — self-bootstrapping Python venv, auto-port-detection, auto-retry on auto-reset failure.
One-button on the Pi — sudo host/run/pi/update_dongle.sh. Pulls the latest firmware from the repo, builds it on the Pi, flashes whatever's plugged in at /dev/byh_dongle. Use this when the Pi is the controller and the dongle is physically attached to it.
From the web UI — Settings → Debug → Dongle firmware update. The daemon talks to a host-side flash server which drives the same esptool. No SSH session required. This is the "operator updates the dongle from their phone" flow.

Unlike receivers, the dongle has no NODE_ID and no NVS-backed state — rfChannel, rfSystemId, and debugMode are RAM-only and re-pushed by the host on every reconnect. So every flash mode is safe with respect to dongle-side state. There's nothing analogous to receiver provisioning.

TL;DR

# Dev box: build, then flash whatever's plugged in:
devices/utils/build_dongle.sh
devices/utils/flash_dongle.py

# Pi: one-shot pull + build + flash:
sudo /opt/backyardhero/host/run/pi/update_dongle.sh

# Recovery (first-time flash, brand-new chip, corrupt partition table):
devices/utils/flash_dongle.py --full

That's the whole interface. The rest of this page is the why.

Step 1 — Build

devices/utils/build_dongle.sh

What it does:

Reads the current FW_VERSION constant out of devices/os4_dongle/os4_dongle.ino.
Runs arduino-cli compile --fqbn esp32:esp32:lolin_s2_mini.
Copies four .bin files into devices/os4_dongle/bin/:
- os4_dongle_v<N>.bin — the app partition (~340 KB), flashed at 0x10000.
- os4_dongle_v<N>.bootloader.bin — chip bootloader, flashed at 0x1000 during --full.
- os4_dongle_v<N>.partitions.bin — partition table, flashed at 0x8000 during --full.
- os4_dongle_v<N>.boot_app0.bin — OTA "next-app" pointer, flashed at 0xe000 on every flash.
Updates bin/latest.* symlinks pointing at the newest version.

Unlike build_receiver.sh, this script auto-installs its toolchain on first run:

Missing arduino-cli → fetched from Arduino's official installer, installed to /usr/local/bin (with sudo) or ~/.local/bin (without).
Missing esp32:esp32 core → arduino-cli core install.
Missing libraries (RF24, Adafruit NeoPixel, ArduinoJson) → arduino-cli lib install.

Pass --no-auto-install if you'd rather manage the toolchain yourself (typical on a CI runner with locked dependencies).

Why differs from build_receiver.sh? Historical. build_receiver.sh assumes you've already done the arduino-cli setup. build_dongle.sh was added later and inherits the lesson that "the toolchain is the worst part of the experience"; the Pi installer would otherwise need 20 lines of apt-and-arduino-cli setup just so update_dongle.sh could call into it.

Step 2 — Flash with `flash_dongle.py`

devices/utils/flash_dongle.py             # app + boot_app0 (routine)
devices/utils/flash_dongle.py --full      # bootloader + partitions + boot_app0 + app

This wraps esptool with three pieces of value-add:

a) Self-bootstrapping venv

On first run, flash_dongle.py creates devices/utils/.venv and installs pyserial + esptool from devices/utils/requirements.txt. Subsequent runs reuse the venv. No pip install step you have to remember.

If the venv goes stale (you bumped requirements.txt):

rm -rf devices/utils/.venv
devices/utils/flash_dongle.py              # re-bootstraps

b) Port discovery

The dongle's serial port can be at half a dozen different paths depending on platform and what else is plugged in. flash_dongle.py tries them in this order:

/dev/byh_dongle — the stable symlink set up by install.sh on a Pi. Always wins when present.
/dev/serial/by-id/* paths matching Espressif, ESP32, or LOLIN.
/dev/ttyACM* whose USB vendor-ID is 0x303a (Espressif).
The first detected USB-CDC port, with an interactive confirmation prompt.

Override any of that with --port:

devices/utils/flash_dongle.py --port /dev/tty.usbmodem01
devices/utils/flash_dongle.py --port COM5

Pass --yes to suppress the "no obvious dongle found — use COM4?" confirmation prompt (useful in scripts).

c) Auto-retry on auto-reset failure

The lolin_s2_mini's ESP32-S2 runs the user firmware over native USB-CDC, which means esptool's auto-reset signal sometimes doesn't engage cleanly — especially if the dongle is mid-stream sending JSON status frames. The script handles this transparently:

Attempt 1: esptool --before default_reset — the normal path.
Attempt 2: esptool --before no_reset — automatic silent retry after a 1-second pause. On the lolin_s2_mini the chip is often already sitting in a download-ready state after a "failed" default_reset, so this catches the common case without operator interaction.
Attempt 3: Manual BOOT+RESET dance, then esptool --before no_reset again. This is the fallback for an actually-stuck chip.

You'll usually only see attempt 1 succeed, or attempt 2 succeed silently with no extra prompt. The manual prompt is now rare in practice.

Manual ROM bootloader entry (if all three attempts fail)

1. Press and HOLD the BOOT button on the dongle.
2. While still holding BOOT, press and release RESET.
3. Release BOOT.
4. Press Enter at the prompt.

If that still fails: unplug and replug the dongle, close any process holding the serial port (Arduino IDE's Serial Monitor is the usual culprit), or try a different USB-C cable. Charging-only cables look identical and break flashing.

On the Pi: `update_dongle.sh`

When the Pi is the host and the dongle is connected to it, you have a one-button update path that combines everything above:

sudo /opt/backyardhero/host/run/pi/update_dongle.sh

That script runs, in order:

git pull --ff-only in /opt/backyardhero — get the latest firmware source.
devices/utils/build_dongle.sh — compile to devices/os4_dongle/bin/.
devices/utils/flash_dongle.py — flash whatever's plugged into /dev/byh_dongle.

It briefly stops byh-host.service so the bridge isn't holding the serial port during the esptool run, then restarts it after the flash. Total downtime is ~10 seconds.

Why a separate script vs. just update.sh? update.sh updates the host software (the Backyard Hero stack). update_dongle.sh updates firmware on the connected hardware. These are independent — you might want a fresh dongle on an old host, or vice versa.

From the UI: Dongle firmware update panel

Open Settings → Debug → Dongle firmware update. This panel is the "operator can update the dongle from their phone" path:

Pre-staged binaries. The panel reads whatever's in devices/os4_dongle/bin/ and shows the newest os4_dongle_v<N>.bin. If you want to flash a different version, run build_dongle.sh first (or roll back by editing the symlink). The panel currently doesn't upload its own .bin — it always uses the pre-built one already on the host.
Click "Update dongle". The Next.js handler writes a small JSON manifest into /tmp/ota_staging/<job>/ (bind-mounted from the host so the bridge process can see it).
The daemon detects the request and POSTs to the host-side flash server at http://host.docker.internal:9001/flash_dongle. The flash server is part of the tcp_serial_bridge process and runs natively on the host (because the dongle's USB device lives there, not in the container).
The flash server pauses the bridge's serial forwarder so esptool has clean access to the dongle, then forks the same _spawn_esptool helper that flash_dongle.py uses. The UI polls GET /flash_dongle/status every ~200 ms and renders the job's phase.

Phases the UI shows

Phase	What it means
`queued`	Job accepted, worker thread starting.
`connecting`	esptool is trying to reach the chip (auto-reset path).
`connecting` (again)	Silent `no_reset` retry — the chip didn't auto-reset cleanly but we're trying once more before bothering you.
`erasing`, `writing`, `verifying`	esptool's standard progress phases. The progress bar fills as the app `.bin` is uploaded.
`rebooting`	esptool ran `--after hard_reset` and the chip is coming back.
`needs_manual_reset`	Both auto-reset attempts failed. UI shows "Hold BOOT, tap RESET, release BOOT, then click Continue."
`done`	Flash succeeded. The bridge auto-reconnects to the now-rebooted dongle within 1–3 s.
`error` / `aborted`	See the error message; check the job's `log_tail` for the esptool output.

When the UI says "click Continue"

The needs_manual_reset phase only triggers when both the auto-reset and the silent no_reset retry both failed. That's the case where you really do need to physically push the BOOT+RESET buttons. After you have, click Continue — the worker thread wakes up and retries one more time with no_reset. Empirically rare on healthy hardware.

Aborting

Click Abort. The flash server kills the esptool subprocess and tears down the job. Don't abort mid-write — you'll leave the dongle with a half-written app partition and you'll need a --full reflash to recover. Wait for either connecting or needs_manual_reset to abort safely.

Bin directory layout

After a successful build:

devices/os4_dongle/bin/
├── os4_dongle_v16.bin
├── os4_dongle_v16.bootloader.bin
├── os4_dongle_v16.partitions.bin
├── os4_dongle_v16.boot_app0.bin
├── latest.bin               -> os4_dongle_v16.bin
├── latest.bootloader.bin    -> os4_dongle_v16.bootloader.bin
├── latest.partitions.bin    -> os4_dongle_v16.partitions.bin
└── latest.boot_app0.bin     -> os4_dongle_v16.boot_app0.bin

flash_dongle.py always picks the highest-versioned matching .bin. To force a specific binary use --bin. Older versions are kept on disk so you can roll back.

When to use which path

Scenario	Recommended path
You're developing on a Mac / Linux box and have the dongle plugged into your laptop.	CLI: `build_dongle.sh && flash_dongle.py`
You're running on a Pi controller, SSH'd in, want to upgrade the dongle's firmware to whatever the upstream repo currently says.	`sudo update_dongle.sh`
You're a non-technical operator at the launch site, the Pi is the controller, and you got a "please update the dongle firmware to v17" instruction.	UI: Settings → Debug → Dongle firmware update
First-time flash on a brand-new chip you just soldered, or recovery after a partition-table corruption.	CLI: `flash_dongle.py --full`
Dongle is firmly bricked and won't respond to anything.	CLI: `flash_dongle.py --full` after manually mashing BOOT+RESET to force ROM bootloader entry.

Step-by-step: from-source flash on a fresh box

git clone https://github.com/os4-ivmb/backyardhero_pyro.git
cd backyardhero_pyro
devices/utils/build_dongle.sh        # auto-installs arduino-cli + core + libs
ls devices/os4_dongle/bin/           # should see os4_dongle_v<N>.{bin,bootloader.bin,...}
# Plug the dongle in.
devices/utils/flash_dongle.py        # auto-picks the port, auto-retries on reset failure

If you're on a Pi with the installer already run, you can skip the toolchain bootstrap — update_dongle.sh calls all of the above and handles the bridge-stop / bridge-start dance for you.

Confirming the flash worked

After a successful flash the dongle reboots and starts emitting status frames again. On macOS / Linux:

screen /dev/byh_dongle 115200
# or:
screen /dev/tty.usbmodem01 115200

You should see, in the first few lines:

Initializing Radio...
M+ (RF24 Master Hub Online v16: ACK-payloads)

Then, once a second:

{"timestamp":12345,"q":0,"qmax":128,"fw":16,"ch":85,...}

fw is the firmware version that just got flashed. If it matches the os4_dongle_v<N>.bin you built, you're good.

Quit screen with Ctrl-A then k. (Or you may need to release the port if byh-host is supposed to use it next — sudo systemctl restart byh-host does that cleanly.)

You can also verify from the web UI: the Status bar shows the dongle's firmware version next to the protocol indicator.

Troubleshooting

"esptool: Failed to connect to ESP32-S2: No serial data received" — three attempts in a row

You're in the rare manual-reset case. Hold BOOT, tap RESET, release BOOT, then retry. If flash_dongle.py already prompted you for this, just press Enter.

If the manual sequence also fails:

Unplug and replug the dongle to recycle the USB endpoint and re-trigger ROM bootloader mode.
Close anything holding the serial port — Arduino IDE's Serial Monitor, a previous screen session, the host's tcp_serial_bridge (sudo systemctl stop byh-host on the Pi).
Try a different USB-C cable. Some are power-only and won't carry data; they look identical.
Try a different USB port. On Pi 5 specifically, the USB 3 ports (top pair) are sometimes problematic with USB-CDC devices. Move to USB 2 (bottom pair).

"dongle_flash_start: could not reach bridge at http://host.docker.internal:9001"

The host-side flash server isn't reachable from inside the container. This is the bridge process — same one that does TCP forwarding on port 9000. Common causes:

On a Pi, the bridge isn't running. sudo systemctl status byh-host should be active. If not, sudo systemctl restart byh-host.
In dev mode, you started start-dev.sh but its bridge subprocess crashed. Ctrl-C and rerun it; check the bridge output at the top of the terminal.
On macOS / Windows Docker Desktop, the flash server is reachable via Docker's host.docker.internal proxy and should always work. If it doesn't, restart Docker Desktop.

The flash server binds 0.0.0.0:9001 (not just 127.0.0.1) specifically so it's reachable from inside the container via the host.docker.internal:host-gateway mapping. If you've blocked port 9001 in your host firewall, unblock it for the docker bridge interface.

The flash succeeds but the bridge keeps logging "auto-reconnect attempt N failed"

The dongle rebooted (correctly) after the flash but is now stuck in a startup state the bridge isn't expecting. Wait 5–10 seconds — the bridge backs off exponentially up to 5 s between retries. If it's still failing after 30 s, the new firmware likely has a bug that prevents the dongle from finishing init. Roll back:

# Find an older version:
ls devices/os4_dongle/bin/*.bin

# Flash it explicitly:
devices/utils/flash_dongle.py --bin devices/os4_dongle/bin/os4_dongle_v15.bin

The build script claims a library is "already installed" but `arduino-cli compile` still complains

You probably have two arduino-cli config directories — one from brew install arduino-cli and another from build_dongle.sh's auto-install. Pick one and stick with it:

arduino-cli config dump | grep directories

Make sure the directories listed match what build_dongle.sh is using. Usually the fix is unset ARDUINO_CLI_CONFIG_DIR and just deleting the duplicate config.

"fw" version in the dongle status doesn't match what I just flashed

You flashed the wrong binary. flash_dongle.py picks the highest version in bin/ by default. If you wanted to flash an older version, pass --bin <path> explicitly. To roll the latest.* symlinks back, run:

cd devices/os4_dongle/bin
ln -sf os4_dongle_v15.bin latest.bin
ln -sf os4_dongle_v15.boot_app0.bin latest.boot_app0.bin
# (other latest.* are only used for --full)

Flashing a Dongle

Flashing a dongle

TL;DR

Step 1 — Build

Step 2 — Flash with flash_dongle.py

a) Self-bootstrapping venv

b) Port discovery

c) Auto-retry on auto-reset failure

Manual ROM bootloader entry (if all three attempts fail)

On the Pi: update_dongle.sh

From the UI: Dongle firmware update panel

Phases the UI shows

When the UI says "click Continue"

Aborting

Bin directory layout

When to use which path

Step-by-step: from-source flash on a fresh box

Confirming the flash worked

Troubleshooting

"esptool: Failed to connect to ESP32-S2: No serial data received" — three attempts in a row

"dongle_flash_start: could not reach bridge at http://host.docker.internal:9001"

The flash succeeds but the bridge keeps logging "auto-reconnect attempt N failed"

The build script claims a library is "already installed" but arduino-cli compile still complains

"fw" version in the dongle status doesn't match what I just flashed

See also

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Step 2 — Flash with `flash_dongle.py`

On the Pi: `update_dongle.sh`

The build script claims a library is "already installed" but `arduino-cli compile` still complains