Proposal: safe Wi-Fi OTA updates for Tesserae ESP32 panels

[charmmmz]

Motivation

Would you be open to adding opt-in Wi-Fi firmware updates for Tesserae ESP32 panels?

For wall-mounted panels such as the reTerminal E1004, every firmware update currently requires physically connecting USB. That is manageable during development, but becomes inconvenient once a panel is installed in its normal location.

The project already has much of the supporting infrastructure:

• firmware reports its version in the status heartbeat;
• Tesserae can show when an update is available;
• the release pipeline publishes per-board firmware and SHA-256 metadata to Cloudflare R2;
• the firmware already has HTTPS support and the ESP-IDF OTA APIs are available.

The missing pieces are an OTA-capable partition layout, an explicit update command, image verification, and safe rollback.

Suggested shape

One final USB migration

Existing firmware uses one factory application partition with no OTA slots. I would not suggest rewriting that partition table remotely.

Instead, an OTA migration release could be flashed over USB once. It would install an OTA-capable partition table with two application slots. All later releases could then be installed over Wi-Fi.

The current E1004 v1.4.0 merged factory image is around 1.27 MB, while the board configuration conservatively targets devices with at least 8 MB of flash, so two suitably sized application slots appear practical. The real flash size should still be confirmed on hardware before enabling the feature for that target.

Explicit update from the device card

I think this should be user-triggered rather than silent automatic updating:

1. Tesserae shows that a new version is available.
2. The administrator clicks Update over Wi-Fi on the device card.
3. Tesserae records a pending update for that device.
4. On the next status heartbeat, the device receives the target version, board-specific URL, digest, and signature.
5. The firmware streams the image into the inactive OTA slot.
6. It verifies the board, version, SHA-256, and release signature before switching slots.
7. After reboot, the new firmware confirms a successful boot and heartbeat.
8. If it cannot boot successfully, ESP-IDF rollback restores the previous slot.

The UI could report states such as pending, downloading, verifying, rebooting, completed, and rolled back.

Safety constraints

• Keep OTA disabled by default until the administrator explicitly opts in.
• Only offer an image matching the registered device kind.
• Refuse an update when battery power is too low, and prefer external power when detectable.
• Preserve NVS so Wi-Fi credentials and device registration survive.
• Sign release images and embed a verification public key in the firmware. HTTPS plus a digest is useful, but a signature also protects against a compromised download origin and metadata.
• Enable OTA per board only after its flash size and rollback path have been tested on hardware.
• Keep the existing Web Flasher and factory images as the recovery path.

Release pipeline

The current release pipeline publishes a complete factory image for WebSerial flashing. OTA would additionally need an app-only image plus version, board kind, size, SHA-256, and signature metadata. The factory artifact should remain unchanged for first installation and recovery.

Architecture question

The OTA image could either be downloaded directly from the existing public firmware store, with Tesserae only returning signed metadata, or proxied/cached by the local Tesserae server for installations where panels do not have direct internet access.

My initial preference would be:

• the administrator initiates the update through Tesserae;
• Tesserae supplies the approved metadata;
• the device downloads directly from the official HTTPS endpoint;
• a later server-proxy option can support isolated networks.

Would this fit the direction of the project? If so, would you prefer this to begin as an E1004-focused hardware-verified prototype, or as a generic ESP32 OTA framework enabled only for verified board targets?

I would be happy to help prototype it once the desired trust model and server/device boundary are agreed.

[dmellok]

Thank you for the detailed proposal. This fits the direction of the project, and Wi-Fi OTA was already on the roadmap, so most of what you describe is additive rather than a rework. I would be glad to take you up on the offer to prototype it. Below I have tried to settle the two open decisions you raised so there is a clear boundary to build against.

Direction

I agree with the overall shape: opt-in, administrator-triggered updates; a one-time USB migration to an OTA-capable partition table; direct-to-slot streaming with board, version, digest, and signature verification; and ESP-IDF rollback on failed boot. The existing pieces (firmware version in the heartbeat, the update-available check against api.tesserae.ink/firmware/<kind>/latest, and the R2 release pipeline) cover a useful share of the supporting work already.

Download origin and server boundary

I would keep your preferred model:

• the administrator initiates the update through Tesserae;
• Tesserae supplies approved, signed metadata (target version, board kind, R2 URL, size, SHA-256, signature);
• the device downloads the image directly from the existing Cloudflare R2 HTTPS endpoint;
• a server-proxy/cache mode for isolated networks is a later, optional addition rather than part of the first cut.

Cloudflare R2 stays exactly where it is. The only addition on the store side is an app-only OTA image alongside the current factory image, plus the metadata. firmware_check already carries the release url and per-board assets, so this extends what Tesserae serves rather than introducing a new host.

Trust model

I would treat signing as required rather than optional. HTTPS and a digest cover the transport, but a signature is what protects against a compromised origin or tampered metadata. The plan I would fix on:

• an ed25519 release keypair, with the private key held as a CI secret;
• the verification public key embedded in the firmware;
• the device verifying the signature over both the image and the metadata before it switches slots.

Nothing would ship over OTA without this in place.

Scope: generic server, per-board enablement

The server side (the pending-update record, the signed-metadata endpoint, the device-card state machine) is board-agnostic, so I would build it generically and enable OTA per board only after that board's flash size and rollback path have been verified on hardware. E1004 would be the first verified target. That gives both the E1004-focused hardware verification and the generic framework, in the layers where each belongs.

Partition migration

I agree with the one-time USB migration rather than rewriting a live partition table over the air. The one item to confirm first is the actual flash chip on real E1004 hardware; the 1.27 MB image against an 8 MB target leaves room for two application slots, but the chip itself should be confirmed before the feature is enabled for that board.

State reporting

The device can report its OTA state and progress in the normal status heartbeat, and the command descriptor can ride the existing /frame response envelope. No new channel is needed in either direction.

Suggested phasing

1. Pipeline and metadata (server only, low risk): publish the app-only image and signed metadata to R2, have api.tesserae.ink serve it, and add the signature-verification helpers. No device changes.
2. OTA migration firmware (USB, E1004): the OTA-capable partition table and rollback path.
3. The feature: the device-card "Update over Wi-Fi" action, the pending-update descriptor, the device OTA client, and the pending / downloading / verifying / rebooting / completed / rolled-back states. Enabled for E1004 only.
4. Generalisation: other verified boards, plus the optional server-proxy for air-gapped installs.

Your safety constraints (opt-in by default, kind-matched images only, battery gating with a preference for external power, NVS preservation, and keeping the Web Flasher and factory image as the recovery path) are exactly right, and I would build all of them in from the start.

Next step

The release pipeline and signing side depend on access I hold (the R2 account, the CI secrets, and the release private key), so I would keep that in-house: the app-only image, the signed-metadata endpoint, and generating and holding the keypair. I would publish the verification public key for the firmware to embed.

The firmware side is the natural fit for you, and it is where most of the engineering sits: the OTA-capable partition layout, the device OTA client, on-device image and signature verification against the embedded public key, and the rollback path.

Concretely, I would suggest:

• I take the pipeline, signed metadata, and key management (Phase 1), and the device-card UI in Phase 3;
• you take the firmware in Phase 2 and the device OTA client in Phase 3;
• we agree the exact metadata shape and signature format up front, since that is the contract both sides implement.

I can start on Phase 1 so there is real signed metadata for you to verify against early. Does the firmware side suit you?

[charmmmz]

Yes, the firmware side suits me. Thank you for laying out such a clear boundary. I am happy with the proposed split, and please go ahead with Phase 1 so we can integrate against real signed metadata and the published verification key as early as possible.

Before implementation, I would like to lock down three details in the shared contract.

1. Signed payload shape

My preference is to avoid requiring Python and C to independently reproduce canonical JSON. The response can carry:

• payload: base64url of the exact manifest bytes;
• signature: base64url Ed25519 signature over those exact payload bytes.

The decoded manifest would contain at least:

• schema version;
• key id;
• device kind;
• firmware version;
• app-only image URL;
• exact byte size;
• SHA-256 digest.

The firmware verifies the signature over the raw decoded payload first, parses it, checks kind/version/size, then streams the image while computing SHA-256. Matching the downloaded image against the digest inside the signed payload binds the signature to both the metadata and image without buffering the image in RAM.

I checked the ESP-IDF 5.5 bundle used by the firmware. Mbed TLS exposes Ed25519/EdDSA constants, but the PureEdDSA implementation path is currently marked as not implemented. Unless you have a preferred verifier, I would plan to vendor a small reviewed implementation such as Monocypher and pin it with RFC 8032 plus project-specific test vectors.

2. OTA descriptor on /frame

The existing firmware sends If-None-Match and normally receives a bodyless 304 when the displayed frame is unchanged; /frame can also return 204 before the first render exists.

If the descriptor stays on /frame, I suggest that a pending OTA command bypasses both the normal 304 and no-frame 204 paths and returns a 200 JSON envelope containing the OTA descriptor. The firmware will process OTA before fetching or painting the frame and reboot on success, so this will not cause an unnecessary E1004 repaint.

If placing the descriptor in the always-200 /status response is cleaner on the server side, that would also be straightforward for the firmware. I am comfortable with either boundary, but the 304/204 behavior needs to be explicit in the contract.

3. Rollback confirmation

My initial preference is to mark the new app valid after local startup checks succeed: the running partition is correct, NVS is readable, and Wi-Fi initialization/association succeeds. The later Tesserae heartbeat can report completed.

Requiring a successful server heartbeat before marking the app valid could roll back a healthy firmware merely because the self-hosted server is temporarily offline. If you prefer server reachability to be part of the acceptance gate, we should define a bounded retry policy so an unrelated outage does not look like a bad image.

Firmware-side plan

I will keep the work in two reviewable phases:

1. E1004 migration partition layout, NVS preservation, boot/rollback scaffolding, the embedded public key verifier, and deterministic signature test vectors.
2. The OTA descriptor parser, direct R2 streaming into the inactive slot, kind/version/size/digest/signature checks, battery gating, and OTA state reporting.

Before enabling E1004, I will confirm the real flash chip with esptool on hardware and test successful migration, interrupted download, invalid signature/digest, failed first boot, rollback, and preservation of Wi-Fi and registration data.

Once we agree on the envelope and signature bytes, I can build the firmware against fixtures while your Phase 1 pipeline is coming together.

[dmellok]

Thank you, the split works for me, and the firmware boundary is yours. The pipeline, signing keys, and R2 hosting will stay on my side. I will begin Phase 1 tomorrow so that real signed metadata and the published verification key are available for you to build against as early as possible.

I am happy with all three points, with one adjustment on where the descriptor rides.

1. Signed payload shape

Agreed as proposed. The response will carry:

• payload: base64url of the exact manifest bytes;
• signature: base64url Ed25519 signature over those exact payload bytes.

Verification is over the raw decoded bytes before parsing, so neither side reproduces canonical JSON. The decoded manifest will contain:

• schema version;
• key id;
• device kind;
• firmware version;
• app-only image URL;
• exact byte size;
• SHA-256 digest.

Monocypher pinned to RFC 8032 plus project-specific vectors is a sound choice given the mbedTLS PureEdDSA gap. With Phase 1 I will publish the verification key and a set of signed fixtures (valid, wrong key, truncated, digest mismatch) so you can build and test against them before the live pipeline exists.

2. OTA descriptor placement

My preference is to place the descriptor on the always-200 /status response rather than /frame, and I would like that to be the contract.

The reason is fleet coverage. /frame is the image channel, and every device kind polls it expecting either image bytes or a bodyless 304/204. The E1004 firmware would correctly read a 200 JSON envelope there, but the Raspberry Pi, Pico, and CircuitPython clients already in the field poll the same endpoint and would attempt to decode the JSON as an image. /status is already an always-200 JSON channel that every client tolerates, so an additional ota object is ignored by clients that do not understand it and acted upon by those that do.

To bound this cleanly I would gate delivery on a capability flag: firmware advertises OTA support at register/status, declaring the descriptor schema version it speaks, and the server emits a descriptor only to a device that has advertised support. This keeps /frame unchanged, makes OTA opt-in per device kind, and lets me add non-ESP32 updaters later without renegotiating the transport. The existing 304 and 204 behaviour on /frame stays exactly as it is today.

Descriptor shape on /status:

• ota: an object carrying payload and signature as above;
• the key is absent whenever no OTA is pending.

3. Rollback confirmation

Agreed, and local checks only. The new app is marked valid once the running partition is correct, NVS is readable, and Wi-Fi initialisation and association succeed; the later heartbeat reports completed as telemetry rather than as the acceptance gate. Requiring a successful server heartbeat before marking the app valid could roll back healthy firmware during an unrelated server outage. If we later decide server reachability should be part of the gate, we will define it as a bounded-retry option, but the default stays local.

Pipeline-side plan

I will keep Phase 1 to:

• the signing pipeline and key publication;
• the /status descriptor and the register/status capability flag;
• the app-only image hosted on R2, with size and digest bound into the signed payload;
• the signed test fixtures described above.

I will start tomorrow and post here once the verification key and fixtures are published, so you can build the firmware against them while the live pipeline comes together.

[dmellok]

Phase 1 groundwork is on main now (edge, v0.153.0), so there are signed bytes and a key to build against.

• Wire contract: docs/ota/contract.md. The descriptor is {payload, signature} with the manifest fields and the four-step check order we agreed: signature over the raw decoded bytes, then manifest shape and schema version, then device kind and version, then downloaded size and SHA-256.
• Signer and reference verifier: app/ota/. verify.py runs those checks in that order and raises a stable reason code per gate (bad_signature, malformed_descriptor, kind_mismatch, already_current, size_mismatch, digest_mismatch), which the firmware can mirror in its own log lines. sign.py has a python -m app.ota.sign CLI.
• Signed fixtures: tests/fixtures/ota/. Four descriptors (valid, wrong key, truncated, digest mismatch), a sample image, and a test-only key. The README.md there maps each fixture to its expected verdict.

Test verification key (test only; the production key is published separately): 03a107bff3ce10be1d70dd18e74bc09967e4d6309ba50d5f1ddc8664125531b8.

The signed-bytes format above is frozen, so you can build the verifier and the RFC 8032 plus project-specific vectors against these fixtures now. Still to come as the next slice: the /status descriptor delivery and the register/status capability flag. I will post again once the production key and the live /status path are up.

Separately, a heads-up on the firmware tree so we are not both in it at once: I am going to roll #123 (a short stay-awake window after a button press, so rapid next/previous presses do not each pay a fresh wake and Wi-Fi cycle) into the firmware myself today, ahead of the OTA phases. I will flag here once it is in.

[dmellok]

The /status handshake is live in edge (v0.155.0), so you can integration-test end-to-end against a running server now, not just the fixtures.

Capability field (the TBD from earlier): a device advertises OTA support with an ota object in its /register and /status body, naming the schema it speaks:

{ "...heartbeat fields...": "...", "ota": { "schema": 1 } }

Delivery: the server hands back the signed descriptor on the always-200 /status response, and only when the device advertised a schema at least as new as the descriptor's and the descriptor targets the device's kind:

{ "...status fields...": "...", "ota": { "payload": "...", "signature": "..." } }

ota is absent otherwise, and /frame is untouched, so the image channel stays byte-clean for every kind. Re-offering an update the device already applied is harmless: your already_current check skips a descriptor naming the running version, so the server keeps offering a staged descriptor until it is cleared.

Staging (server side, for testing): sign then stage, held in data/core/ota_pending.json and read on the next heartbeat:

python -m app.ota.sign --key-id test-ed25519-1 --device-kind esp32_client \
    --fw-version 1.4.0 --image app.bin --image-url https://host/app.bin \
    --key tests/fixtures/ota/test_signing_key.hex > d.json
python -m app.ota.stage --data-root data --device-id <id> --descriptor d.json

That uses the published test key, so a firmware build carrying the test public key can verify a real descriptor delivered over the wire, with any reachable image_url. docs/ota/contract.md is updated with the handshake, delivery, and staging.

Still on my side: the production key, R2 image hosting, and OTA state reporting. None of those block you building and verifying the descriptor path against the test key now.

[dmellok]

Production signing is live and verified end to end.

Signing + hosting. A Cloudflare Worker signs descriptors with an Ed25519 key held only in a Worker secret and serves the images from R2, co-located so the image it signs is the image it hosts. Its output is byte-identical to the Python signer, so a descriptor from either path verifies the same way. I confirmed the full chain against the real key: a prod-1-signed descriptor verifies against the published key, the image downloads from its image_url, and the downloaded bytes match the signed size_bytes and sha256.

Production key to embed. key_id prod-1:

97f3294e7bf8f2dfb965ac2739e6d2d7ee346c42681a94d87c9189942db3965e

Please have the firmware trust a keyed set {key_id: pubkey} rather than a single hardcoded key. Ship prod-1 as the production anchor; you can also carry test-ed25519-1 (03a107bff3ce10be1d70dd18e74bc09967e4d6309ba50d5f1ddc8664125531b8) on non-production builds if it helps you test against staged descriptors. Because every manifest carries key_id, keys rotate without a re-flash: a future prod-2 can be added while prod-1 is still trusted, then retired once no device depends on it.

One heads-up on the image fetch. Cloudflare's edge returned 403 to a bare/automated user-agent (a Python-urllib fetch was blocked; curl was fine) for the same image URL. So please have the download set a normal User-Agent header. If a minimal ESP client UA still trips it, I will add a WAF skip for the /firmware/* path on my side; either way the device download needs to survive the edge bot filtering, worth a check on real hardware.

The /status handshake (ota: {schema: 1}) and staged delivery are unchanged and already on edge (v0.156.0). Nothing here changes the descriptor or verification contract; it only makes the descriptors real (prod-signed, R2-hosted) instead of test-key fixtures.

[dmellok]

Server-side rollout is wired now (edge v0.157.0), so the full path from a build to a device exists on my end: sign (Worker, prod-1 key) → host on R2 → stage, and now a deliberate per-kind rollout on top. An operator sets a signed build as a kind's release, offers it to a canary device or two first, then promotes it to the rest, with pause and a firmware-version gate so an already-updated device isn't re-offered.

Nothing about the wire contract changed. Delivery is still: the device advertises ota: {schema: 1} on register/status, and the server returns the signed {payload, signature} descriptor on the always-200 /status when the device is eligible. Everything else (payload shape, verification order, key_id, rollback stance) is as in docs/ota/contract.md.

One thing that turned out to be a non-issue: per-board targeting. Each hardware-catalog board (seeed_reterminal_e1001, etc.) is already a distinct device kind server-side, so a descriptor's device_kind matches the exact board a device registered as. No board-id field or coordination needed; the firmware's per-board signing lines up directly.

So the outstanding piece is the device side you own: advertise the capability, read the descriptor from /status, stream the image, verify the signature against the embedded key, apply to the inactive slot, reboot, and roll back on a bad first boot. Whenever a build of that is ready, I can stage a real prod-signed image to a canary device and we can watch one go end to end.

[charmmmz]

A quick firmware-side progress update: I have completed the capacity-independent groundwork locally, starting from the current firmware upstream/main (13164c3, v1.5.0) on a dedicated feature/e1004-wifi-ota branch.

Implemented

• Pinned Monocypher 4.0.3 and added strict unpadded base64url decoding plus Ed25519 verification over the exact decoded payload bytes before JSON parsing.
• Added a keyed trust set: prod-1 is always present; test-ed25519-1 is compiled only when an explicit non-production test flag is set.
• Bound the signed manifest key_id back to the public key that actually verified the payload, so key rotation remains explicit.
• Added the manifest parser and stable rejection reasons for malformed descriptor/manifest, unsupported schema, kind mismatch, and already-current firmware. It also validates and decodes the signed size_bytes and lowercase SHA-256 fields for the later streaming gate.
• Added the agreed ota: {schema: 1} capability to /register and /status, and descriptor handling from the always-200 /status response.
• The enabled Phase-1 path verifies and logs a valid offer but deliberately does not download or write flash yet.

Safety boundary

TESSERAE_OTA_CAPABILITY_ENABLED defaults to 0 in all shipped board environments. Therefore the normal E1004 artifact does not advertise OTA support, and the server will not offer it a descriptor. CI separately compiles the enabled protocol path without publishing that binary. I have not changed the E1004 partition table or performed any flash operation.

Verification completed

• RFC 8032 verification vector passes.
• All four published Tesserae fixtures are covered: valid, wrong key, truncated, and digest mismatch.
• Additional host checks cover malformed base64url, manifest shape/schema, exact device-kind matching, already-current firmware, trailing JSON, and key-id/key binding.
• Normal E1004 build succeeds at about 1.20 MB.
• Capability-enabled, verify-only E1004 build succeeds at about 1.22 MB.
• All eight production board environments build successfully with the new pinned dependency.

I am away from the E1004 at the moment, so the remaining hardware-dependent gate is still intentionally open. Once I am back, I will read the real flash ID/capacity over USB without erasing anything, then define the A/B layout. After that the next slice is direct R2 streaming with a normal User-Agent, inactive-slot writes, exact size/SHA-256 checks, boot validation/rollback, and OTA state telemetry. We can then use your prod-signed canary path for the first end-to-end run.

I will continue from there when I am back later and post another progress update here.

[dmellok]

The verify-only path lines up with the frozen contract, so this is the right shape for the phase. A few confirmations, and a couple of things for when you're back at the E1004.

Contract alignment

Everything here matches docs/ota/contract.md as frozen:

• signature over the raw decoded payload bytes before any parse;
• keyed trust set, prod-1 always present and test-ed25519-1 behind the non-production flag;
• key_id bound back to the key that actually verified, so rotation stays explicit;
• reason codes: your malformed / unsupported-schema / kind-mismatch / already-current set maps onto the verifier's stable codes (malformed_descriptor, kind_mismatch, already_current, then size_mismatch / digest_mismatch at the streaming gate), so device and server logs read the same way when we're debugging a canary;
• ota: {schema: 1} on /register and /status, descriptor read from the always-200 /status.

Nothing on my side has moved since v0.156.0, so what you built against is current.

Safety posture

Agreed, keep it exactly as is: TESSERAE_OTA_CAPABILITY_ENABLED=0 in every shipped board env, the enabled path compiled only in CI and never published, and no partition or flash changes until the real E1004 flash ID is read on hardware. Verify-and-log with no download or write is the correct first cut. And because the server only emits a descriptor to a device that advertised the capability, a shipped E1004 that doesn't advertise can't be offered one either, so the default-off firmware and the capability-gated server back each other up.

Firmware tree / #123

You're clear here: #123 (post-button stay-awake) is on main as fa2fb99, and since you branched from 13164c3 you already have it plus the two CI commits that sign release OTA images via the Worker and probe image_url with a ranged GET. No rebase needed, and we're not both in the tree.

The R2 download (streaming slice)

You've got the normal-User-Agent note. One extra data point: the release CI now probes image_url with a ranged GET rather than HEAD, because the edge bot filter was what 403'd bare agents. A normal UA should be enough on the device; if a minimal ESP client UA still trips it on real hardware, send me the exact UA string and I'll add a WAF skip for /firmware/* so the download always survives the filter. A ranged GET on the device would also let you resume an interrupted download into the inactive slot, which may be handy for your interrupted-download test.

Rollback

Confirmed, local checks only: running partition correct, NVS readable, Wi-Fi init and association, then mark valid; the heartbeat's completed stays telemetry, not the gate.

Canary, whenever you're ready

My end of the chain is live: Worker signing with prod-1, R2 hosting, and a per-kind rollout with canary / promote / pause and a firmware-version gate (v0.157.0). Whenever your apply path can write and boot an inactive slot, I can stage a real prod-1-signed E1004 image to a single canary and we watch one go pending → downloading → verifying → rebooting → completed for real. No rush: ping here once you're back with the board and have the flash ID and A/B layout settled, and I'll have a canary descriptor waiting.

The test matrix (RFC 8032 vector, the four fixtures, and the malformed/edge cases) is the coverage I wanted before pointing this at hardware, so I'm comfortable going straight to a canary once the apply slice lands.

[charmmmz]

The E1004 firmware apply path and the one-time hardware migration are now ready for a real canary.

Hardware result

• The production E1004 reports an ESP32-S3 rev 0.2, 8 MB PSRAM, and a 32 MB flash chip.
• The original layout was NVS at 0x9000 (24 KiB), PHY at 0xf000, and one 4 MiB factory app at 0x10000.
• The migrated layout preserves NVS and PHY at the same offsets, adds otadata at 0x10000, and uses two 4 MiB slots: ota_0 at 0x20000 and ota_1 at 0x420000.
• Secure Boot and Flash Encryption are disabled on this unit.

I backed up the partition table, NVS, and the full used flash range before writing anything. The one-time USB migration then succeeded without erase-all. A semantic NVS readback showed all 59 pre-existing keys unchanged, including Wi-Fi credentials, Tesserae server identity, token, and device id. The only additions were the normal Wi-Fi bssid and channel cache entries created after reconnecting.

The migrated device booted from ota_0, rejoined Wi-Fi, retained its registration, received a frame 304, posted battery and SHT4x telemetry successfully, and returned to its server-driven deep sleep interval.

Firmware apply path

The implementation is pushed here: 25711cc.

It now includes:

• E1004-only capability enablement with the 32 MB A/B layout and ESP-IDF rollback support;
• the frozen raw-payload Ed25519 verification flow with the keyed prod-1 trust anchor;
• descriptor parsing from the always-200 status response;
• a normal download User-Agent: Tesserae-ESP32/ (reTerminal_E1004);
• exact Content-Length and signed size enforcement before writing;
• streaming SHA-256 while writing only the inactive slot;
• ESP-IDF image validation before selecting the next boot partition;
• a conservative 3600 mV minimum battery gate, with unknown readings rejected;
• first-boot confirmation only after NVS, the network stack, Wi-Fi initialization, persisted configuration, and Wi-Fi association succeed. Server reachability is not part of the rollback gate.

The RFC 8032 and published fixture tests pass, and a clean E1004 build succeeds at about 1.24 MB. I also rebuilt and flashed the final capability-enabled baseline to the migrated unit; it is online and posting status successfully.

Ready for canary

The remaining hardware work is the real Wi-Fi transition from ota_0 to ota_1, followed by interrupted download, bad size/digest, and failed-first-boot rollback tests. I have deliberately not invented an OTA telemetry request shape beyond the frozen descriptor contract; the firmware currently emits the stable reason codes in its serial logs, and I can add heartbeat state fields once that exact server-side shape is published.

The device side is ready for a single prod-1-signed E1004 canary whenever convenient. I will keep USB serial attached so we can capture the download, slot switch, first-boot confirmation, and any edge/WAF behavior end to end.

[dmellok]

@charmmmz I pulled feature/e1004-wifi-ota (25711cc) and reviewed the full diff before we stage anything against a production key.

What I verified, with evidence:

• The embedded prod-1 public key matches ota/keys/prod-1.pub byte for byte. As a stronger check, I verified the v1.5.0 seeed_reterminal_e1004 descriptor from our release assets (produced by the real production signer) against your embedded key with an independent Ed25519 implementation; the signature validates. The trust anchor is authentic.
• Unauthenticated bytes never reach the JSON parser. The base64url decoder is strictly canonical (rejects padding, whitespace, and non-canonical unused bits), and manifest parsing rejects trailing content after the JSON object.
• The signed key_id is bound to the key that actually verified the payload, so a manifest cannot claim a key it was not signed with.
• The installer compares the streamed SHA-256 against the signed manifest hash before esp_ota_set_boot_partition, bounds the download by both the signed size_bytes and the slot size, and aborts cleanly on every failure path.
• The rollback gate is purely local (NVS, network stack, persisted config, Wi-Fi association). Server reachability correctly has no vote in boot validity.
• The partition layout keeps NVS at 0x9000, and OTA remains compile-gated off for every board except the E1004.

Three changes I would like before merge:

1. Pin Monocypher by commit hash rather than the 4.0.3 tag. Tags are mutable, and a moved tag would be a silent change of crypto implementation.
2. Set disable_auto_redirect = true in ota_install.c. A redirect to plain http would drop TLS for the image transfer. The signed digest still protects integrity, but no legitimate redirect exists in this flow, so the surface is free to remove.
3. Document that migration begins with esptool flash_id on each unit, in the partition csv comment or the migration notes. Your production unit has the 32 MB chip, but if any E1004 revision shipped with less, ota_1 would sit past the end of flash.

Proposed path to your canary:

1. Open this branch as a PR against dmellok/tesserae-device-firmware with the three items above.
2. On merge I cut a v1.6.0-rc1 pre-release. Release CI now signs every build with prod-1 through a protected environment with a required-reviewer gate, so descriptor-seeed_reterminal_e1004.json is produced automatically, with the image hosted by the signer and the provenance being a reviewed commit on the canonical repo.
3. That descriptor is your canary bundle. Interrupted-download and rollback fault testing on your bench proceeds from there, as you proposed.

This keeps the property that prod-1 only ever signs bytes traceable to reviewed source, which I think matches the contract you designed the device side around.

Thanks for the migration write-up. The 32 MB confirmation unblocked the whole layout.

[dmellok]

OTA status: validated end to end, now fleet-wide. Rolling today's updates into one.

Server side (edge 0.177.0 -> 0.180.0)

• State reporting ingested. The server reads the heartbeat ota report (phase / reason / target_fw / attempt_id / detail), stores the latest on the device's live status, and event-logs each transition (a re-sent terminal report logs once). It renders as a phase chip on the Devices card and the Firmware page. fw_version alone still works; the fields light up once firmware emits them.
• Firmware admin page, now a flat device list: one Check now, then a row per device (installed vs available) with Queue update / Withdraw. Queue fetches + verifies the published descriptor against the prod-1 anchor, sets it as the kind's release, and adds the device to the offer set in one click. The staged canary -> promote -> pause flow (confirmed-canary gate on promote, no auto-promote) moved to a collapsed "Fleet rollout" fold and stays in app.ota.release; both edit the same offer set. Wire contract untouched: descriptor delivery on /status, the version gate, and the signing chain behave exactly as before.
• Discovery: with online mode on, each device shows whether newer firmware is published (api.tesserae.ink/firmware/<kind>/latest, cached hourly, sends the reported version as ?current=); off, the check never runs and the page discloses it.

Release + a bug caught

• 1.6.0 final shipped with the OTA path, signed via CI.
• Factory-image bug (fixed in 1.6.2). The release workflow's factory-merge step hardcoded the app at 0x10000 for every board; on the E1004 A/B table that's where otadata lives, so the 1.6.0/1.6.1 E1004 factory images (USB / webflasher) were unbootable. OTA images were never affected (app-only, signing input untouched). The merge now detects OTA layouts via ota_data_initial.bin and assembles otadata at 0x10000 + app at 0x20000, verified byte-level before publish.
• Parts-addressed flashing. Images now also publish as offset-addressed parts (bootloader / partition table / otadata / app), so a browser flash writes only those regions and never touches NVS at 0x9000, credential-preserving migration works from a browser tab, not just esptool.

End-to-end hardware validation

• Full update through the per-device queue flow on a migrated E1004, watched on serial: descriptor on /status -> 1,239,984 bytes to ota_1 in ~17 s -> pending_confirm -> reboot -> rollback cancelled after local checks -> phase=confirmed on the next heartbeat. ~19 s extra awake. No server dependency in the boot-validity decision, as specced.
• 1.6.3 adds a boot-log line naming the running partition, so A/B switches are visible on serial.

Fleet-wide (1.6.4)

• OTA on all ten board targets: a shared A/B table (two 4 MiB slots, otadata 0x10000, NVS untouched 0x9000) + bootloader rollback via a shared sdkconfig. reTerminals are 32 MB, XIAO/WROOM 16 MB, one layout fits all. Fleet devices need the one-time migration flash; the browser flasher's parts support keeps credentials + registration intact.
• Build gotcha: ESP-IDF doesn't re-apply a changed SDKCONFIG_DEFAULTS chain to an already-generated sdkconfig, so a local build silently lacked rollback until the generated per-env sdkconfigs were deleted and rebuilt. CI generates fresh, so it was never exposed.

Remaining caveat

The device fetches the image from R2 directly, and Cloudflare's bot rules 403'd bare / automated user-agents on /firmware/* in earlier testing. Confirm the firmware sends a real User-Agent (or add a WAF skip for that route) before broad rollout.

---

I've cleaned up my earlier scattered comments from today into this single update. Apologies for the notification spam.

[charmmmz]

I have now exercised the new Firmware page on the real E1004 and found one small state-model/UI issue after a successful update.

Reproduction

• Server: v0.183.0
• Device: reterminal_e1004_838f04 (Seeed reTerminal E1004)
• Auto update: off
• Queued v1.7.1 from the Firmware page
• The OTA completed successfully and the device's next heartbeat persisted fw_version: 1.7.1

The row then showed:

• Installed: v1.7.1
• Available: v1.7.1
• Status: queued v1.7.1

This survives a page reload, so it is not browser caching. The persisted state explains it: ota_releases.json still has the device in canary_device_ids, while device_facts.json correctly reports v1.7.1.

There is no repeated-update risk: _released_ota() already requires the release to be strictly newer than the firmware reported by the device, so v1.7.1 is no longer offered. This looks limited to the Firmware page's effective queue state.

Root cause

In _devices_model(), queued currently means only “the device is in the release offer/canary set”:

update_available = bool(available) and (not fw or is_newer(str(available), fw))
queued = bool(v["is_canary"])

The template checks d.queued before d.update_available / up to date, so retained rollout membership wins even after the device has installed that release. can_withdraw = queued and update_available also hides Withdraw in this completed state.

I do not think the heartbeat path needs to delete the device from canary_device_ids: that set is also useful rollout state for the Advanced/Fleet view. The device-list view can instead distinguish rollout membership from a release that is actually still pending for this device.

Suggested fix

Derive the effective queue from the imported release version (the version the server can really offer), rather than from membership alone or from the possibly newer online Available version:

fw = str(v["fw_version"] or "")
update_available = bool(available) and (not fw or is_newer(str(available), fw))

release_pending = bool(rel_fw) and (
    not fw or is_newer(str(rel_fw), fw)
)
queued = bool(v["is_canary"] and release_pending)

rows.append(
    {
        **v,
        # ...
        "queued": queued,
        "queued_fw": rel_fw if queued else None,
        "available_fw": available,
        # ...
        "can_queue": bool(
            v["capable"] and update_available and not queued and offerable
        ),
        "can_withdraw": queued,
    }
)

Then render the actual queued release:

{% elif d.queued %}
<span class="pill is-accent">
  queued{% if d.queued_fw %} {{ fw(d.queued_fw) }}{% endif %}
</span>

This should give the desired outcomes:

1. selected for v1.7.1 + installed v1.7.0 → queued v1.7.1;
2. selected for v1.7.1 + installed v1.7.1 → up to date;
3. retained membership on an older imported release while online Available is newer → do not label the newer version as queued;
4. promoted releases behave the same way: devices stop displaying queued once their reported firmware reaches the release.

Suggested regression coverage would assert those four view-model cases, plus that the underlying canary membership remains available to the Fleet rollout model.

The actual E1004 update itself was clean; this is just the post-update admin-page state. Happy to put this into a small PR if that would be useful.

[dmellok]

Happy for the PR thank you for all your work.