agent runbook: get `bash autoresearch esp32p4 --parlio` working end-to-end on the attached EV board

[zackees]

Goal

bash autoresearch esp32p4 --parlio runs to completion on the attached ESP32-P4-EV (currently COM25, VID 0x303A PID 0x1001) and prints a normal autoresearch test summary. Today it fails before any test runs. This issue is only about the autoresearch feature — not general compile, not the perf analysis. The success bar is autoresearch driving the device over its JSON-RPC protocol and reporting PARLIO pass/fail.

How to work this issue (agent instructions)

This is a meta-tracking issue. As you make progress, the goal is not to do everything in one shot; it's to discover blockers, file them as new GitHub issues cross-referenced back here, fix them one at a time, and tick them off the checklist below.

Rules of engagement:

1. Each distinct autoresearch blocker → a new GitHub issue. Don't pile fixes into one PR or commit. If you discover that, say, the fbuild daemon doesn't propagate PLATFORMIO_SRC_DIR, file fbuild: daemon ignores PLATFORMIO_SRC_DIR env var from client (autoresearch can't pick sketch) as its own issue, link it as a child of this one ("Tracked in #..."), and fix it on its own branch + PR.
2. Stay in scope. Anything orthogonal to autoresearch (e.g. perf optimization opportunities you spot in the trace data, refactoring ideas, general compile-time issues unrelated to autoresearch's build path) belongs in its own issue — file it, link it, and keep moving. This issue is only the autoresearch happy-path.
3. Touch the right repo. Code fixes go to whichever repo owns the bug:
   • Build/toolchain → https://github.com/FastLED/fbuild
   • Autoresearch driver / FastLED firmware / CI wrapper scripts → https://github.com/FastLED/FastLED
4. Use the local fbuild checkout at C:/Users/niteris/dev/fbuild (already cloned) for fbuild work. Always branch off main, never push to main.
5. Verify every fix end-to-end on the connected hardware via autoresearch before closing a sub-issue. "Compiles" is not enough; "bash compile esp32p4 --examples AutoResearch succeeds" is not enough; the sub-issue is only done when bash autoresearch esp32p4 --parlio gets further than it did before the fix.
6. Append progress here as comments, one per sub-issue closed. Include the autoresearch summary output if available.
7. Don't disturb in-progress local edits. If the fbuild clone has uncommitted changes, git status it first, decide whether they're throw-away formatter churn (drop) or real work-in-progress (stash and restore after).
8. Stop signal: bash autoresearch esp32p4 --parlio --skip-lint exits 0 with at least one PARLIO test reported in the streaming output.

Known blockers (do these in order, file sub-issues as you confirm each)

1. ⏳ fbuild PR #273 needs to merge + ship in a release

• PR: fix(esp32): strip Windows UNC prefix + collect .S/.s assembly sources (FastLED #2507) fbuild#273 (fix/esp32p4-include-paths) — fixes the Windows \?\ UNC prefix mangling and the .S/.s assembly source drop. Without this, autoresearch fails at the build phase before it can even attempt to upload.
• Already verified locally: with that PR's binaries dropped into .venv/Scripts/, bash compile esp32p4 --examples AutoResearch --no-filter succeeds.
• Action: chase the review, get it merged, cut a new fbuild release (likely 2.2.7), then bump FastLED's pyproject.toml fbuild pin to that version. Mirror the pattern from commits 7ae55c0 / adab3f8. Don't sub-issue this one — it's the unblock for everything below.

2. 🟥 fbuild daemon ignores PLATFORMIO_SRC_DIR — autoresearch compiles the wrong sketch

• Symptom: bash autoresearch esp32p4 --parlio sets os.environ[\"PLATFORMIO_SRC_DIR\"] = examples/AutoResearch in ci/autoresearch/phases.py:564, but the long-lived fbuild daemon inherits its env from whoever spawned it first. Result: the daemon compiles examples/Sailboat (platformio.ini default src_dir) instead of AutoResearch, hits unrelated build errors, and autoresearch never reaches the RPC handshake phase.
• Reproducer:

  bash autoresearch esp32p4 --parlio --skip-lint --timeout 60
  

  Confirm by checking the build error output references examples/Sailboat/... not examples/AutoResearch/....
• Two valid fixes:
  a. Pass src_dir through the daemon protocol (preferred — explicit > env-var inheritance).
  b. Kill + respawn the daemon at the start of each autoresearch invocation so it picks up the current env.
• File this as a new fbuild issue before fixing. Title suggestion: fbuild daemon doesn't pick up PLATFORMIO_SRC_DIR from client process env. Confirm whether the fix lives in the daemon protocol or the FastLED-side autoresearch wrapper.

3. 🟥 USB-Serial-JTAG produces no host-visible bytes post-boot — autoresearch RPC can't connect

• Symptom: COM25 enumerates and esptool can flash, but once the AutoResearch sketch boots, no bytes ever reach the host. The autoresearch driver's RPC client times out waiting for the device's REMOTE: responses, and the run aborts with No response with ID 1 within Xs.
• This is the autoresearch-relevant slice of fbuild: bash autoresearch esp32p4 --parlio fails to build (WString.h / soc_caps.h missing, zccache hang) #2507 — fix it here only insofar as autoresearch needs it. (If you find a way to make autoresearch work over a different transport — BLE, flash-dump, anything — that's also a valid fix for this issue. The goal is autoresearch, not specifically serial.)
• What's already been tried (don't repeat verbatim):
  • ARDUINO_USB_MODE=0 alone
  • ARDUINO_USB_MODE=0 + ARDUINO_USB_CDC_ON_BOOT=1 (currently committed)
  • ARDUINO_USB_MODE=1 + ARDUINO_USB_CDC_ON_BOOT=1 (no OTG port enumerates — only one USB cable)
  • 3 s / 5 s / 10 s delays before first Serial.println
  • DTR/RTS variants, dtr=None, rts=None no-touch open
  • Reopening port after expected re-enumeration window
• New angles to try (not yet exhausted):
  • Serial.setTxTimeoutMs(0) so HWCDC writes don't block when no host is reading
  • Drive the JTAG endpoint directly via usb_serial_jtag_driver_install + usb_serial_jtag_write_bytes from ESP-IDF instead of going through Arduino's Serial
  • Try pio device monitor --port COM25 — PIO's monitor may handle the JTAG handshake differently than raw pyserial / the autoresearch RPC client
  • Compare against a minimal Serial.println(\"hi\") sketch (rule out our app being the variable)
  • Investigate whether autoresearch's BLE transport (AutoResearchBle) is a viable alternate path on esp32p4 — the AutoResearch sketch already wires it up
  • On Windows, swap the USB driver via Zadig and try a different CDC-class driver
• File this as a new FastLED issue scoped to "autoresearch can't talk to esp32p4-EV over the dev-board's USB connector." Title suggestion: autoresearch: ESP32-P4-EV RPC client times out — no host-visible Serial bytes after boot. Cross-ref PR debug(parlio): #2493 reproducer + FL_PARLIO_PROFILE markers #2494, issue fbuild: bash autoresearch esp32p4 --parlio fails to build (WString.h / soc_caps.h missing, zccache hang) #2507.

Reference state at the time of filing

• Connected device: ESP32-P4-EV on COM25 (VID 0x303A PID 0x1001)
• FastLED branch with the perf-trace framework and the AutoResearch RPCs (setPerfTraceEnabled, getPerfTraceLog): feat/perf-trace (PR feat(perf-trace): thread-local push/pop perf tracing + PARLIO sweep instrumentation #2508)
• fbuild branch with the include-path fix: fix/esp32p4-include-paths (fix(esp32): strip Windows UNC prefix + collect .S/.s assembly sources (FastLED #2507) fbuild#273)
• Local fbuild binaries (dropped into C:/Users/niteris/dev/fastled7/.venv/Scripts/) already include the fix, so the build half works today — only the autoresearch wrapper + serial RPC remain.

Done definition

• bash autoresearch esp32p4 --parlio --skip-lint exits 0
• At least one PARLIO test reported in the streaming autoresearch output
• Autoresearch summary line posted as a comment here
• All sub-issues filed by the agent have either landed PRs or have a clear "won't fix" note linked back to this issue

[zackees]

Cross-reference: the current complete P4/PARLIO missing-output report is now tracked separately in #2513. This runbook remains the AutoResearch path to reproduce/verify it; the latest local run builds/deploys but still dies at serial/RPC before it can execute a PARLIO output test.

[zackees]

Update after the fbuild release and PARLIO fix:

• fbuild v2.2.7 is released: https://github.com/FastLED/fbuild/releases/tag/v2.2.7
• FastLED pins fbuild==2.2.7 on master via chore(fbuild): bump pin to 2.2.7 #2516
• ESP32-P4 PARLIO no-output root cause is fixed on master via fix(esp32p4): route legacy clockless controllers through PARLIO #2515 and documented on bug(esp32p4): PARLIO produces no LED output on recent master #2513
• bash compile esp32p4 --examples AutoResearch --no-filter succeeds on master dc586bf368 using the fbuild backend

Remaining AutoResearch blocker from a hardware run:

bash autoresearch esp32p4 --parlio --skip-lint --timeout 20

Deploy succeeds with fbuild, but RPC never answers ping. The serial stream repeatedly reports invalid header: 0x47550ff7, then HP_SYS_HP_WDT_RESET, followed by No response with ID 1 within 10.0s. This prevents AutoResearch from reaching the PARLIO test phase, so it should be treated as the remaining ESP32-P4 serial/RPC validation blocker after the PARLIO driver-selection fix.

[zackees]

clud for @zackees — root cause for "blocker 3 / no host-visible bytes" found, and it was not primarily a USB-Serial-JTAG console problem. The app was never booting.

Filed as FastLED/fbuild#278.

What was actually happening on the attached ESP32-P4-EV (COM25, eco2 silicon, rev v1.3)

bash autoresearch esp32p4 --parlio now builds and flashes via fbuild (PR fbuild#273 unblocked the build). But the RPC ping times out. Investigation (raw serial capture + flash dump + esptool) shows two fbuild bugs in the P4 target:

1. Wrong bootloader offset. fbuild flashes the 2nd-stage bootloader at 0x0; the ESP32-P4 ROM loads it from 0x2000. The ROM finds garbage at 0x2000 and reboot-loops printing invalid header: 0x47550ff7 (the bytes f7 0f 55 47 at 0x2000). Those invalid header lines were the device's own ROM output — the host RPC client just echoed them as [async-serial]. Flash dump confirms: valid 0xE9 bootloader at 0x0, partition table at 0x8000, app at 0x10000, garbage at 0x2000.

2. eco5-linked bootloader/app on eco2 chip. After manually re-flashing fbuild's bootloader to the correct 0x2000, the ROM executes it and immediately panics: Guru Meditation Error: Core 0 panic'ed (Illegal instruction) with PC == entry (0x4ffac2c0). fbuild's esp32p4.json links only esp32p4.rom.eco5.* scripts; the chip is esp32p4-eco2. Control test: the stock PlatformIO/Arduino bootloader flashed to 0x2000 boots normally on the same board (prints ESP-IDF logs), and then rejects fbuild's eco5-linked app with Segment 5 ... overlaps bootloader stack → No bootable app partitions.

So "no host-visible bytes post-boot" in the original report was the app never starting — the device was stuck in the ROM invalid header loop the whole time. The ARDUINO_USB_MODE=1 console-routing concern is real but secondary; it only matters once a bootable app exists.

Implications for the checklist

• The serial/RPC path is not the first blocker — fbuild#278 (bootloader offset + eco revision) is. bash autoresearch esp32p4 --parlio can't reach the RPC handshake until fbuild produces an eco2-bootable image.
• Suggested fbuild fixes in Arduino Zero / MKR1000 : first byte of clockless pixel data not sent #278: bootloader offset 0x0 → 0x2000; select ROM .ld by detected chip revision (eco2), link the app to match.

Device state note

The board is currently left with the PlatformIO bootloader at 0x2000 + fbuild's (incompatible) app at 0x10000, so it reboot-loops. Re-running bash autoresearch/fbuild deploy restores fbuild's (also non-booting) layout. A clean boot needs the fbuild#278 fix or a stock-toolchain build.

Context

This investigation was unblocking hardware validation of the PARLIO show-timing fix in #2519 (umbrella #2518). That fix's memory/time reductions remain validated by the flash-backed timing sketch already posted in #2518; live autoresearch validation is gated on fbuild#278.

[zackees]

Closing as superseded. The fbuild fixes shipped, P4 PARLIO perf work landed (#2553/#2555/#2557/#2559/#2564/#2565), and remaining sub-items have their own tickets (e.g. #2541 Serial routing, #2540 FL_PRINT, #2542 runSimdTests gating). Use those for follow-up rather than this umbrella.