FUG-71: solve device heap pressure (release BLE + zero-copy envelopes + RAM tooling) - #45
Open
issuefleet[bot] wants to merge 11 commits into
Open
FUG-71: solve device heap pressure (release BLE + zero-copy envelopes + RAM tooling)#45issuefleet[bot] wants to merge 11 commits into
issuefleet[bot] wants to merge 11 commits into
Conversation
FUG-71 Solve device heap pressure
Claim in fughilli/splanc#44 is that the connection drop is due to heap exhaustion blocking TLS sessions. There are 512 KiB of RAM on the ESP32-C6. It is absurd that it is having heap/arena pressure for this complexity of an application. First, build tooling for inspecting the RAM allocation of the device. This tooling should perform static analysis on the binaries (with a new .ram_chart subtarget on the firmware binary macro; add this tooling as a branch in the embedded repo and pin as you iterate). Then, use this tooling to determine where the RAM is being spent. My hypothesis is that a lot of RAM is being spent for static allocations of micropb decode/encode buffers. This is a waste of space--we should have a zero-copy form of protocol buffer encoding/decoding which doesn't need these buffers. Build out that support as a dedicated PR and then stack the investigation on top (it's a large feature, I know. just do it). Investigate any other places where the RAM is being spent stupidly/poorly. Be ruthless! Stack PRs as necessary to make them sensible to review. Test on HITL diligently. |
|
issuefleet
Bot
changed the title
issuefleet
Bot
changed the title
issuefleet
Bot
changed the title
issuefleet
Bot
force-pushed
the
agent/fug-71-solve-device-heap-pressure
branch
2 times, most recently
from
August 7, 2026 04:56
78e5693 to
251b489
Compare
Build tooling to see where the C6's internal SRAM goes, so the device heap-pressure work is driven by data rather than guesses. - fw_memaudit.py: fix the static-RAM total (it was counting flash-mapped rodata — incl. the ~2 MB .flash_rodata_dummy reservation — as RAM, inflating the figure ~15x). Now classify sections by VMA against the HP-SRAM window, add a heap-headroom summary (static footprint vs the 512 KiB SRAM, the ceiling on runtime free heap), and a --compare mode that diffs a --json snapshot per symbol (the lever for iterating a cut). - ram_chart.bzl: a `ram_chart` macro that wraps a firmware_binary's `elf` output group in a runnable auditor, giving every image a `.ram_chart` sibling (a `bazel run` target, like the flash siblings). - Wire //firmware/player_app:esp32c6.ram_chart. - docs/design/ram-budget.md: methodology + the measured baseline (154.7 KiB static; rx 32K / FX_ARENA 24K / ARENA 16K / FX_TEX_PREV 8K dominate) + the runtime draws the static chart can't see (BLE never released, heap-allocated task stacks, mbedTLS sessions) + the roadmap. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The `.ram_chart` audit + on-device measurement showed the TLS heap exhaustion behind FUG-69's mid-solve wss drop is NOT primarily the micropb buffers — it's the Bluedroid BLE stack. Improv onboarding brings BLE up at boot and it was never torn down, holding ~43 KiB of heap permanently, even long after the device is on WiFi and BLE is pure overhead. Measured on a C6: BLE init draws 43 KiB (heap 125.7K -> 82.3K). Release it once provisioned + STA-only: - improv_ble_end(): BLEDevice::deinit(true) tears down host + controller AND hands their arenas back (esp_bt_controller_mem_release); null the characteristic pointers so the guarded set_* helpers no-op. improv_ble_begin is now idempotent, doubling as the re-arm. improv_ble_set_name no-ops while down (the name still persists in NVS + re-applies on the next begin). - main.cpp: call improv_ble_end() in the post-join demote (right where the soft-AP is already dropped), before the cert re-sign so ledmapper_selfsign runs with the reclaimed headroom. A re-onboarding watchdog re-raises soft-AP + BLE if the LAN is lost for >60 s (arduino auto-reconnect heals transients), so the device stays re-provisionable without a power cycle; a subsequent reconnect demotes + re-releases via the shared path. Verified on the HITL rig (same-DUT flash+provision): on join the device logs `[ble] released; heap 43560 -> 76736 (+33176)`, re-signs the cert with 106 KB free, and steady-states at ~82 KB free heap — up from ~48 KB (BLE resident). Full e2e (provision + wss + time-sync + rename) passes. docs/design/ram-budget.md updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The heap-allocated FreeRTOS task stacks (loop 24K, httpd_ssl 28K, render 8K) are among the biggest single heap draws we control. Log each task's stack high-water (smallest free-stack ever reached) in the periodic [player] status, so the reserved-vs-used headroom — reclaimable heap — is visible on the rig. Capture the render task handle from xTaskCreate and look up the httpd task by name. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pb::ServerMessage is a oneof sized to its largest arm, and StoredMapChunk
carries a Vec<u8, 1024> inline — so EVERY ServerMessage temporary on the
loop/httpd task stacks in lm_player_handle was ~1 KiB, even for a tiny
reply. But the player only ever PRODUCES that message (the phone decodes
it), in one place.
Encode ServerMessage{stored_map_chunk} straight into the output buffer in
handle_get_stored_map (hand-rolled, matching the existing arena/effects
walkers), decoding the GetStoredMap request the same way to avoid a
by-value ClientMessage on that path too. The firmware profile then shrinks
StoredMapChunk.data to an 8-byte stub (host keeps 1 KiB for the generated
encoder the conformance test drives), dropping ServerMessage to the Welcome
arm's ~496 B. envelope_size_test pins it so an arm can't silently re-bloat.
ffi_test + conformance pass; -c opt image builds.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SetCountingPattern.blocks is Vec<ColorBlock, 32>, and each ColorBlock carries an inline Vec<f64, 4> rgb — ~1.5 KiB, the fat arm that sized EVERY by-value pb::ClientMessage on the loop/httpd task stacks (the oneof is sized to its largest arm), even for a bare hello. Decode that arm with the hand-rolled walker instead (like the arena / effects arms): handle_set_counting_pattern walks the ColorBlock array, reduces each [0,1]-double rgb to 8-bit, and installs the blocks via a new Player::set_counting_blocks (which the generated-decode path — kept for the host session/conformance tests — now also delegates to). The firmware profile then stubs SetCountingPattern.blocks to 1, dropping ClientMessage from 1560 to 472 B (a control-frame arm is now the ceiling). Combined with the StoredMapChunk work, the by-value envelope cost on the handler frame drops ~2.6 KiB → ~1 KiB. ffi_test gains a counting-walker case (lm_counting_color reads back the reduced blocks); envelope_size_test tightened to <=560 B. session/conformance/arena tests green; -c opt builds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ploads) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The rebase onto #56 (FUG-83) removed kApSsid (now the derived g_ap_ssid) and added per-name hostnames + mDNS to the soft-AP bring-up. The re-onboarding watchdog's enter_onboarding still called WiFi.softAP(kApSsid, ...) — now mirror the initial bring-up: setHostname + softAPsetHostname + softAP(g_ap_ssid) + mdns_begin_or_update, so a LAN-loss re-provisioning surface is identical to a fresh boot's. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issuefleet
Bot
force-pushed
the
agent/fug-71-solve-device-heap-pressure
branch
from
August 9, 2026 01:27
251b489 to
fabde23
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FUG-71: solve device heap pressure
Reclaims ~47 KiB of heap so the TLS (
wss) handshake always gets its ~17 KiB contiguous block (device booted with only ~39 KiB free). Rebased onto the latestmain(now incl. #46 streamed uploads, #56 derived AP-SSID/mDNS, #58/#60 FUG-85 Android-flash + rename/wss reliability + HITL suite); wins stack cleanly on top.The reclaims
improv_ble_end()releases it at the post-provision demote; a watchdog re-arms it if the LAN is lost (mirrors FUG-83: derive AP SSID + mDNS/DHCP hostname from the configured device name #56's derived AP SSID + hostnames + mDNS).StoredMapChunk.data→ServerMessage1048 → 496 B;SetCountingPattern.blocks→ClientMessage1560 → 472 B.envelope_size_testpins them.httpd_ssl28 → 20 KiB (~14 KiB), watched by a[stack]high-water log.Tooling & test infra
.ram_chartstatic RAM auditor +docs/design/ram-budget.md.Testing (on the rebase)
ffi_test,envelope_size_test,conformance, player + store — all green.-c optimage builds (my BLE/enter_onboardingintegrates with FUG-85: Android flashing + rename/wss reliability + devices-page UX + HITL suite #60's wss-reliability); HITL full-stack e2e PASSES.Notes on this rebase
reserved_board_ble_macreads the reserved board's own serial); dropped my redundant commit and took FUG-85: Android flashing + rename/wss reliability + devices-page UX + HITL suite #60's.rxreclaim (32 → 12 KB).Follow-up (ram-budget.md)
FX buffers (~36 KiB, reclaimable during a capture) — needs a
#[global_allocator](crate is#![no_std]).🤖 Generated with Claude Code
Closes-Linear: FUG-71 (https://linear.app/fughilli/issue/FUG-71/solve-device-heap-pressure)