feat(library-selection): #218 add bench-fastled-examples Phase 7 warm-cache harness

[zackees]

Summary

Adds a real-FastLED-matrix bench under bench/fastled-examples/ (new workspace-member crate fbuild-bench-fastled-examples), complementing the synthetic per-crate resolve_warm from #216. For a curated subset of FastLED examples (Blink, Pacifica, Animartrix, Audio, BlurBenchmark, Chromancer), runs resolve_cached cold + warm against a fresh KvStore and reports timings. Asserts from_cache=true on warm.

CI lane added in bench-205.yml as workflow_dispatch-only — it needs a FastLED checkout step. No automatic threshold gate yet (capture cross-runner baseline first).

Local numbers

Windows / Ryzen workstation, release build, FastLED master:

example	cold (ms)	warm (ms)	speedup
Blink	923.58	11.36	81.3x
Pacifica	915.98	12.64	72.4x
Animartrix	970.14	11.76	82.5x
Audio	830.51	11.74	70.7x
BlurBenchmark	827.46	10.48	79.0x
Chromancer	844.13	10.89	77.5x

Warm path comfortably clears AC#5 (≤ +50 ms over current fbuild) at ~11 ms per example.

Test plan

• uv run soldr cargo build -p fbuild-bench-fastled-examples clean
• uv run soldr cargo run --release -p fbuild-bench-fastled-examples produces the table above
• uv run soldr cargo clippy -p fbuild-bench-fastled-examples --all-targets -- -D warnings clean
• CI lane (fastled-examples) runs green via workflow_dispatch

Closes #218. Refs #205 Phase 7 (AC#5).

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • New benchmark tool measures library resolution performance for FastLED examples, tracking both cold and warm cache behavior with timing metrics.

• Documentation

  • Updated benchmark documentation with harness specifications, execution commands, sample results, and CI workflow behavior.

[coderabbitai]

Warning

Rate limit exceeded

@zackees has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 33 minutes and 14 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: d6c7c5b6-ee3a-4333-b1f9-2bbdb0045e12

📥 Commits

Reviewing files that changed from the base of the PR and between 6e513a1 and 42ba403.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (7)

• .github/workflows/bench-205.yml
• Cargo.toml
• bench/README.md
• bench/fastled-examples/Cargo.toml
• bench/fastled-examples/README.md
• bench/fastled-examples/src/README.md
• bench/fastled-examples/src/main.rs

📝 Walkthrough

Walkthrough

This PR implements a cold vs. warm cache benchmark harness for FastLED example sketches. It adds a new workspace crate that discovers example .ino files, measures resolver performance through empty and primed cache states, and reports timing results. CI integration enables manual workflow dispatch to run the harness against a checked-out FastLED repository.

Changes

FastLED Examples Benchmark Implementation

Layer / File(s)	Summary
Package Manifest bench/fastled-examples/Cargo.toml	New binary crate fbuild-bench-fastled-examples with target pointing to src/main.rs; depends on fbuild-cache, tempfile, serde_json, and workspace crates.
Benchmark Implementation bench/fastled-examples/src/main.rs	Binary that iterates curated FastLED examples, creates temporary synthetic projects, runs resolve_cached cold then warm against each example with a fresh KvStore, records timings/selected libraries/cache-hit status, outputs Markdown table and optional JSON report via --json flag.
Benchmark Documentation bench/fastled-examples/README.md, bench/fastled-examples/src/README.md	Describes benchmark purpose (cold/warm resolver measurement), measurement methodology, running instructions with default FASTLED_DIR=~/dev/fastled, sample timing results, curated example subset, CI behavior, and cross-references.
Workspace & CI Integration Cargo.toml, bench/README.md, .github/workflows/bench-205.yml	Adds bench/fastled-examples to workspace members; documents harness in bench/README.md; adds workflow_dispatch-only job that checks out FastLED, runs benchmark with FASTLED_DIR=external/fastled, and uploads Markdown/JSON reports as 30-day artifact.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related issues

• FastLED/fbuild#205: This PR directly implements Phase 7 (AC#5) of issue #205, which specifies the FastLED examples warm-cache benchmark harness with cold/warm measurement, optional JSON output, and CI integration.

Possibly related PRs

• FastLED/fbuild#216: Both PRs add warm-cache benchmarks exercising the same resolve_cached path using MiniFramework fixture and KvStore cache-hit measurement.
• FastLED/fbuild#212: This PR introduces the CachedSelection / from_cache API that the new benchmark directly relies on to verify cache behavior.
• FastLED/fbuild#153: Both PRs integrate soldr into GitHub Actions workflows for Cargo execution.

Poem

🐰 A benchmark born to measure cache,
Cold starts swift, warm hits are a blast—
FastLED examples dance through the test,
From cold to warm, resolver knows best!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title directly references issue #218 and clearly describes the main change: adding a bench-fastled-examples Phase 7 warm-cache harness, which matches the primary objective of implementing a harness to measure warm-cache performance on FastLED examples.
Linked Issues check	✅ Passed	The PR implements all core requirements from #218: a workspace crate fbuild-bench-fastled-examples that discovers FastLED examples, runs resolve_cached cold and warm, produces timing tables, asserts cache hits, includes updated documentation, and adds a CI workflow_dispatch lane for FastLED integration testing.
Out of Scope Changes check	✅ Passed	All changes are directly aligned with #218 scope: the new workspace member crate, CI workflow integration, README updates, and Cargo.toml workspace member addition. No unrelated changes are present.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/218-fastled-examples-bench

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/bench-205.yml:
- Around line 74-80: The workflow currently checks out FastLED using
"actions/checkout@v6" with ref: master which is mutable; replace the mutable ref
with a pinned immutable ref (a commit SHA or annotated tag) by changing the
checkout step that references repository: FastLED/FastLED and path:
external/fastled to use ref: <FASTLED_COMMIT_SHA> (or a specific tag) so the
benchmark uses a reproducible FastLED snapshot; ensure the chosen SHA is the
full commit hash for immutability.

In `@bench/fastled-examples/src/main.rs`:
- Around line 194-202: The warm-cache result from timed(resolve_cached(...)) is
being recorded but not enforced: after obtaining (warm, warm_ms) from
resolve_cached, check warm.from_cache and fail the function (return an Err or
propagate an error) when it's false to enforce the AC#5 contract; keep filling
Row as before (example/name, cold_ms, warm_ms, selected from
warm.selection.required_libraries, hit from warm.from_cache) but add a guard
that returns a clear error referencing the example name when warm.from_cache is
not true so warm cache misses do not silently pass.
- Around line 116-145: The loop over EXAMPLES currently swallows errors from
measure_example (in the Err arm) by printing "skip ..." and continuing; change
this so a failed example causes the run to fail: in the Err(err) branch inside
the loop (where measure_example is matched) replace the current
print-and-continue behavior with a failing exit/return (e.g., print the error to
stderr and call std::process::exit(1) or propagate an Err from main) so that any
Err from measure_example aborts the program instead of allowing the benchmark to
pass with missing data.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: ac15b167-4a79-466c-b629-97fa10a7a680

📥 Commits

Reviewing files that changed from the base of the PR and between 86bb2d9 and 6e513a1.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (7)

• .github/workflows/bench-205.yml
• Cargo.toml
• bench/README.md
• bench/fastled-examples/Cargo.toml
• bench/fastled-examples/README.md
• bench/fastled-examples/src/README.md
• bench/fastled-examples/src/main.rs