perf-comparison autoloop: run benchmarks and commit real results.json each iteration

[Copilot]

The repo has 599 TS/Python benchmark pairs under benchmarks/, but nothing invokes benchmarks/run_benchmarks.sh, so benchmarks/results.json on main stays a stub ({ "benchmarks": [], "timestamp": null }). pages.yml copies that stub into the playground, and benchmarks.html renders "No benchmark data available yet."

Changes

All in .autoloop/programs/perf-comparison/program.md:

• Move benchmark execution into Evaluation. Evaluation always runs and its file side-effects are committed on accepted iterations, so this guarantees benchmarks/results.json is regenerated with real data every run. Previously the per-iteration checklist merely asked the agent to run the script and update the file, which is why results.json never got populated.
• Runner bootstrap. Install pandas if missing; install Bun if missing (autoloop runners don't have it) and bun install so TS benchmarks can import from src/.
• Metric now reflects reality. benchmarked_functions is read from the regenerated results.json (count of entries that actually ran to completion) instead of counting files on disk — so the ratchet credits only benchmarks that truly work end-to-end.
• Visible failures, guaranteed metric. Replaced silent || true + output redirection with || echo "WARN: …" >&2 so failures surface in autoloop logs while still guaranteeing the final metric line is emitted for autoloop to parse.
• Trimmed the iteration checklist. Removed the now-redundant "run benchmarks" / "update results.json" steps and added a short note pointing at Evaluation.

Flow after this change

autoloop iteration
  → adds bench_{fn}.ts + bench_{fn}.py
  → Evaluation runs run_benchmarks.sh
  → benchmarks/results.json regenerated with real timings
  → committed to autoloop branch on accept
  → merge to main
  → pages.yml copies results.json → playground/benchmarks/
  → benchmarks.html renders real data

Note on scope

AGENTS.md tells agents not to modify .autoloop/programs/**, but the program definition is exactly what the issue asks to change, so the edit is scoped to that one file.

[mrjf]

@copilot you need to compile the agentic workflow

[mrjf]

@copilot ci failing #154

[Copilot]

@copilot you need to compile the agentic workflow

Ran gh aw compile (v0.65.6) — all 3 workflows compiled cleanly with no resulting diff. The compiled .lock.yml files don't embed program contents; the autoloop workflow loads .autoloop/programs/*.md at runtime, so this PR's edit to perf-comparison/program.md doesn't require a recompile. Only the pre-existing unrelated warning about push-to-pull-request-branch: target: "*" was emitted.

[Copilot]

@copilot ci failing #154

Looked at run 24674003775. The single failing test is a pre-existing flaky property-based test, unrelated to this PR:

(fail) strRemovePrefix — properties > result never starts with prefix (when prefix non-empty)

This PR only touches .autoloop/programs/perf-comparison/program.md (51 +/16 -, no source or test files), so it cannot have caused a failure in tests/stats/string_ops.test.ts. The property at tests/stats/string_ops.test.ts:418 has a logic bug — when str doesn't start with prefix, strRemovePrefix returns str unchanged, and if that str happens to start with prefix characters by coincidence the assertion fails. fast-check eventually finds such a counterexample.

Happy to fix the test in a separate PR if you'd like, but it's out of scope here.

[github-actions]

---

Warning

The push_to_pull_request_branch operation failed: Failed to apply patch. The code changes were not applied.

Evergreen CI Fix

I investigated the failing Test & Lint CI check on this PR.

Finding: This PR only modifies .autoloop/programs/perf-comparison/program.md — no source code or tests were changed. All CI checks pass locally:

Check	Result
bun run typecheck	✅ 0 errors
bun run lint	✅ 0 errors (355 style warnings, exit 0)
bun test --coverage	✅ 4669 pass, 0 fail

Root cause: The CI failure appears to have been transient. The PR branch was created minutes after main received a large merge (PR #120, ~2300 lines of new tests and features). The initial CI run may have encountered a flaky test or infrastructure issue.

Action taken: Pushed an empty commit to trigger a fresh CI run. All checks should pass now.

Generated by Evergreen — PR Health Keeper · ● 17.6M · ◷

[github-actions]

Commit pushed: b4db965

Generated by Evergreen — PR Health Keeper

[github-actions]

Evergreen: Fixed failing CI checks

Fixed 3 failing tests in the Test & Lint check caused by a snake_case/camelCase mismatch in src/reshape/explode.ts.

Root cause: ExplodeOptions interface had readonly ignore_index?: boolean (snake_case) while all tests pass { ignoreIndex: true } (camelCase). The option was silently ignored at runtime, producing wrong index labels.

Fix: Renamed ignore_index → ignoreIndex in:

• ExplodeOptions interface definition (line 39)
• explodeSeries usage (line 92)
• explodeDataFrame usage (line 155)

Tests fixed:

• explodeSeries > resets to RangeIndex when ignoreIndex=true
• explodeDataFrame > resets to RangeIndex when ignoreIndex=true
• Property test: ignore_index produces RangeIndex 0..n-1

Generated by Evergreen — PR Health Keeper · ● 22.9M · ◷