chore(emdash-bot): eval harness + gen-1 cutover (bot next-gen 5/5) — merge gated on evals - #2383
chore(emdash-bot): eval harness + gen-1 cutover (bot next-gen 5/5) — merge gated on evals#2383ascorbic wants to merge 4 commits into
Conversation
Operator tool (pnpm evals) that drives the deployed worker's investigate agent against 26 curated closed issues and scores each verdict against recorded ground truth. Gate: zero confident-wrong (a reproduction asserted on a case with nothing to reproduce). Cannot run in CI; the dataset loader, pre-fix SHA resolution, scorer, and formatter are unit-tested. Adds an optional baseRef to the investigate agent so a run can stand up at a fixing PR's pre-fix commit; cloneRepo and the container checkout now resolve a branch, tag, or bare commit SHA uniformly.
Delete the gen-1 .flue/ workspace and four issue-automation workflows (investigate, reporter-reply, maintainer-reply, bot-cleanup) now that the emdash-bot worker owns issue-side automation. The repro/diagnose/verify skill prompts were already ported into the worker. Clean the dead investigate.yml entry from zizmor.yml and a stale reference in a preview-releases.yml comment (comment only; the workflow is independent and untouched). Keep triage-project-sync.yml: board sync needs org-level Projects v2 write and keys off the retired triage/* labels, so it is a later singleton, not a cutover change -- documented in evals/CUTOVER.md. Rewrite TRIAGE.md's bot sections for the maintainer-directive flow, the reporter preview-confirm loop, and the new bot:* state labels. evals/CUTOVER.md drafts the cutover PR body and holds the eval-gate section to fill after the live pre-flight run.
…omment Nothing reads EventMeta.labelTriggers and the webhook skips labeled/unlabeled events, so no bot:* label fires any event -- comments are the only entry. Remove the labelTriggers from all four events (repro/investigate/implement/fix) and the now-orphaned field, and regenerate machine.json. Correct the normalizeIssues comment: label drift is reconciled by the Orchestrator DO's periodic alarm tick (reconcileLabels), not a cron -- the worker has no cron trigger.
|
ascorbic
changed the title
Scope checkThis PR changes 10,236 lines across 55 files. Large PRs are harder to review and more likely to be closed without review. If this scope is intentional, no action needed. A maintainer will review it. If not, please consider splitting this into smaller PRs. See CONTRIBUTING.md for contribution guidelines. |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-demo-cache | 1ba97ec | Aug 08 2026, 04:51 PM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-playground | 1ba97ec | Aug 08 2026, 04:52 PM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-demo-do | 1ba97ec | Aug 08 2026, 04:52 PM |
@emdash-cms/admin
@emdash-cms/auth
@emdash-cms/auth-atproto
@emdash-cms/blocks
@emdash-cms/cloudflare
@emdash-cms/contentful-to-portable-text
emdash
create-emdash
@emdash-cms/gutenberg-to-portable-text
@emdash-cms/plugin-cli
@emdash-cms/plugin-types
@emdash-cms/registry-client
@emdash-cms/registry-lexicons
@emdash-cms/registry-verification
@emdash-cms/sandbox-workerd
@emdash-cms/x402
@emdash-cms/plugin-ai-moderation
@emdash-cms/plugin-atproto
@emdash-cms/plugin-audit-log
@emdash-cms/plugin-color
@emdash-cms/plugin-embeds
@emdash-cms/plugin-field-kit
@emdash-cms/plugin-forms
@emdash-cms/plugin-webhook-notifier
commit: |
There was a problem hiding this comment.
Pull request overview
This PR completes the emdash-bot “next-gen” cutover by introducing an operator-run investigation eval harness (with scoring + dataset) and removing the legacy gen-1 Actions-based .flue/ workspace/workflows, while updating the worker to support eval-driven checkouts via an explicit baseRef.
Changes:
- Add
infra/emdash-bot/evals/harness (dataset parsing, dispatch/poll client, scoring, formatting, CLI) with unit tests that pin scoring to the worker’s authoritative outcome mapping. - Update the investigation worker execution environment to support fetch-then-detach checkout for full commit SHAs (needed to run confirmed-bug evals at pre-fix commits) and add a
baseRefoverride for investigations. - Remove gen-1 GitHub Actions workflows and the root
.flue/workspace; update triage docs and related comments/config.
Reviewed changes
Copilot reviewed 60 out of 62 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| TRIAGE.md | Updates issue triage docs to the directive-driven investigation bot and bot:* labels. |
| infra/emdash-bot/tsconfig.json | Includes eval harness TypeScript sources in typechecking. |
| infra/emdash-bot/tests/unit/preview.test.ts | Formatting-only change in preview probe test. |
| infra/emdash-bot/tests/unit/exec-env.test.ts | Adds test coverage for fetch+detach behavior when cloning by full SHA. |
| infra/emdash-bot/tests/unit/evals-scorer.test.ts | Adds scorer unit tests and drift pinning to outcomeFromResult. |
| infra/emdash-bot/tests/unit/evals-format.test.ts | Adds unit tests for table/summary/json report formatting. |
| infra/emdash-bot/tests/unit/evals-dataset.test.ts | Adds unit tests for dataset validation + pre-fix ref resolution. |
| infra/emdash-bot/tests/unit/evals-client.test.ts | Adds unit tests for snapshot verdict extraction and polling behavior. |
| infra/emdash-bot/tests/integration/orchestrator.test.ts | Formatting-only change in orchestrator integration test. |
| infra/emdash-bot/package.json | Adds pnpm evals script to run the operator harness. |
| infra/emdash-bot/evals/src/types.ts | Introduces shared harness types (cases, reported results, scoring outputs). |
| infra/emdash-bot/evals/src/scorer.ts | Implements outcome mapping + scoring + gate summary logic. |
| infra/emdash-bot/evals/src/runner.ts | Orchestrates eval selection, GitHub issue fetch, dispatch, poll, and scoring. |
| infra/emdash-bot/evals/src/format.ts | Formats reports (plain-text + JSON artifact) and the gate banner. |
| infra/emdash-bot/evals/src/dataset.ts | Loads/validates dataset and resolves pre-fix checkout refs. |
| infra/emdash-bot/evals/src/client.ts | Implements dispatch/read snapshot APIs and verdict extraction. |
| infra/emdash-bot/evals/README.md | Documents eval harness purpose, scoring, usage, and safety. |
| infra/emdash-bot/evals/dataset.md | Provides human-readable dataset rationale and case list. |
| infra/emdash-bot/evals/dataset.json | Adds the committed 26-case evaluation dataset. |
| infra/emdash-bot/evals/CUTOVER.md | Adds the cutover record template and live eval gate checklist. |
| infra/emdash-bot/evals/bin/run.ts | Adds the operator CLI entrypoint (env parsing, selection, output). |
| infra/emdash-bot/evals/.gitignore | Ignores local eval run result artifacts. |
| infra/emdash-bot/.flue/skills/verify/SKILL.md | Minor markdown emphasis tweaks for verify skill docs. |
| infra/emdash-bot/.flue/skills/investigate/SKILL.md | Minor markdown emphasis tweaks for investigate skill docs. |
| infra/emdash-bot/.flue/skills/fix/SKILL.md | Minor markdown emphasis tweaks for fix skill docs. |
| infra/emdash-bot/.flue/skills/diagnose/SKILL.md | Minor markdown emphasis tweaks for diagnose skill docs. |
| infra/emdash-bot/.flue/lib/webhook.ts | Clarifies webhook normalization comment re: label reconciliation. |
| infra/emdash-bot/.flue/lib/orchestrator.ts | Formatting + preserves a side-effect drain guard behavior. |
| infra/emdash-bot/.flue/lib/machine.ts | Removes label-trigger metadata (moving away from label-triggered events). |
| infra/emdash-bot/.flue/lib/machine.json | Removes label-trigger metadata from generated machine JSON. |
| infra/emdash-bot/.flue/lib/exec-env.ts | Supports cloning/checking out by full SHA via fetch+detach. |
| infra/emdash-bot/.flue/agents/investigate.ts | Adds baseRef initialData support and switches to fetch+detach checkout flow. |
| .github/zizmor.yml | Removes no-longer-needed ignore for deleted workflow. |
| .github/workflows/preview-releases.yml | Updates comment to reflect worker-driven branch pushes (no workflow dependency). |
| .github/workflows/maintainer-reply.yml | Deletes gen-1 maintainer directive workflow (cutover). |
| .github/workflows/bot-cleanup.yml | Deletes gen-1 cleanup workflow (cutover). |
| .flue/workflows/classify-reply.ts | Deletes gen-1 Flue workflow code (cutover). |
| .flue/workflows/classify-maintainer-reply.ts | Deletes gen-1 Flue workflow code (cutover). |
| .flue/tsconfig.json | Deletes gen-1 .flue workspace TS config (cutover). |
| .flue/skills/verify/SKILL.md | Deletes gen-1 skill docs (cutover). |
| .flue/skills/repro-public/SKILL.md | Deletes gen-1 skill docs (cutover). |
| .flue/skills/repro-api/SKILL.md | Deletes gen-1 skill docs (cutover). |
| .flue/skills/repro-admin/SKILL.md | Deletes gen-1 skill docs (cutover). |
| .flue/skills/fix/SKILL.md | Deletes gen-1 skill docs (cutover). |
| .flue/skills/diagnose/SKILL.md | Deletes gen-1 skill docs (cutover). |
| .flue/skills/_INVESTIGATE.md | Deletes gen-1 pipeline reference doc (cutover). |
| .flue/scripts/run-local.ts | Deletes gen-1 local prototype runner (cutover). |
| .flue/README.md | Deletes gen-1 bot README (cutover). |
| .flue/pnpm-workspace.yaml | Deletes gen-1 workspace config (cutover). |
| .flue/package.json | Deletes gen-1 workspace package manifest (cutover). |
| .flue/lib/classifier.ts | Deletes gen-1 classifier implementation (cutover). |
| .flue/lib/capacity.ts | Deletes gen-1 Workers AI retry/timeout helpers (cutover). |
| .flue/fixtures/issue-1080.json | Deletes gen-1 fixture (cutover). |
| .flue/fixtures/issue-1049.json | Deletes gen-1 fixture (cutover). |
| .flue/fixtures/issue-1046.json | Deletes gen-1 fixture (cutover). |
| .flue/fixtures/issue-1042.json | Deletes gen-1 fixture (cutover). |
| .flue/fixtures/issue-1021.json | Deletes gen-1 fixture (cutover). |
| .flue/.gitignore | Deletes gen-1 workspace ignore file (cutover). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| { | ||
| command: `cd ${REPO_DIR} && git checkout '${branch}' && git reset --hard 'origin/${branch}'`, | ||
| command: `cd ${REPO_DIR} && git fetch --depth ${CLONE_DEPTH} origin '${ref}' && git checkout --detach FETCH_HEAD`, | ||
| timeoutMs: 5 * 60_000, |
github-actions
Bot
added
review/awaiting-author
Overlapping PRsThis PR modifies files that are also changed by other open PRs:
This may cause merge conflicts or duplicated work. A maintainer will coordinate. |
![emdashbot[bot]](https://avatars.githubusercontent.com/in/3255304?s=60&v=4){kind=small}
There was a problem hiding this comment.
This is the right capstone for the bot next-gen stack. The PR gates a big-bang gen-1 cutover on a live eval run, deletes only the surfaces that have been replaced by the worker, and documents the deliberate gaps (project-sync pause, global sweep singleton, preview-URL verification). That approach is sound.
I reviewed the eval harness (infra/emdash-bot/evals/), the worker's new baseRef checkout path, the generated machine artifacts, the deleted .flue/ root and workflow files, TRIAGE.md, .github/zizmor.yml, and .github/workflows/preview-releases.yml. The code is mostly clean and the pure parts of the harness are well-tested. I found two small issues worth fixing before merge: the container shell commands in investigate.ts use hardcoded quoting instead of the safe quote() helper used on the isolate side, and a stale comment in machine.ts still points to the gen-1 file this PR deletes.
Neither is a blocker, and I didn't find SQL/auth/localization/query-count regressions in this infra-only change.
Findings
-
[needs fixing]
infra/emdash-bot/.flue/agents/investigate.ts:371The container fetch command wraps
refin literal single quotes ('${ref}'). Ifinput.baseRefis a branch or tag that contains a single quote, the shell word is terminated early and the remainder of the string becomes arbitrary shell syntax. The isolate-sidecloneRepoalready usesquote()fromexec-env.tsfor the same reason; the container side should reuse that helper. Exportquotefrom../lib/exec-env.jsand use it here too.{ command: `cd ${REPO_DIR} && git fetch --depth ${CLONE_DEPTH} origin ${quote(ref)} && git checkout --detach FETCH_HEAD`, timeoutMs: 5 * 60_000, }, -
[needs fixing]
infra/emdash-bot/.flue/agents/investigate.ts:375The
git config http.extraHeadercommand also wraps a dynamic value in literal single quotes ('${PUSH_CAPABILITY_HEADER}: ${pushCapability}').pushCapabilityis generated internally, but a JWT-like value could still contain'in principle, and using the same hardcoded-quote pattern asrefis fragile. Use the sharedquote()helper (exported fromexec-env.ts) consistently.{ command: `cd ${REPO_DIR} && git config http.https://github.com/.extraHeader ${quote(`${PUSH_CAPABILITY_HEADER}: ${pushCapability}`)}`, }, -
[suggestion]
infra/emdash-bot/.flue/lib/machine.ts:272This comment points to
.flue/workflows/investigate.ts, but that gen-1 file is deleted by this cutover PR. The agent contract it refers to now lives in.flue/agents/investigate.ts.// These names map 1:1 to the agent contract in .flue/agents/investigate.ts.
What does this PR do?
Slice 5, final slice of the bot next-gen stack (on #2382): the investigation eval harness and the gen-1 cutover.
infra/emdash-bot/evals/): operator tool driving the deployed worker against 26 curated closed issues (17 confirmed bugs checked out at their pre-fix commit, 5 not-reproducible, 4 needs-info). Gate = zero confident-wrong verdicts (areproducedclaim where nothing was reproducible) and zero harness errors; a missed repro is a miss, never a pass. Scorer is pinned to the authoritativeoutcomeFromResultby a drift test. Supporting worker change: investigations accept abaseRef(branch/tag/SHA, fetch-then-detach, fail-safe)..flue/root workspace and four Actions workflows (investigate,reporter-reply,maintainer-reply,bot-cleanup) — all event classes traced into the worker's webhook routing (reviewed).preview-releases.ymluntouched (independent, still the pkg.pr.new publisher).TRIAGE.mdbot sections rewritten for the directive-driven flow.triage-project-sync.ymlis kept (deletion list is four, not five) — it needs org-level Projects write, a different auth surface than the worker's App token. Consequence: the Auto-Triage board pauses at cutover until a board-sync singleton follows. Rationale in CUTOVER.md.Documented follow-ups (CUTOVER.md, maintainer's call): global
bot/artifacts-*sweep needs ascheduled()cron; the awaiting-reporter-close edge (reap → reopen → confirm fails against a deleted branch; fix sketched); board-sync singleton.Rollback story: Workers version redeploy — the deleted Actions are not the rollback path.
Closes #
Type of change
Checklist
pnpm typecheckpasses — n/a-with-note: same pre-existinginfra/emdash-botbaseline as the stack; zero new errorspnpm lintpasses — oxlint--type-aware --deny-warningsexit 0pnpm testpasses — unit 169/169 (incl. scorer/loader/drift), integration 42/42 (workers pool)pnpm formathas been runAI-generated code disclosure
Screenshots / test output
Try this PR
Open a fresh playground →
A full working EmDash site, deployed from this branch. Each visit gets its own session-scoped sandbox: no login needed and no shared state. Try the admin, edit content, hit the public site.
Tracks
bot-nextgen/05-evals-cutover. Updated automatically when the playground redeploys.