Evaluation Summary Collection

Evaluation Summary Collection

Shell examples run from the target project root. For scripts/forgeflow/ commands, use the helper path from your ForgeFlow checkout, or replace that prefix with "${CODEX_HOME:-$HOME/.codex}/forgeflow/scripts/forgeflow/" for Codex and "$HOME/.claude/forgeflow/scripts/forgeflow/" for Claude Code. Run JavaScript helpers with node and shell helpers with bash. Replace <project> with the actual project folder name before running a placeholder example.

Use this workflow to collect public-safe evaluation summaries from real reviews during field validation. It keeps raw records local while making aggregate evidence easy to compare.

For maintainer pilot trials, use Pilot Public Summary before adding summaries to a broader collection.

Collection Folder

Store shareable summaries outside source control by default:

.forgeflow/<project>/public-summaries/

Create the folder locally:

mkdir -p ".forgeflow/$(basename "$PWD")/public-summaries"

If the project does not already ignore .forgeflow/, add it to local exclude:

printf ".forgeflow/\n.forgeflow-budget.json\n" >> "$(git rev-parse --git-path info/exclude)"

Generate A Summary

After a branch trial and human triage:

PROJECT="$(basename "$PWD")"
STAMP="$(date +%Y%m%d-%H%M)"

scripts/forgeflow/render-evaluation-report.js \
  --outcomes ".forgeflow/${PROJECT}/review-outcomes.jsonl" \
  --context-root .forgeflow \
  --budget-config .forgeflow-budget.json \
  --public \
  --out ".forgeflow/${PROJECT}/public-summaries/${STAMP}-evaluation-summary.md"

Read the generated file before sharing:

sed -n '1,220p' ".forgeflow/${PROJECT}/public-summaries/${STAMP}-evaluation-summary.md"

Metadata To Capture

Keep a separate local note with this metadata for each summary:

summary_file:
project_type: frontend | api | monorepo | docs-config | release-prep | other
runtime: claude-code | codex | both
install_path: update-forgeflow | template-installer | existing-install
workflow_labels_present: no-agent | single-agent | forgeflow
reviewed_changes:
context_budget_status:
first_run_blockers:
notes:

Do not put private branch names, customer names, private URLs, secrets, or source snippets in the shareable summary.

What To Share

Share one of these:

• the generated evaluation-summary.md
• copied aggregate tables from the summary
• screenshots of the summary
• a short written note using the suggested language in Evaluation Sharing

Do not share raw review-outcomes.jsonl, context packets, memory summaries, or telemetry rows unless the receiving audience is allowed to see the underlying project context.

Rollup Checklist

When enough summaries are collected, build a small rollup with:

• number of project types covered
• number of reviewed changes
• confirmation rate range
• false positive rate range
• average review minutes range
• context percent saved range
• budget violation count
• repeated first-run blockers

Use the rollup to decide whether the next fix belongs in install, health, docs, routing, context budgets, or the template installer.