eval_removal_plan

Eval Removal Plan: Migrate to Producer-Owned Measurement

Goal: delete the file/run-based workflow evaluation ("eval") subsystem from mcp-agent-builder-go, with outcome measurement continuing to come from the producing steps' own stored outputs — no mandated route, step, or table.

Non-goal: routing evaluation (routing-evaluation.json, RoutingEvaluatedEvent, deterministic routing, route tracing) is a separate feature and stays. Note this does NOT include routeEvaluations.ts, which despite its name implements eval-step route gating only (see Phase 5) — it is deleted.

Direction: Flexible Contract, Not Mandatory Topology

Durable workflow outputs already land in the database, so Pulse needs measurement data, not a particular plan shape. Any normal workflow step or scheduled collector may produce a run-scoped, evidence-backed measurement; Pulse does not depend on which step or route produced it. Target architecture:

ordinary workflow steps (producers)
        ↓
run-scoped database outputs (source of truth, stays in place)
        ↓
Pulse / reports / Goal Advisor read producer outputs directly;
producers call record_goal_observations only when a value should
enter the generic goal-progress history

Final review rejected the earlier mandatory-migration design (an earlier draft of contract 1.0.43: a measurement-router → measure-outcomes route plus a workflow_metrics table on every workflow). It imposed topology most workflows don't need, referenced the nonexistent add_regular_step tool, assumed table-scoped DB authorization that doesn't exist (steps get database-wide read-write), assumed immutable run identity that producers don't guarantee (RUN_FOLDER reuses iteration-0), and described a metrics table no runtime code implements — Pulse and Goal Advisor consume workflow_goal_metrics / pulse_goal_observations through get_goal_metrics. That mandatory migration was deleted. No replacement contract migration is required: producer-owned measurement is ordinary workflow design guidance, not an unattended pre-run rewrite. Legacy evaluation dependencies that genuinely need redesign are handled explicitly in Builder instead of blocking every workflow schedule.

Replacement contract (flexible placement, no mandated topology):

• Reuse an existing step's DB output when it already measures the outcome.
• Add record_goal_observations to that producer only when the value should enter Pulse's generic goal-progress history.
• Put route-specific measurements in existing route-local steps.
• Use a shared convergence step only when multiple routes genuinely require the same calculation.
• Add a dedicated measurement step only when no existing step can safely own it.
• Make no topology change when existing measurement is already sufficient.
• Flag ambiguous evals for manual migration rather than inventing steps or routes.
• Retain old evaluation artifacts as read-only history unless a separate cleanup migration is approved.

What is still lost (no replacement): re-grading an old run after completion, grading without modifying or influencing the run, hiding learnings from the grader, keeping evaluation cost/failures separate from execution, and comparing multiple evaluator versions against the same archived output. Full removal is only correct if none of these are product requirements.

Phase 0 — Decisions and Deprecation (Do First)

All compatibility decisions are made here, BEFORE any deletion. Deleting tools first would strand older workspace upgrades and saved schedules that still invoke run_full_evaluation.

1. Contract migrations (cmd/server/workflow_version_upgrades.go): per migration that invokes eval tools, decide guard (skip when eval is gone) or retire (if every supported workspace is past that contract version).
2. Saved schedules: rewrite schedule instructions that assume an eval pass (e.g. Mututal-Fund "Weekly Saturday Portfolio Sync" sequences steps "including its eval pass"); confirm no enabled schedule depends on run_full_evaluation output.
3. Workspace data: leave evaluation/ dirs, evaluation_report.json files, costs/evaluation/ ledgers, and eval_results rows inert on disk, or ship a one-shot cleanup migration. Either way, decide before deleting the code that reads them.
4. Backup scopes: drop evaluation / evaluation-plan from the trading and check-form backup/sync folder lists (or accept syncing stale dirs).
5. Deprecate at runtime: disable automatic eval by default and observe for one or two release cycles to surface hidden consumers before deleting anything.

Deprecation mechanics — DisableEval is a plain bool (controller_types.go), and omitted tool args decode to false (planning_exports.go), so "default disabled with explicit opt-in" cannot be expressed with the current flag: omitted and explicit-false are indistinguishable. Pick one before implementing:

• (a) Add EnableEval *bool (nil = default disabled during deprecation, explicit true opts in), or
• (b) Keep DisableEval but explicitly default it to true at every entry point (run_full_workflow, single-step execute paths, scheduler, webhook).

Also hide eval mode in the frontend (workflowMode: 'eval', eval canvas, EvaluationPopup); leave components in place until Phase 5.

Phase 1 — Write the Flexible Measurement Guidance

Guidance only; no topology or contract migration is mandated. Must complete before Phases 2–5 delete what it replaces.

1. Per-legacy-eval placement follows the Direction rules: reuse a producer's DB output, add record_goal_observations only for Pulse history, route-localize, converge only when shared, dedicate a step only when no producer can own it, otherwise change nothing, and flag ambiguity for manual migration.
2. Repoint Goal Advisor and report/cost guidance from eval reports / eval_results to producer outputs and get_goal_metrics.
3. Write the measurement guidance (replacement for evaluation-plan.md): guidance/templates/system/measurement-plan.md — placement rules, run scope and evidence, and legacy artifacts as read-only history.
4. Do not add a contract migration for this retirement. A universal pre-run agent turn cannot safely infer whether legacy measurements remain relevant and must not block unrelated workflow execution. Use Builder for an explicit, operator-visible redesign when a workflow still depends on a retired evaluation artifact.

Phase 2 — Delete Eval-Only Backend Files

Delete wholesale (no callers outside the eval subsystem except the entry points removed in Phase 3):

• agent_go/pkg/orchestrator/agents/workflow/step_based_workflow/evaluation_execution.go (ExecuteEvaluationOnly, MaybeRunAutoEvaluation, report phase)
• agent_go/pkg/orchestrator/agents/workflow/step_based_workflow/evaluation_plan_tool.go (add_evaluation_step, update_evaluation_plan, delete_evaluation_step)
• agent_go/pkg/orchestrator/agents/workflow/step_based_workflow/evaluation_helpers.go (validate_evaluation_plan registration)
• agent_go/pkg/orchestrator/agents/workflow/step_based_workflow/evaluation_types.go
• agent_go/pkg/orchestrator/agents/workflow/step_based_workflow/eval_results_storage.go
• agent_go/pkg/orchestrator/agents/workflow/step_based_workflow/evaluation_score_storage.go
• agent_go/cmd/server/evaluation_score_storage.go
• Eval-only tests: evaluation_output_content_test.go, evaluation_plan_tool_test.go, evaluation_report_enrichment_test.go, evaluation_run_metadata_order_test.go, evaluation_skipped_sentinel_test.go, eval_results_storage_test.go, evaluation_score_storage_test.go (both dirs).

Phase 3 — Remove Engine Integration

• controller_batch_execution.go: delete the auto-eval block after group completion (keep finalizeRunMetadata, which is needed regardless).
• controller_types.go: delete the DisableEval option (or the EnableEval replacement from Phase 0, once nothing reads it).
• webhook_step.go: delete the DisableEval = true line (standalone steps simply never had eval).
• Delete every isEvaluationMode branch (~20 files). Highest-density files: controller_execution.go (10), interactive_workshop_manager.go (7), controller_agent_factory.go (6), controller_message_sequence.go (5), controller_orchestrator.go (4), controller_workshop.go (3), controller.go (3, incl. the field itself), step_config.go (2, collapse to planning/), planning_exports.go (2), plus single branches in code_layout.go, controller_scripted.go, plan_snapshot.go, prompt_health.go, reflection_turn_run.go, run_provenance.go, workflow_continuation_recovery.go, workshop_retry_recovery.go, and cmd/server/workflow_phase_tools.go.
• controller_agent_factory.go: delete TARGET_RUN_PATH read-path grants and the eval medium-tier default.
• controller_execution.go: delete TARGET_RUN_PATH prompt injection and the IsEvaluationMode template var.
• controller_learning_helpers.go: delete the isEvalMode parameter from canReadLearnings / canWriteLearnings / resolveExecutionLearningsAccess / shouldDirectWriteLearnings (keep the routing-step exclusions).
• controller_agent_factory.go: resolveEffectiveDBAccess already ignores its eval parameters — simplify its signature.
• agent_go/pkg/workflowtypes/types.go: delete WorkflowStatusEvalExecution and WorkflowStatusEvalBuilder.
• agent_go/pkg/orchestrator/types/workflow_orchestrator.go: delete the ExecuteEvaluationOnly call site.
• planning_exports.go: delete RegisterRunFullEvaluationTool and the eval controller call site.
• planning_agent.go: delete add_evaluation_step registration.
• schedule_collision_guard.go: delete eval tool entries.

Phase 4 — Server APIs and Accounting

• cmd/server/workflow_review_data.go: delete the evaluations section (EvaluationReport* structs, CostScopeEvaluation reads, evaluation/runs timing load, loadWorkflowEvaluationReports).
• Routes: delete /workflow/pulse-eval-results (server.go, handler handleGetPulseEvalResults in pulse_worklist.go) and /workflow/evaluation-reports (server.go, handler handleGetEvaluationReports in workflow.go).
• cmd/server/workspace_state.go: delete eval_data from overview responses.
• cmd/server/external_tools.go: drop evaluation/* from externalPlanPaths.
• cmd/server/workflow_backup.go: drop evaluation/evaluation_plan.json.
• cmd/server/workflow_manifest.go, report_preview_metrics*, slack_retention.go, webhook_retention.go: delete eval references.
• Cost ledger/observer: delete CostScopeEvaluation and costs/evaluation handling.
• cmd/server/workflow_phase_tools.go: delete the eval-session registration.
• cmd/server/toolset_invariant_test.go: drop the five eval tools from the expected set.
• cmd/server/workflow_version_upgrades.go: execute the Phase 0 decision (guard or retire each eval-invoking migration).
• schemas/auto-improvement.schema.json: remove the deprecated eval_step source, leaving measurement and telemetry.

Phase 5 — Frontend

Delete:

• frontend/src/components/workflow/hooks/useEvaluationPlanData.ts
• frontend/src/components/workflow/hooks/useEvaluationPlanToFlow.ts
• frontend/src/components/workflow/nodes/EvaluationNode.tsx
• frontend/src/components/workflow/nodes/CompactEvaluationNodes.tsx
• frontend/src/components/workflow/EvaluationPopup.tsx
• frontend/src/components/workflow/PulseEvalSummary.tsx
• frontend/src/components/workflow/canvas/evaluationLayout.ts (+ test)
• frontend/src/components/workflow/canvas/routeEvaluations.ts (+ test) — despite the name this is eval-only: it imports EvaluationStep and implements applies_to_routes gating. Keeping it after deleting EvaluationStep breaks compilation.
• frontend/src/utils/evaluationReport.ts (+ test)

Scrub (keep ordinary routing logic, delete eval branches):

• canvas/routeTrace.ts (imports evaluationMatchesRoute; handles isEvaluationStep / evaluation-group nodes)
• canvas/triggerLayout.ts (imports EvaluationStep; handles eval nodes)
• canvas/WorkflowCanvas.tsx, canvas/WorkspaceViewHost.tsx, canvas/workspaceViewData.ts, hooks/usePlanToFlow.ts, nodes/RoutingStepNode.tsx, nodes/StepNode.tsx, nodes/index.ts, utils/stepConfigMatching.ts
• components/WorkflowsOverviewPage.tsx: remove evalData state, EvaluationPopup usage, onOpenEval handlers.
• components/workflow/ReportViewer.tsx: remove pulse-eval-results consumption.
• services/api.ts: remove getPulseEvalResults, getEvaluationReports, eval_data handling.
• services/api-types.ts: remove EvaluationPlan, EvaluationStep, EvaluationReportsResponse, and related report types.
• stores/useWorkflowStore.ts: remove 'eval' from workflowMode, evaluationPlan state, loadEvaluationPlan.
• Keep routeTrace.ts / triggerLayout.ts route-tracing logic and routeEvaluations-free routing display — only the eval-gating code goes.
• Rebuild frontend and delete the stale agent_go/static/assets/PulseEvalSummary-*.js bundle.

Phase 6 — Guidance, Prompts, Docs

• Delete: agent_go/cmd/server/guidance/templates/system/evaluation-plan.md, agent_go/cmd/server/guidance/templates/improve/improve-evaluation.md.
• Scrub eval references in: report/improve-report.md, review/review-artifact-drift.md, system/file-layout.md, system/plan-drift-review.md, system/strategy-auditor.md, system/workshop-mode-flow.md, system/workspace-views.md, guidance/guidance.go, cmd/server/instructions.go (incl. Goal Advisor prompt, evaluation/runs/ layout, run_retention_count wording), and interactive_workshop_manager.go prompts referencing eval plans/reports. Repoint each to the Phase 1 measurement guidance.
• Docs: delete docs/workflow/evaluation_system.md; scrub eval mentions in the remaining docs/workflow/*.md (about 20 files reference it).
• This file (docs/workflow/eval_removal_plan.md) should be deleted once the removal is complete.

Phase 7 — Existing Workspaces and Data

Execute the Phase 0 decision: leave evaluation/ dirs, evaluation_report.json files, costs/evaluation/ ledgers, and eval_results rows inert, or run the one-shot cleanup migration. Locally affected: 7 workflows with live eval plans (trading has 18 steps), 122 evaluation_report.json files (~165 MB), eval_results rows in 6 DBs, and eval-created domain tables (HDFC eval_behavior_audit / eval_login_download; trading eval_profile_integrity / eval_runtime_duration / eval_trace_integrity / eval_sc). Schedules that previously got an automatic eval pass keep outcome tracking through their producers' existing outputs (plus record_goal_observations where Pulse history is wanted); no new step or route is required.

Phase 8 — Tests

• ~46 Go test files mention evaluation. Delete the eval-only tests listed in Phase 2; update the rest (toolset invariants, workshop registration, plan snapshot/changelog, execution-only, webhook, scheduler, report metrics tests) to drop eval fixtures and assertions.
• Frontend: delete evaluationLayout.test.ts, evaluationReport.test.ts, routeEvaluations.test.ts, RouteEvaluations.test.tsx, routeTrace.test.ts eval cases, triggerLayout.test.ts eval cases; update store/canvas tests.

Verification Gates

1. Module-scoped builds and tests pass (the repo root holds only go.work; there are no root packages, so bare go build ./... fails):
   • (cd agent_go && go build ./... && go test ./...)
   • (cd workspace && go build ./... && go test ./...)
2. Frontend tsc + unit tests pass; production bundle contains no PulseEvalSummary chunk.
3. Grep gates. These strings must return zero in non-test Go and non-test frontend/src:
   • evaluation_plan, isEvaluationMode, IsEvaluationMode, TARGET_RUN_PATH, ExecuteEvaluationOnly, MaybeRunAutoEvaluation
   • run_full_evaluation, validate_evaluation_plan, add_evaluation_step, update_evaluation_plan, delete_evaluation_step
   • CostScopeEvaluation, EvaluationStep, EvaluationPlan, EvaluationReportsResponse, EvalExecution
   • pulse-eval-results, evaluation-reports, eval_data, EvalData, getEvaluationReports, getPulseEvalResults, applies_to_routes, routeEvaluations, evaluationMatchesRoute
   • measurement-router, measure-outcomes, workflow_metrics, upgrade-measurement-route (the rejected mandatory topology)
   • eval_step, costs/evaluation, evaluation/runs, evaluation_report, eval_results, eval_route_scores, improve-evaluation, evaluation-plan In guidance templates, prompts, schemas, tests, and active docs, the same strings survive only on lines that also say legacy/retired/read-only history (or in assertions pinning retired behavior). Explicitly reviewed exceptions: this plan; dated historical records under docs/bugs/ and docs/audits/; the pulse_v2 proposal docs and the retired design section in persistent_stores_design.md; registry.test.ts (asserts the commands are retired); external_tools_test.go (asserts the retired plan path stays write-protected); declared_execution_mode_strip_test.go (asserts retired files stay byte-identical); slack_review_test.go (asserts retention leaves legacy evidence inert); ManualWorkspaceRefresh.test.ts (asserts the eval refresh is absent); the upgrade-chain guard test and the measurement-guidance render test (assert the rejected topology is absent); reflection_turn_test.go (asserts the retired table stays platform-owned); isEvaluationAgentEvent / inferTrackedExecutionKind (classify historical records only); isPlatformOwnedTable / planDriftReservedTables (keep retired eval tables out of step reach).
4. Allowed survivors: routing-evaluation.json, RoutingEvaluatedEvent, routeTrace.ts / triggerLayout.ts routing logic, RoutingEvaluationCounts, cron "evaluation" wording in scheduler.go, ToolResponseEvaluation (external mcpagent context-editing type — later removed 2026-09-22 with the whole context-editing feature), pulse-review evaluation argument aliases (legacy compat).
5. End-to-end: full workflow run, scheduled run, and webhook run complete with no eval phase; review/costs/timing pages render without an eval section; Goal Advisor and reports read producer outputs and goal observations with no mandated measurement route, step, or table.