Skip to content

Warn on gate-target mismatch when recording a formulation #214

Description

@kateebonner

Important

Problem

The formulation validator only checks that target is a string, so the agent can lock a formulation whose target contradicts the problem it belongs to — a real CZ-gate workspace shipped with target X, and the formulation tab faithfully rendered an "x-gate" chip for a CZ run. Nothing at record time even hints at the mismatch.

Approach

Add a cheap gate-token mismatch heuristic at formulation record time, surfaced through the existing soft-warning channel in the formulate tool's reply — a warning, never a block, because problem names are free-form and workspaces get repurposed.

Approaches Considered

  • Soft warning via mismatch heuristic (chosen) — catches the observed failure with zero false-positive cost; the agent self-corrects in the same turn.
  • Hard validation failure — rejected: slugs are free-form; repurposed/renamed workspaces would be blocked on legitimate targets.
  • Prompt-only fix in the score — rejected: prompts drift and give no durable guardrail; this failure already slipped past the current prompt.

Scope

  • In: token extraction from problem name/slug; comparison against normalized target; warning appended to the formulate tool response alongside existing soft warnings; unit tests.
  • Out: blocking validation — free-form names make false positives certain.
  • Out: reconciling against the actual solve script — separate issue (linked below).
  • Out: UI changes — the chip is correct once the data is correct.

Assumptions / Open Qs

  • Assumes a small closed set of recognizable gate tokens (X, Y, Z, H, CZ, CX/CNOT, CCZ, iSWAP, …) is enough; names without a recognizable token never warn.
  • Open: should the warning also fire on trajectory-type mismatch (name says "state-prep", formulation says gate)?

Acceptance Criteria

  • Recording a formulation with target X in a problem whose name contains a cz token returns a warning naming both values.
  • Recording a matching target (e.g. CZ in a cz-* problem) produces no warning.
  • Problem names with no recognizable gate token (e.g. transmon-pulse-design) never warn, whatever the target.
  • The warning is soft: the entity is still recorded and the stage still completes.
  • Unit tests cover token extraction, aliasing (CNOT/CX), and the no-token case.

Key Decisions

  • Reuse the existing formulation soft-warning path (the same channel that carries spec §3.2 warnings) rather than a new mechanism.
  • Normalize both sides before comparing: lowercase, strip -gate/gate suffixes, alias table for common synonyms.
  • Compare against the problem name and slug; the system entity is not consulted (it doesn't know the gate).

Source

  • Found while diagnosing a wrong "x-gate" chip in the formulation tab: workspace cz-gate-transmon-2 (2026-07-24) has target: "X" recorded by the formulate tool per its event log; 17 sibling CZ workspaces record CZ correctly, so the miss was one agent flub with no guardrail.
  • Companion issue: Reconcile the formulation entity against what the solve actually ran #215 (formulation ↔ solve reconciliation) — covers everything after launch; this issue covers the capture window.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions