0.2c — Route per-iter PNGs through LivePulsePlotCallback (AbstractIntermediateCallback)

[Rchari1]

Implements #21 (Phase 0′, task 0.2c). Depends on 0.2a (DirectTrajOpt.jl#114 + Piccolo.jl install path).

What this does

Retires the hand-rolled plot_pulse/CairoMakie.save per-iter PNG path (the PR #14 interim) in the bundled solve template. The live plot now flows through Piccolo's LivePulsePlotCallback — an AbstractIntermediateCallback — installed on the Ipopt solve via IpoptOptions(intermediate_callback=…). Satisfies the Phase-0prime DoD: "β's live plot now flows through AbstractIntermediateCallback."

• templates/solve_template.jl
  • LivePulsePlotCallback(qtraj, prob.trajectory; every=PLOT_EVERY, save_dir=".") installed via options=IpoptOptions(intermediate_callback=live_plot); emits iter_<N>.png through the callback abstraction (inspector unchanged — its regex already accepts unbounded digits).
  • AMICODE_ITER text telemetry stays intact on the raw Ipopt callback — it needs the rich IPM state (obj_value/inf_pr/inf_du) the agnostic (primal,iter) contract can't carry. DTO composes the raw callback with intermediate_callback, so both fire per iter.
  • Dropped save_control_plot + the end-of-solve fallback plot (LivePulsePlotCallback fires at iter 0 + every PLOT_EVERY → always ≥1 frame) and the now-redundant callback_update_trajectory_factory (the callback reconstructs the trajectory from the primal itself).
• AGENTS.md — documents LivePulsePlotCallback/AbstractIntermediateCallback as the blessed per-iter plot idiom; instructs the agent never to hand-roll plotting.

Live inspector is ipopt-only for v1 (Q74). Repo grep confirms no plot_pulse/save_control_plot/CairoMakie.save remains in the template.

Validation (gpu-dev-v2, end-to-end)

Ran the migrated template against a dev-coupled DTO#114 + Piccolo install-path env: exit 0 · 11 iter_*.png frames · 61 AMICODE_ITER lines · result.toml written · DONE emitted. Per-iter PNGs flow through the callback and AMICODE_ITER survives.

⚠️ Release coordination (blocks final green)

Cannot go green end-to-end in CI until DTO 0.2a is released (v0.9.7, the intermediate_callback field) and the bundled packages/extension/julia Manifest (currently DTO v0.9.6) is bumped to resolve it. Chain: merge DTO#114 → register v0.9.7 → (Piccolo install-path PR) → bump amicode's julia Manifest. Fast-lane CI here is unaffected (it doesn't execute the template; that's the slow tier).

🤖 Generated with Claude Code

[jack-champagne]

Read this against the pinned Julia env + the depot source, and the end-state is the right one — routing per-iter PNGs through LivePulsePlotCallback is exactly the 0.2c target. I confirmed the callback renders the convention-correct way (extract_pulse(qtraj, traj) → plot_pulse(pulse; …) in PiccoloMakieExt, not raw plot(traj,[:u])), and it zero-pads iter_00006.png, so the inspector's iter_(\d+) watcher still orders frames fine. The AGENTS.md idiom section is clean.

The problem is sequencing: this is the 0.2c consumer landing ahead of its 0.2a producer. The bundled packages/extension/julia Manifest pins DirectTrajOpt 0.9.6, and I checked the depot — intermediate_callback is a MadNLPOptions-only field there (the Ipopt path never invokes an AbstractIntermediateCallback; wiring it into Ipopt is literally #19). So IpoptOptions(intermediate_callback = live_plot) throws at construction against the env we actually ship — the bundled template is dead-on-arrival — and nothing in CI catches it (the fast job excludes **/slow/**, schema-roundtrip is non-Piccolo, boot-smoke is opencode-only, and test/slow/template.test.ts self-skips without AMICO_TEST_JULIA_PROJECT). It also runs against D10, which says hand-roll the β PNG from the existing Ipopt callback and treat the abstraction as fast-follow because it's absent from DTO.

Three changes before merge (inline):

1. [blocking] gate on the DTO v0.9.7 release (#19) and bump packages/extension/julia/Manifest.toml to it — ideally in this PR so the template + the pin move in lockstep (same rule as the #28/#30 rename).
2. [blocking] keep the hand-rolled plot_pulse/save fallback documented in AGENTS.md until that pin lands — don't ban the one idiom that runs on 0.9.6, which is also the one D10 prescribes for β.
3. [important] add a slow-tier CI gate that runs the bundled template ≥1 iteration against packages/extension/julia (same hole as #45), so "template doesn't run on the pin" can't merge green again.

Verify against v0.9.7 once it's pinned (not blocking the above, but don't rely on them until checked): that Ipopt actually invokes the callback at iter 0 (so dropping the end-of-solve fallback is safe for sub-PLOT_EVERY solves), and that the raw callback (AMICODE_ITER) and intermediate_callback (PNG) both fire per iter on Ipopt.

Taste / non-blocking: the live frames lose the bounds=true shading the old plot_pulse(qcp; bounds=true) gave — LivePulsePlotCallback hardcodes plot_pulse(pulse; title) with no bounds knob, so a researcher watching a solve can't see the drive railing against its amplitude limit anymore. Minor, but it's a real downgrade in the live view.

[Rchari1]

Addressed the review — converted to draft, since this is hard-blocked on the DTO v0.9.7 release.

What landed now (c860d28): AGENTS.md softened to the dual-state idiom you asked for (prefer LivePulsePlotCallback once DTO ≥ 0.9.7 is pinned; hand-roll from the raw Ipopt callback otherwise — no longer bans the 0.9.6 path).

Lockstep plan (the remaining blocking items move together):

1. Merge DirectTrajOpt.jl#114 → register DTO v0.9.7 (the Ipopt intermediate_callback field).
2. In this PR, add the packages/extension/julia/Manifest.toml bump to 0.9.7 — so the template flip + the pin land in one PR (0.1a — @amicode/schema: single source of truth for the five config/artifact schemas (closes #15-core) #28/0.1c — amico-validate CLI + CI fast-tier schema gate (closes #17) #30-style), never out of step on main.
3. Slow-tier CI gate (review item 3): I'll add a test:slow job that runs the bundled template ≥1 iteration against packages/extension/julia with the bump — it can't be green before the pin (the template throws at construction on 0.9.6), so it ships in the same lockstep commit and guards "template doesn't run on the pin" from then on.

The flipped template is already validated end-to-end against a dev-coupled DTO#114 on gpu-dev-v2 (11 iter_<N>.png + AMICODE_ITER + result.toml + DONE); the slow gate will pin that on the released version. Marking ready once v0.9.7 is registered.

[jack-champagne]

Correction (retract the prior note): DTO 0.9.7 is the correct pin — the callback fix was folded into 0.9.7 after all. Release is imminent (tonight). When 0.9.7 registers: bump packages/extension/julia/Manifest.toml to 0.9.7 + un-draft. Hand-rolled fallback stays until the pin lands.

[jack-champagne]

DTO 0.9.7 is out — bumped packages/extension/julia/Manifest.toml DirectTrajOpt 0.9.6 → 0.9.7 via re-resolve (diff is DTO-only; the julia_version/stdlib artifacts from resolving on a different Julia patch were reverted). Verified locally: IpoptOptions(intermediate_callback = …) now constructs on 0.9.7 (the 0.9.6 throw is gone) and the Piccolo/DTO/CairoMakie stack precompiles clean. Marked ready for review.

Re-review still owes (from the prior pass, needs a real solve): (1) confirm Ipopt actually invokes the callback at iter 0 and both callbacks (LivePulsePlotCallback + cb_log) fire; (2) the template dropped the end-of-solve guarantee frame — a solve shorter than PLOT_EVERY could leave the inspector blank; (3) bounds=true shading loss is non-blocking.

[Rchari1]

Re-review items verified on the released DTO 0.9.7 (thanks for the Manifest bump). Ran the bundled template against packages/extension/julia on a clean box — fresh Pkg.instantiate() resolved DirectTrajOpt v0.9.7 from the registry (no dev-couple), so this is the real pinned env.

Item 1 — Ipopt invokes the callback at iter 0, and both callbacks fire. Normal solve (max_iter=60), exit 0:

• frames: iter_00000.png … iter_00060.png (11, every 6) — the iter-0 frame is present, so Ipopt does invoke intermediate_callback at iter 0.
• AMICODE_ITER = 61 lines (iters 0–60) — the raw cb_log fires every iter alongside LivePulsePlotCallback. Both composed callbacks stream per-iteration. result.toml + DONE written.

Item 2 — end-of-solve guarantee frame (dc602ea). Short solve (max_iter=3 < PLOT_EVERY=6), exit 0:

• frames: iter_00000.png (random init) + iter_00003.png (the guard frame — final pulse). Without the guard, only the iter-0 frame would exist and the inspector would sit on the initial guess. The guard re-invokes LivePulsePlotCallback once at every=1 with the final primal — still through the callback, no bespoke plot_pulse.

Item 3 — bounds=true shading (non-blocking). plot_pulse does take an explicit per-drive bounds vector, so the fix is a kwargs passthrough on LivePulsePlotCallback (upstream, PiccoloMakieExt) + a Piccolo release before the template can wire it — tracked as a follow-up, not in this slice.

CI is green on the fast lanes. Ready for another look.

[jack-champagne]

Verified end-to-end on DTO 0.9.7 — ran the vetted template (X gate, 3-level transmon) via amico-run's Julia project:

• Both callbacks fire, incl. iter 0: 61 AMICODE_ITER lines (iter=0 → 60) + 11 iter_*.png frames starting at iter_00000.png. So the raw Ipopt callback (cb_log) and the LivePulsePlotCallback (intermediate_callback) both fire per-iter — the Q74 composition works.
• Guarantee frame (dc602ea) runs clean: re-invokes LivePulsePlotCallback at the final primal; iter_00060.png written as the converged pulse. Short-solve blank-inspector risk closed.
• Converges: fidelity 0.99999, result.toml + DONE + pulse.jld2 emitted, julia exit 0.
• Watcher compat: ITER_PNG_RE = /^iter_(\d+)\.png$/ (run_dir_reader.ts) parses the zero-padded frame names — the inspector will render them.

All prior review comments now addressed: AGENTS.md fallback (c860d28), 0.9.6 constructor throw → DTO 0.9.7 pin (3ab9d32), end-of-solve guarantee frame (dc602ea), render path (confirmed). Resolving the threads.

Non-blocking leftover: a cosmetic Δt has no bounds warning prints even with timesteps_all_equal=true set — noise, not a defect.

[jack-champagne]

All four review comments addressed and verified end-to-end on DTO 0.9.7 (both callbacks fire incl. iter 0, guarantee frame writes the converged pulse, fidelity 0.99999, watcher parses the frames). Threads resolved, CI green.