benchmarks: convergence suite v1 (Ipopt + MadNLP on X gate)#93
Open
jack-champagne wants to merge 4 commits into
Open
benchmarks: convergence suite v1 (Ipopt + MadNLP on X gate)#93jack-champagne wants to merge 4 commits into
jack-champagne wants to merge 4 commits into
Conversation
Adds `benchmark/convergence/` exercising HarmoniqsBenchmarks.jl v0.2.0's new convergence API: `InfidelityConvergence`, `ipopt_capture`, and `compare_convergence`. Two `@testitem`s on a shared bilinear X-gate state-transfer problem (N=51, terminal cost + control regularizers), both asserting `converged(...) == true` against a `target_infidelity = 1e-3`, `feas_tol = 1e-6` criterion. Ipopt path uses `ipopt_capture()` to read final `iter_count` / `inf_pr`; MadNLP falls back to the post-solve evaluator's `constraint_violation` as the primal-infeasibility proxy (no MadNLP capture analogue yet). Adds a dedicated `.github/workflows/benchmark-convergence.yml` workflow (Julia 1.12, 30 min timeout, artifact upload) so this suite runs without piggybacking on the timing-only benchmark workflow from PR #75. Demo problems (atoms / spin-qubit / bosonic) intentionally deferred to follow-up PRs once this harness is proven green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Strips aligned `=` padding from the new `benchmark/convergence/*.jl` files
to satisfy the Formatter CI job (which fails on any diff vs.
`JuliaFormatter.format(".")`). No semantic changes — both testitems still
pass locally on Julia 1.12.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The convergence suite under `benchmark/convergence/` uses `@testitem` blocks that depend on `HarmoniqsBenchmarks` — which isn't in the package's main test environment. `@run_package_tests` was picking those up and failing with `ArgumentError: Package HarmoniqsBenchmarks not found in current path` on Julia 1.10/1.11/1.12. Filter out anything whose path includes a `benchmark` component. Match the path component (via `splitpath`) rather than a substring so future test files like `foo_benchmark.jl` aren't accidentally excluded. Mirrors the same filter added in PR #75 (initial benchmark suite).
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This was referenced May 20, 2026
Tracking `rev = "main"` makes convergence-suite CI non-reproducible: any commit to HBJ silently changes the schema/extractors for every old DTO CI re-run, and cached CI runs diverge from cold ones. Pin to HBJ's current main HEAD (c38418cb7f — alloc analyzer + Piccolo-aligned CI), matching the pin pattern used in `benchmark/Project.toml` for the initial benchmark suite. Drop in favor of `[compat]` once HBJ registers in General (PR JuliaRegistries/General#155879).
jack-champagne
added a commit
that referenced
this pull request
May 20, 2026
After the deterministic+sweep rework no testitems use :experimental anymore, and the sweeps add ~5 seconds in aggregate on top of a ~10-minute matrix — not worth a parallel filter framework. Drops both INCLUDE_EXPERIMENTAL and INCLUDE_ROBUSTNESS env vars and the ti.tags closure. The robustness sweeps now run on every PR (along with their deterministic baselines). Keeps a `/benchmark/` path filter following Piccolissimo.jl's convention — the benchmark subtree has its own Project.toml + deps + workflow, so its @testItems shouldn't be discovered by `@run_package_tests`. This also preemptively unbreaks PR #93's CI (which fails today because benchmark/convergence/convergence.jl is being picked up). Verified locally on Julia 1.12: 355 pass / 1 broken / 0 fail. TimeConsistency sweep: pass_rate = 1.0 (20/20) NonlinearKnotPoint sweep: pass_rate = 0.8 (16/20, threshold 0.65) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
benchmark/convergence/— a small DTO benchmark environment thatexercises HarmoniqsBenchmarks.jl v0.2.0's new convergence API
(
InfidelityConvergence,ipopt_capture,compare_convergence).Scope is deliberately tight (one IPOPT + one MadNLP testitem on the same
X-gate state-transfer); atoms-demo / spin-qubit-demo / bosonic-demo problems
land in separate follow-up PRs once this harness is proven green.
Depends on: HarmoniqsBenchmarks.jl
main(v0.2.0 + convergence schema, not yet registered).What's included
benchmark/convergence/Project.toml— isolated env with HBJ (main),Ipopt, MadNLP, NamedTrajectories.
DirectTrajOptpulled in as a path depto
../..so we test the package under PR HEAD, not General.benchmark/convergence/problem_utils.jl—_make_xgate_prob(bilinear4D rep + terminal cost + control regularizers) and
_build_convergence_result(rebuilds aBenchmarkResultwith aConvergenceCriterionattached).benchmark/convergence/convergence.jl— two@testitems:ipopt_capture()extracts finaliter_count+inf_pr, buildsInfidelityConvergencewithtarget_infidelity = 1e-3,feas_tol = 1e-6, assertsconverged(...).post-solve
evaluator.constraint_violationas the primal-infeasibilityproxy (no MadNLP capture analogue yet).
benchmark/convergence/README.md— usage instructions..github/workflows/benchmark-convergence.yml— Julia 1.12, 30 mintimeout, JLD2 artifact upload.
Local results
JLD2 artifacts produced:
xgate_convergence_ipopt_N51_<commit>.jld2xgate_convergence_madnlp_N51_<commit>.jld2Test plan
benchmark-convergenceworkflow green on this PRconverged(result.convergence) == truecompare_convergence([result])returns one row per converging benchmarkOut of scope (follow-up PRs)
ipopt_capture()