Skip to content

Refactor example_db: stored ground truth, human-friendly JSON, no static fixture#696

Merged
GiggleLiu merged 15 commits intomainfrom
issue-680-example-db-refactor
Mar 18, 2026
Merged

Refactor example_db: stored ground truth, human-friendly JSON, no static fixture#696
GiggleLiu merged 15 commits intomainfrom
issue-680-example-db-refactor

Conversation

@GiggleLiu
Copy link
Contributor

Summary

Closes #680

  • SimpleGraph/DirectedGraph serialize as {"num_vertices":N,"edges":[[u,v],...]} instead of petgraph internals
  • ModelExampleSpec is pure data (Box<dyn DynProblem> + optimal config + value) — no solver calls at construction time
  • RuleExampleSpec uses thin closures with pre-stored solution pairs — runs reduce_to() only, no solver
  • Static examples.json removed — generated on demand via export_examples binary for paper export
  • Self-consistency tests verify evaluate_json(optimal_config) == stored optimal_value
  • Typst paper updated for new JSON format and single-optimal-config model examples

Test plan

  • All 2377 tests pass (cargo test --features "ilp-highs example-db" -- --include-ignored)
  • Clippy clean (cargo clippy --features "ilp-highs example-db" -- -D warnings)
  • Self-consistency test catches real bugs (found and fixed invalid MIS config on Petersen graph)
  • make paper builds PDF successfully (requires Typst)
  • ILP cross-validation tests (deferred to follow-up issue)

🤖 Generated with Claude Code

GiggleLiu and others added 11 commits March 18, 2026 12:21
Replace petgraph's internal serde format with minimal representations:
- SimpleGraph: {"num_vertices": N, "edges": [[u,v],...]}
- DirectedGraph: {"num_vertices": N, "arcs": [[u,v],...]}

Update all .inner. graph accessor patterns in the Typst paper (24 sites).
Regenerate examples.json fixture with new format.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace closure-based ModelExampleSpec with pure data: Box<dyn DynProblem>
instance, optimal_config, and optimal_value (serde_json::Value).

- Add evaluate_json to DynProblem trait
- Simplify ModelExample: remove SampleEval, samples field
- Migrate all 44 model spec files to new format
- No solver runs at spec construction time

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove all solver-calling helpers from specs.rs (direct_best_example,
direct_satisfying_example, direct_ilp_example, etc.). Add
rule_example_with_witness helper. Migrate all 35 rule spec files to
use hardcoded solution pairs instead of solver calls.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- build_example_db() now computes from specs directly (fast, no solver)
- Delete static fixture file and regeneration binary
- Add export_examples binary that writes to docs/paper/data/ (gitignored)
- Update paper pipeline: make paper runs export_examples first
- Update Typst paper for new ModelExample format (single optimal_config
  instead of Vec<SampleEval>)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove normalization helpers and fixture-vs-computed verification tests
(tautological now that build == compute). Add model_specs_are_self_consistent
test (evaluate_json must match stored optimal_value) and
rule_specs_solution_pairs_are_consistent test.

Fix MIS/SimpleGraph/One optimal_config (was invalid IS on Petersen graph)
and MIS/SimpleGraph/i32 optimal_value (was 12, correct is 10).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Clarify that expected outcome is stored as ground truth and
cross-validated by ILP. Allow dead_code on ModelExampleSpec
(id field is used by test code).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Resolve conflicts:
- example_db/fixtures/examples.json: deleted (our branch removed it)
- strong_connectivity_augmentation.rs: adopt main's new instance, convert to new spec format
- longestcommonsubsequence_ilp.rs: adopt main's new LCS constructor, compute correct witness

Convert 8 new model files from main's old `build:` closure format to new
`instance: Box::new(...)` + `optimal_config` + `optimal_value` format:
hamiltonian_circuit, longest_common_subsequence, sequencing_with_release_times_and_deadlines,
staff_scheduling, string_to_string_correction, sum_of_squares_partition,
consecutive_sets, minimum_cardinality_key.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR standardizes graph JSON serialization, removes the embedded example-db fixture workflow in favor of in-code “ground truth” specs (optimal configs/values + rule witnesses), and adds helper APIs to solve type-erased problems via reduction to ILP (used by the CLI and tests).

Changes:

  • Implement custom serde JSON formats for SimpleGraph / DirectedGraph ({num_vertices, edges} / {num_vertices, arcs}) and update affected tests/CLI fixtures.
  • Replace example-db fixtures + “compute” pipeline with spec-driven construction using DynProblem, optimal_config, optimal_value, and pre-stored rule witness pairs.
  • Add DynProblem::evaluate_json and ILPSolver::solve_via_reduction, and switch CLI ILP solving to use the new API.

Reviewed changes

Copilot reviewed 110 out of 112 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/topology/graph.rs Custom Serialize/Deserialize for SimpleGraph using {num_vertices, edges}
src/topology/directed_graph.rs Custom Serialize/Deserialize for DirectedGraph using {num_vertices, arcs}
src/unit_tests/topology/graph.rs Add JSON roundtrip/format tests for new SimpleGraph schema
src/unit_tests/topology/directed_graph.rs Add JSON roundtrip/format tests for new DirectedGraph schema
src/unit_tests/models/graph/multiple_choice_branching.rs Update test JSON inputs to new directed graph schema
src/registry/dyn_problem.rs Add evaluate_json + require Metric: Serialize; use variant_to_map helper
src/solvers/ilp/solver.rs Add solve_dyn and solve_via_reduction for type-erased solving via ILP reductions
src/export.rs Replace samples/optimal with optimal_config/optimal_value; remove SampleEval
src/example_db/specs.rs Redefine model specs as DynProblem + optimal ground truth; add rule_example_with_witness
src/example_db/model_builders.rs Build exported ModelExample from ModelExampleSpec (instance JSON + optimal data)
src/example_db/rule_builders.rs Remove compute-time rule example tests; build rule examples directly from specs
src/example_db/mod.rs Remove fixture loading/computation split; build DB directly from specs
src/unit_tests/export.rs Refactor export tests (shared sample DB + temp dir helper; new model example fields)
src/unit_tests/example_db.rs Update tests to new model fields; add self-consistency / witness validation tests
src/rules/travelingsalesman_qubo.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/travelingsalesman_ilp.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/spinglass_qubo.rs Switch canonical rule example specs to explicit SolutionPair witnesses
src/rules/spinglass_maxcut.rs Switch canonical rule example specs to explicit SolutionPair witnesses
src/rules/sat_minimumdominatingset.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/sat_maximumindependentset.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/sat_ksat.rs Switch canonical rule example specs to explicit SolutionPair witnesses
src/rules/sat_coloring.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/sat_circuitsat.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/qubo_ilp.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/minimumvertexcover_minimumsetcovering.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/minimumvertexcover_maximumindependentset.rs Switch canonical rule example specs to explicit SolutionPair witnesses
src/rules/minimumsetcovering_ilp.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/minimumfeedbackvertexset_ilp.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/minimumdominatingset_ilp.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/maximumsetpacking_qubo.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/maximumsetpacking_ilp.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/maximummatching_maximumsetpacking.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/maximummatching_ilp.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/maximumindependentset_maximumsetpacking.rs Switch canonical rule example specs to explicit SolutionPair witnesses
src/rules/maximumindependentset_maximumclique.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/maximumclique_maximumindependentset.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/maximumclique_ilp.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/longestcommonsubsequence_ilp.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/ksatisfiability_subsetsum.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/ksatisfiability_qubo.rs Switch canonical rule example specs to explicit SolutionPair witnesses
src/rules/knapsack_qubo.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/ilp_qubo.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/factoring_ilp.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/factoring_circuit.rs Replace computed witness generation with explicit SolutionPair witness
src/rules/coloring_qubo.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/coloring_ilp.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/circuit_spinglass.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/circuit_ilp.rs Switch canonical rule example spec to explicit SolutionPair witness
src/rules/binpacking_ilp.rs Switch canonical rule example spec to explicit SolutionPair witness
src/models/set/set_basis.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/set/minimum_set_covering.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/set/minimum_cardinality_key.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/set/maximum_set_packing.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/set/exact_cover_by_3_sets.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/set/consecutive_sets.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/set/comparative_containment.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/misc/sum_of_squares_partition.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/misc/string_to_string_correction.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/misc/staff_scheduling.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/misc/shortest_common_supersequence.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/misc/sequencing_within_intervals.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/misc/sequencing_with_release_times_and_deadlines.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/misc/paintshop.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/misc/multiprocessor_scheduling.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/misc/minimum_tardiness_sequencing.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/misc/longest_common_subsequence.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/misc/factoring.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/undirected_two_commodity_integral_flow.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/traveling_salesman.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/strong_connectivity_augmentation.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/steiner_tree.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/spin_glass.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/partition_into_triangles.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/multiple_choice_branching.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/minimum_vertex_cover.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/minimum_sum_multicenter.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/minimum_multiway_cut.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/minimum_feedback_vertex_set.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/minimum_dominating_set.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/maximum_matching.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/maximum_independent_set.rs Convert canonical model examples to instance + optimal_config/optimal_value
src/models/graph/maximum_clique.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/maximal_is.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/max_cut.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/length_bounded_disjoint_paths.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/kcoloring.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/isomorphic_spanning_tree.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/hamiltonian_path.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/hamiltonian_circuit.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/directed_two_commodity_integral_flow.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/bounded_component_spanning_forest.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/biconnectivity_augmentation.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/biclique_cover.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/graph/balanced_complete_bipartite_subgraph.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/formula/sat.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/formula/ksat.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/formula/circuit.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/algebraic/qubo.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/algebraic/ilp.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/algebraic/closest_vector_problem.rs Convert canonical model example to instance + optimal_config/optimal_value
src/models/algebraic/bmf.rs Convert canonical model example to instance + optimal_config/optimal_value
problemreductions-cli/tests/cli_tests.rs Update CLI JSON fixtures to new graph schema
problemreductions-cli/src/dispatch.rs Use solve_via_reduction; update graph JSON parsing assumptions in tests
examples/regenerate_fixtures.rs Remove fixture regeneration example binary
examples/export_examples.rs Add example-db export binary for paper data generation
docs/paper/reductions.typ Update paper to load generated examples JSON and new model/rule fields
Makefile Remove fixture regeneration target; paper build now generates examples data on demand
Cargo.toml Swap example binary registration to export_examples
.gitignore Ignore generated docs/paper/data/ artifacts; stop whitelisting fixture JSON
.github/ISSUE_TEMPLATE/problem.md Clarify that examples are stored as ground truth and ILP-cross-validated
.claude/skills/final-review/SKILL.md Update final-review checklist guidance
src/unit_tests/models/graph/multiple_choice_branching.rs Update embedded JSON fixtures to match new directed graph schema

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +345 to +351
for spec in specs {
let actual = spec.instance.evaluate_json(&spec.optimal_config);
assert_eq!(
actual, spec.optimal_value,
"Model spec '{}': evaluate(optimal_config) = {} but stored optimal_value = {}",
spec.id, actual, spec.optimal_value
);
Comment on lines +451 to +459
for pair in &example.solutions {
assert!(
!pair.source_config.is_empty(),
"Rule {label} has empty source_config"
);
assert!(
!pair.target_config.is_empty(),
"Rule {label} has empty target_config"
);
@codecov
Copy link

codecov bot commented Mar 18, 2026

Codecov Report

❌ Patch coverage is 99.43289% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.25%. Comparing base (6053e8f) to head (e67e42d).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/unit_tests/example_db.rs 93.47% 6 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #696      +/-   ##
==========================================
+ Coverage   97.23%   97.25%   +0.01%     
==========================================
  Files         316      316              
  Lines       41105    41058      -47     
==========================================
- Hits        39970    39930      -40     
+ Misses       1135     1128       -7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@GiggleLiu GiggleLiu merged commit 9623130 into main Mar 18, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refactor example_db: human-friendly JSON format and test-from-fixture workflow

2 participants