Repository navigation
Reproducing the Paper
The headline result: GEAR solves 421 of the 505 instances within 180 seconds (251 of 265 in gams_world, 170 of 240 in minlplib_linear; counted from logs/gear_results.tsv). The released logs and tables let you check the numbers without running anything, and the commands below re-run the solver.
An instance is solved when both hold:
- The returned point satisfies every constraint and variable bound of the original model to an absolute tolerance of 1e-6.
- The primal value P and the certified bound D satisfy
(P - D) / max(1, |P|, |D|) <= 1e-4.
An instance that is proved infeasible counts as solved. Solved counts for GEAR and for the 12 AMPL solvers use this same rule (baselines/README.md).
The paper's 421 of 505 is the run in logs/ (logs/gear_results.tsv: 420 certified and 1 certified infeasible). It used logs/configs/int3p_crownmerge.yaml, a time limit of 180 s and one NVIDIA RTX 5090, one instance at a time. The released code is the cleaned version of the solver with the f3 configuration (exp_configs/NLP_solver/f3.yaml, the API preset gear-f0); a re-run does not use the YAML file of the logs.
Two runs of the released code on one RTX 5090 solved 419 and 422 of the 505 instances under the same scoring rule. kriging_peaks-full200 and wastewater05m1, both certified in the paper's run, are not certified by the released code in either run. On kriging_peaks-full200 the search processes fewer boxes per step than in the paper's run, for two reasons: a change in CUDA graph capture that no longer releases cached GPU memory before capture (being fixed), and the floating-point enclosure that the released configuration enables, which needs more memory per box. The cause on wastewater05m1 is not yet identified.
logs/gear_results.tsv has one row per instance. The count of instances with certified equal to 1 (column 4) is the GEAR count of the paper run. The per-solver counts of the AMPL baselines are in baselines/results/per_instance_long.tsv and baselines/results/report.md.
The loop below runs the first two instances of benchmarks/instances.tsv (N=2) with a 60 s limit and writes a log and a result file per instance under runs/gear/:
export PYTHONPATH=$PWD/solver:$PWD/solver/complete_verifier PYTHONNOUSERSITE=1 N=2; tail -n +2 benchmarks/instances.tsv | head -n $N | while IFS=$'\t' read -r family name; do mkdir -p runs/gear/$family; (cd solver/complete_verifier && ../../env/bin/python nlp_example.py --problem_name $name --benchmark $family --benchmark_root ../../benchmarks --config exp_configs/NLP_solver/f3.yaml --device cuda --use_beta False --override_timeout 60 --batch_size 1 --result_json ../../runs/gear/$family/$name.json) > runs/gear/$family/$name.log 2>&1; done; ls runs/gear/*; grep -h '^problem_name=' runs/gear/*/*.log; grep -h '^problem_name=' runs/gear/*/*.log | grep -c 'status=verified'The paper setting is N=505 and --override_timeout 180 (the loop above runs N=2 with 60 s). The last command counts the instances with status=verified. The scoring rule above additionally needs the strict check, which is the replay entry of each result file. Run the instances one at a time on one GPU.
logs/gear_results.tsv has one row per instance (family, instance, sense, certified, status, P, D, relative_gap, time_s). Compare your P, D and certified flag per instance.
- Compare the number of certified instances and the certified flag, P and D per instance. Run times depend on the GPU and the host, so compare times per instance only between runs on the same machine.
- Local paths in the released logs are shortened (
benchmarks/,runs/,<gear>/,<env>/,<root>,~,<host>). -
logs/MANIFEST.tsvandlogs/SHA256SUMSgive sizes and checksums of every log and archive.
logs/<solver>.tar.xz holds the final runs of BARON, Couenne, SCIP, LindoGlobal, Gurobi, Xpress, LGO, Knitro, IPOPT, SNOPT, MINOS and CONOPT, with the exact options in baselines/commands.md. Per-instance results and the scoring script are in baselines/results/ and baselines/harness/report.py. Re-running them needs the commercial licenses of the respective solvers.