Skip to content

Troubleshooting

Lemutisme edited this page Nov 7, 2026 · 1 revision

Troubleshooting

The first run is slow

The first run of an instance writes a .vnnlib.compiled cache next to its .vnnlib file in benchmarks/, and on a GPU the first solve compiles GPU kernels (Triton). Run one small instance first (Quick Start) and do not time it. The benchmarks/ directory must therefore be writable. The .compiled files are not part of SHA256SUMS.

Out of GPU memory

  • The status MEMORY_LIMIT in the Python API (Result.termination) means the search stopped on memory.
  • Lower the number of boxes per batch with batch=gr.Batch(domains=...) (solver.batch_size) and set solver.auto_enlarge_batch_size to false so the runner does not grow it again (Configuration).
  • Lower bab.domain_storage_gpu_mb, the memory in which stored boxes stay on the GPU; boxes beyond it are kept in host memory.
  • Lower solver.batched_crown_auto_enlarge_max_vram_ratio to leave memory free.
  • Close other processes on the GPU. Result.stats.peak_gpu_mb shows the peak of the last solve.
  • The largest instances of the benchmark need the most memory; on a smaller GPU than the one used for the paper, some instances may stop earlier.

CUDA, PyTorch and Triton versions

  • The tested environment is PyTorch 2.8.0 built for CUDA 12.8 (torch==2.8.0 from the cu128 index, see Installation). The check at the end of Installation prints the PyTorch version and whether CUDA is available.
  • A Triton version that does not match the PyTorch build fails when the first kernel is compiled. Use the Triton version that torch==2.8.0 installs.
  • The fused kernel is used only on CUDA in float64. If it fails, GEAR switches it off and continues with the standard implementation, which is slower.

No GPU

Use --device cpu (or device="cpu") for small tests. Expect much longer run times, and do not compare CPU results with the GPU results in logs/.

Results differ from the released logs

Compare certified flags and instance counts, not times (Reproducing the Paper). Check Result.provenance for the preset, device and commit, and the Configurations: note in Output and Logs.

The CPU solvers

The default preset calls an LP solver on the CPU and IPOPT in parts of the search (Python API). The CPU solvers a solve used are listed in Result.provenance.

Clone this wiki locally