ConstraintLLM Open-Source Release

Project Overview

This repository provides the open-source release package of the paper ConstraintLLM: A Neuro-Symbolic Framework for Industrial-Level Constraint Programming.

ConstraintLLM is a neuro-symbolic framework for industrial-level constraint programming. It combines large language models with symbolic solvers for automatic CP model generation and evaluation. This release includes:

• a minimal runnable implementation of the ConstraintLLM evaluation pipeline;
• the released IndusCP dataset under data/induscp/;
• the released retrieval knowledge base under data/knowledge_base.json;
• support for DIRECT, COT, RAG, CARM, and TOT methods.

In this release, CARM is the open-source method name for the constraint-aware retrieval module described in the paper.

Paper page and download link:

• https://aclanthology.org/2025.emnlp-main.809/

Project Structure

open_source_release/
├── src/constraintllm/        # Core pipeline implementation
├── data/induscp/             # IndusCP test set
├── data/knowledge_base.json  # Retrieval corpus for RAG/CARM/TOT
├── scripts/                  # Convenience run scripts
├── configs/                  # Configuration templates
├── outputs/                  # Experiment summaries and detailed results
├── logs/                     # Runtime logs generated by parallel runs
├── README.md
├── README_CN.md
└── LICENSE

Installation

Linux / macOS

Create and initialize a virtual environment with:

bash setup_venv.sh
source .venv/bin/activate

Windows

Create and initialize a virtual environment with:

setup_venv.bat

Solver and Java Requirement

This project executes generated PyCSP3 models and therefore requires a working Java runtime for the underlying solver stack.

Recommended environment:

• Python 3.10 or newer
• Java 8 or newer
• java available in PATH

Quick check:

java -version
python --version

If Java is missing, install an OpenJDK distribution first. Without Java, model execution may fail even if the LLM endpoint is correctly configured. Java 8 or newer is typically sufficient; Java 11 or newer is recommended for a more stable environment.

Model Endpoint Configuration

Before running experiments, set an OpenAI-compatible model endpoint:

export CONSTRAINTLLM_API_BASE="https://api.openai.com/v1"
export CONSTRAINTLLM_API_KEY="YOUR_OPENAI_API_KEY"
export CONSTRAINTLLM_MODEL="gpt-5"

Methods

This release provides the following methods:

• DIRECT: directly generate PyCSP3 code from the problem statement.
• COT: generate PyCSP3 code with a chain-of-thought style coding prompt.
• RAG: retrieve reference examples by embedding similarity between the current problem statement and the retrieval corpus.
• CARM: retrieve reference examples by constraint-tag overlap.
• TOT: retrieve references, generate multiple modelling plans, turn them into candidate programs, and keep the best candidate under a lightweight quick check.

The bundled knowledge base at data/knowledge_base.json is intended for retrieval in RAG, CARM, and TOT. It is not part of the IndusCP test set itself.

Quick Start

Run a minimal COT example on the first problem:

PYTHONPATH=src python -m constraintllm.run_csp --method COT --max-items 1

Run a minimal RAG example:

PYTHONPATH=src python -m constraintllm.run_csp --method RAG --max-items 1

Run a minimal CARM example:

PYTHONPATH=src python -m constraintllm.run_csp --method CARM --max-items 1

Run a minimal TOT example:

PYTHONPATH=src python -m constraintllm.run_csp --method TOT --max-items 1

Full Experiment Examples

Run full experiments with the convenience scripts:

export CONSTRAINTLLM_API_BASE="https://api.openai.com/v1"
export CONSTRAINTLLM_API_KEY="YOUR_OPENAI_API_KEY"
export CONSTRAINTLLM_MODEL="gpt-5"

bash scripts/run_cot.sh
bash scripts/run_rag.sh
bash scripts/run_carm.sh
bash scripts/run_tot.sh

Run with parallel shards:

export CONSTRAINTLLM_PARALLEL_WORKERS=4
bash scripts/run_rag.sh

Output Files

Experiment outputs are written under outputs/:

• *_summary.json: aggregate experiment statistics for one run or one shard
• *_details.json: per-case detailed outputs, scores, and generated code

Runtime logs are written under logs/:

• parent log: the launcher process
• shard logs: one log file per parallel shard

Known Limitations

• This is a research release, not a production deployment package.
• Throughput and stability depend heavily on the model service and its concurrency capacity.
• Large models may become very slow under high parallel load.
• Some reference outputs in the released data may still require additional manual verification.
• The retrieval knowledge base is released for research and reproduction use. It should not be treated as part of the IndusCP test set.
• The knowledge base provenance is documented at:
  • docs/knowledge_base_license_review.md

Citation

If you use this project, please cite the following paper:

@inproceedings{shi2025constraintllm,
  title={ConstraintLLM: A Neuro-Symbolic Framework for Industrial-Level Constraint Programming},
  author={Shi, Weichun and Liu, Minghao and Zhang, Wanting and Shi, Langchen and Jia, Fuqi and Ma, Feifei and Zhang, Jian},
  booktitle={Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  pages={16010--16030},
  year={2025}
}