PDDLCoder

Agentic LLM-Formalizer for the autonomous generation of applicable PDDL files from abstract natural language descriptions.

Overview

LLMs are unreliable when asked to generate plans directly, often producing logically inconsistent or non-applicable action sequences. LLM-Formalizers avoid this by translating natural language into the Planning Domain Definition Language (PDDL) and delegating planning to a symbolic planner, yielding plans that are verifiable relative to the generated model. Existing formalizers, however, rely on rigid generation pipelines, partial formal specifications, or human feedback — and are typically evaluated on intermediate properties such as syntactic validity rather than on whether the resulting plans actually solve the task.

PDDLCoder addresses these limitations with an agentic approach. Given only abstract NL descriptions of a planning domain and problem — without predicate signatures, action parameters, preconditions, or effects — a ReAct agent zero-shot creates, analyzes, and refines PDDL 1.2 domain and problem files using external verification tools, then solves them with Fast Downward. Instead of following a fixed refinement schedule, the agent treats syntax errors, unsolvable models, and semantically infeasible plans as different failure states and adaptively chooses the appropriate tool at each step, with no human feedback or predefined PDDL involved. The generated plan is then mapped to the target environment's action schema and verified for applicability by executing it step by step in a gym environment.

Complementing the method, this project is evaluated on NL-pddlgym, a benchmark of 711 planning problems across 23 domains that pairs each NL description with an executable pddlgym environment, enabling automated verification of plan applicability instead of proxy metrics. On its held-out test set (106 problems across 4 unseen domains), PDDLCoder generates applicable plans for 89.6% of problems with DeepSeek v4 Flash — compared to at most 45.3% for adapted versions of previous LLM-Formalizers and 74.5% for direct LLM-Planners.

How It Works

The agent is given a natural language task description and access to 8 tools, which it can call freely for up to 50 iterations:

• File tools — create_pddl_file, read_pddl_file, edit_lines (edits are applied to a temporary file first and rejected if VAL reports new syntax errors)
• Feedback tools — get_syntax_mistakes_domain, get_syntax_mistakes_problem (VAL syntax checks with line annotations), translate_pddl (Fast Downward translation as a stricter check), generate_plan (Fast Downward lama-first, capped at 1 min / 4 GB, with curated unsolvability hints), get_plan_feedback (LLM feedback on the physical/logical feasibility of the plan)

After the loop, the final PDDL files are validated and solved. The resulting plan is mapped to the action schema of the target environment by an LLM (MapAgent) and executed step by step in the corresponding pddlgym environment — a problem only counts as solved if every action is applicable and the terminal state satisfies the goal.

NL-pddlgym

The evaluation benchmark (provided by the nl-pddlgym package, sourced from a pddlgym fork) comprises 711 problems across 23 domains, each pairing NL domain/problem descriptions with an executable gym environment. The test set used by eval.py consists of 106 problems from 4 held-out domains (Elevator, Hanoi, Ring and Peg, Satellite) that never appear in the train/validation splits, leaving the remaining domains available for prompt optimization.

Pipelines

Pipeline	Description
pddl_coder	ReAct agent with the 8 tools listed above
rigid	Fixed step-by-step pipeline: generate domain → generate problem → fix syntax → fix unsolvability → incorporate plan feedback
chain_of_thought	CoT baseline without tools; generates a plan directly

Requirements

For a local setup (see below for the Docker alternative):

• Python 3.12 and uv
• VAL: the Parser binary must be on your PATH (requires SWI-Prolog to build)
• Fast Downward 24.06.1 checked out as a sibling directory of this repo (../fast-downward-24.06.1, path is hardcoded in src/eval/fast_downward.py); run all commands from the repo root

Installation

uv sync --locked

Model Access

Available models (see src/inference/__init__.py): deepseek_v4_flash (via OpenRouter) and gemma_4_31b, gpt_oss_120b, qwen_36_35b_a3b, glm_47_flash, llama_4_scout (local inference).

• OpenRouter: set the OPENROUTER_API_KEY environment variable or place the key in a .openrouter-key file in the repo root
• Local models: serve an OpenAI-compatible API (e.g. with vLLM) and set SERVER_ADDRESS (defaults to http://localhost:8000/v1); the committed .default-key file is used as the API key

Docker

The provided docker-compose.yml builds an image that already contains VAL and Fast Downward and runs eval.py as its entrypoint:

docker compose build pddlcoder

OPENROUTER_API_KEY=<your-key> docker compose run --rm pddlcoder \
    --pipeline pddl_coder --model deepseek_v4_flash

The results/, logs/, pddl/ and plans/ directories are bind-mounted, so all outputs land in the corresponding host directories.

The compose file also defines one vLLM service per local model behind the vllm profile (requires an NVIDIA GPU with the container toolkit; set HF_TOKEN for gated models):

docker compose --profile vllm up -d gpt_oss_120b

Each service exposes the OpenAI-compatible API on host port ${VLLM_PORT:-8000}, so only one model service can use a given port at a time. Once the server is healthy, run the evaluation on the host (the default SERVER_ADDRESS is http://localhost:8000/v1) or in the container via docker compose run --rm -e SERVER_ADDRESS=http://gpt_oss_120b:8000/v1 pddlcoder ....

Running Evaluation

Evaluation runs over the full NL-pddlgym test set (106 problems):

# PDDLCoder with DeepSeek v4 Flash, plan mapped before feedback (full pipeline)
uv run eval.py --pipeline pddl_coder --model deepseek_v4_flash --map_plan_for_feedback

# Rigid pipeline with a local model
uv run eval.py --pipeline rigid --model gemma_4_31b

# CoT baseline
uv run eval.py --pipeline chain_of_thought --model deepseek_v4_flash

# Disable specific tools for ablation (pddl_coder only; names are lowercase)
uv run eval.py --pipeline pddl_coder --model deepseek_v4_flash --ablate_tools get_plan_feedback,generate_plan

# Print model interactions to console and stream tokens as they arrive
uv run eval.py --pipeline pddl_coder --model deepseek_v4_flash --verbose --stream

Tool names for --ablate_tools: create_pddl_file, read_pddl_file, edit_lines, get_syntax_mistakes_domain, get_syntax_mistakes_problem, translate_pddl, generate_plan, get_plan_feedback.

Other flags:

• --map_plan_for_feedback — map the generated plan to the target action schema before producing plan feedback (off by default; the final plan is always mapped before applicability checking)
• --optimize — run GEPA prompt optimization on the train/validation splits and save the program as optimized_<model>.json
• --optimized_program <path> — load a previously optimized program

Outputs

• results/*.csv — per-problem metrics: failure stage in error (empty on success), runtime, input/output tokens, per-tool call counts, and paths to the generated domain/problem/plan/log files
• logs/ — per-problem debug logs with full model interactions
• pddl/generated/, plans/ — generated PDDL files and plans

Tests

uv run pytest