Static benchmarks ask what a model knows. ChemWorld asks how it experiments when the answer is hidden.
ChemWorld is a replayable causal physical-chemistry environment for training and evaluating experimental agents. It separates a physical causal world substrate, an experimental interaction runtime, and task/evaluation contracts. Under partial observability, finite budgets, and operational constraints, agents choose interventions and measurements, form hypotheses, and revise their strategies from feedback generated by executable mechanisms.
ChemWorld provides the environment and evaluation contracts; it does not retrain a hosted Agent during an evaluation campaign. Agent intelligence may live in externally trained weights, while within-campaign adaptation is measured through memory, belief, and action updates.
The same public task can run across different hidden kinetic, phase-behavior, or process worlds. This makes it possible to test whether an agent identifies and adapts to changing rules instead of memorizing one optimal recipe. Bayesian optimization, reinforcement learning, world models, LLM tool agents, and humans share the same versioned interaction and replay contracts.
ChemWorld does not claim to predict arbitrary real reactions. Its first goal is to make experimental decision making scalable, comparable, and falsifiable; its long-term bridge question is whether virtual training reduces the number, risk, and cost of experiments needed to adapt to independent models, real data, and narrow physical systems.
Website: https://sunyrain.github.io/ChemWorld/
| Layer | Role |
|---|---|
| Physical Causal World Substrate | Typed state, executable physical laws, instruments, constraints, and controlled world shifts |
| Experimental Interaction Runtime | Operations, transactions, measurements, lifecycle, resource ledgers, and trajectories |
| Task and Evaluation Contract | Public objectives, information/action permissions, budgets, scoring, and scenario distributions |
| ChemWorld Interface | Optional Student and Agent Observatory views over the same environment API |
| ChemWorld Bridge | A validation roadmap for independent backends, real datasets, and physical systems |
The agent, trainer, model weights, and private agent memory sit outside the three environment layers. Core,
Diagnostic, and Extended refer only to evaluation-suite roles, not separate engines.
ChemWorld supports Python 3.11 and 3.12.
git clone https://github.com/sunyrain/ChemWorld.git
cd ChemWorld
python -m pip install -e ".[dev]"The default development extra collects the complete suite and skips optional RL modules when their
backend is absent. RL development and training use python -m pip install -e ".[dev,rl]", which also
installs the supported Stable-Baselines3/Torch stack.
Run and independently verify a complete trajectory:
chemworld tasks list
chemworld run --task reaction-to-assay --agent random --seed 0
chemworld verify --constitution --submission runs/<trajectory>.jsonl
chemworld evaluate --submission runs/<trajectory>.jsonlEvaluation replays the trajectory and recomputes metrics rather than trusting a score supplied by the agent.
import gymnasium as gym
import chemworld
env = gym.make("ChemWorld", task_id="reaction-to-assay", seed=0)
observation, info = env.reset(seed=0)
action = {"operation": "add_reagent", "amount_mol": 0.01}
check = env.unwrapped.validate_action(action)
if check["valid"]:
observation, reward, terminated, truncated, info = env.step(action)
env.close()Runnable notebooks for all six serious research tasks are collected in
notebooks/task_demos/. They compare public feedback across candidate
interventions and paired opaque worlds without reading hidden state or prescribing an Agent training method.
Unmeasured array values are NaN (null in JSONL). Read observed_mask or observed_keys before using an
observation field.
python -m apps.task_lab.server --port 8876- Agent Observatory: http://127.0.0.1:8876/agent/
- Student Lab: http://127.0.0.1:8876/student/
The documentation site is static; the visual interface currently runs locally. Classical agents work offline. Online model credentials are read from the local process environment and must not enter the repository or evaluation artifacts.
Pre-v0.5 algorithm numbers are retained only as protocol and failure-mode diagnostics; they are not current rankings.
The chemworld-physical-chemistry-v0.5-candidate backend has deterministic controls for all 15 registered tasks,
including exact task-contract hashes, replay-bound transitions, and a fail-closed public boundary. Backend validation
and clean-source attestation describe the environment only; they do not imply an algorithm ranking, a completed
evaluation campaign, or real-chemistry validation.
All 15 tasks have executable complete-experiment contracts, but only two
flagship tasks have formal multi-world Participant results. Those tasks have
completed both a no-dossier descriptive campaign and a paired
opaque / nominal / misindexed anonymous-material study. Correct information
has confirmed positive value in electrochemistry and an inconclusive effect in
crystallization. A targeted wrong prior changes early behavior in both tasks,
but neither passes the joint recovery rule.
configs/current.json is the authoritative status surface. It reports backend-contract
validation, clean-release attestation, formal-evaluation readiness, and publication readiness as separate states.
Historical RC28 Gate A passed on its frozen source, but its current-source
binding is stale. Participant Gates B–E, private confirmation, and external
bridge evidence remain incomplete. Current benchmark_ready=false and
publication_ready=false; the evidence does not support broad SOTA,
mechanism-adaptation, provider-causal, or real-world-transfer claims.
See Evidence and current status before citing results, and Flagship experiments for exact numbers and evidence links.
| Goal | Page |
|---|---|
| Understand the research thesis | Why ChemWorld |
| Read the normative system model | System Model |
| See how causal worlds change | Causal Worlds |
| Explore Showcase Worlds | Worlds |
| Inspect Confirmatory Benchmark Tasks | Confirmatory Tasks |
| Choose an agent interaction level | Agent Tracks |
| Build an agent | Getting Started |
| Design an evaluation | Benchmark Design |
| Understand the real-world roadmap | Real-world Bridge |
Software quality gates and backend audits establish checkout integrity; they do not authorize a scientific benchmark or real-world claim. Maintainer-only release procedures are intentionally kept out of this user guide. Contributors should use the repository development and evidence guide for project maintenance workflows.