ADToolbox is developed by a joint team from East China Normal University (ECNU) and the Shanghai Innovation Institute (SII).
Luosong Guo · Haochun Gao · Xiang Yang · Dinghao LIn · Yi Luo
Zhenyu Hou · Ao Gao · Chaoyang Li · Jiangming Shi† · Zhizhong Zhang† · Jingyu Gong† · Yuan Xie†
† Project leaders.
Our work focuses on reusable tools for driving video retrieval, dynamic 3D reconstruction, scene editing, and policy evaluation, making these capabilities easier to use independently and integrate into research workflows.
ADToolbox brings failure evidence, scenario design, video retrieval, dynamic reconstruction, editing and policy evaluation behind a consistent interface. Use one tool from Python, call it from a terminal, or expose it to an MCP client. Each call has explicit inputs and inspectable outputs.
The library includes 14 independent tools with real implementations, and the heavy compute engines ship with the repository: a dynamic Gaussian reconstruction engine with editing extensions, a closed-loop simulation engine, and a policy training stack live under projects/. Orchestration and next-step selection belong to your own application — ADToolbox deliberately does not include an agent framework.
git clone https://github.com/Max-luo-song/tool_box.git
cd tool_box
python -m venv .venv
source .venv/bin/activate
python -m pip install -e .
adtoolbox list
adtoolbox run edit_config_generator --input examples/edit_config_generator/input.json
adtoolbox run failure_case_filter --input examples/failure_case_filter/input.jsonThese examples compile a real edit document and screen synthetic box geometry. They need no API key, GPU or model download. The curation example also writes a local JSONL case entry. External-data examples use explicit path templates.
from adtoolbox.tools.edit_config_generator import run
result = run({
"coordinate_system": "camera_local_meters",
"operations": [{"operation": "remove", "instance_id": 0}],
})
assert result["status"] == "success"
print(result["result"]["edit_config"])| Tool | Capability | Runtime |
|---|---|---|
failure_analyzer |
Failure evidence analysis | llm |
scene_requirement_generator |
Scenario requirements | llm |
scene_conceiver |
Scenario conception | llm |
video_retriever |
Driving video retrieval | native + engine |
instance_discovery |
Traffic instance tracking | optional vision / engine |
reconstruction |
Dynamic scene reconstruction | bundled engine |
edit_plan_builder |
Scene edit planning | llm |
edit_config_generator |
Edit configuration compiler | native + llm |
scene_editor |
Scene editing and rendering | bundled engine |
scene_validator |
Scene consistency checks | native + optional llm |
recovery_advisor |
Execution diagnostics | native |
policy_runner |
Closed-loop policy evaluation | bundled engine |
failure_case_filter |
Failure case curation | native |
policy_trainer |
Supervised policy fine-tuning | bundled engine |
Native tools run with the base package. LLM tools require your configured model endpoint. Bundled engine tools drive real engine source included under projects/; you still prepare the engine's Python environment, model weights and data. Native video retrieval ranks supplied captions; the bundled projects/video_search pipeline provides full semantic indexing and retrieval.
- One contract. Versioned JSON results, typed inputs, explicit artifacts and useful errors.
- Independent environments. Reconstruction, tracking, simulation and training can use separate runtimes.
- Portable scene edits. Trajectory, rotation, scale, removal, addition and replacement share one functional interface.
- Evidence with limits. Visual plausibility, geometric screening and official policy metrics remain distinguishable.
Configure only what you use. Start with runtime configuration, then follow the engine guides. Verify a bundled engine tree and print its setup checklist:
python scripts/setup_project.py reconstruction
python scripts/setup_project.py simulationAn edited reconstruction can be converted into a simulation scene with scripts/convert_scene.py, and a simulated rollout can be exported as supervised fine-tuning data with projects/simulation/scripts/export_policy_training_data.py. Existing integrations can also use the JSON subprocess bridge.
For model tools, export ADTOOLBOX_API_KEY, ADTOOLBOX_BASE_URL and ADTOOLBOX_MODEL. Text evidence and explicitly attached frames are sent to that endpoint. The library does not automatically load .env files.
python -m pip install -e '.[mcp]'
adtoolbox doctor --config configs/runtime.local.json
adtoolbox-mcp --config configs/runtime.local.jsonSee MCP setup, coordinate and data contracts, and per-tool examples.
This is an alpha toolkit release. Portable implementations, model transport and process/protocol contracts are covered by tests. The bundled engines are verified for source completeness and integration wiring; clean GPU reconstruction, edited-scene fidelity, closed-loop metrics and policy training outcomes depend on your hardware, checkpoints and data. Validation status records the distinction.
Box-overlap screening does not establish collision responsibility or road compliance. Sampled-frame review does not certify physics. Fine-tuning requires valid labels; a mined failure is not automatically a training example.
See contributing and security. Useful next steps are reproducible engine profiles, measured editing examples and additional portable scene adapters.
Standalone toolkit code is licensed under Apache-2.0. The bundled engines under projects/ are derived from Apache-2.0 upstream projects and retain their original licenses and attribution; see NOTICE, third-party notices and the engine guides. Model weights and datasets are not distributed.