Two-stage JEV control of a Franka Panda in MuJoCo.
Measured simulator state → task intent → Cartesian motion and gripper commands.

Interactive showcase · Todo list · Demo presentation · Quick start · Evaluation · 中文

RoboJEV is a small, inspectable robotics laboratory. JEV receives structured simulator state, not images, selects an immediate intent, then selects X/Y/Z directions and a gripper command. A Cartesian controller executes the action using real MuJoCo contacts. Each task has independent physical success checks; model answers cannot declare success.

Completed

• Implement two-stage JEV control: intent selection followed by XYZ and gripper commands.
• Integrate MuJoCo and Franka Panda with structured state observations and physical contact.
• Demonstrate pick & place, surface pushing, and stacking on a fixed pedestal with real JEV decisions.
• Complete 100 evaluation episodes across five tasks in two frozen campaigns, with an independent rule baseline and documented failures.
• Publish demonstration videos, reproducible code, bilingual documentation, and an interactive showcase.

Next steps

• Add loose-fit peg insertion and gate obstacle pick & place with independent physical failure checks.
• Extend simulation experiments to further manipulation tasks and scene configurations.
• Explore additional simulation platforms and evaluate the framework across simulators.
• Adapt the framework for real robotic arms and validate control on physical hardware.
• Integrate OpenAI API calls as an additional model backend.
• Integrate Claude Code API calls as an additional model backend.

Pick & place	Surface push	Pedestal stack	Peg insertion	Gate pick & place
MP4 · seed 1000	MP4 · seed 1000	MP4 · seed 1000	MP4 · seed 0	MP4 · seed 2

Pick & place	Surface push	Pedestal stack	Peg insertion	Gate pick & place
10/10 succeeded. No natural failure video.	10/10 succeeded. No natural failure video.	2/10 failed; no failure recording in the original campaign. Failure evidence	10/10 succeeded. No natural failure video.	MP4 · seed 0

All five tasks appear in the same order in both parts. Videos show real JEV decisions and follow simulation time; API waits are omitted. The original three success videos use separate demonstration seeds; insertion and gate recordings come directly from formal evaluation. Stack uses a fixed pedestal. Its two validation failures were not recorded as videos in the original campaign; their evidence remains in the report.

Fixed seeds 0–9 per task and policy; 100/100 episodes complete across the original 60-trial and new 40-trial frozen campaigns. Every completed failure remains in the denominator.

JEV Wilson 95% is a confidence interval for the success rate under the evaluated conditions. With only ten trials per task, even 10/10 successes leaves substantial uncertainty; it does not guarantee future success.

The independent rule baseline uses the same physical scene and success checks. It is never a fallback for JEV. See the protocol and failure analysis and machine-readable summary. Ten seeds per task is a small sample, not a claim of general-purpose manipulation.

New challenge recordings: insertion success, gate success, and gate failure. Insertion had no natural failure. Gate failures exposed arm-link collisions while lowering, repeated wrong-direction requests exhausting the decision budget, and one inconsistent model response. See all seven failed trials and measured boundaries.

The interactive comparison now shows the completed September 27 campaign: 40 fixed JEV trials, 20 paired scenes, and 40/40 offline verifications with zero network requests. It is separate from the five-task, 100-episode evaluation above. Original input remains the default; full geometry is a simulator-only comparison.

No success-rate improvement was observed. Single-gate paired outcomes: both succeed 2, full geometry only 0, original only 1, both fail 7. Double-gate outcomes: both fail 10. Exact McNemar two-sided p=1.00 for each task; ten pairs are insufficient to establish equivalence or general harm from extra geometry.

Physics pauses during API retries; received answers are preserved, including invalid choices. All 113 transport errors were recovered. The 38 pre-recharge HTTP 402 responses are recorded separately as account blocks, with no unresolved slots or infrastructure failures in the final outcome counts. Ten model-response validation failures remain in the denominator. Six original-trial recordings show successes and collision failures; neither double-gate group had a successful trial in this campaign. See the full paired analysis, recovery protocol, and auditable JSON. The September 23 comparison remains a separate archive.

• State: TCP and object poses, velocity, jaw width, contact measurements, geometric relationships and recent actions. No camera input or evaluator history is supplied to JEV.
• Intent: a real JEV choice over task-specific intentions; task instructions and geometric helpers are engineered explicitly.
• Motion: negative / zero / positive for each axis; open / hold / close for the gripper. Nonzero axes form a normalized 1 cm translation in robot-base coordinates, reduced to 4 mm while a grasped peg is within 25 mm of the socket axis. Orientation stays downward; both policies share this step selection.
• Execution: Jacobian damped least-squares IK, joint position control and 2 ms physics steps. Grasping uses friction and contact, with no welded object or scripted object trajectory.
• Validation: malformed or inconsistent responses execute no action. Simulation pauses during API calls. Each two-stage decision makes at least two API requests.

This project demonstrates task-conditioned control from privileged state. It does not provide learned vision, robot training, arbitrary object manipulation, or real-robot deployment.

Linux with Python 3.11 is the tested platform. CPU physics and the rule policy need no GPU or API key. A graphics backend is required only for recording.

git clone https://github.com/lykycy123/RoboJEV.git
cd RoboJEV
conda env create -f environment.yml
conda activate jev-vla-sim
python -m pip install -e '.[test,video]'
python scripts/fetch_panda.py
robojev --task pick_place --policy rule --seed 1000

Alternatively install into an existing Python 3.11 environment with pip install -e '.[test,video]'. fetch_panda.py downloads only pinned robot assets, verifies checksums, and never runs implicitly at simulation startup. GPU support is not required for physics; H100 EGL rendering has been tested.

Install the optional console once, then operate RoboJEV from a local browser without editing configuration files:

python -m pip install -e '.[ui]'
robojev-ui
# open http://127.0.0.1:8767/

The console configures six tasks, including the new staggered double gate, rule or JEV policies, paired batches, seeds, workers and original-state capture for later videos. Original input remains the default. Full geometry is an optional simulator-only ablation; it assumes measurements that are difficult to obtain in reality. The Set up 40-trial comparison button pairs both observation profiles on both gate tasks. See the experiment protocol and completed paired results and six original-trial recordings. For the same-request transport recovery used in the final comparison, use the resilient experiment driver; the console preset configures the paired scenes. The final comparison found no success-rate improvement, with substantial uncertainty from ten pairs per task.

It keeps a local SQLite history under ignored runs/ui/, preserves completed trials when a batch is stopped, and never puts a TypeSafe key in commands or logs. Keys are session-only by default; selecting Remember on this machine stores a private file with restrictive permissions under the user config directory. On a remote Linux server, use ssh -N -L 8767:127.0.0.1:8767 user@host and open the same local URL. The console binds to localhost only.

The console requires Linux or WSL2 because the tested MuJoCo environment and off-screen rendering use Linux graphics backends. Physics can run without a GPU; video generation needs a working EGL or OSMesa backend and can be retried from the Run detail page when a GPU node is available.

For real JEV, obtain a TypeSafe API key, copy .env.example to .env, and set TYPESAFE_API_KEY. The file is ignored by Git. Model calls use jev-1.13.0 at https://api.typesafe.ai/v1/systemone and may incur API charges.

cp .env.example .env
chmod 600 .env
# Edit .env locally; do not put the key into committed files.
robojev --task push --policy jev --seed 1000
robojev --task stack --policy jev --seed 1000 --record-video

Standard HTTPS_PROXY is supported where necessary. No cluster, account or private proxy settings are required by the public code. The original jev-sim command remains an alias for robojev; the Python package is jev_vla_sim.

The new peg_insert and obstacle_pick_place tasks add loose-fit insertion and transport through a narrow gate over a low crossbar. The insertion uses a 20 mm peg, 30 mm socket and at least 30 mm depth. Gate traversal requires a 40 mm cube to clear a 120 mm crossbar between posts with a 70 mm opening. See physical boundaries and the challenge report.

robojev --task peg_insert --policy rule --seed 1000
robojev --task obstacle_pick_place --policy jev --seed 1000
python scripts/evaluate_suite.py --tasks peg_insert obstacle_pick_place --workers 4 --capture-video-state
# Render original success/failure trial states with one EGL process, without calling JEV again.
python scripts/render_captured_episode.py runs/robojev-evaluation/RUN_ID
python scripts/export_challenge_results.py runs/robojev-evaluation/RUN_ID

Challenge videos come from the fixed-seed evaluation itself. The earliest success and earliest natural failure are selected per task; no failure is manufactured when all ten JEV trials succeed. Captured physical states, original model probabilities and failure evidence are preserved. Videos omit API waits and append a labeled two-second outcome still. Original three-task videos retain their separate demonstration seeds and source fingerprints.

# Original CPU-only paired evaluation: 3 tasks × 10 seeds × 2 policies = 60 episodes.
python scripts/evaluate_suite.py --tasks pick_place push stack --workers 4

# Resume incomplete work with identical source/configuration; completed failures stay failures.
python scripts/evaluate_suite.py --resume runs/robojev-evaluation/RUN_ID

# One task, explicit evaluation seeds; no video unless requested.
robojev --task stack --evaluate --seeds 0 1 2

# Sequential recording in one rendering process at a time.
python scripts/demo_suite.py --seed 1000

pytest -q
ruff check src tests scripts

Complete run logs, API responses and videos are written under ignored runs/. Summary files contain timings, failure reasons and usage. Record videos separately from parallel evaluation: concurrent EGL rendering proved unreliable in the tested cluster setup.

Task specifications · Cluster usage · Publication policy

RoboJEV code is released under Apache-2.0. Robot assets come from MuJoCo Menagerie, pinned to commit 822c2d8f877dd166c5b7d3c9f7e3c3b6589473b7, and retain their Apache-2.0 license. Simulation uses MuJoCo; model inference uses TypeSafe JEV. See THIRD_PARTY.md.

This is an independent integration and experiment suite, not an official product of the upstream providers. Task decomposition, structured state and Cartesian control are established techniques; no novelty claim is made for the two-stage pattern.

The final fixed-slot comparison is complete: 40/40 evaluable outcomes and 40/40 offline verifications. API retries pause physics, retain received answers and preserve completed outcomes. Account blocks and transport errors have their own audit counts, rather than becoming robot failures. The earlier 18-slot retry remains in the historical status record; it is not pooled with this campaign or the five-task 100-episode evaluation. See the final report.