A loadable MJCF can still be the wrong experiment: a small free-fall acceptance gate

[FenjuFu]

The XML parser can tell you whether a scene is legal. It cannot tell you whether you built the experiment you intended.

I tested a small acceptance gate for generated MuJoCo scenes: ask Spark-X2.6-4B for a falling sphere, compile the returned MJCF, and compare the resulting motion with a simple physical expectation before accepting it. The model failed at the first gate twice. The more useful follow-up was that two deliberately altered, loadable scenes still failed the physical task.

What actually ran

The requested scene was a 0.05 m radius, 0.2 kg sphere released from z=1 m, with gravity -9.81 m/s², a free joint, Euler integration and a 1 ms timestep. At 0.2 s it should still be above the ground. I used a 2 mm maximum trajectory-error tolerance against z(t) = 1 - 0.5 * 9.81 * t**2.

• First API response: MuJoCo rejected an unsupported <ball> element at line 5. No simulation trajectory exists for this output.
• One repair request: I supplied the actual compiler error and explicitly described the worldbody → body → freejoint/geom hierarchy. The response still used <ball>, now at line 3, and compilation failed again.
• Separately constructed reference: both 0.2 kg and 2 kg variants compiled and reached z=0.802819 m. The analytic value is 0.803800 m; the maximum discrepancy was 0.000981 m, within the chosen tolerance.
• Deliberate boundary mutations: removing the freejoint or setting gravity to zero both compiled, but left the ball at z=1.000000 m, a 0.196200 m error against the requested task. Negative mass was rejected by the compiler.

The fixed body and zero-gravity cases are valid configurations for other experiments. They fail this free-fall contract. The reference and its mutations were constructed separately to test the gate; they are not successful model generations.

Reproduce the saved-output experiment

This runs the exact returned XML and the five constructed fixtures below, without needing an API key. It prints compiler acceptance, contract acceptance, and the measured height/error. Expected rejections count as successful reproduction of the observed behavior.

python -m venv .venv
# Windows:
.venv\Scripts\python -m pip install mujoco==4.0.0 numpy==2.5.4
.venv\Scripts\python reproduce.py
# On Linux/macOS, use .venv/bin/python instead.

Save this complete program as reproduce.py:

import mujoco
import numpy as np

GENERATED = '''<mujoco>
  <worldbody>
    <geom name="ball_geom" type="sphere" size="0.05" mass="0.2"/>
    <freejoint name="ball_free"/>
    <ball name="ball" pos="0 0 1"/>
  </worldbody>
  <body>
    <geom name="ground_geom" type="plane" size="1 1 0.001" sizepos="0 0 0"/>
    <geom name="light_geom" type="light" origin="0 0 0"/>
  </body>
  <actuator>
    <joint name="ball_free_joint" type="free"/>
  </actuator>
  <option timestep="0.001" integrator="Euler" gravity="0 0 -9.81"/>
  <default>
    <geom contype="0" conaffinity="0"/>
  </default>
</mujoco>'''

REVISED = '''<mujoco>
  <worldbody>
    <ball name="ball" pos="0 0 1">
      <freejoint name="ball_free"/>
      <geom name="ball_geom" type="sphere" size="0.05" mass="0.2"/>
    </ball>
    <geom name="ground_geom" type="plane" size="1 1 0.001" sizepos="0 0 0"/>
    <light name="render_light"/>
  </worldbody>
  <option timestep="0.001" integrator="Euler" gravity="0 0 -9.81"/>
</mujoco>'''

REFERENCE = '''<mujoco model="freefall_contract">
  <option timestep="0.001" integrator="Euler" gravity="0 0 -9.81"/>
  <worldbody>
    <light pos="0 0 3"/>
    <geom name="ground" type="plane" size="2 2 0.1"/>
    <body name="ball" pos="0 0 1">
      <freejoint name="ball_free"/>
      <geom name="ball_geom" type="sphere" size="0.05" mass="0.2"/>
    </body>
  </worldbody>
</mujoco>'''

def check(xml, mass):
    try:
        m = mujoco.MjModel.from_xml_string(xml)
    except ValueError as e:
        return False, False, str(e).splitlines()[0]
    d = mujoco.MjData(m)
    b = mujoco.mj_name2id(m, mujoco.mjtObj.mjOBJ_BODY, "ball")
    if b < 0:
        return True, False, "missing ball body"
    mujoco.mj_forward(m, d)
    errors = []
    for i in range(201):
        expected = 1 - 0.5 * 9.81 * d.time**2
        errors.append(abs(d.xpos[b, 2] - expected))
        if i < 200:
            mujoco.mj_step(m, d)
            mujoco.mj_forward(m, d)
    ok = (m.nq == 7 and m.nv == 6
          and np.isclose(m.body_mass[b], mass)
          and np.allclose(m.opt.gravity, [0, 0, -9.81])
          and np.isclose(m.opt.timestep, 0.001)
          and np.isclose(d.time, 0.2) and d.ncon == 0
          and np.isfinite(errors).all() and max(errors) < 0.002)
    return True, bool(ok), f"z={d.xpos[b,2]:.6f}, max_error={max(errors):.6f} m"

cases = [
    ("Spark first", GENERATED, 0.2, (False, False)),
    ("Spark repair", REVISED, 0.2, (False, False)),
    ("Reference 0.2 kg", REFERENCE, 0.2, (True, True)),
    ("Reference 2 kg", REFERENCE.replace('mass="0.2"', 'mass="2"'), 2, (True, True)),
    ("No freejoint", REFERENCE.replace('<freejoint name="ball_free"/>', ''), 0.2, (True, False)),
    ("Zero gravity", REFERENCE.replace('gravity="0 0 -9.81"', 'gravity="0 0 0"'), 0.2, (True, False)),
    ("Negative mass", REFERENCE.replace('mass="0.2"', 'mass="-0.2"'), 0.2, (False, False)),
]
for name, xml, mass, expected in cases:
    result = check(xml, mass)
    print(name, result)
    assert result[:2] == expected, (name, result)

Actual output:

Spark first (False, False, 'XML Error: Schema violation: unrecognized element')
Spark repair (False, False, 'XML Error: Schema violation: unrecognized element')
Reference 0.2 kg (True, True, 'z=0.802819, max_error=0.000981 m')
Reference 2 kg (True, True, 'z=0.802819, max_error=0.000981 m')
No freejoint (True, False, 'z=1.000000, max_error=0.196200 m')
Zero gravity (True, False, 'z=1.000000, max_error=0.196200 m')
Negative mass (False, False, 'Error: mass, inertia or density are negative in geom')

Generation setup and limits

The two real requests used the hosted spark-x2.6-4b endpoint, temperature=0, max_tokens=1800, and thinking={"type":"disabled"}. The first user message was:

Return ONLY JSON with one key "xml", containing a complete minimal MuJoCo MJCF document. Task: a sphere named ball, radius 0.05 m, mass 0.2 kg, initially at (0,0,1) m, falling freely from rest under gravity (0,0,-9.81) m/s^2. Use a freejoint named ball_free. Use timestep 0.001 s and Euler integrator. Include a plane ground at z=0, and a light for rendering. No actuators, damping, viscosity, wind, external files, plugins or initial velocity. Sphere geom name ball_geom. A 0.2 s simulation must remain above ground; we will compare height to z=1-0.5*9.81*t^2, allowing numerical integration error. Give MJCF only in the JSON string, no Python and no prose.

The repair used the first response and the actual compiler failure as conversation context, plus explicit nesting guidance. Neither output was manually fixed before its compile test.

Local execution: Windows amd64, Python 3.12.10, MuJoCo 4.0.0, NumPy 2.5.4, Intel Core Ultra 5 125H. The checked-out MuJoCo 4.0.0 tag resolves to ac60eae30fa7e0d31315c38d9476752fb81be67e; execution used the official Python wheel. The model card was checked at revision 64ca8f2a295b8ceb2757b9d4749a6558c4759e5b; the hosted deployment's weight revision was not independently available.

This is two responses in one conversation, plus five constructed fixtures, not a model accuracy estimate. The physical test covers only 0.2 seconds of simple free fall, before impact. It does not validate collision response, complex mechanisms, controllers, or arbitrary MJCF. The fixed 2 mm tolerance is specific to this timestep and horizon. A useful next extension would be separate task-specific checks for contacts and constraints, with explicit units and tolerances.