Skip to content

benchmark: add cross-backend physics_step throughput coverage for genesis, isaacgym, and mujoco_warp #251

Description

@TATP-233

Background

We already have a grouped set of physics-step benchmark entrypoints under benchmark/:

  • benchmark/benchmark_physics_step_mj_step.py
  • benchmark/benchmark_physics_step_genesis.py
  • benchmark/benchmark_physics_step_isaacgym.py
  • benchmark/benchmark_physics_step_mujoco_warp.py

But we still lack a clear, repo-level benchmark target / support claim for comparing physics_step throughput across backends, especially for genesis, isaacgym, and mujoco_warp.

Goal

Add benchmark coverage and a stable reporting path for cross-backend physics_step throughput, including at least:

  • genesis
  • isaacgym
  • mujoco_warp
  • baseline MuJoCo path (benchmark_physics_step_mj_step.py)

Expected Scope

  1. Standardize the measurement target as physics-step throughput / latency, not full env.step throughput.
  2. Ensure the backend-specific benchmark scripts expose comparable output fields and compatible task / batch-size sweeps.
  3. Save results under the shared benchmark/outputs/physics_step/ subtree so results are easy to compare and postprocess.
  4. Make it easy to compare:
    • total throughput
    • per-env throughput
    • latency per benchmark call
  5. Keep task naming aligned with current owner task ids (go1_joystick, go2_joystick, g1_joystick) and avoid stale benchmark-only naming drift.

Validation

  • Run the relevant benchmark_physics_step_* scripts for at least one overlapping task / batch-size sweep.
  • Confirm JSON outputs are produced under benchmark/outputs/physics_step/....
  • Confirm the reported metrics are comparable across backends.

Notes

This issue is about benchmark coverage and evidence for support / performance claims, not about changing the env contract or training entrypoints.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions