Skip to content

action: close two gaps that break the action forward-dynamics cookbook - #156

Closed
lfengad wants to merge 2 commits into
mainfrom
liangf/action-dataset-shared-properties
Closed

action: close two gaps that break the action forward-dynamics cookbook#156
lfengad wants to merge 2 commits into
mainfrom
liangf/action-dataset-shared-properties

Conversation

@lfengad

@lfengad lfengad commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Two independent gaps between the action cookbooks and the framework. Both are hit by cookbooks/cosmos3/generator/action/run_fd_with_cosmos_framework.ipynb; each is one commit.

1. domain_name / viewpoint missing on BaseActionLeRobotDataset

The two action-dataset hierarchies are consumed interchangeably, but only one exposes them:

domain_id domain_name viewpoint
ActionBaseDataset (UMI, hand pose, bridge, fractal, ...)
BaseActionLeRobotDataset (DROID, ...)

The cookbook's create_record_from_dataset helper reads them off whichever dataset it was handed, so it works for UMI and dies on DROID:

AttributeError: 'DROIDMergedLeRobotDataset' object has no attribute 'domain_name'

domain_name returns the embodiment type — already what domain_id derives from (get_domain_id(self._embodiment_type)) — so the two stay consistent by construction. Both backing attributes are assigned unconditionally in __init__, so every subclass is covered.

2. Forward dynamics rejected for hand_pose / libero

get_action_sample_data validated domain_name against EMBODIMENT_TO_RAW_ACTION_DIM. That table is a width lookup, not the list of valid domains (EMBODIMENT_TO_DOMAIN_ID is), and it deliberately omits hand_pose and libero because their raw width is set per-dataset at construction time. Result:

ValueError: invalid domain_name 'hand_pose'; expected one of ['abc_yam', 'agibotworld', ...]

— even for forward dynamics, which never needs the lookup: the caller supplies an action file and the raw width is just its last dimension. The table's own comment says only inverse_dynamics and WAM are unsupported for these domains, and the next line agreed:

assert action_path is not None or raw_action_dim is not None

That assert was already dead code, since the membership check above it guaranteed a non-None width.

Now: validate against EMBODIMENT_TO_DOMAIN_ID, look the width up with .get(), and reject a missing width only for the modes that must synthesise actions. Forward dynamics resolves the width from the action file, keeping the table as a cross-check where it has an entry.

Verification

Both on GB200 (driver 580.126.20), Cosmos3-Nano, running the cookbook's own cells:

  • (1) Reverting the commit reproduces the AttributeError; with it, the DROID section runs to completion — 5/5 autoregressive chunks, stitched to (80, 528, 640, 3) @ 15fps.
  • (2) The hand-pose section previously raised before any sampling; it now completes and writes vision.mp4 (EXIT=0).

Regression check: UMI path unchanged (domain=umi view=wrist_view prompt='cup arrangement').

Note

Committed with --no-verify: pre-commit cannot run in this environment (its tool interpreter symlinks into a container-root uv python that is not readable — PermissionError: /root/.local/share/uv/python/.../libpython3.13.so.1.0). Ran the equivalent checks manually: ruff check and ruff format --check both clean.

Companion PR

NVIDIA/cosmos#302 repairs the DROID cookbook asset and depends on this landing first.

🤖 Generated with Claude Code

…tDataset

The two action-dataset hierarchies are consumed interchangeably, but only
`ActionBaseDataset` (UMI, hand pose, bridge, fractal, ...) exposed `domain_name`
and `viewpoint`. `BaseActionLeRobotDataset` (DROID, ...) did not, so callers that
read those off whichever dataset they were handed hit an AttributeError at
runtime -- e.g. the action forward-dynamics cookbook, whose
`create_record_from_dataset` helper works for UMI but fails on DROID with:

    AttributeError: 'DROIDMergedLeRobotDataset' object has no attribute 'domain_name'

Add both as properties. `domain_name` returns the embodiment type, which is
already what `domain_id` derives from (`get_domain_id(self._embodiment_type)`),
so the two stay consistent. Both backing attributes are set unconditionally in
`__init__`, so every subclass is covered.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@lfengad
lfengad force-pushed the liangf/action-dataset-shared-properties branch from 0b8b860 to 4a52c5f Compare August 4, 2026 07:05
…cal action width

`get_action_sample_data` validated `domain_name` against `EMBODIMENT_TO_RAW_ACTION_DIM`,
but that table intentionally omits `hand_pose` and `libero` -- their raw action width is
set per-dataset at construction time. The table is a width lookup, not the list of valid
domains; `EMBODIMENT_TO_DOMAIN_ID` is. So any hand-pose or LIBERO inference died with:

    ValueError: invalid domain_name 'hand_pose'; expected one of ['abc_yam', 'agibotworld', ...]

even for forward dynamics, which does not need the lookup at all: the caller supplies an
action file and the raw width is simply its last dimension. The comment on the table says
as much -- it only rules out inverse_dynamics and WAM -- and the very next line agreed:

    assert action_path is not None or raw_action_dim is not None

That assert was already dead, since the membership check above it guaranteed a non-None
width.

Validate against `EMBODIMENT_TO_DOMAIN_ID`, look the width up with `.get()`, and only
reject a missing width for the modes that must synthesise actions. Forward dynamics now
resolves the width from the action file, with the table kept as a cross-check when it has
an entry.

Verified against the action forward-dynamics cookbook's hand-pose section, which now runs
to completion instead of raising.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@lfengad lfengad changed the title action/datasets: expose domain_name and viewpoint on BaseActionLeRobotDataset action: close two gaps that break the action forward-dynamics cookbook Aug 4, 2026
@lfengad

lfengad commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator Author

Superseded by #159.

Split on review: the domain_name/viewpoint properties this PR added to BaseActionLeRobotDataset were a design preference, not a requirement — the cookbook can read the embodiment off EMBODIMENT_TYPE and the viewpoint off the sample, so that side is handled in NVIDIA/cosmos#302 with no framework change and no cross-repo release ordering.

What genuinely needs a framework fix is the domain_name validation in inference/action.py, which rejected hand_pose/libero even for forward dynamics. #159 carries that alone, tightened to an explicit set so no_action is not let through as a side effect.

@lfengad lfengad closed this Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant