Policy mode: opposite gripper instructions produce almost the same actions

[zcxixixi]

Hi Cosmos team,

We have been testing policy mode with the community checkpoint fwd4xl/cosmos3-nano-policy-liberoall-5k. We found that changing an instruction from “close the gripper” to “open the gripper” barely changes the predicted actions. An empty instruction produces the same opening/closing choice as well.

We used two fixed LIBERO images: A, with the gripper in free space, and B, with the gripper near the milk. Each image contains the external and wrist camera views. We supplied no demonstrations. The exact instructions were:

• close the gripper while keeping the robot arm stationary
• open the gripper while keeping the robot arm stationary
• An empty string, retaining the action JSON wrapper.

For each image we repeated the comparison with seeds 195, 196 and 198. The close/open runs have identical images, initial action and future-image noise, and position IDs; only one text token changes. Each run predicts 16 actions using 30 denoising steps.

Image	Close instruction	Open instruction	Empty instruction
A: gripper in free space	+0.990981	+0.990983	+0.995309
B: gripper near the milk	−0.993130	−0.992725	−0.992805

These are normalized gripper-command means over 16 actions, averaged across the three seeds. Approximately +1 means open and −1 means close; these are not measured finger widths. In A all 16 actions predict opening, even under the close instruction. In B all 16 predict closing, even under the open instruction.

The paired close/open gripper MAE is 0.001450 in A and 0.001537 in B. Keeping the close instruction and changing the random seed produces larger MAE: 0.005157 in A and 0.004171 in B. Keeping the close instruction and changing image A to B produces a gripper MAE of 1.984111.

We also inspected the actual internal tensors rather than inferring this only from the actions. At the first denoising step, swapping close/open changes the normalized hidden vectors entering the action head by about 1.01% relative RMS. Changing A to B changes them by about 45.22%. The actual gripper Flow-velocity output changes by 0.003047 RMS for close/open, versus 1.944389 RMS for A/B. Flow velocity is a denoising intermediate, not physical robot velocity. Native attention recomputation and replay through the real action head both matched the recorded outputs exactly.

We do not think this means the model ignores all language. As a positive comparison, we used the same actual LIBERO_GOAL image with open the middle drawer of the cabinet and turn on the stove. These produce clearly different predicted action chunks: normalized 10D action RMSE 0.429937 and 0.471817 for two matched seeds. Swapping their GEN rotary positions only changes actions by RMSE 0.00232–0.00730, so GEN position changes alone do not explain the large instruction-dependent difference. The instructions still differ in text length, and different predicted actions do not establish successful task execution.

Our current conclusion is that these standalone gripper instructions have very weak control over the policy's actions in these two visual states, even though their differences reach the internal action representations. We have not identified a faulty layer or proved a training-data shortcut.

Is this expected after LIBERO task-specific post-training—for example, because these primitive instructions are outside the supported instruction distribution? Is there an input or sampling convention we should check, or an official LIBERO policy checkpoint and runner you recommend for the same comparison?

The experiment data and input-image package contains the input images, plots, per-seed results, full predicted action arrays, internal readout comparisons and control records. The JSON and CSV data are directly readable; the ZIP is provided as Base64 text with decoding instructions in the attachment README. This is a small offline test with three image states, not a task-success benchmark or a statistical sample of 44 independent tasks. There was no weight update in these experiments.

Environment: RTX A6000; BF16; Python 3.10.12; PyTorch 2.14.0 / CUDA 13.0; Diffusers 0.41.0.dev0 at fef717ffb01f407d2637584ed936c16db908587a; cosmos-framework at cf5d68c00d97ccd2480a2320ed652b92dec63102; checkpoint revision 64798337c642c53f9e22332554ae92f76a86cf04. We used a custom Diffusers runner, domain libero, 10D frame-wise-relative rot6d actions, concatenated views, resolution tier 256, FPS 20, guidance 1.0 and the framework's native Flow UniPC scheduler with shift 1.0. The runner sets the JSON idle_frame field to 0 out of 16.. We have not yet cross-validated this runner numerically against the official framework policy server, so we are reporting the behavior of this checkpoint/runtime combination, not a confirmed defect in every Cosmos3 model.

[fwd4]

Thanks for reporting this. Weak language sensitivity appears to be a common failure mode among open-source VLA/WAM policies after task-specific SFT. We observed similar behavior in the LIBERO-finetuned checkpoints of Cosmos3, π0.5, and LingBot-VA.

With identical observations and paired random seeds, switching between opposite gripper instructions produced the following results:

Model	Same gripper direction under close/open	Close/open gripper MAE
Cosmos3	180/180	0.002037
π0.5	180/180	0.001036
LingBot-VA*	180/180	0.001888

The table compares the first 10 actions across six states and three seeds. Cosmos used four diffusion steps, guidance 1, shift 3, and a 16-action chunk; the same-direction result also held over all 16 actions.

LingBot-VA was evaluated with temporal history reset, so its result should not be generalized to continuous execution.

This is consistent with the broader problem of policies relying on familiar visual context more strongly than language instructions. It does not establish that these models ignore all language or that every checkpoint exhibits the same failure.

Relevant approaches include:

• Counterfactual Action Guidance (CAG): a CFG-inspired inference method that strengthens language conditioning using a vision-only reference branch.
• CAST: counterfactual instruction/action augmentation to improve language grounding during training.
• CounterAlign: counterfactual supervision for learning instruction-grounded rewards in offline RL.

These are promising mitigation directions, though we have not yet validated them on this Cosmos checkpoint.

[zcxixixi]

Thanks for the detailed reply and for the pointers (CAG / CAST / CounterAlign). Your suggestion matches what I am seeing, so I will try an adjustment along those lines.

What I found since the first post (same checkpoint and runner; fixed-seed case studies, not success rates):

• Behaviour (LIBERO-Object task 7, "pick up the milk…"): with the milk moved 15 cm the arm picks the cream cheese. If I then swap cream cheese and butter, it picks the butter at the old cream-cheese spot, so the choice follows a location rather than the instruction. On a LIBERO-Goal scene "bowl" vs "wine bottle" does change the target, so the language is not ignored in general.
• Inside the network: changing only the instruction changes the action hidden vectors (0.15% after layer 1, 3.22% after layer 36) but not the grasped object. Swapping the layer-36 text K/V of two equal-length target names did not change the target in 8 closed-loop runs; earlier layers are untested.
• Attention: at layer 36, first denoising step, one scene, the action queries put 0.77% of their attention on the 122 text tokens (about 91% on the first token) and about 0.001% on the target-name tokens. This is a single layer and run, and attention weight is not influence.

Next: I will measure this for all 36 layers, then try a CAG-style boost of the target-name tokens and check whether the target follows the name without breaking cases that already work. If it helps, I will open a PR.

Data and code: https://github.com/zcxixixi/cosmos3-policy-experiments