Action-State Deviation in GR00T-VLA Fine-Tuning Data Collection

[huan-zz3]

-1. Hope someone give me some advices. The contents below are my insufficient ideas.

---

0. Problem Summary

In the automated data collection system based on Isaac Lab, there is a significant deviation between the recorded action (joint target positions sent to the simulator) and the state (actual joint positions of the simulator) during state transitions. This data is used to fine-tune the NVIDIA GR00T-VLA model, and step-like jumps in action directly affect the quality of action chunking training for the VLA model.

---

1. Development Environment

1.1 Simulation and Hardware

Item	Value
Simulation Platform	NVIDIA Isaac Lab 2.2.1 + Isaac Sim 5.0.0
Robot Configuration	Dual 7-DOF arms + OmniHand 19-DOF right hand + lift joint (1-DOF)
Python Version	≥ 3.10
PyTorch	2.3.0 (tied to Isaac Sim version)
Data Storage Format	CSV

1.2 Current IK and Controller Configuration

Component	Value
IK Solver	Isaac Lab DifferentialIKController (SVD / Damped Least Squares)
Actuator Type	ImplicitActuatorCfg (PhysX implicit PD position control)
PD Parameters - Arm Joints	stiffness: 819 - 38328, damping: 0.33 - 15.33, friction: 0.01
PD Parameters - Hand	stiffness=None, damping=None → uses USD defaults (not explicitly tuned)
Control Mode	Joint position control (set_joint_position_target())
Joint Limits	soft_joint_pos_limit_factor = 1.0, IK results clipped within soft limits

1.3 Frequency and Timing Parameters

Stage	Frequency	Description
Physics Simulation	200 Hz (dt = 0.005s)	PhysX physics substeps
Effective Simulation Frequency	~100 Hz	Main loop calls sim.step() twice per frame
IK Solving	~20 Hz	Solves joint targets every 50ms
Data Saving	~20 Hz	Saves one frame every 50ms, aligned with IK frequency
State Update	~30 Hz	Updates all_current_state and camera images every 33ms

1.4 Core Dependency Modules of This Project

Module	Role
Isaac Lab Articulation	Robot articulation management, provides set_joint_position_target(), write_data_to_sim(), data.joint_pos
Isaac Lab DifferentialIKController	Current IK solver, maps EE pose to joint space via Jacobian pseudo-inverse
Isaac Lab ImplicitActuator	Implicit PD actuator, computes torques internally by the physics engine to track target positions

---

2. Objectives

2.1 Project Objective

Collect high-quality (state, action) sequence data for fine-tuning the NVIDIA GR00T-VLA (Vision-Language-Action) model.

GR00T-VLA is NVIDIA's embodied intelligence foundation model that generates robot control actions from visual observations and natural language instructions. Our collection system serves as the data production end.

2.2 Data Requirements

GR00T-VLA training imposes the following key requirements on data:

Requirement	Description
Action Format	Absolute joint positions (unit: rad). GR00T training preprocessing automatically converts to relative action: action_rel = action_abs - state_current
Action Chunking	The model predicts a sequence of 16~30 future actions at once (delta_indices = range(16)), requiring continuity and smoothness within the chunk
Temporal Alignment	action[t] is the control command at time t, which should cause state[t+1] to be close to that action. Inter-frame timing deviation should not exceed ~50ms
Data Format	GR00T-enhanced LeRobot v2: Parquet files (state/action columns) + H264 video + meta/modality.json (field split definitions)

---

3. Current Progress

3.1 Current Status of Data Collection System

The system has been fully implemented and is running stably. Main features:

Feature	Status	Description
State Machine Driven	✅	Based on RobotStatePlugin architecture, fully defines states like pick → move → place → reset
IK Real-time Control	✅	Differential IK, 20Hz solving frequency, with joint limit clipping
Format Storage	✅	CSV
Independent Action Recording	✅	Independent buffer records action and state to avoid confusion
Asynchronous Image Saving	✅	Does not block the collection loop

3.2 Data Flow Overview

[State Machine generates target EE pose]
         │
         ▼
[IK Solver: EE pose → joint target angles]   ← This is the source of action
         │
         ▼
[Write to physics simulation: set_joint_position_target() + write_data_to_sim()]
         │              │
         ▼              ▼
   Record action      Physics simulation advances
   to buffer          (sim.step × 2)
                        │
                        ▼
                Read actual joint positions   ← This is the source of state
                        │
                        ▼
         Both paired as same frame written to CSV

3.3 Specific Definitions of Action and State

Source and Meaning of Action:

• Source: The state machine specifies a target end-effector (EE) pose (position + quaternion) at each step
• Solving: Differential IK receives the EE pose and solves for joint angles that can reach the target pose from the current joint state
• Contains: 8 arm joints + lift joint + 19 hand joints = 27 joint target positions
• Writing: Set to the simulator via set_joint_position_target(), simultaneously recorded in the all_robot_arm_action_joint_pos buffer
• Meaning: This is the "commanded joint target position for the simulator," not the "actual position the joint has reached"

Source and Meaning of State:

• Source: Read from Isaac Lab Articulation's data.joint_pos after each sim.step() completes
• Contains: Actual joint positions with the same dimensionality as action
• Meaning: This is the "actual joint position of the simulator after the current physics step"

Key Difference:

• Action = Desired target (where the joint ideally should be)
• State = Actual result (where the joint physically currently is)

3.4 Example Output Data

Taking episode 0009 as an example, the CSV contains approximately 501 frames with the following column structure:

Column Group	Contents	Meaning
Metadata	frame_id, timestamp, state_machine, language_instruction	Frame number, timestamp, state machine state, language instruction
Image Paths	body_image_path, wrist_image_path	Camera image file paths
State - EE	state_x/y/z/roll/pitch/yaw/pad	End-effector pose (7D)
State - Arm	lift_Link_joint, right_joint1~7 (8 columns)	Actual arm joint positions
State - Arm Velocity	*_vel (8 columns)	Arm joint velocities
State - Hand	hand_right_thumb_cmc_roll, etc. (19 columns)	Actual hand joint positions
Action - Arm	lift_Link_joint_action, right_joint1_action~7_action (8 columns)	Arm joint target positions
Action - Hand	hand_right_*_action (19 columns)	Hand joint target positions

---

4. Issues Encountered

4.1 Observed Phenomenon

In data comparison plots, action and state exhibit the following characteristics:

• Steady-state tracking phase (arm moving slowly or stationary): action and state curves largely overlap, with minimal deviation

• State transition phase (e.g., switching from "move to cup" to "grasp cup"):

  1. Action suddenly jumps to a value significantly deviating from the current state in a single frame, forming a step-like discontinuity
  2. Action then gradually converges over subsequent frames (as the EE target progressively approaches)
  3. State catches up to action gradually, with the two realigning after tens of frames

A distinct "flared" shape can be observed in the data plots: action jumps like a step function, while state approaches slowly like an exponential curve.

4.2 Root Cause Analysis

First Layer: Frame-by-Frame Independence of the IK Solver

The current Differential IK is a "memoryless" differential IK:

• Each solve only considers the difference from the "current EE pose" to the "target EE pose"
• It does not perceive the previous solve result and does not generate smooth trajectories
• When the EE target pose from the state machine jumps, the IK directly computes a joint configuration that can achieve the new pose
• The IK solver does not perceive joint velocity/acceleration constraints, and its output may deviate significantly from the current joint state

Second Layer: Gradual Tracking by the PD Controller

After the joint targets computed by IK are written to the simulator via set_joint_position_target(), the simulator's implicit PD controller gradually pushes the actual joint positions toward the target:

• PD torque: τ = Kp × (q_desired - q_actual) - Kd × v_actual
• Physical inertia determines that state can only follow action in a gradual manner
• The hand joint PD parameters are None (using USD defaults), possibly not tuned, with uncertain tracking performance

Third Layer: Dual sim.step() Timing

The main loop calls sim.step() twice per frame, with the IK control loop executing between the two steps. This may cause a phase difference between action and state sampling time points, further amplifying the deviation.

---

5. Possible Solutions

5.1 Direction One: Add Smoothing During Collection

Core concept: Do not change the IK solver; instead, add a smoothing mechanism after the IK output so that the targets sent to the simulator transition continuously rather than instantaneously.

Solution A1: EE Pose Space Interpolation (Recommended)

Principle: Instead of directly setting all_robot_arm_target_pose to the new target EE pose during state transitions, perform trajectory interpolation (e.g., 5th-order polynomial, S-curve, or minimum-jerk trajectory) from the current EE pose to the new target pose, transitioning gradually over tens of frames. The IK solver receives continuously transitioning EE poses, and the output joint targets naturally become smooth.

Advantages:

• Most physically reasonable: Real robot motion planners interpolating trajectories between targets is standard practice
• Minimal changes: Only need to modify the target pose update logic, without affecting the IK solver or simulator
• Generated data closest to real robot collection

Implementation Points:

• Record the current EE pose and target EE pose at state transitions
• Use spline/Bezier curve interpolation to generate intermediate EE pose sequences
• Control interpolation time (e.g., 500ms ~ 1000ms) to enable smooth arm movement to the new position
• Handle hand joints similarly

Solution A2: Joint-Space Low-Pass Filtering

Principle: Apply a first-order low-pass filter to the IK solution joint_pos_des:

smooth_target = α × new_target + (1-α) × prev_target

Where α is the filter coefficient (e.g., adjustable between 0.3~0.5, smaller = smoother).

Advantages: Simplest implementation, only requires modifying 1~2 lines of code.

Risk: Filtered joint targets may not precisely reach the expected EE pose, causing deviation between the actual end-effector pose and the state machine expectation.

Solution A3: Hard Constraint on Joint Variation

Principle: Limit the maximum change in joint target per frame Δq_max, truncating when exceeded:

Δq = new_target - prev_target
if |Δq| > Δq_max:
    smooth_target = prev_target + sign(Δq) × Δq_max
else:
    smooth_target = new_target

Advantages: Conforms to real robot velocity/acceleration limits, physically meaningful.

Risk: Requires per-joint parameter tuning (different joints have different motion capabilities).

5.2 Direction Two: Replace the IK Solver

Core concept: The current Differential IK is a frame-by-frame differential IK that does not produce smooth trajectories. Replace it with a kinematic solver that has built-in trajectory planning capabilities.

Solution B1: Isaac LULA (Already Integrated, Medium-term Solution)

Status: This project already has LULA-required YAML files configured under assets/robot_rise_lift/ (including cspace mapping, collision spheres, joint default configuration), indicating LULA is already integrated into the current environment.

LULA's Capabilities:

• LulaKinematicsSolver: IK solving, supports warm start (using current joint positions as initial guess, reducing jumps)
• LulaCSpaceTrajectoryGenerator: Joint-space trajectory planning, spline interpolation connecting waypoints, time-optimal
• LulaTaskSpaceTrajectoryGenerator: Task-space trajectory planning
• RmpFlow: Reactive motion policy, real-time smooth guidance to target, built-in collision avoidance

Migration Plan:

1. Replace DifferentialIKController with LulaKinematicsSolver
2. Use current joint positions as warm start parameters
3. Or directly use LulaTrajectoryGenerator to generate complete trajectories

Limitations: NVIDIA has officially stated that "cuMotion has entirely subsumed LULA"; LULA will no longer receive new features.

Solution B2: Isaac cuMotion TrajectoryOptimizer (Long-term Solution)

Positioning: NVIDIA's next-generation motion planning library, completely replacing LULA.

Core Capabilities:

• TrajectoryOptimizer: Optimization-based global trajectory planning (not available in LULA), generating collision-free, smooth trajectories satisfying kinematic constraints
• TrajectoryFollower: Smooth tracking of planned trajectories, fundamentally eliminating jumps
• GraphBasedMotionPlanner: Collision-aware graph search path planning
• Full GPU parallelism (CUDA acceleration), excellent performance in multi-environment scenarios
• Supports joint position/velocity/acceleration constraints

Built-in integration in Isaac Sim 6.0+ , accessible via the isaacsim.robot_motion.cumotion extension. XRDF configuration files are backward compatible with LULA YAML.

Example Code Framework:

from isaacsim.robot_motion.cumotion import CumotionRobot, CumotionWorldInterface, TrajectoryOptimizer
from isaacsim.motion_generation import TrajectoryFollower, RobotState

# Plan trajectory from current position to target position
optimizer = TrajectoryOptimizer(cumotion_robot=robot_config, robot_joint_space=joint_names)
trajectory = optimizer.plan_to_goal(q_initial, cspace_target)

# Use TrajectoryFollower for smooth tracking
follower = TrajectoryFollower()
desired_state = follower.forward(estimated_state, None, t=current_time)
robot.set_dof_position_targets(positions=desired_state.joints.positions)

Limitations: cuMotion is not integrated into the current project; XRDF configuration files need to be set up from scratch.

---

Appendix A: GR00T-VLA Data Format Reference

A.1 Dataset Structure (GR00T-enhanced LeRobot v2)

my_dataset/
├── meta/
│   ├── info.json              # Dataset schema (features, fps, robot_type)
│   ├── modality.json          # GR00T-specific, defines state/action field splits
│   ├── stats.json             # Normalization statistics (mean/std/min/max)
│   ├── relative_stats.json    # GR00T-specific, statistics for relative actions
│   ├── episodes.jsonl         # Episode boundary information
│   └── tasks.jsonl            # Natural language task descriptions
├── data/chunk-000/
│   └── episode_000000.parquet  # Tabular data (state, action, timestamp)
└── videos/chunk-000/
    └── observation.images.ego_view/
        └── episode_000000.mp4  # H264 encoded video

A.2 modality.json Example

{
    "state": {
        "left_arm":  {"start": 0,  "end": 7},
        "right_arm": {"start": 7,  "end": 15},
        "gripper":   {"start": 15, "end": 17}
    },
    "action": {
        "left_arm":  {"start": 0,  "end": 7},
        "right_arm": {"start": 7,  "end": 15},
        "gripper":   {"start": 15, "end": 17}
    },
    "video": {
        "front": {"original_key": "observation.images.front"}
    },
    "annotation": {
        "human.action.task_description": {}
    }
}

[vick-yu]

Hi there,

Thanks for the incredibly detailed write-up — the problem is clearly articulated and the analysis is thorough.

One quick note: this discussion would be better placed on the Isaac Lab repository rather than here on Isaac Sim. Your question is fundamentally about Isaac Lab's DifferentialIKController,
ImplicitActuatorCfg, and the data collection pipeline built on Isaac Lab 2.2.1 — the Isaac Lab maintainers and community are much better positioned to help with this.

I'd suggest reposting at https://github.com/isaac-sim/IsaacLab/discussions so it gets the right eyes on it.

That said, a few quick thoughts on your analysis before you move it over:

Your Direction One (A1 — EE pose interpolation) is the right call. The "step-like jump" you're seeing is expected behavior — Differential IK is a single-frame solver with no trajectory memory, so when your state machine jumps the EE target, the
joint targets jump too. Interpolating in task space before feeding the IK solver is the cleanest fix and produces the most realistic training data.

On Direction Two: Before migrating away from DifferentialIKController, consider that the smoothing problem lives upstream of the IK solver. Even LULA or cuMotion will produce jumps if the input target pose jumps discontinuously. The real fix is
generating smooth EE trajectories for the solver to track, regardless of which solver you use. That said, if you eventually want collision-aware planning, cuMotion is worth exploring as a longer-term upgrade.

One thing to watch: Your dual sim.step() per frame with IK solving between them could introduce a one-step phase lag between recorded action and state. When you repost, it might be worth asking the Isaac Lab folks whether the recommended
pattern for VLA data collection aligns action/state timestamps differently.

Good luck with the GR00T fine-tuning!

[huan-zz3]

Thank you very much for your suggestion. My main work today is to explore the feasibility of migrating the project to coMotion.
And I have pushed a same issue in IsaacLab in the same time, please forgive my unreasonable issue of this prj.