[bug] Ray Caster Does Not Move with the Robot in Isaac Lab

[MiraiMoe]

Describe the bug

I encountered an issue when using Isaac Lab's Ray Caster: it does not move along with the robot. I enabled debug_vis and observed that the point cloud remains static, even though my robot moves as expected.

My robot is modified from the official cf2x (as shown in the screenshot). I can control it properly using API functions. However, while the camera moves with the robot, the Ray Caster remains fixed.

Steps to reproduce

1. Extract and use the provided model: crazyflie_final_new.zip.
2. Replace the usd_path in my code with:

usd_path=r"/home/username/crazyflie_final_new.usd"

3. Run the provided script and observe the issue.
4. Enable debug_vis and notice that the Ray Caster's point cloud does not move, while the robot moves correctly.
5. The camera follows the UAV correctly, but the Ray Caster remains static.

Sensor Configuration

Here is my sensor configuration:

camera = CameraCfg(
    prim_path="{ENV_REGEX_NS}/Robot/crazyflie/body/front_cam",
    update_period=0.1,
    height=480,
    width=640,
    data_types=["rgb", "distance_to_image_plane"],
    spawn=sim_utils.PinholeCameraCfg(
        focal_length=24.0, focus_distance=400.0, horizontal_aperture=20.955, clipping_range=(0.1, 1.0e5)
    ),
    offset=CameraCfg.OffsetCfg(pos=(0.15, 0.0, 0.15), rot=(-0.707, 0.707, 0, 0), convention="world"),
)

height_scanner = RayCasterCfg(
    prim_path="{ENV_REGEX_NS}/Robot/crazyflie/body",
    update_period=1 / 60,
    offset=RayCasterCfg.OffsetCfg(pos=(0, 0, 0)),
    attach_yaw_only=True,
    pattern_cfg=patterns.LidarPatternCfg(
        channels=100, vertical_fov_range=[-90, 90], horizontal_fov_range=[-90, 90], horizontal_res=1.0
    ),
    debug_vis=True,
    mesh_prim_paths=["/World/ground"],
)

---

Expected Behavior

The Ray Caster should move along with the robot, similar to the camera.

---

System Info

• Isaac Sim Version: [4.5.0]
• OS: [e.g. Ubuntu 22.04]
• GPU: [e.g. RTX 4070Ti]
• CUDA: [e.g. 12.4]
• GPU Driver: [e.g. 550.120]

---

Code to Reproduce

Here is my full code that demonstrates the issue, I wrote this code based on the tutorial of Isaac Lab:

# Copyright (c) 2022-2025, The Isaac Lab Project Developers.
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause

"""
This script demonstrates how to add and simulate on-board sensors for a robot.

We add the following sensors on the quadruped robot, ANYmal-C (ANYbotics):

* USD-Camera: This is a camera sensor that is attached to the robot's base.
* Height Scanner: This is a height scanner sensor that is attached to the robot's base.
* Contact Sensor: This is a contact sensor that is attached to the robot's feet.

.. code-block:: bash

    # Usage
    ./isaaclab.sh -p scripts/tutorials/04_sensors/add_sensors_on_robot.py --enable_cameras

"""

"""Launch Isaac Sim Simulator first."""

import argparse

from isaaclab.app import AppLauncher
# add argparse arguments
parser = argparse.ArgumentParser(description="Tutorial on adding sensors on a robot.")
parser.add_argument("--num_envs", type=int, default=1, help="Number of environments to spawn.")
# append AppLauncher cli args
AppLauncher.add_app_launcher_args(parser)
# parse the arguments
args_cli = parser.parse_args()
args_cli.enable_cameras = True
# launch omniverse app
app_launcher = AppLauncher(args_cli)
simulation_app = app_launcher.app

"""Rest everything follows."""

import torch

import isaaclab.sim as sim_utils
from isaaclab.assets import ArticulationCfg, AssetBaseCfg
from isaaclab.scene import InteractiveScene, InteractiveSceneCfg
from isaaclab.sensors import CameraCfg, ContactSensorCfg, RayCasterCfg, patterns
from isaaclab.utils import configclass
from isaaclab_assets import CRAZYFLIE_NEW_CFG, CRAZYFLIE_CFG
from isaaclab.terrains import TerrainImporterCfg, TerrainImporter, TerrainGeneratorCfg, HfDiscreteObstaclesTerrainCfg
from isaaclab.actuators import ImplicitActuatorCfg

@configclass
class SensorsSceneCfg(InteractiveSceneCfg):
    """Design the scene with sensors on the robot."""

    # ground plane
    ground = AssetBaseCfg(prim_path="/World/defaultGroundPlane", spawn=sim_utils.GroundPlaneCfg())

    # lights
    dome_light = AssetBaseCfg(
        prim_path="/World/Light", spawn=sim_utils.DomeLightCfg(intensity=3000.0, color=(0.75, 0.75, 0.75))
    )

    terrain = TerrainImporterCfg(
        prim_path="/World/ground",
        terrain_type="plane",
        collision_group=-1,
        physics_material=sim_utils.RigidBodyMaterialCfg(
            friction_combine_mode="multiply",
            restitution_combine_mode="multiply",
            static_friction=1.0,
            dynamic_friction=1.0,
            restitution=0.0,
        ),
        debug_vis=False,
    )

    # robot
    robot: ArticulationCfg = ArticulationCfg(
        prim_path="{ENV_REGEX_NS}/Robot",
        spawn=sim_utils.UsdFileCfg(
            usd_path=r"/home/username/crazyflie_final_new.usd",
            activate_contact_sensors=True,
            rigid_props=sim_utils.RigidBodyPropertiesCfg(
                disable_gravity=False,
                max_depenetration_velocity=10.0,
                enable_gyroscopic_forces=True,
            ),
            collision_props=sim_utils.CollisionPropertiesCfg(
                collision_enabled=True,
            ),
            mass_props=sim_utils.MassPropertiesCfg(mass=0.3),
            articulation_props=sim_utils.ArticulationRootPropertiesCfg(
                enabled_self_collisions=False,
                solver_position_iteration_count=4,
                solver_velocity_iteration_count=0,
                sleep_threshold=0.005,
                stabilization_threshold=0.001,
            ),
            copy_from_source=False,
        ),
        init_state=ArticulationCfg.InitialStateCfg(
            pos=(0.0, 0.0, 3),
        ),
        actuators={
            "dummy": ImplicitActuatorCfg(
                joint_names_expr=[".*"],
                stiffness=0.0,
                damping=0.0,
            ),
        },
    )
    # sensors
    camera = CameraCfg(
        prim_path="{ENV_REGEX_NS}/Robot/crazyflie/body/front_cam",
        update_period=0.1,
        height=480,
        width=640,
        data_types=["rgb", "distance_to_image_plane"],
        spawn=sim_utils.PinholeCameraCfg(
            focal_length=24.0, focus_distance=400.0, horizontal_aperture=20.955, clipping_range=(0.1, 1.0e5)
        ),
        offset=CameraCfg.OffsetCfg(pos=(0.15, 0.0, 0.15), rot=(-0.707, 0.707, 0, 0), convention="world"),
    )
    height_scanner = RayCasterCfg(
        prim_path="{ENV_REGEX_NS}/Robot/crazyflie/body",
        update_period=1 / 60,
        offset=RayCasterCfg.OffsetCfg(pos=(0, 0, 0)),
        attach_yaw_only=True,
        pattern_cfg=patterns.LidarPatternCfg(
            channels=100, vertical_fov_range=[-90, 90], horizontal_fov_range=[-90, 90], horizontal_res=1.0
        ),
        debug_vis=True,
        mesh_prim_paths=["/World/ground"],
    )



def run_simulator(sim: sim_utils.SimulationContext, scene: InteractiveScene):
    """Run the simulator."""
    # Define simulation stepping
    sim_dt = sim.get_physics_dt()
    sim_time = 0.0
    count = 0
    # Fetch relevant parameters to make the quadcopter hover in place
    body_ids = scene["robot"].find_bodies("body")[0]
    robot_mass = scene["robot"].root_physx_view.get_masses().sum()
    gravity = torch.tensor(sim.cfg.gravity, device=sim.device).norm()

    # Simulate physics
    while simulation_app.is_running():
        # Reset
        if count % 500 == 0:
            # reset counters
            sim_time = 0.0
            count = 0
            # reset dof state
            joint_pos, joint_vel = scene["robot"].data.default_joint_pos, scene["robot"].data.default_joint_vel
            scene["robot"].write_joint_state_to_sim(joint_pos, joint_vel)
            scene["robot"].write_root_pose_to_sim(scene["robot"].data.default_root_state[:, :7])
            scene["robot"].write_root_velocity_to_sim(scene["robot"].data.default_root_state[:, 7:])
            scene["robot"].reset()
            # reset command
            print(">>>>>>>> Reset!")
            # apply action to the robot (make the robot float in place)
        thrust = torch.zeros(scene["robot"].num_instances, 1, 3, device=sim.device)
        moment = torch.zeros(scene["robot"].num_instances, 1, 3, device=sim.device)
        thrust[:, 0, 2] = robot_mass * gravity * 1.1
        moment[:, 0, :] = 0
        scene["robot"].set_external_force_and_torque(thrust, moment, body_ids=body_ids)
        scene["robot"].write_data_to_sim()
        # perform step
        sim.step()
        # update sim-time
        sim_time += sim_dt
        count += 1
        # update buffers
        scene["robot"].update(sim_dt)
        # print information from the sensors
        print("-------------------------------")
        print(scene["camera"])
        print("Received shape of rgb   image: ", scene["camera"].data.output["rgb"].shape)
        print("Received shape of depth image: ", scene["camera"].data.output["distance_to_image_plane"].shape)
        print("-------------------------------")
        print(scene["height_scanner"])
        print("Received max height value: ", torch.max(scene["height_scanner"].data.ray_hits_w[..., -1]).item())
        print("-------------------------------")


def main():
    """Main function."""

    # Initialize the simulation context
    sim_cfg = sim_utils.SimulationCfg(dt=0.005, device=args_cli.device)
    sim = sim_utils.SimulationContext(sim_cfg)
    # Set main camera
    sim.set_camera_view(eye=[10, 10, 10], target=[2.5, 0.0, 0.0])
    # design scene
    scene_cfg = SensorsSceneCfg(num_envs=args_cli.num_envs, env_spacing=5.0)
    scene = InteractiveScene(scene_cfg)

    # Play the simulator
    sim.reset()
    print("[INFO]: Setup complete...")
    # Run the simulator
    run_simulator(sim, scene)


if __name__ == "__main__":
    # run the main function
    main()
    # close sim app
    simulation_app.close()

crazyflie_final_new.zip

Additional context

• The Ray Caster is attached to "/World/envs/env_.*/Robot/crazyflie/body" but does not inherit motion.
• The camera follows the UAV correctly, but the Ray Caster remains fixed.
• I suspect the issue might be due to how the Ray Caster's transformation is updated.

---

Checklist

• I have checked that there is no similar issue in the repo (required)
• I have checked that the issue is not in running Isaac Sim itself and is related to the repo

---

Acceptance Criteria

• The Ray Caster follows the robot’s movement correctly
• The issue is reproduced and resolved in an upcoming update

[RandomOakForest]

Thank you for posting this. It may be something missing in the sensor/robot configuration. The team will review.

[MiraiMoe]

A few days ago, I encountered an issue in Isaac Lab where the raycaster attached to my UAV did not move with the UAV body—the points remained static. At that time, I checked the perspective of the camera attached to the UAV in Isaac Sim, and it appeared to move correctly with the UAV. Based on this observation, I initially suspected that the problem was with the raycaster.

However, today I inspected the depth images obtained using the Isaac Lab API and visualized them as grayscale images. To my surprise, I found that the depth camera also did not move as expected. This suggests that the issue might not be limited to the raycaster but could be related to my UAV setup.

Could this be a problem with my robot configuration? Any insights would be greatly appreciated.

[MiraiMoe]

I've resolved this issue—it was due to a mistake in my code. Initially, I wrote:

scene[robot].write_data_to_sim()
scene[robot].update(sim_dt)

However, to properly update the sensor data, it should be:

scene.write_data_to_sim()
scene.update(sim_dt)

After correcting this, everything is now working as expected.

[tccoin]

Hi I am having the same issue here but with isaac sim 5.0 and isaac lab 2.2.0. Can any one help me to check how to solve it? I have tried the solution @MiraiMoe provided above but it's not working in my case.

I notice that even though the robot moves, the coordinate of base doesn't move in the gui. Is it because the translate of xform is recorded in usd and root_state_w is only updated on gpu fabric? @RandomOakForest

This can be reproduced by this code. I modified the original demo to move the robot with write_root_pose_to_sim():

# Copyright (c) 2022-2025, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause

import argparse

from isaaclab.app import AppLauncher

# add argparse arguments
parser = argparse.ArgumentParser(description="Example on using the raycaster sensor.")
parser.add_argument("--num_envs", type=int, default=1, help="Number of environments to spawn.")
# append AppLauncher cli args
AppLauncher.add_app_launcher_args(parser)
# parse the arguments
args_cli = parser.parse_args()

# launch omniverse app
app_launcher = AppLauncher(args_cli)
simulation_app = app_launcher.app

"""Rest everything follows."""

import numpy as np
import torch

import isaaclab.sim as sim_utils
from isaaclab.assets import AssetBaseCfg
from isaaclab.scene import InteractiveScene, InteractiveSceneCfg
from isaaclab.sensors.ray_caster import RayCasterCfg, patterns
from isaaclab.utils import configclass
from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR

##
# Pre-defined configs
##
from isaaclab_assets.robots.anymal import ANYMAL_C_CFG  # isort: skip


@configclass
class RaycasterSensorSceneCfg(InteractiveSceneCfg):
    """Design the scene with sensors on the robot."""

    # ground plane
    ground = AssetBaseCfg(
        prim_path="/World/Ground",
        spawn=sim_utils.UsdFileCfg(
            usd_path=f"{ISAAC_NUCLEUS_DIR}/Environments/Terrains/rough_plane.usd",
            scale=(1, 1, 1),
        ),
    )

    # lights
    dome_light = AssetBaseCfg(
        prim_path="/World/Light", spawn=sim_utils.DomeLightCfg(intensity=3000.0, color=(0.75, 0.75, 0.75))
    )

    # robot
    robot = ANYMAL_C_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot")

    ray_caster = RayCasterCfg(
        prim_path="{ENV_REGEX_NS}/Robot/base/lidar_cage",
        update_period=1 / 60,
        offset=RayCasterCfg.OffsetCfg(pos=(0, 0, 0.5)),
        mesh_prim_paths=["/World/Ground"],
        ray_alignment="yaw",
        pattern_cfg=patterns.LidarPatternCfg(
            channels=100, vertical_fov_range=[-90, 90], horizontal_fov_range=[-90, 90], horizontal_res=1.0
        ),
        debug_vis=not args_cli.headless,
    )


def run_simulator(sim: sim_utils.SimulationContext, scene: InteractiveScene):
    """Run the simulator."""
    # Define simulation stepping
    sim_dt = sim.get_physics_dt()
    sim_time = 0.0
    count = 0

    triggered = True
    countdown = 42

    # Simulate physics
    while simulation_app.is_running():

        if count % 500 == 0:
            # reset counter
            count = 0
            # reset the scene entities
            # root state
            # we offset the root state by the origin since the states are written in simulation world frame
            # if this is not done, then the robots will be spawned at the (0, 0, 0) of the simulation world
            root_state = scene["robot"].data.default_root_state.clone()
            root_state[:, :3] += scene.env_origins
            root_state[:, :2] += 2
            scene["robot"].write_root_pose_to_sim(root_state[:, :7])
            scene["robot"].write_root_velocity_to_sim(root_state[:, 7:])
            # set joint positions with some noise
            joint_pos, joint_vel = (
                scene["robot"].data.default_joint_pos.clone(),
                scene["robot"].data.default_joint_vel.clone(),
            )
            joint_pos += torch.rand_like(joint_pos) * 0.1
            scene["robot"].write_joint_state_to_sim(joint_pos, joint_vel)
            # clear internal buffers
            scene.reset()
            print("[INFO]: Resetting robot state...")
        if count % 10 ==0:
            root_state = scene["robot"].data.root_state_w.clone()
            root_state[:, :3] += scene.env_origins
            root_state[:, :2] += 0.2
            scene["robot"].write_root_pose_to_sim(root_state[:, :7])
            scene["robot"].reset()
            scene.reset()
        # Apply default actions to the robot
        # -- generate actions/commands
        targets = scene["robot"].data.default_joint_pos
        # -- apply action to the robot
        scene["robot"].set_joint_position_target(targets)
        # -- write data to sim
        scene.write_data_to_sim()
        # perform step
        sim.step()
        # update sim-time
        sim_time += sim_dt
        count += 1
        # update buffers
        scene.update(sim_dt)

        # print information from the sensors
        print("-------------------------------")
        print(scene["ray_caster"])
        print("Ray cast hit results: ", scene["ray_caster"].data.ray_hits_w)

        if not triggered:
            if countdown > 0:
                countdown -= 1
                continue
            data = scene["ray_caster"].data.ray_hits_w.cpu().numpy()
            np.save("cast_data.npy", data)
            triggered = True
        else:
            continue


def main():
    """Main function."""

    # Initialize the simulation context
    sim_cfg = sim_utils.SimulationCfg(dt=0.005, device=args_cli.device)
    sim = sim_utils.SimulationContext(sim_cfg)
    # Set main camera
    sim.set_camera_view(eye=[3.5, 3.5, 3.5], target=[0.0, 0.0, 0.0])
    # design scene
    scene_cfg = RaycasterSensorSceneCfg(num_envs=args_cli.num_envs, env_spacing=2.0)
    scene = InteractiveScene(scene_cfg)
    # Play the simulator
    sim.reset()
    # Now we are ready!
    print("[INFO]: Setup complete...")
    # Run the simulator
    run_simulator(sim, scene)


if __name__ == "__main__":
    # run the main function
    main()
    # close sim app
    simulation_app.close()

Thank you!!