Policy server cuda, robot client on cpu - tensor mismatch

[wvangils]

System Info

Lerobot latest (git clone), running on Ubuntu Linux

Information

• One of the scripts in the examples/ folder of LeRobot
• My own task or dataset (give details below)

Reproduction

Started a PolicyServer on a remote machine with GPU. The configuration for the RobotClient is as follows:

import threading
from lerobot.robots.so101_follower import SO101FollowerConfig
from lerobot.cameras.opencv.configuration_opencv import OpenCVCameraConfig
from lerobot.async_inference.configs import RobotClientConfig
from lerobot.async_inference.robot_client import RobotClient
from lerobot.async_inference.helpers import visualize_action_queue_size
from lerobot.async_inference.configs import PolicyServerConfig
from lerobot.async_inference.policy_server import serve

camera_cfg = {
    "top": OpenCVCameraConfig(index_or_path=2, width=320, height=240, fps=30),
    "wrist": OpenCVCameraConfig(index_or_path=4, width=320, height=240, fps=30)
}

robot_cfg = SO101FollowerConfig(
  port="/dev/ttyACM0",
  id="ciabot_follower_arm",
  cameras=camera_cfg
)

client_cfg = RobotClientConfig(
    robot=robot_cfg,
    #server_address="localhost:8080",
    server_address="20.240.218.100:8080",
    policy_device="cuda",
    policy_type="smolvla",
    pretrained_name_or_path="wvangils/BlockPickerV4",
    chunk_size_threshold=0.5,
    actions_per_chunk=500,  # make sure this is less than the max actions of the policy
)

client = RobotClient(client_cfg)
task = "Pick up the blue cilinder and place it in the yellow cup."

if client.start():
    # Start action receiver thread
    action_receiver_thread = threading.Thread(target=client.receive_actions, daemon=True)
    action_receiver_thread.start()

    try:
        # Run the control loop
        client.control_loop(task)
    except KeyboardInterrupt:
        client.stop()
        action_receiver_thread.join()
        # (Optionally) plot the action queue size
        visualize_action_queue_size(client.action_queue_size)

Expected behavior

The RobotClient works on CPU and the PolicyServer is an A100 device, reachable on port 8080. I get this error when trying to perform inference for a Smolvla model:

python run_smolvla_async.py 
INFO 2025-10-17 14:20:19 a_opencv.py:179 OpenCVCamera(2) connected.
INFO 2025-10-17 14:20:20 a_opencv.py:179 OpenCVCamera(4) connected.
INFO 2025-10-17 14:20:20 follower.py:104 ciabot_follower_arm SO101Follower connected.
INFO 2025-10-17 14:20:20 t_client.py:113 Initializing client to connect to server at 20.240.218.100:8080
INFO 2025-10-17 14:20:20 t_client.py:132 Robot connected and ready
INFO 2025-10-17 14:20:20 t_client.py:155 Sending policy instructions to policy server
INFO 2025-10-17 14:20:43 t_client.py:446 Control loop thread starting
INFO 2025-10-17 14:20:43 t_client.py:272 Action receiving thread starting
Exception in thread Thread-1 (receive_actions):
Traceback (most recent call last):
  File "/home/cmotions/miniconda3/envs/ai-arm-smolvla-034/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/home/cmotions/miniconda3/envs/ai-arm-smolvla-034/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/tmp/lerobot/src/lerobot/async_inference/robot_client.py", line 285, in receive_actions
    timed_actions = pickle.loads(actions_chunk.data)  # nosec
  File "/home/cmotions/miniconda3/envs/ai-arm-smolvla-034/lib/python3.10/site-packages/torch/storage.py", line 530, in _load_from_bytes
    return torch.load(io.BytesIO(b), weights_only=False)
  File "/home/cmotions/miniconda3/envs/ai-arm-smolvla-034/lib/python3.10/site-packages/torch/serialization.py", line 1549, in load
    return _legacy_load(
  File "/home/cmotions/miniconda3/envs/ai-arm-smolvla-034/lib/python3.10/site-packages/torch/serialization.py", line 1807, in _legacy_load
    result = unpickler.load()
  File "/home/cmotions/miniconda3/envs/ai-arm-smolvla-034/lib/python3.10/site-packages/torch/serialization.py", line 1742, in persistent_load
    obj = restore_location(obj, location)
  File "/home/cmotions/miniconda3/envs/ai-arm-smolvla-034/lib/python3.10/site-packages/torch/serialization.py", line 698, in default_restore_location
    result = fn(storage, location)
  File "/home/cmotions/miniconda3/envs/ai-arm-smolvla-034/lib/python3.10/site-packages/torch/serialization.py", line 636, in _deserialize
    device = _validate_device(location, backend_name)
  File "/home/cmotions/miniconda3/envs/ai-arm-smolvla-034/lib/python3.10/site-packages/torch/serialization.py", line 605, in _validate_device
    raise RuntimeError(
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

Is it not possible to serve the instructions from a GPU-machine to a client with only CPU support?

[suessmann]

@wvangils hey, encountered the same issue. We've made a simple monkey-patch: added after L383 of lerobot/src/lerobot/async_inference/policy_server.py the following line list(action_tensor.cpu()). The error has gone away.

[milong26]

Is this right?

# lerobot/src/lerobot/async_inference/policy_server.py
# line after 383
        """5. Convert to TimedAction list"""
        action_chunk = self._time_action_chunk(
            observation_t.get_timestamp(), list(action_tensor.cpu()), observation_t.get_timestep()
        )

[zuoxingdong]

fixed by #2792