async inference doesn't generate actions.Observation #0 has been filtered out

[milong26]

I tried to use async inference,

I run this in server: python -m lerobot.scripts.server.policy_server --host="xx,xx,xx,xx" --port=9000
And have models in server.

run this locally:
PYTHONPATH=src python src/lerobot/scripts/server/robot_client.py --config_path=async_evaluate/template.yaml

the yaml looks like:

server_address: xx.xx.xx.xx:9000

robot:
  type: so100_follower
  port: /dev/ttyACM0
  id: congbi
  cameras:
    side:
      type: intelrealsense
      serial_number_or_name: aaaaaaaaaaa
      width: 640
      height: 480
      fps: 30
      use_depth: false
    wrist:
      type: opencv
      index_or_path: 6
      width: 640
      height: 480
      fps: 30

task: "pick up the cube and place it into the box."
policy_type: smolvla
pretrained_name_or_path: outputs/train/bbb/checkpoints/last/pretrained_model
policy_device: cuda

actions_per_chunk: 50
chunk_size_threshold: 0.5
aggregate_fn_name: weighted_average
debug_visualize_queue_size: true


# dataset:
#   repo_id: eval/eval_firts50_baseline
#   num_episodes: 1
#   episode_time_s: 60
#   push_to_hub: false
#   reset_time_s: 0

Then I saw these in server(they look similar)

INFO 2025-07-13 12:59:54 y_server.py:175 Received observation #0 | Avg FPS: 5.95 | Target: 30.00 | One-way latency: 2.17ms
INFO 2025-07-13 12:59:54 y_server.py:191 Observation #0 has been filtered out
INFO 2025-07-13 12:59:54 /helpers.py:354  Starting receiver
INFO 2025-07-13 12:59:54 y_server.py:175 Received observation #0 | Avg FPS: 5.95 | Target: 30.00 | One-way latency: 1.89ms
INFO 2025-07-13 12:59:54 y_server.py:191 Observation #0 has been filtered out
INFO 2025-07-13 12:59:54 /helpers.py:354  Starting receiver
INFO 2025-07-13 12:59:54 y_server.py:175 Received observation #0 | Avg FPS: 5.95 | Target: 30.00 | One-way latency: 1.80ms
INFO 2025-07-13 12:59:54 y_server.py:191 Observation #0 has been filtered out
INFO 2025-07-13 12:59:54 /helpers.py:354  Starting receiver
INFO 2025-07-13 12:59:54 y_server.py:175 Received observation #0 | Avg FPS: 5.95 | Target: 30.00 | One-way latency: 2.52ms
INFO 2025-07-13 12:59:54 y_server.py:191 Observation #0 has been filtered out
INFO 2025-07-13 12:59:54 /helpers.py:354  Starting receiver
INFO 2025-07-13 12:59:55 y_server.py:175 Received observation #0 | Avg FPS: 5.95 | Target: 30.00 | One-way latency: 2.25ms
INFO 2025-07-13 12:59:55 y_server.py:191 Observation #0 has been filtered out
INFO 2025-07-13 12:59:55 /helpers.py:354  Starting receiver
INFO 2025-07-13 12:59:55 y_server.py:175 Received observation #0 | Avg FPS: 5.95 | Target: 30.00 | One-way latency: 2.10ms
INFO 2025-07-13 12:59:55 y_server.py:191 Observation #0 has been filtered out
INFO 2025-07-13 12:59:55 /helpers.py:354  Starting receiver
INFO 2025-07-13 12:59:55 y_server.py:175 Received observation #0 | Avg FPS: 5.95 | Target: 30.00 | One-way latency: 1.77ms
INFO 2025-07-13 12:59:55 y_server.py:191 Observation #0 has been filtered out
INFO 2025-07-13 12:59:55 /helpers.py:354  Starting receiver
INFO 2025-07-13 12:59:55 y_server.py:175 Received observation #0 | Avg FPS: 5.95 | Target: 30.00 | One-way latency: 1.81ms
INFO 2025-07-13 12:59:55 y_server.py:191 Observation #0 has been filtered out
INFO 2025-07-13 12:59:55 /helpers.py:354  Starting receiver
INFO 2025-07-13 12:59:55 y_server.py:175 Received observation #0 | Avg FPS: 5.95 | Target: 30.00 | One-way latency: 1.92ms
INFO 2025-07-13 12:59:55 y_server.py:191 Observation #0 has been filtered out
INFO 2025-07-13 12:59:55 /helpers.py:354  Starting receiver
INFO 2025-07-13 12:59:55 y_server.py:175 Received observation #0 | Avg FPS: 5.95 | Target: 30.00 | One-way latency: 1.84ms
INFO 2025-07-13 12:59:55 y_server.py:191 Observation #0 has been filtered out
INFO 2025-07-13 12:59:55 /helpers.py:354  Starting receiver

By the way I saw reduce layer to 16 so the server got policy message and load policy correcly.

And to test I add an Error om getAction function

            # Create and return the action chunk
            actions = async_inference_pb2.Actions(data=actions_bytes)
            raise KeyError("generate an action")
            self.logger.info(
                f"Action chunk #{obs.get_timestep()} generated | "
                f"Total time: {(inference_time + serialize_time) * 1000:.2f}ms"
            )

But nothing happened. So I think getAction wasn't used at all.

And locally terminal said:

INFO 2025-07-13 20:59:54 t_client.py:471 Control loop (ms): 166.87
INFO 2025-07-13 20:59:55 t_client.py:213 Sent observation #0 | 
INFO 2025-07-13 20:59:55 t_client.py:471 Control loop (ms): 168.18
INFO 2025-07-13 20:59:55 t_client.py:213 Sent observation #0 | 
INFO 2025-07-13 20:59:55 t_client.py:471 Control loop (ms): 169.37
INFO 2025-07-13 20:59:55 t_client.py:213 Sent observation #0 | 
INFO 2025-07-13 20:59:55 t_client.py:471 Control loop (ms): 165.41
INFO 2025-07-13 20:59:55 t_client.py:213 Sent observation #0 | 
INFO 2025-07-13 20:59:55 t_client.py:471 Control loop (ms): 165.57
INFO 2025-07-13 20:59:55 t_client.py:213 Sent observation #0 | 
INFO 2025-07-13 20:59:55 t_client.py:471 Control loop (ms): 166.10

(My server time is incorrect, I run them at the same time.)

And of course no action made by my robot.;-;

How to solve it?

lerobot: d2645cb

[milong26]

Btw, I got this in local terminal when I stopped the program. said;

qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "/home/qwe/anaconda3/envs/lerobot/lib/python3.10/site-packages/cv2/qt/plugins" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: xcb, eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx, webgl.

I don't think it caused the problem though

[fracapuano]

Hey @milong26, thank you so much for using the async stack. I think I can say I agree with you: I am not sure this caused your issue either

In short

How I think you could solve this issue:

• Try to run the server for longer? From your logs (thank you so much for adding them!) it looks like you're running the server (minus setup) for ~1s only

An actual answer, 😅

Why I believe this is the solution to your issue:

• On the client side, observations (and actions) are marked with a timestep (t=0,1,2,3,...) to be easily reconciliated after round trips.
• The timestep counter is defined at the client level as an environment variable, and is incremented only when actions are performed (see robot_client.control_loop_action() in the robot_client).
• In short: one starts with observation 0, sends it for processing, receives the first action chunk, steps through it, updating the timestep. Meaning, acts, sends observation 1, acts, sends observation 2, ...
• On the server side, we want to limit the computational load to the essential. As of now (but this could totally change, one can make very compelling arguments on why the current setup can be considered suboptimal):
  • We keep a history of all the observations processed. That is, we store the timestep of all the observations which have undergone processing (i.e., have been used for inference, a ✨neural network✨ touched them)
  • Whenever the server receives a new observation from the client, it checks if the timestep is among those that have already been used for inference. If so, the incoming observation is "filtered out", and not used for inference, with the exception of when that observation comes with a boolean flag (which we call must_go). In that case, we always process the incoming observation. Think of it as a "skip all checks, make GPU go brrr"
  • If you run the policy server not long enough (i.e., before the first incoming observation is successfully processed and becomes an action chunk), then the timestep is never updated, and you exclusively see timestep-0 observations coming in

[milong26]

Thanks! It runs correctly now. If anyone met same problem here is my experience:

1. check the first observation must_go ,it should be true
2. if it is, _enqueue_observation should return true.
3. function GetActions should predict action chunks and send to local
4. in robotclient.py,send_action should send action instruction to robot.

But I find local send action is less smooth than not using async inference.
I will test later.

By the way. Do I need to stop server when I stopped locally?
Since when I first run

server listening ...
local def start(self): send model config to server -> server local model ->
local send observation to server -> server get obs -> server predict action chunk -> server send action chunk to local -> local send action to robot
...
local stopped.

if I don't stop server, will the server reload model again?

[eliasab16]

@milong26 have you encountered the following error while running async inference on pi0?
"Error in StreamActions: Currently not implemented for PI0"

INFO 2025-08-21 19:35:24 y_server.py:150 Time taken to put policy on cuda: 62.2355 seconds
INFO 2025-08-21 19:35:24 ort/utils.py:74 <Logger policy_server (NOTSET)> Starting receiver
INFO 2025-08-21 19:35:25 y_server.py:175 Received observation #0 | Avg FPS: 0.00 | Target: 30.00 | One-way latency: 55.49ms
INFO 2025-08-21 19:35:25 y_server.py:205 Running inference for observation #0 (must_go: True)
ERROR 2025-08-21 19:35:25 y_server.py:245 Error in StreamActions: Currently not implemented for PI0
INFO 2025-08-21 19:35:25 ort/utils.py:74 <Logger policy_server (NOTSET)> Starting receiver
INFO 2025-08-21 19:35:25 y_server.py:175 Received observation #0 | Avg FPS: 1.92 | Target: 30.00 | One-way latency: 51.12ms

[milong26]

@eliasab16 I used smolvla not pi0 so I didn't encounter this
I serached, in modeling_pi0.py

    @torch.no_grad()
    def predict_action_chunk(self, batch: dict[str, Tensor]) -> Tensor:
        """Predict a chunk of actions given environment observations."""
        raise NotImplementedError("Currently not implemented for PI0")

so you can't use async inference with pi0 under this version i think.

[kk2kl]

I tried to use async inference,

I run this in server: python -m lerobot.scripts.server.policy_server --host="xx,xx,xx,xx" --port=9000 And have models in server.

run this locally: PYTHONPATH=src python src/lerobot/scripts/server/robot_client.py --config_path=async_evaluate/template.yaml

the yaml looks like:

server_address: xx.xx.xx.xx:9000

robot:
type: so100_follower
port: /dev/ttyACM0
id: congbi
cameras:
side:
type: intelrealsense
serial_number_or_name: aaaaaaaaaaa
width: 640
height: 480
fps: 30
use_depth: false
wrist:
type: opencv
index_or_path: 6
width: 640
height: 480
fps: 30

task: "pick up the cube and place it into the box."
policy_type: smolvla
pretrained_name_or_path: outputs/train/bbb/checkpoints/last/pretrained_model
policy_device: cuda

actions_per_chunk: 50
chunk_size_threshold: 0.5
aggregate_fn_name: weighted_average
debug_visualize_queue_size: true

dataset:

repo_id: eval/eval_firts50_baseline

num_episodes: 1

episode_time_s: 60

push_to_hub: false

reset_time_s: 0

Then I saw these in server(they look similar)

INFO 2025-07-13 12:59:54 y_server.py:175 Received observation #0 | Avg FPS: 5.95 | Target: 30.00 | One-way latency: 2.17ms
INFO 2025-07-13 12:59:54 y_server.py:191 Observation #0 has been filtered out
INFO 2025-07-13 12:59:54 /helpers.py:354  Starting receiver
INFO 2025-07-13 12:59:54 y_server.py:175 Received observation #0 | Avg FPS: 5.95 | Target: 30.00 | One-way latency: 1.89ms
INFO 2025-07-13 12:59:54 y_server.py:191 Observation #0 has been filtered out
INFO 2025-07-13 12:59:54 /helpers.py:354  Starting receiver
INFO 2025-07-13 12:59:54 y_server.py:175 Received observation #0 | Avg FPS: 5.95 | Target: 30.00 | One-way latency: 1.80ms
INFO 2025-07-13 12:59:54 y_server.py:191 Observation #0 has been filtered out
INFO 2025-07-13 12:59:54 /helpers.py:354  Starting receiver
INFO 2025-07-13 12:59:54 y_server.py:175 Received observation #0 | Avg FPS: 5.95 | Target: 30.00 | One-way latency: 2.52ms
INFO 2025-07-13 12:59:54 y_server.py:191 Observation #0 has been filtered out
INFO 2025-07-13 12:59:54 /helpers.py:354  Starting receiver
INFO 2025-07-13 12:59:55 y_server.py:175 Received observation #0 | Avg FPS: 5.95 | Target: 30.00 | One-way latency: 2.25ms
INFO 2025-07-13 12:59:55 y_server.py:191 Observation #0 has been filtered out
INFO 2025-07-13 12:59:55 /helpers.py:354  Starting receiver
INFO 2025-07-13 12:59:55 y_server.py:175 Received observation #0 | Avg FPS: 5.95 | Target: 30.00 | One-way latency: 2.10ms
INFO 2025-07-13 12:59:55 y_server.py:191 Observation #0 has been filtered out
INFO 2025-07-13 12:59:55 /helpers.py:354  Starting receiver
INFO 2025-07-13 12:59:55 y_server.py:175 Received observation #0 | Avg FPS: 5.95 | Target: 30.00 | One-way latency: 1.77ms
INFO 2025-07-13 12:59:55 y_server.py:191 Observation #0 has been filtered out
INFO 2025-07-13 12:59:55 /helpers.py:354  Starting receiver
INFO 2025-07-13 12:59:55 y_server.py:175 Received observation #0 | Avg FPS: 5.95 | Target: 30.00 | One-way latency: 1.81ms
INFO 2025-07-13 12:59:55 y_server.py:191 Observation #0 has been filtered out
INFO 2025-07-13 12:59:55 /helpers.py:354  Starting receiver
INFO 2025-07-13 12:59:55 y_server.py:175 Received observation #0 | Avg FPS: 5.95 | Target: 30.00 | One-way latency: 1.92ms
INFO 2025-07-13 12:59:55 y_server.py:191 Observation #0 has been filtered out
INFO 2025-07-13 12:59:55 /helpers.py:354  Starting receiver
INFO 2025-07-13 12:59:55 y_server.py:175 Received observation #0 | Avg FPS: 5.95 | Target: 30.00 | One-way latency: 1.84ms
INFO 2025-07-13 12:59:55 y_server.py:191 Observation #0 has been filtered out
INFO 2025-07-13 12:59:55 /helpers.py:354  Starting receiver

By the way I saw reduce layer to 16 so the server got policy message and load policy correcly.

And to test I add an Error om getAction function

            # Create and return the action chunk
            actions = async_inference_pb2.Actions(data=actions_bytes)
            raise KeyError("generate an action")
            self.logger.info(
                f"Action chunk #{obs.get_timestep()} generated | "
                f"Total time: {(inference_time + serialize_time) * 1000:.2f}ms"
            )

But nothing happened. So I think getAction wasn't used at all.

And locally terminal said:

INFO 2025-07-13 20:59:54 t_client.py:471 Control loop (ms): 166.87
INFO 2025-07-13 20:59:55 t_client.py:213 Sent observation #0 | 
INFO 2025-07-13 20:59:55 t_client.py:471 Control loop (ms): 168.18
INFO 2025-07-13 20:59:55 t_client.py:213 Sent observation #0 | 
INFO 2025-07-13 20:59:55 t_client.py:471 Control loop (ms): 169.37
INFO 2025-07-13 20:59:55 t_client.py:213 Sent observation #0 | 
INFO 2025-07-13 20:59:55 t_client.py:471 Control loop (ms): 165.41
INFO 2025-07-13 20:59:55 t_client.py:213 Sent observation #0 | 
INFO 2025-07-13 20:59:55 t_client.py:471 Control loop (ms): 165.57
INFO 2025-07-13 20:59:55 t_client.py:213 Sent observation #0 | 
INFO 2025-07-13 20:59:55 t_client.py:471 Control loop (ms): 166.10

(My server time is incorrect, I run them at the same time.)

And of course no action made by my robot.;-;

How to solve it?

lerobot: d2645cb

@milong26 may I ask how you managed to solve this error please? I am facing the same error now, the robot does not generate action when running policy_server and robot_client.py

[kk2kl]

Thanks! It runs correctly now. If anyone met same problem here is my experience:

1. check the first observation must_go ,it should be true
2. if it is, _enqueue_observation should return true.
3. function GetActions should predict action chunks and send to local
4. in robotclient.py,send_action should send action instruction to robot.

But I find local send action is less smooth than not using async inference. I will test later.

By the way. Do I need to stop server when I stopped locally? Since when I first run

server listening ...
local def start(self): send model config to server -> server local model ->
local send observation to server -> server get obs -> server predict action chunk -> server send action chunk to local -> local send action to robot
...
local stopped.

if I don't stop server, will the server reload model again?

@milong26 could you explain in more details how you implement these in your code please?

[milong26]

@kk2kl Try to run the server for longer? From your logs (thank you so much for adding them!) it looks like you're running the server (minus setup) for ~1s only this method works.

Or you can put your object to another place, since in one trial, I saw action generated but arm didn't work.
There're two other methods which may help u check

1. write a function(such as listen to keyboard) to clear action queues and set must_go to true.
2. watch the output of policy_server and robot client. Make sure
   1. first observation must_go =true, server should generate some actions and sent them to client.
   2. client pick actions from queue one bye one, and robot follow the action.
   3. there should be some like policy_server_1757343164.log in log/ folder.

By the way my function looks like(you can ask ai to write one too)

# in init
        # 监听键盘输入->的时候清空action缓存
        threading.Thread(target=self._listen_clear_key,daemon=True).start()
        # 等待时间
        self.pause_until=0
        self.use_skip_step=config.use_skip_step

# listen to keyboard,in robot_clinet.py
    def _listen_clear_key(self):
        def on_press(key):
            try:
                if key==keyboard.Key.right:
                    self.clear_action_queue()
                if key==keyboard.Key.left:
                    self.running=False
            except AttributeError:
                self.logger.info("键盘出问题")
        with keyboard.Listener(on_press=on_press) as listener:
            listener.join()

    def clear_action_queue(self,pause_seconds:float=3.0):
        with self.action_queue_lock:
            # while not self.action_queue.empty():
                # self.action_queue.get_nowait()
                # 直接新建队列
                self.action_queue = Queue()
        self.must_go.set()
        self.logger.info("清空了actionqueue，顺便设置了mustgo，停个几秒")
        # 要设置control_loop里面暂停
        self.pause_until=time.time()+pause_seconds

    def control_loop(self, task: str, verbose: bool = False) -> tuple[Observation, Action]:
        """Combined function for executing actions and streaming observations"""
        # Wait at barrier for synchronized start
        self.start_barrier.wait()
        self.logger.info("Control loop thread starting")

        _performed_action = None
        _captured_observation = None

        while self.running:

            # 右键清空，暂停几秒
            if time.time()<self.pause_until:
                time.sleep(0.1)
                continue

[kk2kl]

@milong26 thanks a lot for your reply, I will try to make modification according to your suggestion. I have another question which is not related to " async inference", when I try to run the server and client in same machine, it can work properly without error, but when I run server and client in different machine, I got error as shown below, even though I pass the path for locally stored policy model, the error shows that the path is treated as huggingface repo_id.

I am running the remote server in jupyter notebook with GPU, and local Window computer with CPU:

server(remote) and client in different machine:

python ./src/lerobot/scripts/server/policy_server.py --host=0.0.0.0 --port=8011

python src\lerobot\scripts\server\robot_client.py ^
--server_address=10.XX.0.XXX:8011

python src\lerobot\scripts\server\robot_client.py ^
--server_address=10.XX.0.XXX:8011 ^
--robot.type=so101_follower ^
--robot.port=COM4 ^
--robot.id=follower_arm ^
--robot.cameras="{ handeye: {type: opencv, index_or_path: 1, width: 640, height: 360, fps: 30}, fixed: {type: opencv, index_or_path: 2, width: 640, height: 360, fps: 30}}" ^
--task="pick and place the cube" ^
--policy_type=act ^
--pretrained_name_or_path="/home/myuser_name_placeholder/diffusion1" ^
--policy_device=cuda ^
--actions_per_chunk=50 ^
--chunk_size_threshold=0.5 ^
--aggregate_fn_name=weighted_average ^
--debug_visualize_queue_size=True

error:

debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"Exception calling application: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96:

server and client in same machine (this can run without error):

python ./src/lerobot/scripts/server/policy_server.py --host=127.0.0.1 --port=8011

python src\lerobot\scripts\server\robot_client.py ^
--server_address=127.0.0.1:8011 ^
--robot.type=so101_follower ^
--robot.port=COM4 ^
--robot.id=follower_arm ^
--robot.cameras="{ handeye: {type: opencv, index_or_path: 1, width: 640, height: 360, fps: 30}, fixed: {type: opencv, index_or_path: 2, width: 640, height: 360, fps: 30}}" ^
--task="pick and place the cube" ^
--policy_type=act ^
--pretrained_name_or_path="C:\Users\Downloads\act_3" ^
--policy_device=cuda ^
--actions_per_chunk=50 ^
--chunk_size_threshold=0.5 ^
--aggregate_fn_name=weighted_average ^
--debug_visualize_queue_size=True

May I ask the policy model should be stored in the server machine or the client machine? (I stored policy model in jupyter notebook server device, so I passed the argument: "/home/myuser_name_placeholder/diffusion1" ), but the path is treated as huggingface repo_id :(

Could you give some suggestion and possible solution please? Thank you!

[milong26]

@kk2kl I run similar to you one in remote server one in local.

server connection

1. on remote server, I run CUDA_VISIBLE_DEVICES=2 python -m lerobot.scripts.server.policy_server --host="10.10.xx.xx" --port=9000 not 0.0.0.0,but 0.0.0.0 is for any quest i think it's ok but it may be better use right one?
2. on local pc connecting robot, I run python robot_client where server_address: 10.10.xx.xx:9000 will told it location of server.

from the error debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"Exception calling application: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: I think it happened even before connection?

model savepath

the policy model should be stored in the server machine or the client machine?

Download models except .safetensors(you can download also but no use just waste network) to local file and make sure they have same path relative to parent folder.

• download is because local code need config.json or something to send observation.
• same path is because server will try to find the policy according to the path.
• relative to parent folder is like:

in local terminal:
python robot_clinet.py --pretrained_name_or_path: outputs/train/should_find_the_path_of_model(incluing pretrained_path)

server: xxxx/lerobot$ you can reach model through cd outputs/train/should_find_the_path_of_model(incluing pretrained_path)
local: xxxx/leorbot$ same.

lerobot is repository of https://github.com/huggingface/lerobot. You can start from any other folder just make sure server and local can reach model folder use same pretrained_name_or_path

By the way if you don't want it reated as huggingface repo_id and started downloading, use HF_HUB_OFFLINE=1 in terminal before python xxx.py

[kk2kl]

@milong26 many thanks for your help, I successfully implemented the remote inference for Smolvla, and I really learned a lot of stuff from you.

Sorry for asking so many questions but I want to ask you how to let the Lerobot generalize well on unseen scenarios or unseen object position, using limited/small amount of dataset for performing task such as pick and place of object? I tried few types of policies (Act, Diffusion, Smolvla), but each of them when I change the initial position of object slightly (even move object a few centimeters only), the robotics arm failed to perform pick and place, and it is infeasible for me to record thousands of episodes for each position of object..........

Let say on the table, there are thousands of possible initial positions of object, or let said I rotate the object a few angle only, how to use limited/small amount of dataset (assuming less than 100 episodes) to train a policy model which is intelligent enough to perform pick and place when object is located in different positions?

I know there are some simulation tutorials which allow user to initialize random positions of objects (object spawn region) for training, but I am curious is there any easier solution for this issue?

In order to achieve my objective, should I perform additional modification on policy codes, apply other techniques like object tracking, advanced Computer Vision Techniques or Yolo etc?

[milong26]

@kk2kl

finish one task

I tried few types of policies (Act, Diffusion, Smolvla), but each of them when I change the initial position of object slightly (even move object a few centimeters only),
How many episodes u collected for training/fine-tuning? In tuorial of fine-tune smolvla it said
In this dataset, we recorded 50 episodes across 5 distinct cube positions. For each position, we collected 10 episodes of pick-and-place interactions.

for me, I collected picking up dataset on a 50cm*50cm Padding cloth(I don't know how to describe, just a defined working space on table). picking up objects ranged from 2cm to 5cm for 100 episodes. Highest success rate can be around 70% when evaluating: initialize the object on different position and different pose.

So if you only want to finish one task using pure code , make sure dataset is high quality: no failed episode, the route is clear, fix the robot and working space, fix camera.

My data quality is still low cuz my friend only used 50 episodes and reach 70%+.

generalization of smolvla

I made some experiments, known that with pure smolvla, it must lack generalization for it's only 0.45B, i tried to let it pick up unknown object but low chance, then do complex task with unknown objects it can't succeed at all ... You can try to find some way to upgrade smovla, which is called reserach. :D

By the way Im glad to think about the problem u asked me, because i'm still new to this, wish we can make more progress with it.

[kk2kl]

@milong26 may I ask how to run multiple-gpus for training and inference for Lerobot?

I tried 2 commands, but both got error:

First Trial:

CUDA_VISIBLE_DEVICES=1,2
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
torchrun --nproc_per_node=2
src/lerobot/scripts/train.py
--dataset.repo_id=datasetid/name
--policy.type=pi05_openpi
--output_dir=outputs_pi05_2/train/so101_test4_pi05_2
--job_name=pi05_2
--policy.repo_id=pepijn223/pi05_base_fp32
--policy.pretrained_path=repoid/name
--policy.compile_model=true
--policy.gradient_checkpointing=true
--policy.push_to_hub=true
--policy.scheduler_decay_steps=3000
--policy.device=cuda
--wandb.enable=true
--policy.dtype=bfloat16
--steps=50000
--num_workers=4
--batch_size=4

Second Trial

CUDA_VISIBLE_DEVICES=1,2
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
python -m accelerate.commands.launch
--num_processes=2
src/lerobot/scripts/train.py
--dataset.repo_id=datasetid/name
--policy.type=pi05_openpi
--output_dir=outputs/train/so101_test4_pi05_2
--job_name=pi05_2
--policy.repo_id=pepijn223/pi05_base_fp32
--policy.pretrained_path=repoid/name
--policy.compile_model=true
--policy.gradient_checkpointing=true
--policy.push_to_hub=true
--policy.scheduler_decay_steps=3000
--policy.device=cuda
--wandb.enable=false
--policy.dtype=bfloat16
--steps=50000
--num_workers=4
--batch_size=4
--resume=true

Error in Second Trial

I tried these 2 commands, but both commands only utilize single gpu, not using multiple gpus,
@milong26 is there any solution which can run Multiple GPUS support and bypass these errors?

[milong26]

@kk2kl lerobot hasn't written distributed training till now. see #1666 luckily it said it will go on after dataset v3 and i saw dataset v3 is updated.

I ruined the torch.cuda.is_available() when I didn't set CUDA_VISIBLE_DEVICES=0/1/2 before so I only tried using single gpu to train and evaluate. You can search how others finish this.