Optimize Inference Loop by Moving Observation Capture Inside Condition Block#3
Conversation
shantanuparab-tr
commented
Sep 22, 2025
shantanuparab-tr
commented
- Moved get_observation() and image preprocessing inside the inference condition.
- Now observations and image transformations are only triggered when a new action chunk is needed.
- This reduces unnecessary image capture and avoids frame drops on low-power or poorly configured machines (e.g., USB bandwidth bottlenecks).
- Ensures better performance and stability during evaluation and deployment.
shantanuparab-tr
added
bug
There was a problem hiding this comment.
Pull Request Overview
This PR adds support for Trossen AI robot integration and optimizes the inference loop by moving observation capture inside condition blocks. The changes enable training and evaluating Pi-0 policies on Trossen AI stationary bimanual setups.
- Adds a new training configuration for Trossen AI robot support with proper camera mapping
- Implements a complete client bridge for connecting Trossen AI hardware to OpenPI policy servers
- Moves observation capture and image preprocessing inside inference conditions to reduce unnecessary processing
Reviewed Changes
Copilot reviewed 5 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/openpi/training/config.py | Adds training configuration for Trossen AI robot with LoRA fine-tuning |
| pyproject.toml | Enables direct git references for dependency management |
| examples/trossen_ai/pyproject.toml | Sets up isolated environment for Trossen AI client with specific LeRobot version |
| examples/trossen_ai/main.py | Implements bridge between Trossen AI hardware and OpenPI policy server |
| examples/trossen_ai/README.md | Provides comprehensive setup and usage documentation |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| cameras = list(self.robot._cameras_ft.keys()) | ||
| for cam in cameras: | ||
| image_hwc = observation_dict[cam] | ||
| #convert BGR to RGB |
There was a problem hiding this comment.
Comment should start with a capital letter and have proper spacing: '# Convert BGR to RGB'
| #convert BGR to RGB | |
| # Convert BGR to RGB |
| elif self.test_mode == "autonomous": | ||
| joint_features = list(self.robot._joint_ft.keys()) | ||
| action_dict = {k: full_action[i] for i, k in enumerate(joint_features)} | ||
| self.robot.send_action(action_dict) | ||
| else: | ||
| logger.error(f"Unknown mode: {self.test_mode}. No action executed.") |
There was a problem hiding this comment.
The action_dict construction assumes full_action has the same length as joint_features, but there's no validation. This could cause an IndexError if the action dimensions don't match the expected joint count.
| #convert BGR to RGB | ||
| image_resized = cv2.resize(image_hwc, (224, 224)) | ||
| image_rgb = cv2.cvtColor(image_resized, cv2.COLOR_BGR2RGB) |
There was a problem hiding this comment.
[nitpick] The image processing steps (resize to 224x224, BGR to RGB conversion) should be extracted into a separate method to improve readability and reusability.
…n Block (#3) optimize get_observation call
…n Block (#3) optimize get_observation call
…n Block (#3) optimize get_observation call
