feat: add more scenarios to o3de manipulation benchmark

[jmatejcz]

Purpose

#433

• Make creation of new tasks faster
• Add more tasks for o3deTestBenchmark
• Add more simulation configs for o3deTestBenchmark

Proposed Changes

• ManipulationTask for common logic
• Made tasks more generic and parametrized them so they can operate on different types of objects -> this will result in much more variants of a task available, for exampe: instead of move carrot to the left side -> move {object types} to the left side, where object types is passed as argument
• Refactored tasks -> MoveObjectToLeftTask, GroupObjectsTask
• New type of tasks -> BuildCubeTowerTask, PlaceObjectAtCoordTask, RotateObjectTask(not aplicable right now)
• New scenes configs, renamed them to specify what objects are present.
• Unit tests for helper functions and tasks to ensure proper score calculation
• Packets of scenarios, that are subjectively grouped into 5 level of difficulty: trivial, easy, medium, hard, very_hard (can and probably will be adjusted). For now there are 10 trivial, 42 easy, 23 medium, 38 hard and 47 very hard scenarios.
• Resetting arm to base position
• Launching new binaries

Issues

• Not able to test manually if scores are calculated properly in different scenarios, as number of possible scenarios grows significantly. Depending on unit tests to ensure that scores are calculated properly. These tests must cover all cases.

• Not enough trivial scenarios (like move object to coords when few objects). Harder scenarios like creating structures from many objects can be created in various number of ways. It isn't the same for easy scenarios where the scenario should include 1, maybe 2 moves and scene setup that is not complicated. If you have ideas for other easy tasks, let me know in comments!

• For now ManipulatorMoveTo tool does not allow changing orientation of the gripper, which makes some tasks like RotateObjectTask, or scenes with rotated objects not usable.

• Corn entities are to small for gripper, always falling out, which result in them being avoided when defining tasks

• When gripper is holding an object it can place it into other object

Testing

Setup

1. Setup the repository
2. Install dependencies listed in:
   https://github.com/RobotecAI/rai/blob/main/docs/demos/manipulation.md

and:

poetry install --with openset
vcs import < demos.repos
rosdep install --from-paths src/examples/rai-manipulation-demo/ros2_ws/src --ignore-src -r -y
colcon build --symlink-install
source setup_shell.sh

3. Download GameLauncher binary: humble or jazzy

4. Populate src/rai_bench/rai_bench/o3de_test_bench/configs/o3de_config.yaml with:

binary_path: /path/to/your/GameLauncher
level: RoboticManipulationBenchmark
robotic_stack_command: ros2 launch examples/manipulation-demo-no-binary.launch.py
required_simulation_ros2_interfaces:
  services:
    - /spawn_entity
    - /delete_entity
  topics:
    - /color_image5
    - /depth_image5
    - /color_camera_info5
  actions: []
required_robotic_ros2_interfaces:
  services:
    - /grounding_dino_classify
    - /grounded_sam_segment
    - /manipulator_move_to
  topics: []
  actions: []
robotic_stack_command: ros2 launch examples/manipulation-demo-no-binary.launch.py
required_simulation_ros2_interfaces:
  services:
    - /spawn_entity
    - /delete_entity
  topics:
    - /color_image5
    - /depth_image5
    - /color_camera_info5
  actions: []
required_robotic_ros2_interfaces:
  services:
    - /grounding_dino_classify
    - /grounded_sam_segment
    - /manipulator_move_to
  topics: []
  actions: []

Run examples

Run the benchmark with example scenarios. I do not recommend running full packets as this will take some time.
In src/rai_bench/rai_bench/examples/o3de_test_benchmark.py there are prepared packets of scenarios.
You can swap running all_scenarios for for example 3 trivial scenarios -> t_scenarios[:3] here:

    benchmark = Benchmark(
        simulation_bridge=o3de,
        scenarios=all_scenarios,
        logger=bench_logger,
        results_filename=results_filename,
    )

then:

python src/rai_bench/rai_bench/examples/o3de_test_benchmark.py

what to look for:

• check if all packets are running
• check how are they different, if you think the grading should be changed or some scenarios are not suited for the level, let me know in the comment
• logs and results can be found in src/rai_bench/rai_bench/experiments/
• check if arm resets to base position properly after every scenario

Tests

Run unit tests:

pytest tests/rai_bench/tasks/

• Check if they all pass
• Check the tests, if you can think of cases that are not covered, please let me know in comment.

[boczekbartek]

@jmatejcz Thank you for the PR
I can see that "This branch is out-of-date with the base branch" - could you please "Update with rebase"?

[jmatejcz]

@jmatejcz Thank you for the PR I can see that "This branch is out-of-date with the base branch" - could you please "Update with rebase"?

done

[boczekbartek]

@jmatejcz
Thank you for this PR. I tried example commands, but didn't manage to run the benchmark. I think some nodes might not start correctly.

Quick note: In step 2, before running colcon build I did vcs import < demos.repos; rosdep ... to download rai-manipulation-demo.

I configured the path the the demo binary as described. When I run this command:

python src/rai_bench/rai_bench/examples/o3de_test_benchmark.py

I can see that simulator started, but some ros2 nodes seem to be missing:

2025-03-18 22:41:00 robo-pc-005 Agent logger[234244] WARNING Waiting for missing services ['/grounding_dino_classify', '/grounded_sam_segment'] out of required services: ['/grounding_dino_classify', '/grounded_sam_segment', '/manipulator_move_to']
2025-03-18 22:41:00 robo-pc-005 Agent logger[234244] INFO required services: ['/grounding_dino_classify', '/grounded_sam_segment', '/manipulator_move_to']
2025-03-18 22:41:00 robo-pc-005 Agent logger[234244] INFO required topics: []
2025-03-18 22:41:00 robo-pc-005 Agent logger[234244] INFO required actions: []
2025-03-18 22:41:00 robo-pc-005 Agent logger[234244] INFO available actions: {'/move_action', '/panda_arm_controller/follow_joint_trajectory', '/panda_hand_controller/gripper_cmd', '/execute_trajectory'}
2025-03-18 22:41:00 robo-pc-005 Agent logger[234244] WARNING Waiting for missing services ['/grounding_dino_classify', '/grounded_sam_segment'] out of required services: ['/grounding_dino_classify', '/grounded_sam_segment', '/manipulator_move_to']

Here is a full log:
log.log

I think one of the reasons is that I get this error. Did you also encounter it?

$ ros2 launch examples/manipulation-demo-no-binary.launch.py
...
[grounded_sam-2] [ERROR] [1742334299.934854130] [grounded_sam]: Could not load model

Also, in the logs of the benchmark I can see a lot (402 in total) of logs like:

Could not create Scenario from task: Manipulate objects, so that ........

Are they expected? If they are expected - should user know all of them?

[boczekbartek]

@jmatejcz added some high-level comments/questions to the code

[jmatejcz]

@jmatejcz Thank you for this PR. I tried example commands, but didn't manage to run the benchmark. I think some nodes might not start correctly.

Quick note: In step 2, before running colcon build I did vcs import < demos.repos; rosdep ... to download rai-manipulation-demo.

I configured the path the the demo binary as described. When I run this command:

python src/rai_bench/rai_bench/examples/o3de_test_benchmark.py

I can see that simulator started, but some ros2 nodes seem to be missing:

2025-03-18 22:41:00 robo-pc-005 Agent logger[234244] WARNING Waiting for missing services ['/grounding_dino_classify', '/grounded_sam_segment'] out of required services: ['/grounding_dino_classify', '/grounded_sam_segment', '/manipulator_move_to']
2025-03-18 22:41:00 robo-pc-005 Agent logger[234244] INFO required services: ['/grounding_dino_classify', '/grounded_sam_segment', '/manipulator_move_to']
2025-03-18 22:41:00 robo-pc-005 Agent logger[234244] INFO required topics: []
2025-03-18 22:41:00 robo-pc-005 Agent logger[234244] INFO required actions: []
2025-03-18 22:41:00 robo-pc-005 Agent logger[234244] INFO available actions: {'/move_action', '/panda_arm_controller/follow_joint_trajectory', '/panda_hand_controller/gripper_cmd', '/execute_trajectory'}
2025-03-18 22:41:00 robo-pc-005 Agent logger[234244] WARNING Waiting for missing services ['/grounding_dino_classify', '/grounded_sam_segment'] out of required services: ['/grounding_dino_classify', '/grounded_sam_segment', '/manipulator_move_to']

Here is a full log: log.log

I think one of the reasons is that I get this error. Did you also encounter it?

$ ros2 launch examples/manipulation-demo-no-binary.launch.py
...
[grounded_sam-2] [ERROR] [1742334299.934854130] [grounded_sam]: Could not load model

Also, in the logs of the benchmark I can see a lot (402 in total) of logs like:

Could not create Scenario from task: Manipulate objects, so that ........

Are they expected? If they are expected - should user know all of them?

yes it seems like you can't properly load grounded dino and grounded sam models, which results in:
WARNING Waiting for missing services ['/grounding_dino_classify', '/grounded_sam_segment'] out of required services: ['/grounding_dino_classify', '/grounded_sam_segment', '/manipulator_move_to']

try removing any existing weights:

rm -rf build/ install/ log/

then once again:

colcon build --symlink-install
source setup_shell.sh
python src/rai_bench/rai_bench/examples/o3de_test_benchmark.py

[jmatejcz]

@jmatejcz Thank you for this PR. I tried example commands, but didn't manage to run the benchmark. I think some nodes might not start correctly.

Quick note: In step 2, before running colcon build I did vcs import < demos.repos; rosdep ... to download rai-manipulation-demo.

I configured the path the the demo binary as described. When I run this command:

python src/rai_bench/rai_bench/examples/o3de_test_benchmark.py

I can see that simulator started, but some ros2 nodes seem to be missing:

2025-03-18 22:41:00 robo-pc-005 Agent logger[234244] WARNING Waiting for missing services ['/grounding_dino_classify', '/grounded_sam_segment'] out of required services: ['/grounding_dino_classify', '/grounded_sam_segment', '/manipulator_move_to']
2025-03-18 22:41:00 robo-pc-005 Agent logger[234244] INFO required services: ['/grounding_dino_classify', '/grounded_sam_segment', '/manipulator_move_to']
2025-03-18 22:41:00 robo-pc-005 Agent logger[234244] INFO required topics: []
2025-03-18 22:41:00 robo-pc-005 Agent logger[234244] INFO required actions: []
2025-03-18 22:41:00 robo-pc-005 Agent logger[234244] INFO available actions: {'/move_action', '/panda_arm_controller/follow_joint_trajectory', '/panda_hand_controller/gripper_cmd', '/execute_trajectory'}
2025-03-18 22:41:00 robo-pc-005 Agent logger[234244] WARNING Waiting for missing services ['/grounding_dino_classify', '/grounded_sam_segment'] out of required services: ['/grounding_dino_classify', '/grounded_sam_segment', '/manipulator_move_to']

Here is a full log: log.log

I think one of the reasons is that I get this error. Did you also encounter it?

$ ros2 launch examples/manipulation-demo-no-binary.launch.py
...
[grounded_sam-2] [ERROR] [1742334299.934854130] [grounded_sam]: Could not load model

Also, in the logs of the benchmark I can see a lot (402 in total) of logs like:

Could not create Scenario from task: Manipulate objects, so that ........

Are they expected? If they are expected - should user know all of them?

to the second part of the question, the logs about Could not create Scenario from task: are expected but should be in debug level, i will adjust that. Thank you for the note.

[boczekbartek]

@jmatejcz

rm -rf build/ install/ log/

I tested with a fresh install. Probalby gdino weights got corrupted for some reason... but it worked now. Thank you!

to the second part of the question, the logs about Could not create Scenario from task: are expected but should be in debug level, I will adjust that. Thank you for the note.

Could you tell me a bit more about what this log message means?

[jmatejcz]

@jmatejcz

rm -rf build/ install/ log/

I tested with a fresh install. Probalby gdino weights got corrupted for some reason... but it worked now. Thank you!

to the second part of the question, the logs about Could not create Scenario from task: are expected but should be in debug level, I will adjust that. Thank you for the note.

Could you tell me a bit more about what this log message means?

Every task validates if given simulation config is suitable by checking if required objects are present and if any of them is placed incorrectly:
https://github.com/RobotecAI/rai/blob/jm/feat/o3de-bench-more-tasks/src/rai_bench/rai_bench/o3de_test_bench/tasks/manipulation_task.py#L81-L104
If not this means scenario wont be created out of these 2.

This is especially useful when scenarios are created automatically like stated in README. This means you can pass a list of tasks, list of simulation configs and get every possible combination of task x sim_config as scenarios.

[boczekbartek]

@jmatejcz I've added some comments to the code, but didn't mange to check all of it yet. I'll continue

[boczekbartek]

Approved.

• tested on ubuntu 22.04 & ubuntu 24.04
• added minor changes to this PR:
  • added config from the PR description to the repo
  • adjusted rai_bench/README
  • refactored /reset_manipulator service call to use ROS2Connector api