Skip to content

feat: updated docker setup for manipulation demo #641

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 27 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
34e22c8
Fixed: Vision weights were placed at incorrect path upon re-download.
Cytrus14 Jun 17, 2025
6aa277a
Added cuda availability check to GDBoxer
Cytrus14 Jun 18, 2025
b77b5dd
WIP dockerfile
Cytrus14 Jun 20, 2025
768a7b6
Increased timeout for GDino and GSAM from 5.0s to 60.0s
Cytrus14 Jun 23, 2025
7d0d2ae
Added fastrtps config for communication between the docker container …
Cytrus14 Jun 23, 2025
a6c3277
Replaced apt-get with apt, updated poetry to version 2.1.3
Cytrus14 Jun 23, 2025
b8aec43
Update docs for docker setup and the robotic arm demo
Cytrus14 Jun 24, 2025
6e4b18c
Removed branch switching from the dockerfile
Cytrus14 Jun 24, 2025
702d195
Update docs/setup/setup_docker.md
Cytrus14 Jun 25, 2025
4e72628
fixed broken link in docs
Cytrus14 Jun 25, 2025
e2f6f9f
Specified that ROS 2 must be source during the setup of communication…
Cytrus14 Jun 26, 2025
14d5f4b
Clarified that RAI git repository must be cloned to build the docker …
Cytrus14 Jun 26, 2025
7c32345
Move fastrtps_config.xml to the 'docker' directory
Cytrus14 Jun 26, 2025
208d4be
RAI repository is now copied from the host instead of being cloned fr…
Cytrus14 Jun 26, 2025
eb57419
Specified that a GPU can be used within the docker container for fast…
Cytrus14 Jun 26, 2025
a47ebc0
Specified that host and docker ROS 2 distributions should match
Cytrus14 Jun 26, 2025
7b88d37
Update command for running tests in docker
Cytrus14 Jun 26, 2025
07814d9
Fixed typo in docs
Cytrus14 Jun 26, 2025
8afa035
Fixed fastrtps file path in dokcerfile
Cytrus14 Jun 26, 2025
302aad2
Added sudo to apt install in docs
Cytrus14 Jun 30, 2025
329020c
Updated command for executing multiple tests in docs
Cytrus14 Jun 30, 2025
d6c81ff
Added warning about ROS2 version to docker setup docs
Cytrus14 Jun 30, 2025
7a4d5bd
Made setting the FASTRTPS_DEFAULT_PROFILES_FILE env variable more exp…
Cytrus14 Jun 30, 2025
5fef90e
Clarified that the env variable should be exported
Cytrus14 Jun 30, 2025
2e54d09
Added an instruction to docs, on how to resolve a potential GPU error…
Cytrus14 Jul 23, 2025
c7f703d
Added ROS 2 import inside docker to robotic arm demo docs
Cytrus14 Jul 23, 2025
4c63430
Added step with setting the OpenAI API key, clarified that the vendor…
Cytrus14 Jul 24, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
182 changes: 182 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,182 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# ROS
build/
devel/
install/
log/

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/ dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
dist/

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock

# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/

# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/

logs/
.vscode/

!examples/imgs/*.md

src/examples/*-demo
artifact_database.pkl

imgui.ini

src/rai_bench/rai_bench/experiments
src/rai_interfaces

# Demo assets
demo_assets/
31 changes: 23 additions & 8 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -13,27 +13,42 @@
# limitations under the License.

ARG ROS_DISTRO=jazzy

FROM osrf/ros:${ROS_DISTRO}-desktop-full

ARG DEPENDENCIES=core_only
ENV DEBIAN_FRONTEND=noninteractive

# Check whether the $DEPENDENCIES ARG has a valid value
RUN /bin/bash -c 'if [ "${DEPENDENCIES}" != "core_only" ] && [ "${DEPENDENCIES}" != "all_groups" ]; then \
echo Error: invalid DEPENDENCIES value. Valid values "core_only", "all_groups" && exit 1; \
fi'


# Install dependencies
RUN apt-get update && apt-get install -y \
RUN apt update && apt install -y \
python3 \
python3-pip \
git \
wget
wget \
zip

# Install Poetry
RUN curl -sSL https://install.python-poetry.org | python3 - --version 1.8.4
RUN curl -sSL https://install.python-poetry.org | python3 - --version 2.1.3
ENV PATH="/root/.local/bin:$PATH"

# Clone and setup RAI
# Copy the RAI repository
COPY . /rai
WORKDIR /rai
RUN git clone https://github.com/RobotecAI/rai.git .

# Install Python dependencies with Poetry
RUN poetry install --with nomad,openset
# Set up the environment to use Fast DDS
ENV FASTRTPS_DEFAULT_PROFILES_FILE=/rai/docker/fastrtps_config.xml
ENV RMW_IMPLEMENTATION=rmw_fastrtps_cpp

RUN /bin/bash -c 'if [ "${DEPENDENCIES}" = "core_only" ]; then \
poetry install; \
else \
poetry install --all-groups; \
fi'

# Install ROS dependencies
RUN /bin/bash -c '. /opt/ros/${ROS_DISTRO}/setup.bash && \
Expand Down
17 changes: 17 additions & 0 deletions docker/fastrtps_config.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
<?xml version="1.0" encoding="UTF-8" ?>
<profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles" >
<transport_descriptors>
<transport_descriptor>
<transport_id>CustomUdpTransport</transport_id>
<type>UDPv4</type>
</transport_descriptor>
</transport_descriptors>
<participant profile_name="participant_profile" is_default_profile="true">
<rtps>
<userTransports>
<transport_id>CustomUdpTransport</transport_id>
</userTransports>
<useBuiltinTransports>false</useBuiltinTransports>
</rtps>
</participant>
</profiles>
110 changes: 109 additions & 1 deletion docs/demos/manipulation.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,10 @@ manipulation techniques.

Make sure ROS 2 is sourced. (e.g. `source /opt/ros/humble/setup.bash`)

### Local Setup

#### Setting up the demo

1. Follow the RAI setup instructions in the [quick setup guide](../setup/install.md#setting-up-developer-environment).
2. Download additional dependencies:

Expand All @@ -39,7 +43,7 @@ manipulation techniques.
colcon build --symlink-install
```

## Running the Demo
#### Running the demo

!!! note "Remain in sourced shell"

Expand Down Expand Up @@ -75,6 +79,110 @@ manipulation techniques.

To change camera in the simulation use 1-7 keys on your keyboard once it's window is focused.

### Docker Setup

!!! note "ROS 2 required"

The docker setup requires a working Humble or Jazzy ROS 2 installation on the host machine. Make sure that ROS 2 is sourced on the host machine and the `ROS_DOMAIN_ID` environment variable is set to the same value as in the [Docker setup](../setup/setup_docker.md#2-set-up-communications-between-docker-and-host-optional)

!!! warning "ROS 2 distributions"

It is highly recommended that ROS 2 distribution on the host machine matches the ROS 2 distribution of the docker container. A distribution version mismatch may result in the demo not working correctly.

#### 1. Setting up the demo

1. Set up docker as outlined in the [docker setup guide](../setup/setup_docker.md). During the setup, build the docker image with all dependencies (i.e., use the `--build-arg DEPENDENCIES=all_groups` argument)
and configure communication between the container and the host ([link](../setup/setup_docker.md#2-set-up-communications-between-docker-and-host-optional)).

2. On the host machine, download the latest binary release for the Robotic Arm Demo:

```shell
./scripts/download_demo.sh manipulation
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

downloaded binary is specific to ros2 version. Is it possible to run this example in such setup:
host: ros2 jazzy
docker: ros2 humble?

I think communication-wise it might not work, so I would resolve it by mentioningin the docs that we advise against such setup with 2 different ros2 versions. Then download script will work fine

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 6ff5bd1

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the warning should also be added to docker docs

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in ad28165

```

3. Run the docker container (if not already running):

```shell
docker run --net=host --ipc=host --pid=host -e ROS_DOMAIN_ID=$ROS_DOMAIN_ID -it rai:jazzy # or rai:humble
```

!!! tip "NVIDIA GPU acceleration"

If the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) is set up on your host machine, you can use the GPU within the RAI docker container for faster inference by adding the `--gpus all` option:

```shell
docker run --net=host --ipc=host --pid=host -e ROS_DOMAIN_ID=$ROS_DOMAIN_ID --gpus all -it rai:jazzy # or rai:humble
```
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may be worth mentioning that if this error is reported:

docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

the docker may need to be restarted.

sudo systemctl restart docker

Copy link
Contributor Author

@Cytrus14 Cytrus14 Jul 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 2e54d09


Sometimes, passing GPUs to the docker container may result in an error:

```shell
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].
```

Restarting the docker service should resolve this error:

```shell
sudo systemctl restart docker
```

4. (Inside the container shell) Download additional ROS 2 dependencies:

```shell
vcs import < demos.repos
rosdep install --from-paths src/examples/rai-manipulation-demo/ros2_ws/src --ignore-src -r -y
```

5. (Inside the container shell) Build the ROS 2 workspace:

```shell
source /opt/ros/${ROS_DISTRO}/setup.bash
colcon build --symlink-install
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
colcon build --symlink-install
source /opt/ros/jazzy/setup.bash
colcon build --symlink-install

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added and modified to be ROS 2 distribution agnostic in c7f703d

```

6. (Inside the docker container) By default, RAI uses OpenAI as the vendor. Thus, it is necessary
to set the `$OPENAI_API_KEY` environmental variable. The command below may be utilized to set
the variable and add it to the container's `.bashrc` file:

```shell
export OPENAI_API_KEY=YOUR_OPEN_AI_API_KEY
echo "export OPENAI_API_KEY=$OPENAI_API_KEY" >> ~/.bashrc
```

!!! note AI vendor change

The default vendor can be changed to a different provider via the [RAI configuration tool](../setup/install.md#15-configure-rai)

#### 2. Running the demo

!!! note Source the setup shell

Ensure ROS 2 is sourced on the host machine and the `ROS_DOMAIN_ID` environment variable is set to the same value as in the [Docker setup](../setup/setup_docker.md#2-set-up-communications-between-docker-and-host-optional). Ensure that every command inside the docker container is run in a sourced shell using `source setup_shell.sh`.

1. Launch the Robotic Arm Visualization on the host machine:

```shell
./demo_assets/manipulation/RAIManipulationDemo/RAIManipulationDemo.GameLauncher
```

2. (Inside the container shell) Launch the Robotic Arm Demo script inside of the docker container:

```shell
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A step needs to be added to adjust config.toml which will make it more friendly for newcomers not having to figure it out by themselves. For example, if an OpenAI models are to be used, the user will need to export OpenAI key. In addition, user may need to set up langfuse or langsmith. Even though both set as false, it seems that langsmith was initialized under the hood and the script failed with following error in my setup

component_container-6] [INFO] [1753127065.678174334] [container0]: Instantiate class: rclcpp_components::NodeFactoryTemplate<depth_image_proc::PointCloudXyzrgbNode>
[INFO] [python-2]: sending signal 'SIGINT' to process[python-2]
[INFO] [static_transform_publisher-3]: sending signal 'SIGINT' to process[static_transform_publisher-3]
[INFO] [robot_state_publisher-4]: sending signal 'SIGINT' to process[robot_state_publisher-4]
[INFO] [move_group-5]: sending signal 'SIGINT' to process[move_group-5]
[INFO] [component_container-6]: sending signal 'SIGINT' to process[component_container-6]
[python-2] Traceback (most recent call last):
[python-2]   File "/rai/src/rai_extensions/rai_open_set_vision/scripts/run_vision_agents.py", line 17, in <module>
[python-2]     from rai.agents import wait_for_shutdown
[python-2]   File "/rai/src/rai_core/rai/__init__.py", line 15, in <module>
[python-2]     from .agents import AgentRunner, ReActAgent
[python-2]   File "/rai/src/rai_core/rai/agents/__init__.py", line 16, in <module>
[python-2]     from rai.agents.langchain import BaseStateBasedAgent, ReActAgent
[python-2]   File "/rai/src/rai_core/rai/agents/langchain/__init__.py", line 15, in <module>
[python-2]     from .agent import BaseState, LangChainAgent, newMessageBehaviorType
[python-2]   File "/rai/src/rai_core/rai/agents/langchain/agent.py", line 26, in <module>
[python-2]     from rai.agents.langchain.callback import HRICallbackHandler
[python-2]   File "/rai/src/rai_core/rai/agents/langchain/callback.py", line 20, in <module>
[python-2]     from langchain_core.callbacks import BaseCallbackHandler
[python-2]   File "/root/.cache/pypoetry/virtualenvs/rai-framework-LG7o02pr-py3.12/lib/python3.12/site-packages/langchain_core/callbacks/__init__.py", line 23, in <module>
[python-2]     from langchain_core.callbacks.manager import (
[python-2]   File "/root/.cache/pypoetry/virtualenvs/rai-framework-LG7o02pr-py3.12/lib/python3.12/site-packages/langchain_core/callbacks/manager.py", line 22, in <module>
[python-2]     from langsmith.run_helpers import get_tracing_context
[python-2]   File "/root/.cache/pypoetry/virtualenvs/rai-framework-LG7o02pr-py3.12/lib/python3.12/site-packages/langsmith/run_helpers.py", line 45, in <module>
[python-2]     from langsmith import client as ls_client
[python-2]   File "/root/.cache/pypoetry/virtualenvs/rai-framework-LG7o02pr-py3.12/lib/python3.12/site-packages/langsmith/client.py", line 70, in <module>
[static_transform_publisher-3] [INFO] [1753127065.690098357] [rclcpp]: signal_handler(signum=2)
[python-2]     from langsmith import env as ls_env
[python-2]   File "/root/.cache/pypoetry/virtualenvs/rai-framework-LG7o02pr-py3.12/lib/python3.12/site-packages/langsmith/env/__init__.py", line 3, in <module>
[python-2]     from langsmith.env._runtime_env import (
[python-2]   File "/root/.cache/pypoetry/virtualenvs/rai-framework-LG7o02pr-py3.12/lib/python3.12/site-packages/langsmith/env/_runtime_env.py", line 10, in <module>
[python-2]     from langsmith.utils import get_docker_compose_command
[python-2]   File "/root/.cache/pypoetry/virtualenvs/rai-framework-LG7o02pr-py3.12/lib/python3.12/site-packages/langsmith/utils.py", line 43, in <module>
[python-2]     from langsmith import schemas as ls_schemas
[python-2]   File "/root/.cache/pypoetry/virtualenvs/rai-framework-LG7o02pr-py3.12/lib/python3.12/site-packages/langsmith/schemas.py", line 716, in <module>
[python-2]     class TracerSessionResult(TracerSession):
[python-2]   File "/root/.cache/pypoetry/virtualenvs/rai-framework-LG7o02pr-py3.12/lib/python3.12/site-packages/pydantic/v1/main.py", line 178, in __new__
[python-2]     annotations = resolve_annotations(namespace.get('__annotations__', {}), namespace.get('__module__', None))
[python-2]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[python-2]   File "/root/.cache/pypoetry/virtualenvs/rai-framework-LG7o02pr-py3.12/lib/python3.12/site-packages/pydantic/v1/typing.py", line 398, in resolve_annotations
[python-2]     value = ForwardRef(value, is_argument=False, is_class=True)
[python-2]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[python-2]   File "/usr/lib/python3.12/typing.py", line 897, in __init__
[robot_state_publisher-4] [INFO] [1753127065.690390213] [rclcpp]: signal_handler(signum=2)
[python-2]     code = compile(arg_to_compile, '<string>', 'eval')
[python-2]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[python-2] KeyboardInterrupt
[move_group-5] [INFO] [1753127065.690654607] [rclcpp]: signal_handler(signum=2)
[component_container-6] [INFO] [1753127065.690880424] [rclcpp]: signal_handler(signum=2)
[move_group-5] terminate called after throwing an instance of 'std::runtime_error'

Copy link
Contributor Author

@Cytrus14 Cytrus14 Jul 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarified that the vendor may be changed and added a step to set the OpneAI API key in 4c63430

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding the error in the script, I had trouble reproducing it. However, I've noticed that similar exceptions can be thrown if one does not build the ROS 2 workspace with colcon build --symlink-install. Could you check whether the build step was not accidentally omitted?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. I probably had missed a step. Re-ran the demo again and I couldn't repro the issue so we are good here.

ros2 launch examples/manipulation-demo.launch.py
```

3. (Inside the container shell) Open a new terminal for the docker container (e.g., `docker exec -it CONTAINER_ID /bin/bash`) and launch the streamlit interface:

```shell
streamlit run examples/manipulation-demo-streamlit.py
```

Alternatively, run the simpler command-line version:

```shell
python examples/manipulation-demo.py
```

## How it works

The manipulation demo utilizes several components:
Expand Down
Loading