Skip to content

5 Robot Deployment

Jump to bottom
Sarah Lück edited this page Sep 15, 2026 · 2 revisions

Running a Robot

There are several ways to execute a robot. This section describes the execution methods that have been tested during development.

Regardless of the execution method, the robot must be configured to communicate with a DESTINY repository, either a locally running instance or the production repository. The repository connection is configured via the robot's .env file. Instructions for configuring a robot to connect to a local DESTINY repository are provided in this Section, while the configuration for connecting to the production repository is described in here.

In addition, enhancement batches must be available for the robot to process. When working with a local DESTINY repository, enhancement requests can be created as described here.

Running a Robot locally

The simplest way to execute a robot is directly from a local Python virtual environment.

For example, to start the Relevance Robot:

 uv run python -m relevance_robot

If you implemented your robot with the polling runner you need to shut it down with Ctrl+C. The batch runner will exit after all available batches have been processed.

Running a Robot via a GitLab Schedule on a local GitLab Runner

A robot can also be executed automatically via a GitLab Pipeline Schedule while still running on your local machine. This is especially suitable for robots using the BatchRunner.

Set-up a local GitLab Runner

First, install the GitLabRunner on your local machine. Give the runner a suitable tag, e.g. local-runner, choose the appropriate operating system, and follow the instructions for registering the runner.

When prompted:

  1. Run the registration command as **sudo**. This registers the runner in system mode. The runner service subsequently executes jobs as the gitlab-runner user.

  2. When prompted Enter an executor:, select:

    shell

The Shell executor runs the CI jobs directly on the local machine. This allows the workflow to access resources available on that machine, such as locally stored machine-learning models and, for local testing, a locally running DESTINY repository.

After registration, enable and start the GitLab Runner service:

sudo systemctl enable gitlab-runner 
sudo systemctl start gitlab-runner

Check that the service is running:

sudo systemctl status gitlab-runner

The runner registration can be verified with:

sudo gitlab-runner verify

A successful verification should report:

Running in system-mode. Verifying runner... is valid

The runner should also appear as online in the GitLab project's runner settings.

Note: Because the runner executes jobs as the gitlab-runner user, software and files required by the CI pipeline must be accessible to this user.

Writing the GitLab CI Configuration

The GitLab Runner is configured via a .gitlab-ci.yml file located in the root directory of the robot project. This file specifies how the robot should be executed whenever a GitLab pipeline is triggered.

A Shell runner creates a fresh checkout of the repository in a dedicated build directory before executing the pipeline. Consequently, the pipeline cannot rely on the Python virtual environment or locally generated files (e.g. trained machine learning models) contained in the original working directory. Instead, these resources have to be recreated or copied into the build directory as part of the pipeline.

The different parts of the example are explained below.

Pipeline and job configuration

stages:
  - run

run-robot-workflow:
  stage: run
  
  tags:
    - local-runner

  resource_group: robot-workflow

This means that the pipeline has one stage called run, and the job run-robot-workflow belongs to that stage. The tags field specifies which GitLab Runner should execute the job. The resource_group ensures that only one instance of the Relevance Robot is executed at any given time. If a new scheduled pipeline is triggered while a previous execution is still running, the new job is queued until the previous one has completed.

Setting up the Python environments

Before the robot is executed, the required Python environment is created:

before_script:
  - export PATH="$HOME/.local/bin:$PATH"
  - uv sync
  - uv run python --version

  - cd query_robot
  - uv sync
  - cd ../cluster_relevance_robot
  - uv sync
  - cd ../cluster_label_robot
  - uv sync
  - cd ..

The first uv sync sets up the environment for the outer project, which contains shared functionality such as base_robot. The subsequent commands create the environments for the individual robots.

The uv.lock files are committed to the repository and ensure that the same dependency versions are installed when the pipeline runs on another machine.

Creating the robot environment files

The robot credentials are not stored in the repository. Instead, they are configured as GitLab CI/CD variables and used to create the .env files required by the individual robots.

For example:

- |
  cat > query_robot/.env << EOF
  ROBOT_ID=${QUERY_ROBOT_ID}
  ROBOT_SECRET=${QUERY_ROBOT_SECRET}
  DESTINY_REPOSITORY_URL=${DESTINY_REPOSITORY_URL}
  ENV=local
  EOF

The corresponding .env files are created for the cluster-based robots.

Providing machine-learning models

The trained machine-learning models are not stored in the Git repository. They are stored separately on the machine running the GitLab Runner.

The location of the model directory is specified using MODEL_SOURCE:

variables:
  MODEL_SOURCE: "/path/to/destiny-robot-models"

The pipeline copies the required models into the GitLab checkout:

- mkdir -p cluster_relevance_robot/models
- rsync -a --exclude='.cache/' "${MODEL_SOURCE}/cluster_relevance_robot/models/" cluster_relevance_robot/models/

- mkdir -p cluster_label_robot/models
- rsync -a --exclude='.cache/' "${MODEL_SOURCE}/cluster_label_robot/models/" cluster_label_robot/models/

MODEL_SOURCE is therefore machine-specific and must be adjusted when the pipeline is set up on another machine.

The gitlab-runner user must have read access to the model directory. For example, on a Linux machine, read and directory-traversal permissions can be granted using ACLs:

setfacl -R -m u:gitlab-runner:rX /path/to/destiny-robot-models

The permission should be granted only to the directory containing the model artefacts rather than to the user's entire home directory.

rsync is used instead of a simple cp -r because it allows directories to be copied recursively while selectively excluding files or directories that are not required. In particular, the Hugging Face .cache directories do not need to be copied when the locally downloaded model files themselves are available. This also avoids copying unnecessary cache data into the temporary GitLab checkout.

Executing the robots

The script section specifies the command that starts the robot. The rules section restricts execution of the job to pipelines triggered by a GitLab Pipeline Schedule.

Set-up the scheduled pipeline on GitLab

Once the GitLab Runner and the pipeline configuration have been set up, the robot can be executed automatically using a GitLab Pipeline Schedule.

In GitLab, navigate to:

Build → Pipelines → Pipeline Schedules

Choose a description, a time zone and a suitable cron interval (e.g. */10 * * * * to execute the robot every 10 minutes).

Nest, add the configuration variables required by the robot under Variables. These correspond to the values normally stored in the robot's .env file (e.g. ROBOT_ID, ROBOT_SECRET, and DESTINY_REPOSITORY_URL).

Note: Variable values are stored literally by GitLab. Do not enclose values in quotation marks. For example, UUIDs such as ROBOT_ID should be entered as

cb73f9a3-af2a-4bfa-9502-f61d5fe18b11

Finally, click Create pipeline schedule.

The scheduled pipeline is now configured and will execute according to the specified cron interval. During development, the schedule can also be started manually by clicking the Run (▶) button next to the pipeline schedule.

Clone this wiki locally