2 Robot Implementation

Creating a New Robot

The workflow is identical for all robots; only the enhancement generation step differs between implementations. By separating the common workflow from the robot-specific enhancement logic, new robots can be implemented with minimal effort while reusing the existing infrastructure.

To achieve this, two base classes have been implemented:

• BaseRobot executes the complete workflow within a single process. It is intended for robots whose enhancement generation can be performed on the same machine that communicates with the DESTINY repository.
• ClusterRobot separates the workflow into two stages to support execution on high-performance computing (HPC) clusters or other distributed computing environments. In this case, downloading references and uploading enhancements are performed on a machine with access to the DESTINY repository, while the computationally intensive enhancement generation can be executed independently on a compute cluster.

The following sections describe both base classes and explain their implementation in more detail.

BaseRobot

The BaseRobot provides the common infrastructure shared by robots executed on one machine. It consists of the main robot implementation, two execution runners, and several utility modules.

The robot.py module implements the common robot workflow, including downloading references, uploading enhancements, and interacting with the DESTINY repository. The enhancement generation itself is intentionally left abstract. New robots implement this functionality by overriding the generate_enhancement() method.

Two execution modes are provided through separate runner classes:

• PollingRunner continuously polls the DESTINY repository for new enhancement batches. If no work is available, it waits for a configurable interval before polling again.
• BatchRunner processes all currently available enhancement batches and terminates once no further work is available.

Several utility modules support the common functionality of all robots:

• config.py defines the robot configuration using Pydantic settings. It loads configuration values from the environment (typically via a .env file) and provides a central location for robot-specific settings.
• client.py creates and configures the client used to communicate with the DESTINY repository, including authentication and HTTP settings.
• auth.py implements the authentication mechanisms required to access protected repository endpoints.
• logging.py configures the logging infrastructure used throughout the robot framework.
• version.py provides utilities for reading and exposing the robot version from the project metadata.

Implementation

Implementing a new robot requires very little robot-specific code:

• robot.py implements the enhancement logic by subclassing BaseRobot and providing an implementation of generate_enhancement().
• main.py initializes the robot, loads the configuration, creates the required clients, and serves as the application's entry point.
• config.py defines the robot-specific configuration. It can include additional settings such as model locations, classifier parameters, or custom runtime options.
• pyproject.toml defines the project metadata and robot version, which is reported to the DESTINY repository during execution.

Minimal example implementations are provided in the repository. The query_robot demonstrates the smallest possible robot implementation, while the dummy_classifier_robot shows how a machine learning model can be integrated into the framework.

Assuming that a local DESTINY repository is running, the robot folder contains a .env file and enhancement batches are available, the query robot can be started with

uv run python -m query_robot 

The dummy_classifier_robot needs a model to be trained, instructions can be found in models. Once this is done the robot is started from the root with

uv run python -m dummy_classification_robot 

ClusterRobot

The ClusterRobot extends the robot framework to support computationally intensive enhancement generation on high-performance computing (HPC) clusters.

In the preparation stage, the robot downloads enhancement batches from the DESTINY repository, prepares the input files required for processing, and submits a Slurm job to the compute cluster. The actual enhancement generation is performed entirely by the cluster job.

In the wait stage, the robot waits for the submitted Slurm job to finish. It continuously queries the Slurm API to check the status of the submitted job and proceeds to the upload stage once the job has completed.

In the upload stage, the robot collects the output produced by the cluster job, converts it into DESTINY enhancements, and uploads the results back to the DESTINY repository.

The ClusterRobot consists of the following components:

• robot.py implements the three main workflows:

  • prepare_all(), which downloads enhancement batches, prepares the input data, and submits the cluster job.
  • wait_all(), which waits for all submitted Slurm jobs to finish. It continuously polls the Slurm API for the status of each submitted job and proceeds once the jobs have completed successfully.
  • upload_all_finished(), which processes completed cluster jobs and uploads the generated enhancements.

  Similar to the BaseRobot, the robot-specific functionality is left abstract. New robots must implement two methods:

  • write_input(), which converts downloaded references into the input format required by the cluster job.
  • parse_output(), which reads the output generated by the cluster job and converts it into DESTINY enhancements.

• main.py provides the command-line entry point for the robot. It exposes three commands, prepare, wait and upload, which execute the corresponding stages. Together, these commands provide functionality similar to the BatchRunner of the BaseRobot.

• batch.py manages the communication between the preparation and upload stages. For every downloaded enhancement batch, it creates a dedicated working directory containing:

  • the generated input files,
  • the output produced by the cluster job,
  • Slurm log files, and
  • a metadata.json file containing information about the batch and its processing status.

• slurm.py provides a wrapper for submitting and monitoring Slurm jobs. If Slurm is not available, it can also execute the processing script locally, simplifying development and testing.

• config.py defines the robot configuration. In addition to the common settings, it specifies the workspace used to store batch directories (by default, the workspace directory inside the robot project). It also defines the batch lease, which specifies how long the repository should reserve a claimed enhancement batch before it must be renewed. The default lease duration is 1 day and 12 hours.

• client.py creates and configures the client used to communicate with the DESTINY repository, including authentication and HTTP settings.

• version.py provides utilities for reading and exposing the robot version from the project metadata.

Implementation

To implement a new cluster robot, only a small number of robot-specific files are required:

• robot.py, implementing the write_input() and parse_output() methods.
• slurm.sh, together with any additional scripts required to execute the enhancement generation on the compute cluster.
• main.py, providing the command-line entry point.

A tiny implementation of a cluster robot can be found in the dummy_robot.

Assuming that a local DESTINY repository is running, the robot folder contains a .env file and enhancement batches are available, the three parts of the dummy robot can be started from the root with

uv run python -m cluster_robot.dummy_robot.main prepare
uv run python -m cluster_robot.dummy_robot.main wait
uv run python -m cluster_robot.dummy_robot.main upload