Repository navigation
1 Robot Concept
All robots follow the same high-level workflow:
- Poll the DESTINY repository for available enhancement batches.
- Download the references contained in the batch.
- Generate enhancements for each reference.
- Upload the generated enhancements.
- Repeat the process continuously or terminate after a single batch, depending on the execution mode.
Robots communicate with the DESTINY repository through the enhancement API. References are not downloaded individually but in enhancement batches, allowing robots to process groups of references.
Robot Repository Object Storage
| | |
|---- Poll -----------> | |
|<--- Batch + URLs ---- | |
| | |
|---- Download -------------------------------> |
|<--- References ----------------------------- |
| | |
| Generate Enhancements |
| | |
|---- Upload Results ------------------------> |
| | |
|---- Complete -------> | |
|<--- Acknowledged ---- | |Robots periodically contact the DESTINY repository to request new work. If an enhancement batch is available, the repository assigns it to the requesting robot and returns the metadata required for processing.
If no work is available, the robot waits for a configurable amount of time before sending the next polling request.
The polling response contains a reference storage URL (reference_storage_url). Rather than transferring all references through the API, the repository stores them temporarily in object storage.
The robot downloads the complete batch from this location before starting the enhancement generation. Using temporary object storage keeps the API lightweight and allows efficient transfer of large datasets.
After processing all references, the robot uploads the generated enhancements to the result storage URL (result_storage_url) provided by the repository.
Once the upload has completed successfully, the robot notifies the repository that processing has finished. The repository validates the uploaded data and incorporates the generated enhancements into the DESTINY repository.
Currently, robots cannot explicitly reject an assigned enhancement batch.
For some applications, this functionality would be beneficial, e.g if the generation of enhancements is computationally intensive.
At present, once a robot successfully polls for an enhancement batch, it is expected to process and complete that batch. If the robot does not report completion, the batch claim eventually expires and the repository makes the batch available for processing again. After three unsuccessful processing attempts, the batch is marked as failed.
When a robot claims an enhancement batch, the claim is associated with a lease that prevents other robots from processing the same batch concurrently. The repository only makes a claimed batch available again after the lease has expired.
Once an enhancement batch has been claimed, the robot is expected to process the references and upload the generated enhancements within the lease period. The repository only makes a batch available for automatic retry after its lease has expired. Therefore, if a robot or cluster job fails, the batch remains reserved and is not offered to another robot until the lease duration has elapsed.
The DESTINY repository uses a default batch lease of 10 minutes. Since HPC jobs may spend a considerable amount of time waiting in the scheduler queue or executing, the ClusterRobot overrides this default and requests a lease of 1 day and 12 hours (36 hours) when claiming an enhancement batch.
An alternative strategy is to use a shorter lease and periodically renew it (PATCH/v1/robot-enhancement-batches/{robot_enhancement_batch_id}/renew-lease/) while the job is waiting in the scheduler queue or being processed.