Skip to content

7 Run a local Destiny Repo

Jump to bottom
Sarah Lück edited this page Sep 15, 2026 · 2 revisions

Setting up a local DESTINY repository

In order to test a self-made robot it is helpful to set-up a locally (e.g. on your laptop) running DESTINY-repository.

The instructions - however a bit complicated - to do so are found in the README.md of the repo. Here, I copy-pasted these instructions with less explanations.

First set-up

This set-up is only done once. After done it once, you can go directly to the next section.

First, you need to download and set-up the repo. Pre-requisite is that you can connect to github via command line, i.e. added your ssh-key into github. Then clone the repo and cd into it:

git clone git@github.com:destiny-evidence/destiny-repository.git
cd destiny-repository

uv is used for dependency management and managing virtual environments. If it is not yet installed you can install it with the uv installer script:

curl -LsSf https://astral.sh/uv/install.sh | sh

Once uv is installed install the dependencies

  uv sync

and copy the example .env file.

cp .env.example .env

Run a local version of the DESTINY-repo

A local version of the DESTINY-repo contains a small database seed and elastic-search exposed by APIs. Especially the ElasticSearch is very memory hungry. This means you might need to clean up space, e.g. close tabs in Firefox and/or close windows in VS Code.

Starting all services

The following commands start all services

 docker compose up -d
 docker compose --profile app up --build

See the README.md for an explanation.

The DESTINY API is exposed at http://127.0.0.1:8000

Initial database setup (run once)

After starting the DESTINY services for the first time, initialize the database schema and import the example data:

uv run alembic upgrade head

PGPASSWORD=localpass psql \
  -U localuser \
  -h 0.0.0.0 \
  -p 5432 \
  -d destiny_dev \
  -f ".db_seed/local.sql"
  • uv run alembic upgrade head applies all database migrations and creates the schema.
  • The psql command imports the seed data from .db_seed/local.sql.

These commands only need to be executed:

  • after creating a fresh database,
  • after deleting the PostgreSQL volume,
  • or when setting up the repository on a new machine.

Create and receive batches

The previous setup allows you to run a robot locally and communicate with the DESTINY repository. However, the robot will not receive any work until enhancement requests have been created.

Enhancement requests assign references to a specific robot and group them into batches. The robot retrieves these batches from the DESTINY repository, processes the references, and uploads the generated enhancements.

The script create_enhancement_requests.py creates enhancement requests for all references currently indexed in Elasticsearch and assigns them to the configured robot. Run it with:

uv run python create_enhancement_requests.py config_local.json

The Docker setup configures MinIO to use the Docker-internal hostname fs. When running the robot directly on the host machine (rather than inside Docker), this hostname cannot be resolved.

To make the hostname fs resolve to the locally running MinIO service, add the following entry to your /etc/hosts file:

echo "127.0.0.1 fs" | sudo tee -a /etc/hosts

This command only needs to be executed once on each machine.

Adding more references to the db-dumb

For some robots, it is useful to test their functionality against realistic reference data in a local DESTINY Repository instance. Test references can be added directly to the local PostgreSQL database together with the enhancements required by the robot, such as abstracts.

After modifying the database directly, the Elasticsearch reference index must be rebuilt so that the newly added references are available through the repository's search API.

We will need a set if uuids (to set as reference_id and enhancement_id). To generate these, run the following in a terminal (Change x in range(x) depending on the number of uuids needed.):

python3 -c "import uuid; [print(uuid.uuid4()) for _ in range(6)]"

This randomly generates uuids like:

From the destiny-repository directory, connect to the PostgreSQL database running in Docker:

docker compose exec db psql -U localuser -d destiny_de

The following SQL commands can then be executed directly in the psql prompt.

First, add the references to the reference table:

INSERT INTO public.reference (visibility, id, created_at, updated_at) VALUES 
('public', 'ff99d28b-a240-4341-8062-f53b93880857', now(), now()), 
('public', '66b788d1-8753-486e-a7c1-629ba23948d7', now(), now());

The references themselves do not contain abstracts. Abstracts are stored as enhancements associated with the corresponding reference.

For example:

INSERT INTO public.enhancement (
    visibility,
    source,
    reference_id,
    enhancement_type,
    robot_version,
    content,
    id,
    created_at,
    updated_at,
    derived_from
)
VALUES (
    'public',
    'manual',
    'ff99d28b-a240-4341-8062-f53b93880857',
    'abstract',
    NULL,
    '{"process": "closed_api", "abstract": "This study proposes short- and medium-term measures to increase water supply in Gurgaon.", "enhancement_type": "abstract"}',
    '37c877bf-2cbb-4f27-8709-6a4c5a9f9c96',
    now(),
    now(),
    NULL
    ),
    (
    'public',
    'manual',
    '66b788d1-8753-486e-a7c1-629ba23948d7',
    'abstract',
    NULL,
    '{"process": "closed_api", "abstract": "We analyse whether disaster-contingency budget mechanisms can improve government capacity to respond to increasing climate related shocks in South Africa, using projected fiscal impacts and policy scenarios.", "enhancement_type": "abstract"}',
    '5f951762-e101-422b-b24f-89af6ab29e49',
    now(),
    now(),
    NULL
    );

To verify that the abstracts were inserted successfully, run:

SELECT reference_id, content->>'abstract' AS abstract FROM public.enhancement WHERE enhancement_type = 'abstract

Exit the psql prompt with:

\q

Because the references and enhancements were inserted directly into PostgreSQL, the Elasticsearch reference index is not automatically updated. Rebuild the index using the reference-index repair endpoint:

curl -s -X POST \ "http://127.0.0.1:8000/v1/system/indices/reference/repair/?rebuild=true"

Query the reference search endpoint to verify that the newly inserted references have been added to the Elasticsearch index:

curl -s \ "http://127.0.0.1:8000/v1/references/search/?q=*:*&page=1" \ | python3 -m json.tool

The newly created reference IDs should appear in the search results.

If there are many references in the local repository, it is more convenient to search directly for one of the inserted UUIDs rather than inspecting the complete first page of results.

Troubleshooting: DESTINY-repo does not start correctly

If the DESTINY-repo was used previously and the repository code has since been updated, the local PostgreSQL database may have an outdated schema. In this case, the Docker containers may appear to be running, but the API can still fail when handling requests.

Check the status of the services with:

docker compose ps

and inspect the application logs with:

docker compose logs app --tail=200

If the logs contain errors such as

UndefinedColumnError: column ... does not exist

the database schema may be behind the current repository version. Check the current and latest database migrations:

docker compose exec app alembic current 
docker compose exec app alembic heads

If current is behind heads, update the database schema with:

docker compose exec app alembic upgrade head

Then restart the application:

docker compose restart app

Test the DESTINY-repo with the toy-robot

To see if the DESTINY-repo is properly set-up you can try if the toy-robot is also working properly and talks to your local repo. Note, we do not want to modify the toy-robot code here, that is why we do not follow all of the instructions in the README.md.

First set-up of the toy-robot

Again, we need to first set-up the robot.

git clone git@github.com:destiny-evidence/toy-robot.git
cd toy-robot
uv sync

and use your favorite editor to create a file named .env in the same directory with the following content

DESTINY_REPOSITORY_URL=http://127.0.0.1:8000
ROBOT_ID="cb73f9a3-af2a-4bfa-9502-f61d5fe18b11"
ROBOT_SECRET="dummy_secret"
ENV=local

This robot ID belongs to robot which is already registered in the local DESTINY repository. Note: To test a home-made robot locally you can copy-paste this .env file to your robot and re-use the toy-robot credentials.

Run the robot locally

Run the robot with

uv run python run_robot.py

If everything, i.e. DESTINY-repo and toy-robot, is set-up properly you should see something like this

2026-07-08 11:15:15,487 - app.main - INFO - Starting Toy Robot polling loop
2026-07-08 11:15:15,487 - app.main - INFO - Polling interval: 30 seconds
2026-07-08 11:15:15,487 - app.main - INFO - Batch size: 2
2026-07-08 11:15:15,487 - app.main - INFO - Starting to poll for robot enhancement batches...
2026-07-08 11:15:15,555 - httpx - INFO - HTTP Request: POST http://127.0.0.1:8000/v1/robot-enhancement-batches/?robot_id=550e8400-e29b-41d4-a716-446655440000&limit=2 "HTTP/1.1 204 No Content"

confirming that the robot speaks to the repo. However, it does not get any batches yet ("HTTP/1.1 204 No Content").

Shut down the robot

A simple Ctrl+C shuts down the robot.

Shut down the locally running DESTINY-repo

In the directory of the destiny-repository run

 docker compose stop

to shut down all dockers.

Clone this wiki locally