Retrieval Bot - Run Spade Integrations for Spade RB

[xmcai2016]

No description provided.

[jcace]

The code to do this is already merged in here data-preservation-programs/RetrievalBot#28

However, the retrieval worker code is not ready yet. Pending this: data-preservation-programs/RetrievalBot#33 to get merged first.

Once that one is merged, we can start running this integration. To do this, we need to add a step to the pipeline to run the Spadev0. I started a branch for this here: data-preservation-programs/RetrievalBot#34

We need to set up a cron task to run it (once per day should be plenty) - not sure how to do this exactly but @xinaxu should know. The code needs to do this:

 ./spadev0_integration spadev0 --sources https://cargo.dag.haus/active_replicas.json.zst

[xinaxu]

Our AWS pipeline should take care of creating the docker image for this new integration
To create a cron task, go to AWS EventBridge and create a schedule

[jcace]

Update: Need to integrate this with the new Spade retrieval bot

[jcace]

Still pending the new Bitswap worker to get merged into Spade RB repo.

[jcace]

For this task, make a new branch on the Spade Retrieval Bot repo. Copy what I've started on this branch: data-preservation-programs/RetrievalBot#34 , it has the .env file templated out and some of the build commands too