v0 Spade CIDs - Retrieval Bot Integration

[jcace]

Add an integration for the Retrieval Bot to receive CIDs from Spade and add them to the task queue.

Initially, this will pull from the Spade deal broker but eventually may go through the Tenant Services.

[jcace]

Constantly updating source of CIDs https://cargo.dag.haus/active_replicas.json.zst

[jcace]

Notes from chat with riba

• JSON is regenerated every hour.
• Piece CID, corresponding Payload CID
  • Cargo always puts a complete dag within a sector
  • https://github.com/filecoin-project/go-dagaggregator-unixfs#grouping-unixfs-structure
  • Everything is organized in a 3 level tree in the DAG
• Just asking for the top CID is kinda pointless, doesnt contain any data. Need to traverse down 3 levels
  • Pull the first CID, open it up, choose a random sub CID
  • repeat 3 times
  • Then, you will have a client CID
• Keep track to make sure we test every SP. Bigger SPs might have tons of content and we won't hit all of them if we never bias for the smaller ones
  • make sure all SPs are tested, even if 99% of CIDs are concentrated in one top SP for instance

[jcace]

WIP at #28

[jcace]

#28 ready for review - this is the Integration to get the Spade CIDs into the task queue.

Additionally, it will need another task to modify the retrieval workers to handle the Spade - type retrieval test.