This is a Rust implementation of the Worker. The previous (Python) version can be found here. A worker is a service that downloads assigned data chunks from persistent storage (currently S3) and processes incoming data queries that reference those data chunks. It can be used in two modes:
- Centralized. In this setup, the assignment is received from a centralized router that is aware of each worker and relies on it (almost) always being available. In this case, the communication between the worker and the router happens directly through HTTP requests.
- Decentralized. In this setup, the assignment comes from a scheduler via a P2P communication protocol. The chunks can be reassigned as the workers join and leave the network.
The full network architecture can be found here.
You can find instructions for how to run your own worker here.