Tethys worker installation

Worker installation instructions

Requirements

To run a single worker you need:

• 4 vCPU
• 16GB RAM
• 1TB SSD
• Stable 24/7 Internet connection, at least 1Gbit
• public IP and two ports open for incoming traffic:
  • one UDP port for p2p communication (default 12345),
  • one TCP port for Prometheus metrics (default 9090).
• 100_000 tSQD tokens (funded to a special vesting contract owned by the Primary Address).
• Some Arbitrum Sepolia ETH (for gas)

The same Primary Address can be used to register multiple workers (if the vesting contract has sufficient funds).

Worker itself is available in two forms:

1. as a Docker image,
2. as source code.

Please refer to the appropriate section for further instructions.

Running a worker Docker image

Prerequisites

• Docker engine (tested with version 25.0.4)
• Docker compose (tested with version 2.24.7)

Steps

1. Create a new directory and save this run_worker.sh script in it.

2. Generate your key file by running the command below in the newly created directory. It will display your peer ID — please copy this ID, as it will be needed for on-chain worker registration.

   $ docker run --rm subsquid/rpc-node:0.2.5 keygen >key
   Your peer ID: <THIS IS WHAT YOU NEED TO COPY>
   

   ⚠️ Note: Please make sure that the generated file is safe and secure (i.e., it will not be deleted accidentally and cannot be accessed by unauthorized parties).

3. Register your worker by logging into the testnet Web APP with the Primary Wallet you provided in the onboarding form. You will need the Peer ID from the previous step. See the detailed instructions here.

4. Make the run_worker.sh script executable and run ./run_worker.sh <DATA_DIR> up -d to start the node. <DATA_DIR> is a directory where the downloaded data will be stored — please make sure it exists and can accommodate up to 1 TB of data. If you want to use a different port than the default 12345, set the LISTEN_PORT environment variable.

5. Check the status of the containers with docker container ls. Check the worker logs using docker logs -f worker-1. After some time the worker should output some info on the downloaded data chunks. See Docker docs for a reference of Docker commands.

Building from the source

1. Install prerequisites (Rust, Git, Protobuf, etc).

   apt install curl git build-essential pkg-config protobuf-compiler libssl-dev libsqlite3-dev
   curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

2. Clone the worker repo.

   git clone --depth 1 --branch v1.0.0-rc3 https://github.com/subsquid/worker-rs.git
   

3. Build the worker.

   cd worker-rs
   mv .env.testnet .env
   cargo build --release
   

4. (optional) Edit parameters in the .env file. The libp2p address and port, specified in the P2P_LISTEN_ADDRS var, should be available for external connections (in the sample config the worker listens on 12345).

5. Run the worker using the command below. It uses nohup, but you may also consider daemonizing it with systemd to tolerate server restarts. A key file will be created at the provided path if you don't have one.

   nohup cargo run --release -- \
       --data-dir <path where you want to store downloaded data> \
       p2p \
       --key <file path where your key is stored>

   It will display your local peer ID in the logs — please copy this ID, as it will be needed for on-chain worker registration.

   You can also add the --port <port> to specify the Prometheus metrics port instead of the default 8000.

Important

Be careful when supplying the path to the existing key. If you supply the wrong path, a new random key will be automatically created there and your node will attempt to operate with a new (unregistered) peer ID — unsuccessfully.

5. Register your worker by logging into the testnet Web APP with the Primary Wallet you provided in the onboarding form. You will need the Peer ID from the previous step. See the detailed instructions here.

6. When the worker gets recognized by the scheduler, it should start downloading data and serving queries.

FAQ

Where do I find PEER_ID?

It is printed when you run keygen (see the steps for running worker using Docker).

It is also in the first line of the worker log output. For the docker setup, list the containers with docker container ls and inspect the worker container logs with docker compose logs | head

The log line you are looking for should look like this

INFO worker_rust::transport::p2p: Local peer ID: <THIS IS WHAT YOU NEED TO COPY>

How do I get tSQD tokens?

See the Tethys annoucement page for details.

I see Failed to download chunk ... operation timed out in the worker logs

Depending on your connection quality, you might want to tune the S3_TIMEOUT and CONCURRENT_DOWNLOADS environment variables. In case you're using our Docker image, edit the run_worker.sh script and add lines setting these vars to the services.worker.environment section. If you encounter this error frequently, try to set S3_TIMEOUT to 180. If it still doesn't help, set CONCURRENT_DOWNLOADS to 1 and S3_READ_TIMEOUT to 30.

Worker crashes with the message Trying to collect logs up to ...

Something is wrong with your local data. Stop the worker, remove the entire data directory and start it again.

I see Insufficient peers... error in the worker logs

Just ignore it

Can I move my worker to another server?

Yes, copy the key file (located in the key file located at KEY_PATH) to the new working directory before starting. You don't need to re-register your worker.

Should I build it from source or run with Docker?

Docker makes for an easier setup. Building from sources is suitable only for experienced Linux/Unix users.

How do I check that my worker is updated to the latest version?

Check the pings endpoint and locate the version by your PeerID.

Which Linux distro is recommended?

We recommend Ubuntu 22.04 LTS

I see error from daemon in stream: Error grabbing logs

This is a docker issue, not a problem with the worker. Look at this GitHub issue and this StackOverflow thread for more context.

How do I check if my worker is up-to-date and running?

Copy your peer ID and look for an entry on this page. If the last ping timestamp is <1 minute ago, and the listed version is the most recent one, you should be good. Alternatively, you can use this Python script.