Skip to content

miner.md

Latest commit

 

History

History
462 lines (334 loc) · 12.5 KB

miner.md

File metadata and controls

462 lines (334 loc) · 12.5 KB
Raw

Miner Setup

This document provides a comprehensive guide on how to set up and run a miner using miner.py. Miners are crucial components of τemplar, responsible for training the model on assigned data subsets and sharing their gradients with peers.

Table of Contents

Introduction

This guide will help you set up and run a miner for τemplar. We'll cover both the recommended Docker Compose method and manual installation for environments where Docker is not preferred.

Prerequisites

  • NVIDIA GPU with CUDA support
    • Minimum H100 recommended
  • Ubuntu (or Ubuntu-based Linux distribution)
  • Docker and Docker Compose
  • Git
  • Cloudflare R2 Bucket Configuration:
    • Bucket Setup:
      1. Create a Bucket: Name it the same as your account ID and set the region to ENAM.
      2. Generate Tokens:
        • Read Token: Admin Read permissions.
        • Write Token: Admin Read & Write permissions.
      3. Store Credentials: You'll need these for the .env file.

Installation

Using Docker Compose (Recommended)

  1. Install Docker and Docker Compose:

    # Update package list
sudo apt-get update

# Install prerequisites
sudo apt-get install \
  ca-certificates \
  curl \
  gnupg \
  lsb-release

# Add Docker’s official GPG key
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

# Set up the repository
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker Engine
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin

# Install Docker Compose
sudo apt-get install docker-compose

  2. Clone the Repository:

    git clone https://github.com/tplr-ai/templar.git
cd templar

  3. Navigate to the Docker Directory:

    cd docker

  4. Create and Populate the .env File:

    Create a .env file in the docker directory by copying the .env.example:

    cp .env.example .env

    Populate the .env file with your configuration. The variables that need to be set are:

    # Add your Weights & Biases API key
WANDB_API_KEY=<your_wandb_api_key>
INFLUXDB_TOKEN=your_influxdb_token


# Cloudflare R2 Credentials - Add your R2 credentials below
R2_GRADIENTS_ACCOUNT_ID=<your_r2_account_id>
R2_GRADIENTS_BUCKET_NAME=<your_r2_bucket_name>

R2_GRADIENTS_READ_ACCESS_KEY_ID=<your_r2_read_access_key_id>
R2_GRADIENTS_READ_SECRET_ACCESS_KEY=<your_r2_read_secret_access_key>

R2_GRADIENTS_WRITE_ACCESS_KEY_ID=<your_r2_write_access_key_id>
R2_GRADIENTS_WRITE_SECRET_ACCESS_KEY=<your_r2_write_secret_access_key>

R2_DATASET_ACCOUNT_ID=dd08f378791881bf6bbb7f161c78a220
R2_DATASET_BUCKET_NAME=edu-dataset
R2_DATASET_READ_ACCESS_KEY_ID=7cadbe19f880785e46898b558ef70ce8
R2_DATASET_READ_SECRET_ACCESS_KEY=9787434d676b05dce69cc4e76c6af74d795b606feafd031944444780d5f72272

R2_AGGREGATOR_ACCOUNT_ID="80f15715bb0b882c9e967c13e677ed7d"
R2_AGGREGATOR_BUCKET_NAME="aggregator"
R2_AGGREGATOR_READ_ACCESS_KEY_ID="aa7ea943895323963edba7323e4e12b9"
R2_AGGREGATOR_READ_SECRET_ACCESS_KEY="8557ba9fc49a5482fff35d591c808bbdf0ec306bf08d9524c2055736aca915d1"

# Wallet Configuration
WALLET_NAME=<your_wallet_name>
WALLET_HOTKEY=<your_wallet_hotkey>

# Network Configuration
NETWORK=finney
NETUID=3
# GPU Configuration
CUDA_DEVICE=cuda:0
# Node Type
NODE_TYPE=validator
# Additional Settings
DEBUG=false

    Replace the placeholders with your actual values.

  5. Update docker-compose.yml:

    Ensure that the docker-compose.yml file is correctly configured for your setup (usually no changes are needed).

  6. Run Docker Compose:

    Start the miner using Docker Compose:

    docker compose -f docker/compose.yml up -d

    This will start the miner in detached mode.

Manual Installation

If you prefer to run the miner without Docker, follow the instructions in the Running Without Docker section.

Running the Miner

Using Docker Compose

Assuming you've completed the installation steps above, your miner should now be running. You can verify this by listing running containers:

docker ps

You should see a container named templar-miner-<WALLET_HOTKEY>.

Running Without Docker

  1. Install System Dependencies:

    # Add Python 3.11 repository
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update

# Install required packages
sudo apt-get install python3.11 python3.11-venv git

*PM2 Support Installation

 # Install required packages
 apt update && apt upgrade -y && apt-get install -y nano git python3-pip jq npm && npm install pm2 -g && pm2 update

  1. Install NVIDIA CUDA Drivers:

    Install the appropriate NVIDIA CUDA drivers for your GPU.

  2. Clone the Repository:

    git clone https://github.com/tplr-ai/templar.git
cd templar

  3. Set Up Python Environment:

    # Create virtual environment
python3.11 -m venv .venv
source .venv/bin/activate

# Upgrade pip
pip install --upgrade pip

# Install PyTorch with CUDA support
pip install torch --index-url https://download.pytorch.org/whl/cu118


# Install uv tool (if needed)
pip install uv

*PM2 Support Installation

 # Install uv and configure venv
 pip install uv && uv python install 3.11 && uv python pin 3.11 && uv venv .venv
 source .venv/bin/activate

 # Install PyTorch with CUDA support
 uv pip install torch --index-url https://download.pytorch.org/whl/cu118\

 # uv sync to install required packages
 uv sync --extra all

  1. Create and Register Wallets:

    # Create coldkey
btcli wallet new_coldkey --wallet.name default --n-words 12

# Create and register hotkey
btcli wallet new_hotkey --wallet.name default --wallet.hotkey miner --n-words 12
btcli subnet pow_register --wallet.name default --wallet.hotkey miner --netuid <netuid> --subtensor.network <network>

  2. Log into Weights & Biases (WandB):

    wandb login your_wandb_api_key

  3. Set Environment Variables:

    Export necessary environment variables or create a .env file in the project root.

    export WANDB_API_KEY=your_wandb_api_key
export INFLUXDB_TOKEN=your_influxdb_token
export NODE_TYPE=your_node_type
export WALLET_NAME=your_wallet_name
export WALLET_HOTKEY=your_wallet_hotkey
export CUDA_DEVICE=your_cuda_device
export NETWORK=your_network
export NETUID=your_netuid
export DEBUG=your_debug_setting

# Gradients R2 credentials
export R2_GRADIENTS_ACCOUNT_ID=your_r2_account_id
export R2_GRADIENTS_BUCKET_NAME=your_r2_bucket_name
export R2_GRADIENTS_READ_ACCESS_KEY_ID=your_r2_read_access_key_id 
export R2_GRADIENTS_READ_SECRET_ACCESS_KEY=your_r2_read_secret_access_key
export R2_GRADIENTS_WRITE_ACCESS_KEY_ID=your_r2_write_access_key_id
export R2_GRADIENTS_WRITE_SECRET_ACCESS_KEY=your_r2_write_secret_access_key

# Dataset R2 credentials
export R2_DATASET_ACCOUNT_ID=dd08f378791881bf6bbb7f161c78a220
export R2_DATASET_BUCKET_NAME=edu-dataset
export R2_DATASET_READ_ACCESS_KEY_ID=7cadbe19f880785e46898b558ef70ce8
export R2_DATASET_READ_SECRET_ACCESS_KEY=9787434d676b05dce69cc4e76c6af74d795b606feafd031944444780d5f72272

# Aggregator R2 credentials
export R2_AGGREGATOR_ACCOUNT_ID="80f15715bb0b882c9e967c13e677ed7d"
export R2_AGGREGATOR_BUCKET_NAME="aggregator"
export R2_AGGREGATOR_READ_ACCESS_KEY_ID="aa7ea943895323963edba7323e4e12b9"
export R2_AGGREGATOR_READ_SECRET_ACCESS_KEY="8557ba9fc49a5482fff35d591c808bbdf0ec306bf08d9524c2055736aca915d1"

export GITHUB_USER=your_github_username

  4. Run the Miner:

    python neurons/miner.py \
  --actual_batch_size 6 \
  --wallet.name default \
  --wallet.hotkey miner \
  --device cuda \
  --use_wandb \
  --netuid <netuid> \
  --subtensor.network <network> \
  --sync_state

*PM2 Support Installation

 pm2 start neurons/miner.py --interpreter python3 --name sn3miner -- \
 --actual_batch_size 6 \
 --wallet.name default \
 --wallet.hotkey miner \
 --device cuda \
 --subtensor.network <network> \
 --sync_state \
 --netuid <netuid>

Configuration

Environment Variables

When using Docker Compose, set the following variables in the docker/.env file:

# Add your Weights & Biases API key
WANDB_API_KEY=your_wandb_api_key
INFLUXDB_TOKEN=your_influxdb_token

# Cloudflare R2 Credentials
R2_ACCOUNT_ID=your_r2_account_id

R2_READ_ACCESS_KEY_ID=your_r2_read_access_key_id
R2_READ_SECRET_ACCESS_KEY=your_r2_read_secret_access_key

R2_WRITE_ACCESS_KEY_ID=your_r2_write_access_key_id
R2_WRITE_SECRET_ACCESS_KEY=your_r2_write_secret_access_key

# Wallet Configuration
WALLET_NAME=default
WALLET_HOTKEY=your_miner_hotkey_name

# Network Configuration
NETWORK=finney
NETUID=3

# GPU Configuration
CUDA_DEVICE=cuda:0

# Additional Settings
DEBUG=false

Note: The R2 permissions remain unchanged from previous configurations.

Hardware Requirements

  • GPU Requirements:
    • Minimum: NVIDIA H100 with 80GB VRAM
  • Storage: 100GB+ recommended for model and data
  • Network: Stable internet connection with good bandwidth

Network Options

  • Mainnet (Finney):
    • Network: finney
    • Netuid: 3
  • Testnet:
    • Network: test
    • Netuid: 223
  • Local:
    • Network: local
    • Netuid: 1

InfluxDB Configuration

Optional InfluxDB configuration variables include:

  • INFLUXDB_TOKEN: Authentication token
  • INFLUXDB_HOST: Custom host address
  • INFLUXDB_PORT: Connection port (default 8086)
  • INFLUXDB_DATABASE: Database name
  • INFLUXDB_ORG: Organization identifier

Example configuration:

INFLUXDB_HOST=custom-influxdb-host.example.com
INFLUXDB_PORT=8086
INFLUXDB_DATABASE=custom-database
INFLUXDB_ORG=custom-org
INFLUXDB_TOKEN=your-influxdb-token

These settings are optional and will fall back to default values if not provided.

Monitoring

Logs

  • Docker Logs:

    docker logs -f templar-miner-${WALLET_HOTKEY}

  • Weights & Biases:

    • Ensure --use_wandb is enabled
    • Monitor training metrics and performance on your WandB dashboard

Performance

Keep an eye on:

  • GPU utilization
  • Memory usage
  • Network bandwidth
  • Training progress
  • Rewards and weights

Troubleshooting

  • CUDA Out of Memory: Reduce --actual_batch_size in your run command.
  • Network Synchronization Issues: Verify your network connection and ensure the correct NETWORK and NETUID are set.
  • Registration Failures: Make sure your wallet is properly registered and funded.

Miner Operations

Model Synchronization

  • The miner synchronizes its model with the latest global state at startup.
  • Attempts to load the latest checkpoint from the validator with the highest stake.

Training Process

  • Data is deterministically assigned based on the miner's UID and the current window.
  • The miner trains on its assigned data and computes gradients.

Gradient Sharing

  • Gradients are compressed and shared with peers via the communication module.
  • The miner gathers gradients from peers, decompresses them, and updates its model.