Workstream I: The Orchestrator (Flow Miner Service)
The mining service is the primary connector of the orchestrator to Bittensor. The orchestrator can be best understood as a key component of Workstream that handles routing actions from an Economic Value Source to the Workstream environment. It is the “main agent” of the system.
In Flow, the orchestrator utilizes automated capabilities to find tasks, break them into chunks and publish them through the Workstream for the personal operators to take up. We can think of the high level functionality in the following steps:
- Register as a miner to multiple subnets on Bittensor
- Breaks subnet work into small chunks by default.
- Dispatches chunks to the personal operators.
- Accepts results; aggregates and submits to subnet validators.
However, to establish a fair service, we must design the system with decentralized properties. For example, a peer-to-peer broadcast mechanism (like a mempool) can be introduced to ensure effective distribution. But to fully understand the miner service, we have to understand its relationship with Bittensor.
Bittensor, is a decentralized blockchain platform that enables the exchange of digital commodities, such as computational power and machine learning models, within subnets. Each subnet has its own incentivized competition mechanism, where participants are rewarded in the network token, $TAO, for their contributions. For example, subnet 25 focuses on protein folding simulations, while subnet 1 provides a distributed conversational AI platform. Anyone can be a consumer of any particular subnet, but there are generally three types of participants:
Miner
- Expert on the subject-matter of the given subnet, attempts to perform a task to the best of their ability such as computing something or providing a response from an LLM. Miners must pay a registration fee which fluctuates given the demand of registering on the subnet. For every registration, the lowest-performing miner is deregistered and unable to mine until they obtain another registration. This model maintains a competitive network, as miners must consistently perform well to stay registered and continue earning rewards.
Validator
- Grades the quality of the work done by subnet miners to determine their incentives, receiving rewards in return.
Owner
- Registers the subnet on the network, updates the codebase to further optimize their unique economy for all users and add value to the ecosystem. Bittensor’s unique mechanism fosters a decentralized, competitive ecosystem, offering endless opportunities for open source innovation. Read their docs for a more technical approach.
After reviewing the roles of participants, we decided to focus on mining due to its lower technical barriers compared to running a subnet as an owner or being a validator. Below is the technical journey of how we hope to use software and automation to improve our mining operations on Bittensor.
We are starting by searching through Bittensor’s subnets to find those that meets the following criteria:
- Consistently profitable
- Low demand for resources
- Scalable Instead of running a local subtensor node, we will used a block explorer/analytics platform called taostats.io. This platform provides essential data, such as miner performance and registration fees, which allows us to efficiently monitor the performance of different subnets. But before we proceed, let’s look into a case study to kickstart our developments.
The following texts were extracted from a blog written by a subnet miner on Bittensor…
“We initially focused on subnet 27, which uses GPU computing power. By analyzing miner performance and reward distribution, we optimized our approach to deploying miners on cloud platforms like Runpod and Latitude.sh
_Subnet 25’s metagraph (sorted by most trustworthy miners)_
One of the key technical challenges we faced was automating the registration process for miners. Each epoch (approximately 72 minutes), only three registration slots were made available on a first-come, first-served basis. The manual process of checking registration prices and competing for slots was inefficient and unsustainable at scale.
_Registration logic flow chart_
To solve this, we developed a registration monitor and registration cost tracker. This system tracked registration prices and sent real-time alerts via webhooks to a Discord server we managed. By monitoring these alerts on our phones, we could quickly identify opportunities to register miners when prices dropped. The registration monitor was the first step in reducing the time spent manually tracking registration availability.
Next, we developed an automated registration script. Since Bittensor wallets require decryption with a password before performing transactions, we implemented a method to manage wallet passwords programmatically. We used environment variables to securely store the wallet passwords, enabling our scripts to run without requiring manual input:
export BT_COLD_PW_<WalletName>
The registration script is not available on GitHub due to its time-sensitive nature.
As we expanded our operations to include more miners, we needed to ensure that our cloud instances were running efficiently. Each cloud instance (whether GPU or CPU-based) incurred costs, so it was essential to track miner status in real time to avoid wasting resources on deregistered miners.
We developed a deregistration monitor that checked the status of each miner’s wallet on the specified subnet. If a miner was deregistered, the system would send an immediate notification to our Discord server, allowing us to reallocate the resources quickly.
In addition to monitoring miner status, we created a system that monitored our cloud provider balances. Using the APIs provided by Datacrunch and Oblivus, we set up an alert system that notified us when the account balances approached depletion. This ensured continuous miner operation without sudden shutdowns due to insufficient funds.
As demand for GPUs surged due to their role in machine learning and AI workloads, it became increasingly difficult to get the necessary hardware for mining. To address this, we built a GPU sniper tool that automatically checked for availability and reserved GPUs using pre-configured SSH keys as soon as they became free. A successful deployment notified users through Discord webhook. This minimized downtime and allowed us to keep our operations running smoothly, despite the limited GPU availability.
Over time, we began fine-tuning the registration script to improve performance and increase the likelihood of securing registrations during high-demand periods. We moved our operations to faster cloud servers, such as those offered by Google Cloud Platform (GCP), to reduce registration lag time and improve script execution speed. We also increased the number of registration attempts by running multiple iterations of the script in parallel across several servers. This brute-force method allowed us to handle competitive registration environments more effectively, as we could attempt multiple registrations in a single epoch, improving our overall success rate.
Through a combination of software development, automation, and resource management, we optimized the Bittensor mining process. Our custom-built monitoring systems reduced the manual effort required to manage miner registration and deregistration, while our automated registration scripts allowed us to scale our operations across multiple subnets efficiently. In addition to the tools already discussed, we developed a registration cost tracker and a reward tracker to help miners better manage their operations. These tools allow users to track the exact amount spent on miner registrations and the amount of net rewards generated.
While the API is still a work in progress, there will soon be the capability to track staking balances at the start of every block, allowing for accurate cost basis tax calculations. We are currently developing a tax tool for Bittensor that will simplify the process of generating detailed tax reports, making it easier for miners to manage their finances and comply with tax regulations. While Bittensor continues to evolve, this experience demonstrates the importance of using software-driven solutions to tackle complex blockchain ecosystems. The journey was not without challenges, but with careful analysis, scripting, and cloud optimization, we were able to significantly improve our mining operations.”
The key property of the value engine is having the ability to produce as much value as it can. The miner service does this by plugging into multiple subnets to relay as much tasks as possible. Mining on multiple subnets in Bittensor is a common strategy to diversify rewards and optimize hardware usage. Each subnet functions as a specialized marketplace (e.g., storage, compute, or AI model training) with its own rules and resource requirements. To adopt this, we’ll need some key requirements.
- Wallet Configuration: A single coldkey (your main wallet) can manage multiple hotkeys (the keys that perform the actual mining). While one hotkey can technically hold multiple UIDs across different subnets, it is standard practice to use separate hotkeys for better organization and security.
- Registration Costs: Every subnet requires a registration fee in TAO. This fee is dynamic, fluctuating based on demand, and is a sunk cost—it cannot be recovered once paid.
- Diverse Hardware: Different subnets have vastly different hardware needs. In our case where the actual “mining” would be done by the Personal Operators who are sub agents, we’ll be looking into cloud infra and local setup specifications that’ll cover: AI/Inference Subnets: Typically require high-end GPUs (e.g., NVIDIA H100s or A100s). Compute/Storage Subnets: May prioritize high-bandwidth internet, large NVMe storage, or CPU cores over raw GPU power.
- We've set up the Miner service
- We've plugged to about 1 (8 left) subnet during the first phase of development
- We've built some Miner Tools: i.e a registration monitor, registration cost tracker, automated registration script, deregistration monitor, a reward tracker and a tax tool for Bittensor
- We are currently building some Personal operator tools: i.e GPU sniper tool
- Running on testnets
- Updating the documentation
One essential part of building this system is that we must be able to measure the unit economics and every engineer should be able to understand how each pieces connect. To get this right, our development will be sequenced. We have to sequence the process in stages that helps us measure the value flow.
Stage 1: Minimal Miner Setup [Now Done ]
We connect to at least 1 subnet on testnet, register and get started. The only objective at this stage, was to get it working. The build should involve these essential parts:
- the connection to bittensor (the wallet setup), plus other tools that helps
- the task decompose and aggregation function
- the dispatch function After this, we’ll build a sample operator agent (maybe spawn like 2 agents) that executes the work from the orchestrator.
If we could complete the whole cycle (preferably on testnets), we could begin to track the metrics and have a good measure of value flow.
Stage 2: Improve Personal operators
- their discovery
- ability to acquire compute (EigenCloud integration)
- building out their “execution systems” and solving other edge cases
Stage 3: Establish the workstream environment:
- providing a discovery algorithm (maybe gossipsub) to the task pipeline
- adding more configuration that aids a more distributed and fair system