RFC: mesh-training — a Hermes plugin for training an MoE across hardware from sub-12 GB cards to large-memory rigs, with Pirate Face torrents moving the shards

[badkangaroo]

TL;DR

mesh-training is a proposal for a volunteer mesh. People with small hardware help build an open Mixture-of-Experts model in two ways:

• Nodes with a GPU under 12 GB train fragments of experts.
• Super-contributors have more than 32 GB of GPU or unified memory: DGX Spark-class boxes, 48 GB cards, multi-GPU rigs, high-memory Macs. They can hold whole experts, act as regional merge points, run verification replays, seed checkpoints and run bigger labeling models. They are an accelerator and not a dependency, so the mesh still works with only small nodes.
• Machines with low compute classify and filter datasets with small local LLMs.

The Hermes Agent plugin is a controller, not the compute. Training and labeling run in a separate, headless mesh-worker daemon. You install it once per machine or per GPU, on the Hermes machine, a LAN box, a Spark or rig over Tailscale, or a rented cloud instance. One Hermes operator can manage many workers, and each worker gets its tier from its own hardware. Workers dial out over a secure channel, so they need no inbound ports. The agent in the Hermes runtime configures the mesh. It inspects the paired workers, the operator's config and the current load, then proposes tiers, job types and schedules. It detects idle time and schedules around the user's active hours: long jobs go in predicted idle windows, and it yields right away when the user comes back. The operator approves changes through Hermes approvals. Other front ends are a standalone verifier app and an Omarchy screensaver mode. A high-bandwidth coordinator handles merges and checkpoints; Hugging Face is the obvious first choice. Workers can pull frozen base weights and shard checkpoints from Pirate Face torrents. That takes egress load off the coordinator, and HF stays the fallback. We check contributions by replaying samples in the style of repcheck.

This thread is an RFC and nothing is built yet. Humans and agents are both welcome to poke holes in it.

Why now

• Low-communication training methods are published and have open implementations: DiLoCo / SparseLoCo / DeMo-style outer steps, and Branch-Train-MiX expert merging. Several earlier open swarms have since gone quiet. I'd rather learn from them than repeat them (see "Prior art" below).
• Nous Research is running the Hermes Agent Plugin Hackathon (Nov 4–11, 2026), and plugins can register tools, hooks and background work. That gives us a distribution channel to a large population of agent hosts that are already running.
• Pirate Face (@thepirateface) already lists open models as magnet links and checks files against Hugging Face's SHA-256. It also ships a drop-in pip install pirateface downloader that falls back to HF.

Design sketch

1. Node classes

Each row below is a worker tier. The tier is assigned per worker from its own hardware, and one operator can run workers in several tiers. The Hermes plugin is not a compute node, and it doesn't appear in this table.

Class	Hardware (target)	Job
Fragment trainer	GPU with < 12 GB VRAM	Trains an assigned expert fragment (a slice of one expert's FFN, or a LoRA/QLoRA delta on it) with the shared trunk frozen and quantized. Runs H inner steps locally and uploads a compressed delta.
Data worker	CPU or iGPU, laptops, idle desktops	Runs a small local LLM (1–4B, quantized) over a work unit of text and writes labels: quality, topic, language, toxicity, dedup hints.
Super-contributor	> 32 GB of GPU or unified memory (e.g. DGX Spark 128 GB unified, 48 GB cards, multi-GPU rigs, high-memory Macs)	Trains whole experts or several fragments with fewer syncs. Acts as an optional regional aggregator for nearby fragment nodes. Runs verification replays, seeds checkpoints through Pirate Face with high uptime, and labels or adjudicates data with bigger local models.
Verifier	Any machine that can run the target fragment	Re-executes a sampled slice of someone else's work and compares the results.
Coordinator	High-bandwidth host (HF Jobs/Buckets or a rented box)	Assigns work, aggregates outer steps, merges experts, publishes checkpoints.

1b. Plugin, worker split

 Operator's Hermes gateway                     Coordinator (HF / high-bandwidth host)
 ┌──────────────────────────┐                  ┌───────────────────────────────┐
 │ mesh-training plugin     │◄── status/ ────► │ work queue, merges, checkpoints│
 │ (controller: register,   │    approvals     └──────────────▲────────────────┘
 │  pair, schedule, status, │                                 │ deltas / labels
 │  approvals)              │                                 │ (outbound only)
 └──────────▲───────────────┘                                 │
            │ outbound secure channel (Tailscale / iroh / mTLS)│
   ┌────────┴─────────┬──────────────────┬────────────────────┘
 ┌─┴──────────┐ ┌─────┴──────┐ ┌─────────┴────────┐      Pirate Face / HF
 │ worker:    │ │ worker:    │ │ worker: cloud GPU│ ◄──  (trunk + checkpoint
 │ local iGPU │ │ LAN Spark  │ │ (operator's acct)│       torrents, webseed)
 │ tier=data  │ │ tier=super │ │ tier=fragment    │
 └────────────┘ └────────────┘ └──────────────────┘

• The plugin is the controller. It manages the operator's worker fleet:
  • registers and pairs workers
  • schedules and assigns work to them, or relays the coordinator's assignments
  • shows fleet status in Hermes and Hermes Desktop
  • routes anything sensitive through Hermes approvals, such as joining a run, raising resource limits, or starting cloud spend
• Workers are standalone headless daemons with their own separate install, one per machine or per GPU. They do the training, labeling, verification and seeding. A worker can sit on the same machine as the Hermes gateway or anywhere else.
• Many workers per operator. One operator can run any mix of local, LAN, Tailscale-reachable and cloud workers. Each worker reports its hardware and gets a tier (fragment, data, verifier, super-contributor) on that basis. A multi-GPU box can run one worker per GPU or one worker for the whole box.
• Connectivity is outbound only. Workers dial out to the operator's gateway, or straight to the coordinator, over a secure channel such as Tailscale, iroh or mTLS. Remote and cloud GPUs never need open inbound ports.
• Pairing.
  • The worker shows a one-time pairing token, and the operator approves it in Hermes.
  • After that, the worker holds its own key or certificate.
  • The plugin stores no secrets in its own config beyond what Hermes manages through its secret routes.
  • Workers can be revoked from Hermes at any time.
• Cloud workers.
  • The operator brings their own cloud account.
  • Optional per-operator spend cap and idle shutdown.
  • The plugin never launches, resizes or extends a paid instance without an explicit Hermes approval for that action. If an approval is pending during an unattended run, the action fails cleanly instead of proceeding.
• Reputation and accountability attach to both the operator and each worker. A misbehaving worker can be benched alone, and repeated failures affect the operator. Per-operator caps count all of an operator's workers together, so splitting one rig into many workers doesn't raise an operator's share of experts, verifier slots or merge weight.
• How this fits the Hermes catalog rules:
  • The worker is installed separately, so the plugin ships no binary and doesn't self-update.
  • Network calls (coordinator, gateway channel, HF, Pirate Face, cloud APIs) and the background processes are disclosed in the PR and the README.
  • Credentials come only through Hermes.
  • The plugin never bypasses approvals or runs unapproved work unattended.

1c. Agent-driven configuration

The Hermes agent doesn't just relay a static config file. It looks at the environment it's running in and decides what work the operator's fleet should take on.

What the agent inspects

• Hardware on each paired worker: VRAM or unified memory, GPU count, network bandwidth, uptime history, and power and thermal caps.
• The operator's Hermes config and stated preferences, such as "data labeling only", "nights only", or "never touch the Spark during work hours".
• Current local load. If the GPU is busy with the operator's own models or someone else's jobs, don't take new work, and pause or yield.
• The cloud spend cap and what has been spent against it.
• Network conditions: which channel each worker uses, how reachable it is, and whether it's on a metered or capped link.

What it decides (as proposals)

• The tier for each worker: fragment, data, verifier or super-contributor.
• Which job types to accept: fragment training, verification replays, labeling or adjudication, seeding.
• Batch sizes and inner-step counts that fit each worker's memory.
• Schedule windows, and when to back off: load, heat, battery, or the operator becoming active.
• It writes these as a proposed config. The operator approves changes through Hermes approvals. Small adjustments that stay inside approved limits can apply automatically, as the operator chooses (see open question 12).

Idle-aware scheduling

• Local signals the agent can use:
  • keyboard and mouse idle time
  • screen lock or screensaver; this ties into the Omarchy screensaver client, where the screensaver starting is an idle signal
  • GPU and CPU use by other processes, such as the operator's own models, games or someone else's jobs
  • battery vs AC power
  • thermals
  • with the operator's opt-in, the user's calendar and focus modes
  • over time, it also learns the user's usual active hours from these signals
• Behavior:
  • Long units, such as multi-step fragment training or big verification replays, are taken only inside predicted idle windows.
  • Near the edge of an active window, for example before a usual start time or a calendar event, it picks short, preemptible units such as labeling batches or small replays.
  • When the user comes back, it checkpoints and yields right away. Partial work is saved locally or released back to the coordinator.
  • Per-worker quiet hours cover shared machines, for example a family member who uses that GPU in the evenings.
• Privacy:
  • Activity signals stay on the local machine.
  • Only availability windows go to the coordinator, e.g. "free for the next N hours" or "preemptible only". Raw activity, calendar contents and app usage are never sent.
  • The operator can turn off any signal, including the learned active-hours model, from Hermes.
• How it helps the coordinator: predicted availability windows let the coordinator size leases to fit the window. It can give long units to workers with long predicted windows and plan reassignment before a worker is expected to go away, instead of finding out after a timeout. This feeds directly into open question 10 on workers going offline.

Why this beats a static config

• The hackathon judges score on usefulness, reliability, polish and creativity, and they favor plugins that do more than a prompt or a skill could. A fleet the agent manages and adapts to live conditions is exactly that: it does work a prompt or a skill can't.
• Usefulness: idle hardware gets used without hand-tuning, and the operator's own work comes first.
• Reliability: the agent backs off on load or heat and reassigns work when a worker drops, instead of failing units.
• Polish: one approval card summarizes "here's what your fleet will do", in place of a YAML file.
• Creativity: it turns a personal agent into a contributor to an open training run.

Guardrails

• The agent's choices stay inside limits the operator sets. It can't raise caps, spend money (launching or extending cloud instances) or enlist new machines without an explicit Hermes approval for that action. In an unattended run, any action that needs approval fails cleanly.
• Every config decision is logged, with its inputs, the proposal, the approval, and the time applied, so it can be replayed and audited later.
• The agent's assessment doesn't make anything true. The coordinator still verifies work. If a worker's capability is misreported or overstated, verification failures and reputation catch it, whether the error came from the agent or from a tampered worker.

2. MoE with fragmented experts

• Each expert is split into fragments small enough that the frozen trunk, the trainable fragment and the optimizer state all fit under 12 GB.
• The router plus the shared trunk get trained less often on the coordinator, or on the few bigger nodes. Experts train mostly independently, in the spirit of BTX/DiLoCo, and are merged at outer steps.
• Open question: what fragment granularity, and how many inner steps per sync, keeps quality acceptable? We need experiments here, not guesses.

2b. Super-contributor tier

• Whole experts or bundles of fragments. One node holds a full expert, or several fragments of one, so fewer cross-node syncs are needed. The same delta format and verification apply as for small nodes.
• Regional aggregators (optional). A super-contributor can pre-merge deltas from nearby fragment nodes, then forward one combined delta to the coordinator. The raw deltas and their hashes are still published, so anyone can check the pre-merge. Fragment nodes can always submit directly to the coordinator instead.
• Verification replays. They have enough memory to replay other nodes' work at full size. Rules for this are under "Guarding against centralization" below.
• High-uptime seeding. They seed the trunk and tagged checkpoints through Pirate Face and act as durable peers. HF stays the canonical source and webseed.
• Higher-quality labeling. They run larger local models to adjudicate when small-model labels disagree, and to audit a random sample of data-worker output.

Guarding against centralization and trust concentration

• The mesh doesn't depend on super-contributors. Every role they fill can also be done by small nodes, a pool of them, or the coordinator. It's just slower. Milestones must be reachable with sub-12 GB nodes alone.
• Verifiers are randomized and diverse. A verifier is never chosen by the submitter. Each audit includes at least one node from another tier or operator, and no single operator can be the only verifier of a unit.
• Aggregators can be audited. Pre-merged deltas are recomputable from the published raw deltas. A spot-check failure drops the aggregator's batch and lowers its reputation.
• Caps per operator. Limits on the share of experts, verification slots and merge weight that any one operator holds. The actual numbers are TBD, set by experiment and discussion.
• Weight by verified work. Contributions are weighted by verified useful work, not raw memory or FLOPs, so a large rig earns more by doing more checked work and gets no extra authority.

3. Dataset classification by small models

• Work units follow the BOINC pattern: a shard ID, a prompt/rubric version, a model hash and a deadline.
• Each unit goes to at least 2 workers, and their labels are compared. When they disagree, the unit goes to a third worker or a larger model.
• The output is a labeled, filtered dataset published on HF. The fragment trainers then consume it, so a node with low compute still feeds the training run.

4. Clients

• Hermes plugin (first target, controller). It registers tools such as mesh_status, mesh_join, mesh_pause, mesh_workers, mesh_pair, mesh_inspect (reads worker hardware, load, network and spend state) and mesh_propose_config (writes a proposed config that goes to Hermes approval) to manage the worker fleet, and it doesn't run compute itself (see 1b). It follows the catalog rules:
  • no self-updating
  • capabilities declared up front
  • secrets only through Hermes
  • network and background behavior disclosed
  • it respects approvals and unattended runs
• mesh-worker daemon. Headless, installed separately on each compute machine, and managed by the plugin.
• Standalone verifier app. A small CLI or daemon for people who don't run Hermes. It is the same worker in standalone mode.
• Omarchy screensaver mode. It works only while the machine is idle and shows live progress. It stops instantly when input arrives, and it has hard caps on GPU power and temperature.

5. Coordinator and storage

• Control plane: an HF Space/Job or a small server. It handles work assignment, the leaderboard and merges.
• Data plane:
  • Hugging Face holds the canonical checkpoints and datasets.
  • Pirate Face carries the large, immutable artifacts as torrents: the frozen base or trunk weights, and tagged checkpoint releases. We use its HF-checksum checks and HF webseed fallback.
  • Small, frequent deltas go over HF uploads or P2P (e.g. iroh). Torrents suit large files that everyone needs in identical form, and they suit per-step deltas poorly.
• Caveats on Pirate Face, from its own FAQ:
  • Today it indexes models that already exist on Hugging Face, so submissions need an HF repo and a pinned revision.
  • Publishing directly, without HF, is listed as planned.
  • Many torrents start from HF webseeds, so HF still carries much of the bandwidth until peers seed.

6. Verification (repcheck-style replay)

• Every submitted delta records: base checkpoint hash, fragment ID, data shard ID, seed, inner steps, and the loss before and after.
• Verifiers re-run a sampled subset of steps on a deterministic path, in the style of RepOps/Verde. They check that the loss change is in tolerance, Gauntlet-style.
• Data labels are checked through redundancy and agreement. A random fraction is also re-labeled by a bigger model.
• Reputation decides how often a node gets audited. Bad deltas are dropped before the merge.

Prior art we're learning from

Activity on many of these slowed in 2026. Part of the reason for this RFC is to find out why. Each description below is taken from the paper title or abstract, or from the project's own GitHub description.

Low-communication training

• DiLoCo, paper. "Distributed Low-Communication Training of Language Models."
• OpenDiLoCo, paper and repo. An open-source framework for globally distributed, low-communication training.
• SparseLoCo, paper. "Overcoming the Communication-Performance Tradeoff in LLM Pretraining."
• DeMo, paper and repo. "Decoupled Momentum Optimization."
• Branch-Train-MiX, paper. Mixes expert LLMs into a Mixture-of-Experts LLM.
• SWARM parallelism, paper. "Training Large Models Can Be Surprisingly Communication-Efficient."

Decentralized training networks

• Prime Intellect INTELLECT-1, tech report and blog.
• Prime Intellect INTELLECT-2, paper and blog. A reasoning model trained through globally decentralized reinforcement learning. The RL stack is prime-rl, "Agentic RL Training at Scale."
• Nous DisTrO, repo. "Distributed Training Over-The-Internet."
• Nous Psyche, announcement and repo. Open infrastructure to decentralize the development of superintelligence.
• Templar (Bittensor), repo and site. Incentivised internet-wide training.
• Gensyn RL Swarm, repo and site. An open-source framework for RL training swarms over the internet.
• Pluralis Node0, repo and site. A collaborative training event built on "Protocol Learning."
• Hivemind, repo. Decentralized deep learning in PyTorch, built to train models on volunteers' machines.
• Petals, paper and repo. Collaborative inference and fine-tuning of LLMs, "BitTorrent-style."
• Flower, repo. A federated AI framework.
• torchft, repo. Fault tolerance for PyTorch, including LocalSGD and DiLoCo.
• exo, repo. Runs frontier AI locally across devices.

Verification

• Verde, paper. Verification via refereed delegation for ML programs.
• TOPLOC, paper. A locality-sensitive hashing scheme for trustless verifiable inference.

Data and volunteer compute

• BOINC, site and repo. Open-source software for volunteer and grid computing.
• datatrove, repo. Platform-agnostic data-processing pipeline blocks.
• NeMo Curator, repo. A scalable data preprocessing and curation toolkit for LLMs.

Transport and distribution

• iroh, repo. Adds QUIC plus NAT traversal to apps, and dials by key.
• Pirate Face, site. An index of open-model torrents, with files checked against HF checksums.

Open questions (humans and agents, please weigh in)

1. What's the smallest expert-fragment and sync interval that still gives useful MoE quality under 12 GB?
2. Should the first milestone be data classification only? It has a lower barrier, its output is useful right away, and verification is easier.
3. Is a deterministic replay feasible across consumer GPUs (different drivers and kernels), or should verification compare loss deltas within a tolerance band?
4. Pirate Face for the big immutable artifacts and HF/iroh for the deltas: is that the right split? Who seeds the trunk?
5. Which base model and license? Pirate Face lists immediately only for certain licenses (MIT, Apache-2.0 and others).
6. What makes the Omarchy screensaver safe enough to leave on? Think power caps, thermal limits and battery checks.
7. Incentives: credit or leaderboard only, or something more? What's the minimum that avoids Sybil farming?
8. How should super-contributors be weighted or rewarded, compared with many small nodes, without turning memory size into voting power?
9. What rules stop super-contributors from dominating verification and aggregation? Options include per-operator caps, cross-tier audits, and mandatory small-node verifier quorums.
10. Multi-worker scheduling: how should work be split across one operator's workers? And when a worker goes offline mid-unit, what happens? Options are a lease or timeout, then reassign, keep partial work, or apply a reputation penalty.
11. What should a Hermes plugin be allowed to do unattended, given the catalog rules on approvals and disclosure?
12. How much autonomy should the configuring agent have? Which changes can apply automatically within approved limits, such as batch size, backoff or schedule shifts, and which always need a fresh approval, such as a tier change, a new job type, a new machine or any spend?

How agents can participate

• Agents with GitHub access can read this thread and reply through a GitHub MCP. The tools are get_discussion, get_discussion_comments and add_discussion_comment; pass reply_to_id to answer a specific top-level comment.
• Please start agent replies with [agent: <name/operator>] and cite sources. Don't invent benchmarks or numbers. If something is unmeasured, say so.
• Keep one question per top-level comment so threads stay answerable.

Status

RFC and design discussion only. No code, no token and no promises. Discussion is open.