Proposal #82: External Test Lab & Community DevNet — M1 Report

[paranjko]

Period: 2026-07-10 — 2026-08-09 · Proposal: #82 (discussion) · Next unlock: 2026-08-13

Workstream A — DevNet infrastructure (M1: Setup)

M1 requirement	Status
DevNet design agreed	Done — architecture note
≥5 nodes online	Done — node0–node4 deployed, one provider each, chain gonka-devnet-community; live state on the status site
Infrastructure blockers documented	Done — see below

Delivered beyond the M1 bar:

• Deployment runbook 1.0.0-alpha.1 (net-deployment-runbook/) — reproducible baseline on testnet-0.2.14 (phased CLI: prepare → identities → qualify-ml → genesis → join → verify), governed 0.2.14 → 0.2.15 upgrade rehearsal, single-node stop/start/verify/reset, network reset contract. The M2 artifact "runbook sufficient to reproduce a node" (ROLE-JOIN.md) is already published.
• Operator handoff flow — an external operator can qualify, deploy, and register a node without receiving coordinator secrets; the coordinator confirms on-chain registration and grants funding/ML permission.
• Public observability — status site (per-node cards from committed chain state, honest OFFLINE on reset) and Grafana dashboards: 24h network view, 7-day inference view, triage overview.
• Authenticated access — api.gonka-dev.net/v1: OpenAI-compatible, chain-accounted DevShard v4 gateway (v3/v4 approved via on-chain governance), automatic escrow rotation across epochs, API keys self-issued through a Telegram bot from a finite pool, no artificial per-key quotas.

All images and binaries are pinned by digest; DevNet keys are fully isolated from mainnet.

Blockers encountered (documented and resolved)

• Blackwell GPU support. The RTX Pro 2000 ML host (compute capability 12.0) is not supported by the MLNode images pinned to the 0.2.14 baseline. Resolved with a pinned hardware-runtime exception: a newer upstream MLNode runtime for that host only, while the chain release and model overlay stay on the baseline (documented in the runbook release profile).
• PoC validation timing on DAPI 0.2.14. The stock gateway could submit its first MLNode distribution in the same block as a newer store commit, missing the validation snapshot. Resolved with a ten-block PoC validation delay as an explicit test-lab genesis parameter (documented in the runbook).
• Provider port limitations (Novita.ai). Evaluated as pay-as-you-go GPU capacity: usable for ML hosts, but does not expose the ports a full network node requires. Kept as reserve ML capacity (remaining balance to be attached to a network node's ML host later).

No hard procurement blockers were encountered during Month 1. One recurring friction: some datacenters and GPU neoclouds require KYC before activating servers, which adds delay to provisioning and slows down multi-provider deployment.

Workstream B — Testing lab (M1: Setup and QA onboarding)

M1 requirement	Status
Initial test strategy	Done — testing/test-strategy.md
QA search and selection	Done — first engineer hired and active
Operating model	Done — task board, issue templates (defect / validation request / DevNet access), SECURITY.md private disclosure path

Hiring. 60+ CVs screened, 8 candidates interviewed, one QA / infrastructure engineer hired, started on July 28. As part of onboarding he took on the DevNet deployment itself: the runbook, observability, and gateway work above; onboarding continues in Month 2. The search for the second engineer restarts in Month 2, likely with paid listing promotion; one candidate is in the pipeline and receives a test assignment once the DevNet stabilizes.

Incidents

The network spent most of the period in initial deployment and stabilization; instability during bootstrap was expected and was resolved as part of deployment work rather than tracked as incidents. Public monitoring is live (status site with per-node state and validator map, dashboards).

Spending (budget lines per proposal §14)

Line	Monthly cap	Spent (Month 1)	Notes
DevNet machines	$5,000	≈ $1,550 ($1,065 + €420)	5 nodes + dedicated ML host + domain, itemized below
Burst GPU rental	$6,500	$100	Novita.ai pay-as-you-go GPU credit (evaluated for ML hosts; balance reserved)
Tooling and reporting	$250	$0
External Testing Engineers	$8,000	$2,000	first QA engineer salary (paid by Aug 11)
Contingency	$2,250	$0
Total	$22,000	≈ $3,650

Infrastructure breakdown (paid invoices; receipts available for review on request):

Date	Provider	Configuration	Role	Billing period	Amount
07-19	DatabaseMart	GPU VPS, RTX Pro 2000 16 GB	node4 ML host	1 month	$119.00
07-19	OneProvider	dedicated CPU (EPYC/64 GB), Portland US	node4 network node + public edge	1 month	$77.50
07-24	DatabaseMart	dedicated, RTX A5000 24 GB	node0 (genesis)	1 month	$254.77
07-27	OneProvider	dedicated + Tesla T4 16 GB, Helsinki FI	node1	1 month	$331.63
07-27	Novita.ai	pay-as-you-go GPU credit	burst ML capacity (evaluated; balance reserved)	—	$100.00
07-28	GIGAGPU	dedicated, RTX 3090 24 GB, UK	node3	1 month	$270.03
07-29	NameSilo	gonka-dev.net registration	domain	1 year	$11.85
07-29 + 08-05	LeaderGPU (LeaderTelecom)	dedicated, RTX 4090 24 GB, NL	node2	5 weeks	€420.00

Month 2 plan (M2 targets)

• Scale to 9+ MLNodes across target regions; publish the regional layout summary (#3, #4).
• Publish smoke and regression checklists (#5).
• Run the second QA engineer hiring round; we aim to close the hire, prioritizing fit over speed (#6).
• Begin validation work as testable artifacts are handed off (validation requests).