Proposal #82: External Test Lab & Community DevNet — M2 Report #1764
paranjko
started this conversation in
Governance Proposal Reports
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Period: 2026-08-10 — 2026-09-09 · Proposal: #82 (discussion)
Executive summary
Month 2 moved the Lab from setup into active infrastructure, Host lifecycle testing, release qualification, and broker compatibility work.
The rented fleet reached nine GPU hosts, but the 9+ validated MLNodes online target remains Partial: some hosts were deliberately used for destructive JOIN / reset / restore tests, two new NVIDIA hosts were still being prepared at cutoff, and one AMD host remains experimental.
The second QA / infrastructure testing engineer started on September 7 after ~160 responses, 13 interviews, and 2 finalists.
Validation produced concrete results: independent JOIN / recovery testing, immutable DevShard candidate artifacts, the active DevShard v5 / Gonka v0.2.16 qualification campaign, broker compatibility tooling, and a Mainnet gateway v4 regression affecting Kimi-K2.6 non-stream requests (gonka-ai/gonka#1680). The bug was fixed upstream and successfully retested.
On September 9, repeated
node5JOIN testing caused a high-severity Community DevNet consensus halt. Recovery and investigation continued after cutoff in #143.M2 targets
Community DevNet
During M2 the Lab renewed the original fleet and added four GPU hosts, reaching nine GPU hosts plus one network-only host across the US, Finland, the Netherlands, the UK, and a staged host in Russia.
The scale KPI remains Partial because rented capacity is not the same as validated online MLNode capacity. Some hosts were repeatedly rebuilt for JOIN / recovery testing; two new NVIDIA hosts were still in preparation; the AMD RX 9070 XT host remains experimental. See regional layout and live state on gonka-dev.net.
The public JOIN path also advanced from coordinator-driven setup toward an independently usable Host lifecycle: network-derived bootstrap / Join Profile handling, ML qualification under the resolved release, retained diagnostics, backup / restore work, and safer failure reporting. Real-server validation remains active in #28 and #117.
External Testing Team
The second QA hiring round completed with ~160 responses, 13 interviews, 2 finalists, and one selected engineer starting September 7. The new engineer received no payments during M2.
The public M1 strategy is now complemented by smoke and regression catalogues. They expose stable test coverage without publishing campaign-specific credentials, private data, or security-sensitive procedures.
Validation and findings
The main active release-assurance campaign is #49 — DevShard v5 and Gonka v0.2.16 qualification. By cutoff, the Lab had prepared immutable candidate definitions, verifiable artifacts / attestations, separate core and DevShard identities, feature scenarios, and controlled rollout / comparison work. No completed readiness verdict is claimed yet.
Testing of the Coreteam scenarios published in feature has started. Qualification is in progress, no final release-readiness verdict has been issued.
Independent JOIN testing produced reproducible defects and regression inputs. Separately, gonka-ai/gonka#1680 identified a Mainnet gateway v4 regression where Kimi-K2.6 non-stream requests could fail with
502 nonce_finished=falsewhile streaming and v3 remained healthy. The fix was merged into gateway v4 and successfully retested onmainnet-v0.2.15-v4.0.1.M2 also published broker-side OpenAI compatibility artifacts: Chat Completions host-compat guidance / shim, a
POST /v1/responsesadapter, and installable skills packaging. Seebroker-compat/andskills/.Incident — GNK-LAB-2026-0001
On September 9, repeated independent JOIN testing for
node5halted block production ongonka-devnet-community; the last block was 306552 at 2026-09-09T18:59:22Z.Preliminary evidence indicates a consensus-key mismatch during JOIN. With another validator offline, remaining voting power fell below the CometBFT quorum.
Impact: DevNet operations requiring new blocks and JOIN verification halted. No Mainnet impact or loss of funds was detected. Recovery / investigation: #143.
Spending
Invoice amounts are the source of truth; original invoice currency is preserved. Raw invoices remain private because they may contain personal, payment, account, or infrastructure data.
Infrastructure breakdown
node4-mlrenewalnode0renewalnode1renewalnode4late-billed prorated servicenode5, JOIN / recovery targetnode2renewalnode3renewalnode6, stagednode8, experimentalnode7, stagedMonth 3 plan
All reactions