External Test Lab & Community DevNet — M3 Report

[paranjko]

Proposal: #82 (discussion).

Executive summary

Month 3 brought Community DevNet back into usable operation after the September consensus incident and expanded its role as an independent release-testing environment. Work covered Host JOIN and recovery, protocol upgrades, DevShard deployment, real inference, broker compatibility, and monitoring.

The Lab aligned protocol and model targets with the core-team test environment to support parallel investigation on comparable configurations. The model lineup was updated to Qwen/Qwen3-4B-Instruct-2507 and Qwen/Qwen2.5-7B-Instruct.

The DevShard execution layer is deployed on DevNet, with approved v4.1, v4.2 and the upcoming v6 candidate.

The team took responsibility for maintenance and development of the mainnet network and inference Grafana dashboards, alongside corresponding DevNet dashboards.

M3 targets and carry-over work

Host JOIN, backup and restore — delivered for the tested flows; hardening continues. Independent real-server runs on node5/node6, consensus-key repair and recovery work are recorded in #28, #117 and #152.

Regional hardware — 12 inventoried machines and 11 NVIDIA GPU hosts. (inventory).

Smoke automation — delivered; scheduling remains open. The repository now contains the gonka-check harness and end-to-end guide. Scheduled and post-deployment execution remains tracked in #216.

Recovery and onboarding guides — published; the ongoing incident log remains open. Published artifacts include the recovery procedure, Host onboarding guide and incident #143. The recurring public incident-log format and discovery path remain tracked in #7.

Release qualification — active, scoped results published. Work includes escrow distribution and settlement checks, v6 local scenarios and live DevNet load evidence; remaining HA and fault cases are tracked separately.

Two-engineer testing team — active.

Community DevNet and DevShard layer

The public status site, regional inventory and hardware inventory document the stand and its constraints. The fleet spans multiple providers and GPU generations. The former AMD host was replaced with an NVIDIA RTX 3090; the former node4-ml GPU host is now named node9.

Two additional machines, RTX 4090 and RTX 5090, were procured in October to replace the Tesla T4 hosts under the node1/node7 identities.

DevNet progressed to Gonka 0.2.16 and the MLNode 3.0.16 fleet. Following PoC validation and validator-set problems during the transition, ten consecutive epochs passed the recorded signing, voting-power distribution and PoC self-vote checks (#217).

The Lab published source-bound, digest-pinned DevShard test artifacts, including v4.2/v6 gateways and the wrapping required for the upstream feature plan.
See v6 stand preparation, release switching and budget handover.

Testing and findings

Host lifecycle and recovery

Independent JOIN, backup, reset and restore runs identified reproducible defects. Fixes addressed consensus-key registration and recovery, interrupted identity handling, portable operator execution, preflight/state-sync behavior and retained diagnostics. The September incident is closed in #143; the consensus-key defect is documented in #141. Broader lifecycle and independent-upgrade validation remains tracked on the task board.

Escrow distribution and settlement

• Participant distribution: the study reproduced all 221 real escrows in a mainnet epoch slot for slot, then simulated 1,000 escrows per model at group sizes 16, 32 and 64. The results distinguish inclusion probability from slot share and retain the model-specific weights (#186, tooling).
• Same-epoch parameter change: on October 4, an escrow created with group size 5 settled using its own five slots after governance changed the global size to 9. A new escrow used nine slots; both settled in the same epoch, payouts plus refunds matched deposits, and the parameter was restored to 5. This was one run through gateway A on DevShard v5 with Qwen/Qwen3-0.6B, not a v6 settlement result (#187).
• PoC admission regression: changing epoch length exposed a gateway fence calculated from block-height modulo rather than the actual epoch start. The defect blocked requests during inference and missed part of the intended PoC fence; the fix and regression coverage are in #197 / PR #205.

DevShard v6: regression, Gherkin and load

The upstream manual plan is written in Gherkin, with 24 scenarios and 42 expanded cases. The Lab mapped those scenarios to an isolated local test environment and recorded PASS, FAIL, partial and not-run outcomes (#231). That environment uses real DevShard processes with mock chain and model components; its results do not establish live-network acceptance.

Local catch-up and miss-rate testing found reproducible failures. Revalidation of successive upstream candidates showed improvements but retained ghost/orphaned-work failures; related upstream investigations and fixes include gonka#1993 and gonka#2003. The separate logprobs-switch regression passed its scoped checks; host-to-gateway payload behavior was verified indirectly, as stated in the results (#246).

On October 15, two sequential five-minute SSE runs on live DevNet outside regular and confirmation PoC completed without errors, using Qwen/Qwen3-4B-Instruct-2507 (run results):

• Gateway B: 1,202 / 1,202 successful requests over 302.19 seconds — 3.98 requests/s.
• Gateway A: 1,125 / 1,125 successful requests over 307.54 seconds — 3.66 requests/s.

JSON and SSE smoke passed before both runs. These are measured closed-loop throughput points, not a fixed-rate 4 RPS guarantee. Earlier failed or blocked windows remain in the record. Phase, capacity, escrow and nonce checks are required before subsequent runs.
Full live Gherkin qualification, shared-storage/two-replica HA and agreed host-pause/fault exercises remain open (#224, #225, #242, #254).

Broker compatibility and reusable tests

The Lab continued broker-facing OpenAI compatibility work, building on the Chat Completions host-compat shim guidance and Responses adapter specifications first reported in M2. See broker-compat/ and skills/.

The new gcheck harness adds public-path readiness checks, small authenticated smoke runs, chain checks, retained per-run verdicts, escrow-slot studies and group-size-change accounting evidence.

Incident analysis tool

The Lab published gonkactl-trace, an experimental read-only investigation tool developed around the September incident #143 and the post-recovery investigation tracked in #153. The laboratory uses it daily to analyse the test stand. It collects chain evidence and produces standalone HTML reports showing validator participation, voting power, signatures and available PoC evidence.

The distributed 1.0.0-beta.2 release includes binaries for Linux, macOS and Windows, checksums and an installer. It also supports mainnet investigation, for example gonkactl-trace report mainnet HEAD~10. It does not modify the network or establish incident causes automatically; conclusions depend on the available historical evidence, and collected reports must be reviewed before sharing (usage and limitations).

Monitoring and dashboard maintenance

The team took over maintenance and development of the mainnet network and inference Grafana dashboards.

Corresponding DevNet network and DevNet inference dashboards support ongoing testing.

Spending — September 10–October 9

Monthly budget and spending:

• DevNet machines: $5,000 cap; M3 expenses $2,183.96 + €299.00.
• Burst / test-server capacity: $6,500 cap; spent €136.49 on a test server.
• Tooling and reporting: $250 cap; spent €72.96 on analytics infrastructure.
• External Testing Engineers: $8,000 cap; spent 8,000 USDT for two engineers.
• Contingency: $2,250 cap; $0 separately reported.

M3 total: $2,183.96 + €508.45 + 8,000 USDT, against the $22,000 monthly budget.

Prior-period adjustment: $92.52 for node4’s August service, omitted from M2.

Total reported, including the prior-period adjustment: $2,276.48 + €508.45 + 8,000 USDT.

Infrastructure breakdown

• September service — OneProvider: node4, usage-billed network-only server — $87.33.
• September 19 — DatabaseMart: node9, formerly node4-ml, RTX PRO 2000 renewal — $119.00.
• September 21 — DatabaseMart: node0, RTX A5000 renewal — $254.77.
• September 21 — DatabaseMart: former node5, RTX A4000 renewal — $279.00.
• September 21 — GIGAGPU: node3, RTX 3090 renewal — $270.03.
• September 21 — OneProvider: node1, Tesla T4 renewal — $331.63.
• September 21, invoice date — LeaderGPU: node2, RTX 4090 renewal — €299.00.
• October 5 — GIGAGPU: node8 upgrade from RX 9070 XT to RTX 3090 — $14.93.
• October 6 — GIGAGPU: node8, RTX 3090 renewal — $270.03.
• October 6 — HOSTKEY: node6, RTX PRO 2000 renewal — $231.02.
• October 6 — DatabaseMart: replacement node5, RTX A5000 — $110.65.
• October 6 — OneProvider: node7, Tesla T4 renewal — $215.57.

Prior-period correction:

• August service — OneProvider: node4, usage-billed network-only server, omitted from M2 — $92.52.

node4 is billed by usage. Daily charges are deducted from the provider account, and monthly invoices summarize the service consumed. Expenses are recorded from these invoices; automatic account top-ups are not counted separately.

The replacement-node5 invoice totals $296.65: $186.00 was covered by a refund credit from the old server and $110.65 by new cash.

Month 4 priorities

The Lab’s primary focus is testing gateways and DevShards, alongside protocol updates and broker integrations.

• Continue DevShard v6 regression, load, Gherkin and fault-recovery testing on local stands and Community DevNet.
• Validate gateway and DevShard compatibility, version transitions, escrow lifecycle and high-availability behavior.
• Reproduce reported defects, verify upstream fixes and publish scoped qualification results.
• Rehearse protocol upgrades and check their effects on validators, PoC and inference.
• Expand broker compatibility testing.
• DevShard v7 testing as testable artifacts become available.