Replies: 3 comments 2 replies
-
|
Hi! Thanks for proposal. A couple thoughts:
I'd probably go with https://huggingface.co/Qwen/Qwen3-0.6B on early phase. For testnet with small models, it doesn't really matter if model is 0.6B or 7B. So when it's testnet with small models, any model/GPUs should be good. I'd suggest to go with cheapest possible GPUs
Based on previous comment, requirements can be lowered. More small GPUs is better that few bigger ones to simulate mainnet
I'd include Network Nodes with multiple MLNodes
I'd say test team also will have to play with such primitives as setting up PoC validation threshold and inference validation threshold. Would be useful to contribute directly in documentation too
Who will have access to this resources? Only testing team?
I think some agreements with current dashboard maintainers would required? Or which dashboards would be used? Will such testnet be publicly available? E.g. can some host experiment with joining testnet before joining mainnet? Will users not from QA team be able to experiment with inference, smart contracts, etc. ? From my perspective the idea overall is good and should be quite helpful. I'd try to clarify the public access (i think it's better to make it fully permissionless) |
Beta Was this translation helpful? Give feedback.
-
|
Thanks Gleb, these are very helpful points. I agree, we'll start with Qwen3-0.6B or a similar model and lower the GPU requirements accordingly. Since the point of the DevNet is to simulate distributed behavior rather than throughput, I'd rather use the cost savings to run a larger number of smaller nodes. I've also lowered the DevNet infrastructure cap. Good point on Network Nodes with multiple MLNodes as well. I will clarify the intended topology. On thresholds and documentation: agreed as well. The testing team should not only validate releases, but also document practical operational primitives such as PoC validation threshold, inference validation threshold, and lessons learned. This should become part of the public runbooks, and where it makes sense we'll contribute these findings directly to the official Gonka documentation. For burst GPU resources, I’ll clarify access and usage. My current thinking is that access should be coordinated by the testing team / project owners during the pilot, mainly for release candidates and model tests, with usage and costs reported publicly. For dashboards, I hadn’t thought about it that way, but it is an interesting point and may save some time and funds. We will contact the current dashboard maintainers to see if they can help. If not, we will use a dedicated Grafana-style dashboard for DevNet health and node status as planned. On public access: yes, host onboarding rehearsal before mainnet is one of the intended DevNet use cases, and non-QA users experimenting with inference and smart contracts is the direction we want. During the pilot, I would gate this through a lightweight request process, mainly to manage abuse, access, and node stability while monitoring and onboarding docs are still being built. Broader participation is the target state, and I want to get to fully permissionless access as soon as safety, abuse limits, onboarding docs, and monitoring allow it. I just don't want to promise a specific date within the pilot yet. I've updated the proposal to reflect these points. |
Beta Was this translation helpful? Give feedback.
-
|
Sure, we should use multinode setup, some network nodes will be on CPU only servers, some full nodes. I believe that:
|
Beta Was this translation helpful? Give feedback.
-
|
Ah, yes, if we mean hosts connecting their own machines to the DevNet, then absolutely, that should be welcome from the start. I was thinking more about controlled access to project-managed resources, where cost, abuse, or stability risks are involved. |
Beta Was this translation helpful? Give feedback.
-
|
Escrow contract is on chain
The contract holds 88,000 USDT + 80,000 GNK and pays them out on the fixed schedule from this proposal. No admin, no migration: recipients and amounts can never change. Governance keeps one lever: a one-time Verify (needs docker): git clone https://github.com/paranjko/testlab-devnet-escrow && cd testlab-devnet-escrow
./build.sh && sha256sum artifacts/milestone_escrow.wasm
# must match: inferenced query wasm code-info 107 |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
External Test Lab & Community DevNet
4-month pilot proposal for community-owned testing infrastructure and QA capacity
1. Executive Summary
This proposal requests funding for a 4-month pilot of External Test Lab & Community DevNet: a community-owned testing function for Gonka protocol upgrades, DevShards, inference flows, host/broker operations, and geographically distributed network behavior before governance decisions and production rollout.
2. Problem Statement
Today, Gonka lacks a dedicated community-owned validation layer for important network changes. The External Test Lab and Community DevNet are proposed to close this gap by providing practical testing capacity, shared infrastructure, and public evidence before releases, governance decisions, or production rollout.
The External Test Lab and Community DevNet will:
Validate protocol upgrades, DevShard releases, broker/inference flows, integrations, and other critical network changes before they move forward.
Test distributed-network behavior that is hard to verify in local or internal environments, including latency, synchronization, propagation, and regional instability.
Use available testing capacity for proactive bug hunting, regression checks, and investigation of known weak points when no release candidate is waiting for validation.
Provide a neutral testing path for work delivered by external teams and ecosystem contributors before it is accepted, funded further, or used in production.
Help validate vulnerability reports from researchers, audits, or programs such as HackerOne through reproduction, impact assessment, regression testing, and confirmation after remediation.
Provide a shared environment where trusted hosts and teams can safely test proposals, integrations, DevShard scenarios, protocol behavior, and early implementation ideas.
Produce clearer testing evidence for governance participants, including test plans, dashboards, runbooks, issue trackers, defect reports, security-sensitive disclosure handling, and release-readiness summaries.
This adds a missing validation layer for the Gonka ecosystem while complementing Core Team testing.
3. Roadmap Alignment
This proposal directly implements two projects from the Gonka Network Development Roadmap.
**Track 4. Network reliability and observability — Project 2. External testing lab
**The roadmap defines an external testing lab for Gonka changes before broad rollout, including changes from Protocol Maintainers, funded external teams, and ecosystem contributors.
This proposal implements that project through the External Testing Team, test plans, smoke and regression checks, defect reports, public issue tracking, and release-readiness reports.
**Track 7. Public sandbox and consumer-GPU testnet — Project 1. Public testing sandbox
**The roadmap defines a separate test environment for experiments with models, parameters, integrations, DevShard scenarios, protocol-level behavior, validation, settlement, and upgrade testing before mainnet.
This proposal implements that project through Community DevNet: a small, always-on, geographically distributed network for protocol, node, DevShard, operational, integration, and distributed-behavior testing**.**
4. What We Are Building
The project has three connected components.
5. Community DevNet Infrastructure
Target size: 9–13 always-on inference machines plus required network nodes, where feasible within the monthly DevNet infrastructure cap.
Topology: part of the DevNet runs as Network Nodes with multiple attached MLNodes, so that realistic multi-MLNode host configurations can be reproduced and tested.
Indicative distribution: North America East, North America West, United Kingdom, Germany, France, Finland, and Asian locations depending on network quality and hosting availability.
Model profile: DevNet inference nodes are expected to run lightweight instruct models, such as Qwen/Qwen3-0.6B, or equivalent models.
Inference nodes: NVIDIA GPU machines with 16 GB VRAM, compatible with the project’s CUDA 13.0 container/runtime stack.
Goal: protocol and distributed behavior testing.
Monitoring: public dashboard for node availability.
Operations: infrastructure lead from the host/DevOps community responsible for provisioning, monitoring, and maintenance.
Public access. The DevNet is intended to serve the broader community, not only the testing team. During the pilot, access for external participants is granted through a lightweight request process — primarily to manage abuse, access control, and node stability while monitoring and onboarding documentation are still being built. Intended external use cases include:
hosts rehearsing onboarding and node operations before joining mainnet;
developers and users experimenting with inference, smart contracts, and integrations outside the QA team's test plan.
Broader participation is the target state: we intend to move to fully permissionless access as soon as safety, abuse limits, onboarding documentation, and monitoring allow it.
6. Burst GPU Testing Budget
Used only when needed for release candidates, large model tests, load tests, and model compatibility checks.
Planning assumption: Up to one calendar week (168 hours) of rental per month.
For Kimi-class testing, the requirement estimate assumes an ML Node with either 4× NVIDIA B200 or 8× NVIDIA H200, around 640 GB total GPU VRAM, 960 GB+ system RAM, 16-core amd64 CPU, and a Network Node host with 16-core CPU, 64 GB+ RAM, 1 TB NVMe, and at least 100 Mbps networking.
Access and accountability. Burst GPU capacity is provisioned and coordinated by the External Testing Team together with the project owners, primarily for release-candidate validation, model compatibility, and load tests. Requests from Protocol Maintainers and ecosystem teams are accommodated where capacity allows. All burst usage is itemized in the monthly public report: purpose, hours used, and cost per test run, with test logs published alongside.
7. External Testing Team
The proposal funds two external QA / Infrastructure Testing Engineers.
Scope of work
Contribute to quality standards and test strategy for the network
Define acceptance criteria and test plans for core network lifecycle events and components
Build pre-release validation frameworks, including smoke checks and regression coverage for known failure patterns
Deploy and verify distributed node and service stacks in production-like environments
Validate critical cross-system flows end to end, with documented evidence and clear defect escalation — e.g. participant and key flows, inference and proof-of-compute participation, gateway and proxy behavior
Use health signals, chain queries, and logs as primary validation inputs, not just debugging aids
Verify upgrade readiness, rollback feasibility, and post-change health across the stack
Establish, tune, and document practical operational primitives, such as PoC validation threshold and inference validation threshold settings, based on DevNet experience, and contribute these findings directly to the official Gonka documentation where appropriate.
Support DevNet validation and release-readiness assessment before production rollouts
Validate recovery and incident resolution through root-cause analysis and re-testing
Report defects with clear reproduction steps, impact assessment, and release-blocking status
Track and communicate quality metrics that reflect network health and operational reliability
Required capabilities
3+ years in system QA / Test Engineering, SDET, or quality-focused DevOps/SRE
Understanding of blockchain lifecycle, key and auth flows: registration, delegated permissions, fee grants
Experience in operation and validation of blockchain nodes (Cosmos SDK preferred): sync, recovery, RPC queries, network phase behavior
Experience validating distributed systems in production-like environments
Strong test design: test plans, acceptance criteria, smoke/regression/e2e, edge cases, negative testing
Solid Linux, SSH, shell scripting, and log-based verification
Docker & container orchestration - including environment and config reload behavior
Clear defect reporting and documentation — runbooks, test results, sign-off checklists
Comfort with time-boxed validation before critical network events
Preferred / Nice-to-Have
SDET experience: scripted validation, CI pipelines, automated health checks
Blockchain / Web3 QA: testnet operations, bridge testing, wallet/key flows, upgrade regression
Cross-chain testing: EVM testnet validation, withdrawal flows, contract interaction
GPU/ML inference testing: worker health, model serving, artifact delivery
Experience with coordinated multi-node upgrades
API gateway and proxy testing (failures, latency, request tracing)
Decentralized inference or Proof of Compute networks
8. Project Owners and Accountability
9. Transparency and Reporting
Public dashboard for DevNet health and node status.
Public task board for planned, active, and completed testing work.
Public issue tracker for non-sensitive bugs, regressions, and operational findings.
Monthly public report with deliverables, incidents, spending by budget line, remaining balance, unused funds, and next-month plan.
Per-release readiness report before governance vote or production rollout when a release candidate is provided in time.
Security-sensitive findings are reported privately to Core Team first; a public placeholder issue is created where appropriate, and details are disclosed after remediation or agreed disclosure window.
10. Protocol Maintainer Coordination
The External Test Lab is intended to work in coordination with Protocol Maintainers while remaining an external community testing function.
Protocol Maintainers are expected to support initial onboarding by providing technical context, relevant documentation, general guidance, scripts and notes, expected test focus, and clarification of protocol-specific behavior where needed.
This coordination helps the testing team become productive faster and reduces the risk of misinterpreting expected network behavior. At the same time, validation reports remain independently prepared by the External Test Lab and are published for the community.
11. Release Validation Handoff Requirements
Where a governance vote or production rollout is scheduled and testable artifacts are provided in time, the External Test Lab publishes a readiness report covering tested areas, pass/fail results, known risks, and recommendations.
If testable artifacts are provided late or incomplete, the External Test Lab may still perform limited validation, but the report will clearly state the reduced scope, time constraints, and known limitations.
12. Milestones and Acceptance Criteria
Workstream A: DevNet Infrastructure
Workstream B: External Testing Lab
13. KPIs
14. Budget Estimate
This is a planning estimate for a 4-month pilot. Unused burst GPU rental budget may roll over within the pilot; any unused funds at the end of the pilot will be returned to the Community Pool.
Any unused funds will be returned to the Community Pool at the end of the pilot.
* The cap above raw GPU-hour pricing covers reserved or non-interruptible instances required for always-on operation, regional price premiums outside low-cost US marketplaces, CPU-only Network Node hosts for the multi-MLNode topology, storage and egress, and temporary node duplication during upgrade and failover testing. It is a cap, not a spend target: actual spending will be itemized monthly, and unused funds will be returned to the Community Pool at the end of the pilot.
** Nebius listed B200 on-demand pricing at $7.15/GPU-hour and H200 on-demand pricing at $4.50/GPU-hour (June 2026). At these on-demand rates, 4× B200 for 168 hours would cost approximately $4.8k, while 8× H200 for 168 hours would cost approximately $6.05k.
15. Payment Schedule
16. End-of-pilot GNK recognition
The Project Lead and Infrastructure Lead receive no monthly compensation from the pilot budget: all USDT tranches fund infrastructure, tooling, and the External Testing Engineers. Leadership work is recognized only through the one-time GNK allocation below, paid after pilot completion and final report acceptance.
17. Open Source, Ownership and Handoff
All non-sensitive documentation, test plans, runbooks, dashboards, issue templates, and reports will be public by default.
Where code or scripts are created, they will be published under an open-source license compatible with Gonka ecosystem norms unless there is a clear security reason not to.
Infrastructure access will not depend on a single individual. The owner model, emergency access process, and handoff procedure will be documented.
At the end of the pilot, a community-approved team should be able to take over the DevNet and External Testing Lab using the published runbooks, documentation, and access handoff process.
Decision Requested
Approve a 4-month pilot of External Test Lab & Community DevNet with a maximum budget authorization of 88,000 USDT, paid in four monthly tranches of up to 22,000 USDT each, and 80,000 GNK paid after pilot completion and final report acceptance.
Beta Was this translation helpful? Give feedback.
All reactions