Trying DevNet locally: test results, findings and next steps

[Santiagocetran]

Hi everyone! Following up on the home testnet idea, I wanted to share my progress testing DevNet locally. These tests were run on October 9–10, 2026, using develop at commit 6e361cb0b4b631f73d3c5ef86fa03469f9f14ae2. Repository findings below refer to that revision.

My goal was to follow the documentation, understand the setup, and gradually try it across three notebooks and a Raspberry Pi. I tested the individual pieces, investigated the failures and kept the original results alongside successful retries. I haven't completed a full training round through the CLI or across the devices yet.

How I tested

I started on Notebook 1, using a separate checkout and temporary local blockchain with test accounts and no real funds. I followed Documentation/technical/testing/dincli-testing-guide.md, using its isolated testing mode. Where the existing tests did not cover what I needed, I added small experiment checks and recorded those separately.

The connectivity setup looked like this:

flowchart TB
    N1["Notebook 1 — main host<br/>Local test blockchain · port 8545<br/>IPFS file service · port 5001<br/>Local deployment, staking and CPU tests"]
    F["Notebook 1 firewall<br/>Temporary access to ports 8545 and 5001<br/>Only from the three test devices"]
    N2["Notebook 2"]
    N3["Notebook 3"]
    P["Raspberry Pi · ARM<br/>Training test incomplete"]
    N2 --> F
    N3 --> F
    P --> F
    F --> N1

Loading

All four devices shared the same Wi-Fi network. The firewall initially blocked access to the two services. I temporarily allowed each of the other three devices to connect to those ports, checked blockchain access and file uploads/downloads, and removed the rules afterward. This confirmed access to central services, not a distributed training round.

What worked

• 468 Python tests passed, with 133 integration tests excluded. A smaller set of 57 focused checks is included in the 468.
• 40 selected smart-contract tests passed, covering deployment, wiring, upgrades and initialization protection. The suites are in foundry/test/UpgradeValidation.t.sol, foundry/test/DeployPlatform.t.sol and foundry/test/DeployPlatformScript.t.sol; this was not the full Solidity suite.
• The local smoke checks passed: wallet reconnection, blockchain access and block growth, and identical file contents after an IPFS round trip.
• The platform deployed with seven proxies and their administrators. My additional checks verified selected ownership, settings and connections, with 22 successful deployment receipts.
• Separate aggregator and auditor test accounts each bought and staked 10 DIN successfully, with six successful transaction receipts.
• All three other devices passed the central blockchain/IPFS connectivity checks after the temporary firewall adjustment.

I also tested the unchanged reference client and auditor services in cache_model_0/services/client.py and cache_model_0/services/auditor.py, using MNIST, the handwritten-digit dataset. The dataset preparation came from cid_services/distribute_mnist.py.

One client trained on 20,000 handwritten-digit images. On a separate set of 10,000 images, accuracy improved from 11.04% before training, using an experiment-generated model with randomly initialized weights, to 96.82% after training. The unchanged auditor scoring rules accepted the trained model.

This was a CPU worker check using prepared local files. It did not test CLI registration, encrypted data delivery, model upload/submission or approval on the blockchain. I prepared three data partitions but trained only one model.

Repository findings

1. Two platform tests expect an older deployment format

Files: tests/dincli/test_01_platform.py and foundry/script/DeployPlatform.s.sol.

The two original platform tests failed because they expect a single proxyAdmin entry. The current deployment produces separate administrator entries for the contracts.

The errors were AssertionError: missing proxyAdmin and KeyError: 'proxyAdmin'. Deployment itself succeeded, and my additional checks verified selected live state. That does not make the original tests pass: their failures need to remain visible.

This matches the problem already described in #228.

2. Local transactions display public testnet explorer links

Files: dincli/config/din_info.json, under the local network, and dincli/cli/utils.py, in print_tx_info.

During the buy/stake checks, the CLI showed Active Network: local, but printed transaction links for Optimism Sepolia. I verified chain ID 1337 and successful local receipts, so this was misleading output rather than a public-chain transaction.

The local configuration supplies the public explorer address, and the output function uses it. For a local network, printing the transaction hash without that link would be clearer unless a suitable explorer is explicitly configured.

3. The client-service guide describes an older interface

Files: Documentation/technical/services/clients.md, cache_model_0/manifest.json, cache_model_0/services/client.py and dincli/cli/client.py.

The guide names train_client_model_and_upload_to_ipfs, while the current manifest and service use train_client_model. The guide also makes it sound as though the training service handles the upload.

In the current workflow, train-lms prepares inputs and runs the worker, which returns a local model file. The separate submit-lm command uploads and submits it. The service's own description still mentions uploading, so that wording also needs alignment.

4. Model-owner deployment calls don't match the current contracts

Files: dincli/cli/modelownerd/deploy.py, foundry/src/DINTaskCoordinator.sol and foundry/src/DINTaskAuditor.sol.

Reading these files showed that the CLI deployment calls omit the model ID required by the current contract constructors. This is already tracked in #223.

I did not execute this model-deployment path and observe it fail. This is a code comparison against the current Foundry contracts, not a failed deployment result from my experiment. The repository already proposes how to handle the model ID in Developer/tasks/task_061026_21.md.

5. Some steps needed for a complete round are missing from the CLI

Files: dincli/cli/modelownerd/deploy.py, dincli/cli/modelownerd/gi.py, dincli/cli/auditor.py and the command registrations under dincli/cli/.

Comparing the CLI with foundry/src/DINTaskCoordinator.sol, foundry/src/DINTaskAuditor.sol and foundry/src/DinEmission.sol showed missing CLI support for reward/token setup, funding, auditor encryption-key registration, reward claims and releasing registration slots after a round.

One particularly confusing example is dincli/cli/auditor.py: its missing-key message tells the user to run dincli auditor register-encryption-key, but that command is not implemented in the tested version.

The contracts already provide the relevant functions. These are CLI gaps, and not all prevent a round from starting: claims and slot release belong to finishing and repeating the process. Slot release does not independently block the first round with the default unlimited registration setting.

This work is already described in Developer/tasks/task_061026_21.md, with progress in Discussion #231. My findings can support that work rather than create duplicate issues.

Setup observations and corrections

These are useful for reproducing the experiment, but should be kept separate from CLI bugs:

• Participant configuration: dincli/cli/utils.py reads .env in the current working directory. My earlier idea of using .env.local automatically was incorrect. Separate participant directories need to use the supported file/state layout.
• Test isolation: tests/dincli/test_01_platform.py launches Forge using the outer process environment. I used a clean outer environment as well as the isolated harness. The harness's filtering should not be assumed to apply to every subprocess.
• Process cleanup: the generic, non-isolated test setup in tests/dincli/conftest.py can stop existing processes using the test port. I used the documented isolated mode, which tracks its own processes through tests/dincli/services.py, to avoid affecting unrelated workloads.
• Dependencies: cache_model_0/requirements.txt includes GPU packages. For CPU training I used the role requirements under cache_model_0/requirements/ and selected CPU PyTorch packages explicitly. ARM needed CPU PyTorch from its CPU index and the pinned torchvision package from PyPI. The package versions were recorded, but this was not a completely locked environment.
• Contract size: the compiled contract-size check passed, but the build output warned that DINTaskCoordinator had 1,079 bytes of headroom below the size limit. That is worth retaining as a constraint, not reporting as a failed deployment.
• Warnings: the initial Python test environment lacked NumPy and produced a warning; the later fresh training environment included it. npm installation logs also contained audit/deprecation notices. I did not investigate those notices sufficiently to call them confirmed security vulnerabilities.
• Device prerequisites: one notebook had a hostname-resolution problem, another lacked Docker socket access, and the older notebook still needs workload/resource measurements. These are unresolved setup questions, not demonstrated DevNet defects.

I also corrected my initial workload plan after reviewing foundry/src/DINTaskCoordinator.sol: at least three approved models are required before aggregation starts. The smaller trailing-batch allowance does not mean that two models are enough to start. My one successful off-chain scoring result does not meet the three-model requirement.

Problems in my experiment scripts

Two problems came from the custom scripts I used to run the experiments.

Dataset preparation permissions: my first preparation container created root-owned directories, preventing the local script from continuing. Running the containers with my local user's permissions resolved this, and the subsequent training and scoring run passed.

Connectivity checks: my initial probe stopped when a hostname lookup failed and later encountered a temporary port-reuse problem. I updated it to record hostname-resolution errors separately and handle port reuse. The subsequent connectivity checks passed.

These were problems in my test setup, not demonstrated DevNet defects.

Raspberry Pi progress

On the Pi, I confirmed remote access, Docker access, local blockchain/IPFS connectivity, an ARM worker build, installation of PyTorch/torchvision/NumPy, and a successful torchvision NMS operation check.

The training attempt did not produce a completed result. Subsequent checks on October 10 found intermittent undervoltage and disabled memory control; further training was deferred. I haven't confirmed completed training on ARM yet.

The saved worker log also warned that Docker discarded the requested memory limit. The restart cause remains unknown, and I am not treating these host problems as CLI bugs.

Evidence retained

I kept the test output, transaction receipts, connectivity results, training and scoring results, package versions, and Pi diagnostics locally, including the original failures and successful retries. These supporting files have not been published. I can share the scripts if anyone would like to reproduce the tests themselves.

The custom deployment, staking and connectivity checks supplemented the repository tests. The CPU training check used a custom driver to prepare local data and run the unchanged reference client and auditor services. These results describe the experiment; the custom checks are not part of the upstream test suite.

A public evidence bundle would need the relevant scripts, exact commands, package versions and sanitized result excerpts so others could inspect and reproduce the custom checks. Raw wallet/configuration files, IPFS state and logs containing test keys would be excluded.

What I still need to test

My next milestone is one complete round through the CLI on a notebook: model and participant setup, reward funding, three approved model submissions, encrypted audits, both aggregation stages, a final model, reward claims and the applicable registration-slot release.

Once that works, I'd like to repeat it across devices, measure resource use and test repeated rounds and participant disconnections. The CPU result is encouraging, but it is not a minimum-hardware specification or proof that all four devices are ready.

Could you confirm which implementation revision and supported command sequence I should use when the relevant parts of task_061026_21 are ready? I'd like this report to support that work and help make the next person's local setup easier.

cc @robertocarlous @umeradl @abrahamnash

[umeradl]

Thanks @Santiagocetran, this is a great report. It's the first time anyone has run DevNet across several machines on a LAN, and the way you wrote it up (original failures kept next to the retries, a clear line between "I ran this" and "I read this in the code", and setup problems kept apart from DevNet bugs) is what we'd like every test report to look like.

Why this helps

• Independent confirmation of the full-GI gaps. Findings No. 1, No. 4 and No. 5 match what task_061026_21 lists, without you having seen that task while testing. So the task's scope is right, and nobody needs to open duplicate issues.
• First check of the reference services outside the test suite. The unchanged cache_model_0 client and auditor took one MNIST partition from 11.04% to 96.82%, and the auditor accepted the result. So once the CLI gaps are closed, the ML side of a round should work.
• A starting point for the home testnet. The LAN layout, firewall steps, CPU/ARM PyTorch install notes and the Pi problems (undervoltage, Docker dropping the memory limit) are exactly what the next person will run into. That belongs in a guide, not just in a discussion.
• Two new findings. No. 2 and No. 3 weren't tracked anywhere. Both are still present on develop today (5f610c7).

Decisions and fixes, finding by finding

Finding	Status on develop	Decision
No. 1 — test_01_platform.py expects a single proxyAdmin	Still present	Already in scope of task_061026_21 (Phase 1 asserts the per-contract proxyAdmin* keys). Running that suite in CI is #228, in progress in PR #242. No new issue.
No. 2 — local transactions print Optimism Sepolia explorer links	Still present: din_info.json → local.explorer is https://sepolia-optimism.etherscan.io	New issue. Fix: drop explorer from the local entry, and have print_tx_info print only the hash when no explorer is configured. Small and self-contained, so a good first PR.
No. 3 — Documentation/technical/services/clients.md names train_client_model_and_upload_to_ipfs	Still present (lines 21 and 47)	New issue. task_061026_21 only covers the public roles/clients.md, not this technical page. Fix: rename to train_client_model, describe the train-lms → submit-lm split, and fix the upload wording in the cache_model_0/services/client.py docstring. Docs-only.
No. 4 — model-owner deploy omits the model ID constructor arg	Still present	Tracked in #223 / task_061026_21. No new issue.
No. 5 — missing CLI commands (reward funding, register-encryption-key, claims, slot release)	Still present, including the stale hint at auditor.py:40	Tracked in task_061026_21 (dincli commands table) and #223 (BL-28). No new issue.

Setup observations: the CPU/ARM PyTorch install path, the separate per-participant .env directories and the requirements.txt GPU-package note should go into the testing guide once the full round works. I'd hold them until then so they're written against the final command sequence. The DINTaskCoordinator size headroom is a known constraint (we're watching it under #201), and your note confirms it.

Next steps

1. @Santiagocetran: please open the two small issues for No. 2 and No. 3. Link this discussion and paste the relevant section from your report. Either can go to a contributor as a first PR.
2. Your question about revision and command sequence: none of task_061026_21's implementation has landed yet (the spec merged in PR docs(tasks): add task_061026_20 (contract follow-ups) and task_061026_21 (full-GI alignment); close task_021026_19 #232; progress is tracked in Discussion task_061026_21 — Full-GI Alignment: Contracts Check + Richer GI States, dincli Commands, Integration Suite on Foundry, GI Docs (#223) (Umer Majeed) #231). For now, a complete CLI round can't run on any revision: gi start reverts with TC_GIRewardPoolNotFunded because funding has no command. Once the dincli commands and the updated test_04_gi.py land, I'll post the exact commit and command sequence in task_061026_21 — Full-GI Alignment: Contracts Check + Richer GI States, dincli Commands, Integration Suite on Foundry, GI Docs (#223) (Umer Majeed) #231 and ping you here. The updated integration suite will then be the reference sequence for your single-notebook round.
3. Until then: a single-notebook round that uses forge/cast for the missing steps (deploying the task contracts with the model ID, depositRewards, registerEncryptionKey) would already exercise everything else: three approved submissions, encrypted audits, T1/T2. That's optional, only if you want to keep going before the CLI catches up.
4. Evidence bundle: yes please, as a separate folder or gist with keys and wallet files stripped, as you described. It will be useful when we write the multi-device guide.

cc @robertocarlous @abrahamnash

[Santiagocetran]

Thanks for the feedback, Umer! I’ve opened the two issues and included the relevant sections from my report:

• Show the correct transaction information when using the local network #265: Show the correct transaction information when using the local network
• Update the client-service guide to explain training and uploading separately #266: Update the client-service guide to explain training and uploading separately