Trying DevNet locally, round two: Fashion-MNIST with three notebooks and a Raspberry Pi

[Santiagocetran]

Hi everyone! Following up on Discussion #260, I tested Fashion-MNIST on a local DevNet on November 10, 2026. This also provides evidence for #276 — Add a second end-to-end example model beyond MNIST, building on the Fashion-MNIST example and guide added in PR #289.

A complete local round finished across three notebooks and a Raspberry Pi 5. The Pi performed client training and submission, encrypted audits, first-tier aggregation, and reward withdrawals. The final aggregated model reached 82.54% accuracy on 10,000 test images, independently verified on Notebook 1 and the Raspberry Pi.

An earlier round used three compute devices with the Pi checking connectivity only, followed by a standalone Pi training test. The results below are from a new, isolated four-device round, with the Pi participating in the protocol. Its role was no longer an optional observer.

How I tested

The round used an isolated checkout based on commit ed175d0deceddfa2e63175ec7752c800b31c1fbf, with recorded experiment patches and orchestration scripts. Contract sources were unchanged. Existing device checkouts were preserved.

This was one fresh GI 1 on a new local chain and task, with a bounded training workload. It demonstrates the lifecycle with the recorded adaptations; it is not an unchanged-checkout or stock-CLI-only pass.

Devices and roles

Device	Role in the completed round
Notebook 1	SSH orchestration, local blockchain and IPFS, representative and model owner; clients C1–C3, auditors A1–A3, aggregators G1–G4; final evaluation
Notebook 2	Clients C4–C6, auditors A4–A6, aggregators G5–G8
Notebook 3	Clients C7–C9, auditors A7–A9, aggregators G9–G12
Raspberry Pi — Pi 5, 8 GB RAM	Clients C10–C12, auditors A10–A12, aggregators G13–G15; final artifact retrieval and independent evaluation

flowchart TB
    subgraph ROUND["Fashion-MNIST full round — fresh GI 1"]
        subgraph N1["Notebook 1"]
            CONTROL["SSH orchestration<br/>Representative · Model owner"]
            SERVICES["Shared local services<br/>Anvil blockchain · chain ID 1337 · port 18545<br/>Kubo IPFS store · port 15001"]
            WORK1["Docker compute<br/>Clients C1–C3 · Auditors A1–A3<br/>Aggregators G1–G4<br/>Final model evaluation"]
        end
        N2["Notebook 2<br/>Docker compute<br/>Clients C4–C6 · Auditors A4–A6<br/>Aggregators G5–G8"]
        N3["Notebook 3<br/>Docker compute<br/>Clients C7–C9 · Auditors A7–A9<br/>Aggregators G9–G12"]
        PI["Raspberry Pi<br/>Pi 5 · 8 GB RAM · Docker compute<br/>Clients C10–C12 · Auditors A10–A12<br/>Aggregators G13–G15 — tier 1<br/>Final retrieval and evaluation"]
        WORK1 -->|"Host-side transactions and file transfers"| SERVICES
        N2 -->|"SSH tunnel · host-side transport"| SERVICES
        N3 -->|"SSH tunnel · host-side transport"| SERVICES
        PI -->|"SSH tunnel · host-side transport"| SERVICES
    end
    FLOW["12 submitted models → 4 first-tier batches → 1 second-tier batch<br/>Final scoring → 39 reward withdrawals → 27 validator slots released"]
    SERVICES --> FLOW

Loading

The arrows show access to Notebook 1's shared services. Participant host software handled IPFS, encryption and wallet transactions; compute containers had networking disabled. Notebook 1 dispatched remote jobs through SSH. Heavy jobs ran sequentially on each device, with devices working in parallel.

There were 41 fresh signing identities: representative, model owner, 12 clients, 12 auditors and 15 aggregators. Each role had its own identity and participant directory. These were logical participants operated by one person across four physical devices, not 41 independent operators or a decentralization/Sybil-resistance test.

Notebook 1 hosted Anvil and one real Kubo IPFS store. Remote devices used SSH tunnels to loopback-only endpoints. Kubo ran offline without peer replication. Computation was distributed across four devices, while blockchain and storage were centralized. The run used local test accounts and tokens, with no real funds.

Model, data and configuration

The example used the cache_model_1/ manifest-and-services structure and a small CNN with two convolution layers. The images came from Zalando Research's official Fashion-MNIST dataset: 28 × 28 grayscale images in ten categories covering clothing, footwear and bags. Downloads used the project's GitHub mirror through torchvision, with resource checksum verification.

• Twelve disjoint client partitions of 1,024 images each: 12,288 unique training examples, with three clients on every device.
• The same genesis model, initialized with seed 42, for every client.
• Ten local training epochs per client, with differential privacy disabled.
• Four audit batches, each with three models, three auditors and 500 evaluation examples.
• Four first-tier aggregation batches feeding one second-tier batch, with three aggregators per batch.
• Final evaluation on all 10,000 official test images.

Twelve clients gave each device the same training workload while fitting the protocol's groups of three. The actual protocol assignments were accepted unchanged. Some aggregation groups contained multiple identities on the same physical device; three computations in a batch do not imply three different physical hosts.

The manifest supplied service and artifact references, contract addresses, policies and the encryption public key. All 40 task-participant manifest caches matched the registered manifest and owner public key. All 12 auditors' decrypted dataset assignments were checked against on-chain commitments.

Compute ran inside non-root Docker containers with two CPUs, 4 GiB memory, no additional swap allowance, no network access and a 15-minute job deadline. Capabilities were dropped and no-new-privileges was enabled. Worker entrypoint hashes matched across all four devices, and all used CPU PyTorch 2.6.0. Images and packages were architecture-specific: x86 compute used Python 3.12.16 and ARM compute used 3.12.15. The Pi's host CLI used Python 3.13.5. This is not a claim of identical container images across architectures.

The test split was also used for baseline scoring and sampled for audits, so the final evaluation should not be described as an entirely unexposed benchmark.

What completed

1. Platform and task setup: deployment and contract wiring checks, model registration and approval, validator staking, auditor encryption-key registration and a 100-DIN local reward pool.
2. Client training and submission: all 12 clients trained on their assigned devices, uploaded their models to real IPFS and submitted on-chain. Produced files, IPFS identifiers and submissions matched.
3. Encrypted audits: 36 commitments and 36 reveals were verified. All three auditors agreed on each model, including ARM/x86 batches. All 12 models were approved, with canonical median scores ranging from 75 to 85.
4. Aggregation: all 15 assigned aggregators computed on their designated devices. Within every batch, all three outputs matched byte-for-byte and by CID, including batches mixing Pi and notebook computations. Four first-tier results fed the second tier, exercising aggregation across multiple upstream batches.
5. Final scoring and settlement: the round reached GIended (state 28). All 39 eligible participants claimed and withdrew the correct rewards, all 27 validator registration slots changed from 1 to 0, and no validator stake was slashed.

The evidence contains 380 successful transaction receipts and 29 state events. Chain timestamps from deployment through slot release span 17 minutes 36 seconds, excluding preparation and qualification.

Final model and rewards

Measurement	Result
Untrained baseline accuracy	9.56%
Final aggregated-model accuracy	82.54% — 8,254 / 10,000 correct
Independent Raspberry Pi evaluation	82.54% — 8,254 / 10,000 correct
Mean cross-entropy on Notebook 1	0.562539
Macro F1 on Notebook 1	82.4731%
Recorded on-chain integer score	82
Participant claims and withdrawals	39 / 39 completed
Validator slots released	27 / 27
Validator stake slashed	None

The Raspberry Pi retrieved the final artifact through its real Kubo connection. The model was copied to Notebook 1 and rehashed; both devices independently evaluated it. The 100-DIN pool allocated 60 DIN to clients, 20 to auditors, 15 to aggregators and 5 to the treasury.

Both Notebook 1 and the Pi also resumed training from the final weights for four Adam steps using 128 examples. These continuation checks demonstrate a usable model artifact; they are not another settled GI.

Final artifact identifiers:

CID: bafybeihwqbfh347hjgh2ngahjokhnj6gnkum3aatgbqzvymnaq5wwbkfx4
SHA256: abec9c0da70f05c071ee206fc03965c8c590162cc496fc5991e72db75be5e21d

The CID identifies the artifact retained in the local store; it is not a claim that a public IPFS gateway currently serves it.

Failures, fixes and experiment adaptations

The earlier three-device Fashion-MNIST round exposed the failures below. The new four-device round retained those fixes and adaptations; these are not newly observed failures in the latest round. The following points refer to the tested revision and isolated experiment, rather than claiming they remain present on the latest branch.

CID conversion blocked auditor data retrieval

An auditor path supplied a 0x-prefixed 32-byte digest to a CID helper that expected an unprefixed value. The resulting conversion failed during encrypted-data retrieval.

The isolated fix accepts either form and checks the digest length. Regression checks passed. The auditors then completed using the original ciphertext, assignments, keys and commitments; these were not regenerated to bypass the failure.

This failure is already addressed by PR #313, including a caller/helper fix and regression tests. Our run independently confirms it; a duplicate issue is unnecessary.

Worker file permissions blocked the initial training attempt

The experiment worker file was copied with mode 600. Adding execute permission did not make it readable to the non-root container user. The workers exited before training.

Setting an explicit mode of 755 fixed the entrypoint access, and it was checked on the three compute hosts in that earlier round before retrying. No model had been submitted during the failed attempt. This was a worker-packaging problem in the experiment.

Audit keys appeared in normal CLI output

Tracked in #328.

The owner CLI prints the full audit-batch result, including each batch's raw AES key. Our experiment suppressed that output. A focused check using a synthetic key confirmed that the normal logging operation prints it. Anyone with the logs and ciphertext could decrypt that batch without auditor-specific key delivery. This is a log-access exposure; we did not demonstrate a private production-data leak.

Saved audit subsets retained the full parent dataset

Tracked in #329.

The reference owner services save a PyTorch Subset, which serializes its parent dataset too. A synthetic round-trip confirmed that a 500-example assignment still carried a 10,000-example parent, with unassigned examples accessible through .dataset. The evaluator normally visits only the selected rows, but the file contains more data than the assignment requires.

Our experiment materialized the selected examples before encryption, so the completed run avoided this problem. Both reference model services need regression coverage that checks the saved content, not only len(dataset).

Additional adaptations needed to run and inspect the experiment

• Separated model-owner computation in Docker from host-side IPFS and encryption operations.
• Materialized each dataset slice as its own TensorDataset, avoiding serialization of the full parent dataset.
• Suppressed secret-bearing audit-key output in logs.
• Made CPU dependencies, resource limits, seeds, result validation and timeout handling explicit.

The scoring configuration bug is tracked in #330: the auditor looks up scoring_policy, while the manifest uses audit_scoring_policy. In this run, the auditor used its effective defaults, whose numeric gates matched the intended configuration. No threshold was lowered. The actual canonical score was a median, despite a legacy mean label in the manifest. A separate loader check changed the manifest minimum accuracy to 99; both reference models still loaded 50.0. These inconsistencies should be resolved in a reproducible example.

These changes are part of the result's scope. The successful run does not establish that a fresh upstream checkout completes the same sequence without them.

Raspberry Pi: a real protocol participant

The Raspberry Pi 5 with 8 GB RAM completed client, auditor and aggregator roles in this round, using the same client partition size, epoch count, compute limits and protocol requirements as the notebooks. Its role transactions were dispatched through its own CLI environment.

Pi responsibility	Verified result
Clients C10–C12	Three models trained, uploaded, submitted and approved
Auditors A10–A12	Encrypted data retrieval/decryption and audit commit/reveal completed
Aggregators G13–G15	First-tier compute and commit/reveal completed; outputs matched notebook outputs in their batches
Reward settlement	All nine Pi identities claimed and withdrew positive rewards
Validator registration slots	All six Pi auditor/aggregator slots released
Final artifact	Retrieved, independently evaluated and used for further training
Highest sampled temperature	59.8°C
Throttling flags	0x0 in all samples
Existing din-node	Running in all health samples

G15 was assigned first-tier batch 0, and G13/G14 were assigned first-tier batch 2. Second-tier aggregation was assigned to notebooks. Therefore, this round verifies Pi first-tier aggregation, but does not directly test the Pi in a second-tier assignment.

Yes: this Raspberry Pi can participate in DevNet for this model and bounded workload, with the recorded adaptations and central orchestration. The test does not establish autonomous daemon operation, minimum hardware requirements, larger-model capacity, sustained thermal behavior or resilience to participant failures. Temperature observations are sampled readings, not guaranteed absolute peaks.

What this contributes to #276

#276 asks for a second small public-dataset example, a complete local training round and a short document explaining how to run it.

Requirement	Evidence and remaining work
Second model example beyond MNIST	Fashion-MNIST CNN using the manifest-and-services structure
Complete a full round on a local devnet	Completed across three notebooks and a Pi, including multiple first-tier batches and reward settlement, with recorded adaptations
Short instructions for running it	A guide exists from PR #289. The reproduction instructions, patches and scripts should improve that guide, coordinated with PR #315

The result supports the full-round requirement for the adapted example. This discussion alone should not close the issue: the remaining work is to integrate or clearly document the necessary fixes, update the supported command sequence, and verify the example from a fresh checkout.

Evidence and next steps

A sanitized evidence bundle contains 136 files covering transaction receipts, state transitions, manifests, dataset indices and checksums, audit and aggregation results, final metrics, reward/slot checks, environment information, patches and reproduction scripts. It includes Pi protocol participation, health samples and independent final evaluation. Bundle checksums and the known-secret scan passed. Wallet secrets and raw private recovery state are kept separately. The evidence is currently local and has not been published.

The experiment blockchain, IPFS process, SSH tunnels and compute containers were stopped after testing. Existing device checkouts and the Pi's din-node were preserved.

The next useful steps are:

1. Extend the existing Fashion-MNIST example and guide for Add a second end-to-end example model beyond MNIST #276 with the necessary fixes and sanitized evidence, coordinating with PR docs: bring the GI docs in line with the final commands and states (task_061026_21 Part E) #315.
2. Resolve the three follow-up bugs: audit-key logging (#328), audit subset serialization (#329), and scoring-policy configuration (#330). Each issue includes evidence, proposed scope and acceptance criteria. The CID fix is already covered by PR test(integration): run a complete GI on the foundry contracts (task_061026_21 Part D) #313.
3. Test repeated GIs, participant failures, a Pi second-tier assignment, larger workloads and longer thermal runs separately.

Would this be a useful basis for the #276 example and the multi-device testing guide discussed in #260? I'd particularly like feedback on which experiment adaptations should become supported behavior and which should stay in the test harness.

[umeradl]

Carried over from Discussion #260

I'm closing #260 as superseded by this round. These are the open items from it that this thread doesn't cover yet:

1. Setup notes for the multi-device guide (No. 276). In #260 we agreed to hold these until a full round worked. It now does, so they belong in the Fashion-MNIST/multi-device guide. None of them are in Documentation/technical/testing/dincli-testing-guide.md yet:

• CPU training: use the per-role requirements under cache_model_0/requirements/ with CPU PyTorch selected explicitly. The top-level cache_model_0/requirements.txt pulls in GPU packages.
• ARM (Raspberry Pi): install CPU PyTorch from the PyTorch CPU index, and the pinned torchvision from PyPI.
• One participant directory per role. dincli reads .env from the current working directory and doesn't pick up .env.local automatically.
• The LAN layout: temporary firewall rules allowing only the test devices onto the RPC/IPFS ports (8545/5001), removed afterwards. This round used SSH tunnels to loopback-only endpoints instead, which is the better default for the guide.
• Pi host issues from the first round: intermittent undervoltage, and Docker discarding the requested memory limit (memory cgroup disabled). This round's Pi health samples were clean (throttling 0x0, max 59.8°C), so the guide should say which power supply and cgroup setup you used.

2. Reference command sequence. In #260 I promised the exact commit and command sequence once task_061026_21 landed. It has: the full GI integration run on the foundry contracts (PR No. 313) is the reference sequence, and the GI docs (PR No. 315) describe the final commands and states. Both are on develop as of 4b3b900. A fresh-checkout run of the Fashion-MNIST example against that sequence is the remaining check for No. 276.

3. Evidence bundle. It's still local. Please publish the sanitized bundle (136 files, keys and wallet state excluded) as a gist or a folder linked from the No. 276 guide, so the guide's numbers can be checked.

Already resolved from #260, for the record: No. 2, local explorer links (No. 265, closed Oct 22); No. 3, the client-service guide (No. 266, closed Nov 2); No. 4, deploy constructors and No. 5, missing CLI commands (PRs No. 261, No. 302 and No. 314). The proxyAdmin test expectation (No. 1) was part of the full-GI suite work in No. 228.