fix(weights): epoch-aligned emission, real-seal automation, metagraph cache - #71
Conversation
Both D24 participant challenges pin their expected set at the epoch's start block (last_epoch_block) but labeled the set with the pre-coinbase +1 epoch whenever emission fired near an overdue boundary. The label and the covered metagraph then referred to different epochs, so any registration/deregistration at a boundary made the two same-label sets disagree and every seal of a real epoch failed D24 with 409 IncompleteParticipantSet — the reason only block-scale burn bundles ever sealed in prod. - prism-challenge/design-challenge: label emissions with the current chain epoch (subnet_epoch_index) so label and pinned metagraph always agree; same-label sets from both challenges then match by construction. - gateway/db: latest_bundle_epoch prefers chain-scale bundles over the reserved smoke range (>= 8_000_000) so interim burn seals no longer shadow a real sealed epoch on /v1/weights/latest. - deploy: prod-real-seal.sh + base-real-seal.timer seal the current chain epoch (block_b = LastEpochBlock) every 10 min once both challenges' sets exist — the missing operator automation between emission and serving. Regression test: s5_latest_epoch_prefers_chain_scale_over_smoke_range.
--score HOTKEY_HEX:VALUE (repeatable) posts an explicit scored set with everyone else NoScore; --expected-csv reads the D24 participant set from a file (zero chain rpcs) for rate-limit-proof emergency re-emission. default behavior unchanged.
Emitters and sealers re-query the same epoch-start metagraph every tick; cache hits within 15 minutes skip state_getKeysPaged/queryStorageAt while keeping the bulk path, and the cache mutex serializes refreshes so concurrent callers share one RPC.
|
Warning Review limit reached
Next review available in: 28 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe pull request adds metagraph caching, reserves chain-scale epoch selection below 8,000,000, aligns challenge emission labels with subnet epochs, adds production real-epoch sealing, and supports participant and score overrides in the smoke tool. ChangesMetagraph caching
Epoch selection and labeling
Production real-epoch sealing
Smoke weight participant overrides
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant LiveChainClient
participant MetagraphCache
participant ChainRPC
LiveChainClient->>MetagraphCache: request metagraph(netuid, block_hash)
alt cached snapshot is valid
MetagraphCache-->>LiveChainClient: return snapshot
else cache miss or expired snapshot
MetagraphCache->>ChainRPC: fetch metagraph and owner data
ChainRPC-->>MetagraphCache: return snapshot
MetagraphCache-->>LiveChainClient: store and return snapshot
end
sequenceDiagram
participant Timer
participant RealSealer
participant ChainRPC
participant Gateway
Timer->>RealSealer: start sealing run
RealSealer->>ChainRPC: read epoch index and last epoch block
ChainRPC-->>RealSealer: return chain values
RealSealer->>Gateway: submit real-epoch seal
Gateway-->>RealSealer: return seal status
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Identity-hashed Subtensor maps append netuid as LE bytes; printf %04x used BE and every state_getStorage returned null, so the timer never sealed.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
🧹 Nitpick comments (1)
crates/db/src/store.rs (1)
321-332: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winAdd database regression coverage for the reserved epoch boundary.
crates/db/tests/gateway_store.rs:179-227only inserts epoch12. It does not exercise the filteredMAX, smoke-only fallback, or the exact8_000_000boundary.Add these cases and keep their expected results aligned with
crates/gateway/src/sealer.rs:98-108.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/db/src/store.rs` around lines 321 - 332, Add database regression tests in the gateway store test coverage around the epoch selection query, covering a real epoch below 8,000,000 taking precedence over smoke epochs, smoke-only fallback when no real epoch exists, and the exact 8,000,000 boundary. Update expected results to match the selection behavior defined by the gateway sealer logic.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@bins/weights-smoke/src/main.rs`:
- Around line 275-278: Update the validation error in the expected-hotkey check
around load_expected_csv so it identifies the selected participant source
accurately: when --expected-csv is set, report that the hotkey is not in the
expected CSV rather than claiming it is absent from the metagraph, while
preserving the existing behavior for other sources.
- Line 280: Update the argument-processing logic around override_values.insert
to detect when the hotkey already exists and reject duplicate --score entries
before generating leaves, rather than replacing the earlier value. Preserve
insertion for unique hotkeys and return a clear error for any repeated hotkey.
In `@crates/chain-live/src/lib.rs`:
- Around line 163-169: Bound the metagraph cache around the entries insertion in
the relevant cache-reading function: prune entries older than the configured
expiration and enforce a fixed capacity or LRU eviction policy before or after
inserting each new block hash. Preserve cached reuse for valid entries, and add
a test using distinct block hashes that verifies older or excess entries are
evicted.
In `@crates/chain-live/src/tests.rs`:
- Around line 749-771: Strengthen metagraph_cache_singleflight_under_concurrency
by synchronizing all worker threads with a start barrier immediately before
metagraph_at, and make the mocked Keys/refresh response block or otherwise
overlap competing cache misses. Track the mock refresh/RPC calls and assert
exactly one refresh occurred, while preserving the existing result assertions.
In `@deploy/AGENTS.md`:
- Line 99: Update the compound modifier in the “Real-epoch sealer” description
to use “post-burn-seal retirement” instead of “post burn-seal retirement,”
preserving the rest of the operator procedure unchanged.
In `@deploy/scripts/prod-real-seal.sh`:
- Around line 60-65: Update the chain-reading flow around epoch and leb to
obtain one finalized block hash before reading either value, then fetch both
storage entries using that same block reference and request set. Ensure the
resulting epoch and last-epoch-block values are the ones submitted to
/v1/admin/seal, while preserving the existing failure handling for chain reads.
- Around line 68-76: Update the seal request handling around the curl invocation
to capture the HTTP status without using -f, while still distinguishing
transport failures. Exit successfully only when POST /v1/admin/seal returns HTTP
200; classify HTTP 409 as pending without treating it as accepted, and return
nonzero for all other HTTP statuses or curl failures so systemd reports
unsuccessful runs correctly.
- Around line 43-46: Update the rpc_storage decoding branch around raw so the
0x-prefixed hex payload is validated and decoded into an integer before
printing; only return success after decoding succeeds. Ensure invalid storage
results propagate failure instead of allowing the subsequent return 0 to report
an empty successful value.
In `@deploy/systemd/base-real-seal.service`:
- Line 10: Increase TimeoutStartSec in the base-real-seal service configuration
from 120 to 180 seconds, matching the burn-seal service and providing headroom
for startup and request overhead.
---
Nitpick comments:
In `@crates/db/src/store.rs`:
- Around line 321-332: Add database regression tests in the gateway store test
coverage around the epoch selection query, covering a real epoch below 8,000,000
taking precedence over smoke epochs, smoke-only fallback when no real epoch
exists, and the exact 8,000,000 boundary. Update expected results to match the
selection behavior defined by the gateway sealer logic.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 1397ec70-6206-4e3d-adbd-34471415fe11
📒 Files selected for processing (14)
bins/weights-smoke/src/main.rscrates/chain-live/src/lib.rscrates/chain-live/src/tests.rscrates/db/.sqlx/query-31947d13ab560212fcb266d537cbd8b20185c0d70c2d1963c12d2526ac90390f.jsoncrates/db/.sqlx/query-eb526c131a5c57f9f03470ccf507fd91002059c0da3d0e5ff62bc334f957e717.jsoncrates/db/src/store.rscrates/design-challenge/src/orchestrator.rscrates/gateway/src/sealer.rscrates/gateway/tests/sealer.rscrates/prism-challenge/src/orchestrator.rsdeploy/AGENTS.mddeploy/scripts/prod-real-seal.shdeploy/systemd/base-real-seal.servicedeploy/systemd/base-real-seal.timer
💤 Files with no reviewable changes (1)
- crates/db/.sqlx/query-eb526c131a5c57f9f03470ccf507fd91002059c0da3d0e5ff62bc334f957e717.json
| if !expected.contains(&hk) { | ||
| return Err(format!( | ||
| "--score hotkey {raw:?} is not in the metagraph at tip={tip}" | ||
| )); |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Report the selected participant source.
When --expected-csv is set, expected comes from load_expected_csv, not from the metagraph. The current error names the wrong source and can mislead the operator.
Proposed fix
- "--score hotkey {raw:?} is not in the metagraph at tip={tip}"
+ "--score hotkey {raw:?} is not in the expected participant set at tip={tip}"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if !expected.contains(&hk) { | |
| return Err(format!( | |
| "--score hotkey {raw:?} is not in the metagraph at tip={tip}" | |
| )); | |
| if !expected.contains(&hk) { | |
| return Err(format!( | |
| "--score hotkey {raw:?} is not in the expected participant set at tip={tip}" | |
| )); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@bins/weights-smoke/src/main.rs` around lines 275 - 278, Update the validation
error in the expected-hotkey check around load_expected_csv so it identifies the
selected participant source accurately: when --expected-csv is set, report that
the hotkey is not in the expected CSV rather than claiming it is absent from the
metagraph, while preserving the existing behavior for other sources.
| "--score hotkey {raw:?} is not in the metagraph at tip={tip}" | ||
| )); | ||
| } | ||
| override_values.insert(hk, value); |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Reject duplicate --score entries.
override_values.insert(hk, value) silently replaces an earlier value for the same hotkey. Conflicting repeated arguments can therefore produce an unintended seal without an error. Reject duplicate hotkeys before generating leaves.
Proposed fix
- override_values.insert(hk, value);
+ if override_values.insert(hk, value).is_some() {
+ return Err(format!("duplicate --score hotkey in {raw:?}"));
+ }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| override_values.insert(hk, value); | |
| if override_values.insert(hk, value).is_some() { | |
| return Err(format!("duplicate --score hotkey in {raw:?}")); | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@bins/weights-smoke/src/main.rs` at line 280, Update the argument-processing
logic around override_values.insert to detect when the hotkey already exists and
reject duplicate --score entries before generating leaves, rather than replacing
the earlier value. Preserve insertion for unique hotkeys and return a clear
error for any repeated hotkey.
| cache.entries.insert( | ||
| key, | ||
| CachedMetagraph { | ||
| fetched_at: Instant::now(), | ||
| metagraph: mg.clone(), | ||
| }, | ||
| ); |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win
Bound the metagraph cache.
Line 163 inserts every distinct block hash. No code removes expired entries. Each entry retains a complete Metagraph, including all hotkeys. A long-running client that reads distinct pinned blocks can grow this cache without limit.
Prune expired entries and apply a fixed capacity or LRU policy. Add a test that uses distinct block hashes and verifies eviction.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/chain-live/src/lib.rs` around lines 163 - 169, Bound the metagraph
cache around the entries insertion in the relevant cache-reading function: prune
entries older than the configured expiration and enforce a fixed capacity or LRU
eviction policy before or after inserting each new block hash. Preserve cached
reuse for valid entries, and add a test using distinct block hashes that
verifies older or excess entries are evicted.
| #[tokio::test] | ||
| async fn metagraph_cache_singleflight_under_concurrency() { | ||
| let server = MockServer::start().await; | ||
| // Eight concurrent callers share one bulk refresh (mutex held across fetch). | ||
| mount_metagraph_mocks(&server, 1).await; | ||
|
|
||
| let uri = server.uri(); | ||
| tokio::task::spawn_blocking(move || { | ||
| let client = Arc::new(LiveChainClient::connect(&uri).expect("connect")); | ||
| let hash = [0_u8; 32]; | ||
| let mut handles = Vec::new(); | ||
| for _ in 0..8 { | ||
| let c = Arc::clone(&client); | ||
| handles.push(std::thread::spawn(move || c.metagraph_at(&hash))); | ||
| } | ||
| for h in handles { | ||
| let mg = h.join().expect("join").expect("metagraph"); | ||
| assert_eq!(mg.hotkeys.len(), 2); | ||
| } | ||
| }) | ||
| .await | ||
| .expect("spawn_blocking"); | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C3 --glob 'Cargo.toml' 'wiremock'
rg -n -C3 'Barrier|set_delay|Responder|mount_as_scoped' crates/chain-liveRepository: BaseIntelligence/base
Length of output: 159
🌐 Web query:
For wiremock 0.6.0 in Rust, what supported mechanism can block or delay a mocked response so a test can deterministically verify concurrent single-flight requests?
💡 Result:
To introduce a delay in a mocked response using wiremock-rs (including version 0.6.0), you should use the set_delay method on the ResponseTemplate struct [1][2]. This allows you to specify a std::time::Duration that the MockServer will wait before returning the response to the client [1][3]. Example usage: use std::time::Duration; use wiremock::{Mock, MockServer, ResponseTemplate}; use wiremock::matchers::method; // Arrange let mock_server = MockServer::start.await; let delay = Duration::from_secs(5); let template = ResponseTemplate::new(200).set_delay(delay); Mock::given(method("GET")).respond_with(template).mount(&mock_server).await; This mechanism is explicitly designed to simulate network latency and is commonly used to test the behavior of timeout policies in your client applications [1][3]. For verifying concurrent single-flight requests where you need precise control over the execution order or timing beyond a fixed duration, you can implement custom behavior using the Respond trait, which allows you to define dynamic responses based on the incoming Request [4][5]. Additionally, if your goal is to verify that certain expectations are met during asynchronous flows, you can use MockServer::register_as_scoped to obtain a MockGuard, which provides a wait_until_satisfied method to synchronize your test execution with the mock's activity [6].
Citations:
- 1: https://docs.rs/wiremock/latest/src/wiremock/response_template.rs.html
- 2: https://docs.rs/crate/wiremock/latest/source/src/response_template.rs
- 3: https://helpmetest.com/blog/wiremock-rs-rust-http-mocking/
- 4: https://github.com/LukeMathWalker/wiremock-rs/
- 5: https://docs.rs/wiremock/latest/wiremock/
- 6: https://docs.rs/wiremock/latest/i686-pc-windows-msvc/wiremock/struct.MockGuard.html
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate chain-live and tests =="
git ls-files | rg '(^|/)Cargo\.toml$|chain-live/src/tests\.rs$|chain-live'
echo
echo "== tests around concurrency =="
sed -n '720,790p' crates/chain-live/src/tests.rs
echo
echo "== LiveChainClient outline and relevant methods =="
ast-grep outline crates/chain-live/src/lib.rs --match metagraph_at --view expanded || true
rg -n -C4 'fn metagraph_at|metagraph_cache|metagraph_at\(&|Refresh|Mutex|parking|tokio::sync|Barrier' crates/chain-live/src
echo
echo "== wiremock references =="
rg -n 'wiremock|MockServer|expect_requests|ReceivedRequest|set_delay|respond_with' .Repository: BaseIntelligence/base
Length of output: 40565
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== thread spawning in chain-live tests =="
rg -n 'std::thread::spawn|tokio::task::spawn|spawn_blocking|Barrier|Notify|Mutex|metagraph_at' crates/chain-live/src/tests.rs crates/chain-live/src -S
echo
echo "== Cargo wiremock version if present =="
rg -n 'wiremock' -g Cargo.toml -g Cargo.lockRepository: BaseIntelligence/base
Length of output: 13584
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== wiremock package entry in lock =="
sed -n '6628,6658p' Cargo.lock
echo
echo "== LiveChainRpc method call path =="
sed -n '148,230p' crates/chain-live/src/lib.rsRepository: BaseIntelligence/base
Length of output: 4092
Make the single-flight test prove concurrent cache misses.
The threads start without a barrier and the mocked Keys response has no delay, so one thread can complete the mock RPC before another thread enters metagraph_at. Add a start barrier before the workers call metagraph_at, delay the mock response or use a custom blocker that observes the competing cache misses, then assert only one RPC refresh occurred.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/chain-live/src/tests.rs` around lines 749 - 771, Strengthen
metagraph_cache_singleflight_under_concurrency by synchronizing all worker
threads with a start barrier immediately before metagraph_at, and make the
mocked Keys/refresh response block or otherwise overlap competing cache misses.
Track the mock refresh/RPC calls and assert exactly one refresh occurred, while
preserving the existing result assertions.
|
|
||
| A seal older than ~256 blocks can never be verified by the validator (public RPC prunes state) — if `GET /v1/weights/latest` shows `metagraph_block` lagging tip by thousands of blocks, check `systemctl status base-burn-seal.timer` and `/var/log/base-burn-seal.log` on the master. | ||
|
|
||
| **Real-epoch sealer (post burn-seal retirement):** `base-real-seal.timer` (every 10 min) drives [`scripts/prod-real-seal.sh`](scripts/prod-real-seal.sh), which seals the **current chain epoch** with `block_b = LastEpochBlock` (the epoch's start block — exactly the metagraph both challenges pin their leaf sets against, so D24 participant matching holds by construction). The attempt 409s until both challenges have emitted for that epoch; that is the expected steady state. The gateway prefers chain-scale bundles over the reserved smoke range (`>= 8_000_000`), so once a real seal lands it outranks every interim burn bundle — retire the burn timer (`systemctl disable --now base-burn-seal.timer`) after the first real seal verifies end-to-end. Install: |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Hyphenate post-burn-seal.
Use post-burn-seal retirement in the compound modifier. This removes the grammar warning in the operator procedure.
🧰 Tools
🪛 LanguageTool
[grammar] ~99-~99: Use a hyphen to join words.
Context: ...n the master. Real-epoch sealer (post burn-seal retirement): `base-real-seal...
(QB_NEW_EN_HYPHEN)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@deploy/AGENTS.md` at line 99, Update the compound modifier in the “Real-epoch
sealer” description to use “post-burn-seal retirement” instead of “post
burn-seal retirement,” preserving the rest of the operator procedure unchanged.
Source: Linters/SAST tools
| raw="$(printf '%s' "${out}" | jq -r '.result // empty')" | ||
| if [[ -n "${raw}" && "${raw}" != "null" ]]; then | ||
| python3 -c 'import sys; print(int.from_bytes(bytes.fromhex(sys.argv[1][2:]), "little"))' "${raw}" | ||
| return 0 |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate file =="
fd -a 'prod-real-seal\.sh$' . || true
echo "== file excerpt =="
if [ -f deploy/scripts/prod-real-seal.sh ]; then
nl -ba deploy/scripts/prod-real-seal.sh | sed -n '1,130p'
fi
echo "== rpc_storage callers and set options =="
rg -n "rpc_storage|set -e|set -" deploy/scripts/prod-real-seal.sh deploy || trueRepository: BaseIntelligence/base
Length of output: 288
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file='/home/jailuser/git/deploy/scripts/prod-real-seal.sh'
echo "== file stats =="
wc -l "$file"
echo "== file excerpt =="
line_count=$(wc -l < "$file")
end=$((line_count < 130 ? line_count : 130))
i=1
while IFS= read -r line || [ -n "$line" ]; do
printf '%6d\t%s\n' "$i" "$line"
i=$((i + 1))
[ "$i" -gt "$end" ] && break
done < "$file"
echo "== rpc_storage callers and set options =="
rg -n "rpc_storage|set -e|set -" "$file" deploy || true
echo "== Python decode behavior for probe values =="
python3 - <<'PY'
import subprocess, sys
values = ['0x', '0xabc', '0xabcde', '']
for raw in values:
try:
run = subprocess.run([sys.executable, '-c',
'import sys; print(int.from_bytes(bytes.fromhex(sys.argv[1][2:]), "little"))'],
args=(raw,), text=True, capture_output=True)
print(repr(raw), 'returncode=', run.returncode, 'stdout=', repr(run.stdout), 'stderr=', repr(run.stderr[-80:]))
except Exception as e:
print(repr(raw), 'exception=', repr(e))
PYRepository: BaseIntelligence/base
Length of output: 7409
Do not return success after a decode failure.
python3 can fail for an invalid storage result, but return 0 executes afterward. Since rpc_storage callers use || guards and rpc_storage also runs in a subprocess (set -u affects neither || continue nor another function’s failure), an invalid RPC response can be treated as an empty successful value and prevent failover.
Validate the 0x-prefixed hex payload and assign the decoded integer before printing it.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@deploy/scripts/prod-real-seal.sh` around lines 43 - 46, Update the
rpc_storage decoding branch around raw so the 0x-prefixed hex payload is
validated and decoded into an integer before printing; only return success after
decoding succeeds. Ensure invalid storage results propagate failure instead of
allowing the subsequent return 0 to report an empty successful value.
| epoch="$(rpc_storage "0x${K_SUBNET_EPOCH_INDEX}${netuid_hex}")" || { | ||
| echo "$(date -Is) chain read failed (epoch)" | ||
| exit 1 | ||
| } | ||
| leb="$(rpc_storage "0x${K_LAST_EPOCH_BLOCK}${netuid_hex}")" || { | ||
| echo "$(date -Is) chain read failed (last_epoch_block)" |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
rg -n -C 6 'state_getStorage|chain_get(Header|FinalizedHead)|last_epoch_block' crates deploy binsRepository: BaseIntelligence/base
Length of output: 50377
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "== prod-real-seal outline/contents =="
wc -l deploy/scripts/prod-real-seal.sh
cat -n deploy/scripts/prod-real-seal.sh
echo
echo "== chain-live storage/key helpers =="
rg -n -C 8 'SubtensorModule|LastEpochBlock|SubnetEpoch|rpc_storage|state_getStorage|fn state_get_storage_at|chain_getHeader|finalized' crates/chain-live src crates/chain crates/chain-live/src crates/chain/src --glob '!crates/**/tests.rs' --glob '!crates/chain/**/tests.rs' 2>/dev/null | head -n 220Repository: BaseIntelligence/base
Length of output: 19942
Read epoch and last epoch block from the same block.
epoch and leb are read by two independent state_getStorage calls, and each call may fail over to the next CHAIN_ENDPOINTS candidate. An epoch transition can return an old epoch with a new start block, or vice versa, while /v1/admin/seal expects them to be pinned from the same moment as the challenge leaves.
Get one finalized block hash first, then read both storage values at that block from the same request set before submitting epoch and block_b.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@deploy/scripts/prod-real-seal.sh` around lines 60 - 65, Update the
chain-reading flow around epoch and leb to obtain one finalized block hash
before reading either value, then fetch both storage entries using that same
block reference and request set. Ensure the resulting epoch and last-epoch-block
values are the ones submitted to /v1/admin/seal, while preserving the existing
failure handling for chain reads.
| resp="$(curl -fsS -m 60 -X POST -H 'content-type: application/json' \ | ||
| -d "{\"epoch\":${epoch},\"netuid\":${NETUID},\"block_b\":${leb}}" \ | ||
| "${GATEWAY}/v1/admin/seal" 2>&1)" && rc=0 || rc=$? | ||
| if [[ ${rc} -eq 0 ]]; then | ||
| echo "$(date -Is) seal ok epoch=${epoch} block_b=${leb}: ${resp}" | ||
| else | ||
| # 409 (sets incomplete / already sealed) is the expected steady state | ||
| # while waiting on a challenge emission; anything else needs a look. | ||
| echo "$(date -Is) seal pending/failed rc=${rc} epoch=${epoch} block_b=${leb}: ${resp}" |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== file excerpt =="
cat -n deploy/scripts/prod-real-seal.sh | sed -n '1,140p'
echo
echo "== smoke client relevant excerpt =="
if [ -f bins/weights-smoke/src/main.rs ]; then
cat -n bins/weights-smoke/src/main.rs | sed -n '150,210p'
fi
echo
echo "== seal endpoint/search =="
rg -n "admin/seal|sealed bundle|HTTP|StatusCode::200|StatusCode::409|seal" -S .Repository: BaseIntelligence/base
Length of output: 50378
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== docker/systemd usage around script =="
rg -n "prod-real-seal|seal-real|role-master|compose|systemctl|systemd|run-seal|seal" -S deploy || true
echo
echo "== curl usage patterns =="
rg -n "curl -[A-Za-z ].*seal|curl .*/v1/admin/seal" -S deploy || trueRepository: BaseIntelligence/base
Length of output: 34835
Classify the seal response status before exiting.
curl -f treats redirects and any non-error response as success, while the seal contract accepts only POST /v1/admin/seal with HTTP 200 as accepted; HTTP 409 is the expected incomplete/already-sealed steady state. Capture the HTTP status without -f, exit 0 only for 200, treat 409 as pending, and exit nonzero for other statuses or transport failures so the systemd service does not report a successful run when no seal was accepted.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@deploy/scripts/prod-real-seal.sh` around lines 68 - 76, Update the seal
request handling around the curl invocation to capture the HTTP status without
using -f, while still distinguishing transport failures. Exit successfully only
when POST /v1/admin/seal returns HTTP 200; classify HTTP 409 as pending without
treating it as accepted, and return nonzero for all other HTTP statuses or curl
failures so systemd reports unsuccessful runs correctly.
| Type=oneshot | ||
| ExecStart=/opt/base/deploy/scripts/prod-real-seal.sh | ||
| Nice=10 | ||
| TimeoutStartSec=120 |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== candidate files =="
fd -a 'base-real-seal.service|base-real-seal|real.*seal|seal' . | sed 's#^\./##' | head -100
echo
echo "== service file =="
if [ -f deploy/systemd/base-real-seal.service ]; then
nl -ba deploy/systemd/base-real-seal.service
fi
echo
echo "== related real seal scripts =="
fd -a 'real-seal|base-real-seal|seal' deploy scripts . 2>/dev/null | sed 's#^\./##' | head -200Repository: BaseIntelligence/base
Length of output: 834
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== deploy/systemd/base-real-seal.service =="
cat -n deploy/systemd/base-real-seal.service 2>/dev/null || true
echo
echo "== deploy/scripts/prod-real-seal.sh =="
cat -n deploy/scripts/prod-real-seal.sh 2>/dev/null || true
echo
echo "== any systemd timers pointing to base-real-seal =="
cat -n deploy/systemd/base-real-seal.timer 2>/dev/null || true
echo
echo "== deployment/doc references =="
rg -n "base-real-seal|prod-real-seal|real seal|real-seal|TimeoutStartSec" -S deploy docs README* .github 2>/dev/null || trueRepository: BaseIntelligence/base
Length of output: 7442
Give TimeoutStartSec headroom.
prod-real-seal.sh can take up to 30 seconds for each state-storage read and 60 seconds for the gateway seal request. That reaches the current 120-second timeout without accounting for startup, logging, or shell overhead. Set TimeoutStartSec=180 like the burn-seal service, or reduce the request budgets.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@deploy/systemd/base-real-seal.service` at line 10, Increase TimeoutStartSec
in the base-real-seal service configuration from 120 to 180 seconds, matching
the burn-seal service and providing headroom for startup and request overhead.
1 participant
Summary
last_epoch_blockmetagraph) so same-label D24 sets can seal; prefer chain-scale bundles over smoke-range burn seals on/v1/weights/latest.prod-real-sealsystemd timer (10 min) to seal the current chain epoch once both challenges have emitted — permanent operator automation after burn-seal retirement.chain-live::LiveChainClient::metagraph_at(bulk path unchanged; cache-hit skips RPC; mutex serializes refresh so concurrent callers share one fetch). Tests cover hit / TTL expiry / singleflight.Test plan
cargo test -p chain-live metagraph_cache(hit, TTL refresh, concurrent singleflight)cargo clippy -p chain-live --all-targets -- -D warningsbase-real-seal.timeron prod master; confirm burn-seal stays disabled/v1/weights/lateststill serves winner ≈0.8admin/sealSummary by CodeRabbit