fix(deploy): keep on-chain validator off master hosts - #101
Conversation
|
Warning Review limit reached
Next review available in: 50 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe PR adds near-epoch-boundary D24 leaf emission with a 15-second polling interval. It starts the emitter during server startup, documents the emission timing, and separates validator deployment responsibilities from master hosts. ChangesD24 leaf emission
Validator deployment roles
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Main as design-challenge main
participant Orchestrator
participant ChainSchedule
participant EmitLeaves as emit_leaves
Main->>Orchestrator: start run_emitter()
Orchestrator->>Orchestrator: poll emitter_tick()
Orchestrator->>ChainSchedule: load current schedule
Orchestrator->>EmitLeaves: submit missing leaf set
EmitLeaves->>Orchestrator: record successful submission
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Dual base-validator containers with the same 5Gzi wallet were fighting WeightsSetRateLimit/CRV4. Profile validator out of role-master, force-remove it on master deploys, and keep co-located validator only for local-e2e.
Seals stalled when design only emitted on admin award while prism posted every epoch. Add a late-tempo emitter that fills NotAttempted coverage so base-real-seal can advance; award_round still emits scored leaves first.
3517b6d to
6487490
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/design-challenge/src/orchestrator.rs`:
- Around line 314-330: Serialize D24 leaf projection updates between run_emitter
and award_round using one shared async lock or store transaction. Hold it across
score persistence and leaf submission in award_round, and acquire it in
emitter_tick before rechecking emitted_epoch and calling emit_leaves, so stale
NotAttempted leaves cannot be exposed before scored leaves.
In `@deploy/scripts/remote-deploy.sh`:
- Around line 427-428: Update the validator cleanup flow around the docker
compose rm command to fail closed: remove the unconditional “|| true” and
suppressed output, ensure removal errors are reported and terminate the master,
then perform an explicit absence check for the inactive validator container and
likewise exit loudly if it still exists. Preserve the existing validator removal
target and use the surrounding deployment error-handling conventions.
- Around line 422-425: Update the validator cleanup block in the role check to
include gateway alongside the existing challenge and proxy services passed to
docker compose rm. Ensure the cleanup does not suppress failure when gateway
remains, so the redeploy fails if the stale gateway container cannot be removed.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 43146592-0304-4428-826b-1a47ab17cc59
📒 Files selected for processing (8)
bins/design-challenge/src/main.rscrates/design-challenge/src/orchestrator.rsdeploy/AGENTS.mddeploy/compose/env-local.ymldeploy/compose/role-master.ymldeploy/scripts/assert-compose-matrix.shdeploy/scripts/remote-deploy.shdocs/DESIGN_CHALLENGE.md
| let state = chain::gather_schedule_state(self.chain.as_ref(), self.cfg.netuid) | ||
| .map_err(|e| format!("schedule: {e}"))?; | ||
| let epoch = state.subnet_epoch_index; | ||
| if epoch == 0 { | ||
| return Ok(false); | ||
| } | ||
| if self.emitted_epoch.load(Ordering::Relaxed) >= epoch { | ||
| return Ok(false); | ||
| } | ||
| let tempo = u64::from(state.tempo.max(1)); | ||
| // Prefer award_round's scored emit; only fill when ~last 48 blocks remain | ||
| // (~tempo-48 … tempo) so mid-epoch winners are not locked behind NoScore. | ||
| let near_end = state.blocks_since_last_step.saturating_add(48) >= tempo; | ||
| if !near_end { | ||
| return Ok(false); | ||
| } | ||
| self.emit_leaves().await?; |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
# Inspect all callers and the gateway submission contract.
ast-grep outline crates/design-challenge/src/orchestrator.rs --items all
rg -n -C 8 '\b(award_round|emitter_tick|emit_leaves|submit_signed_leaf_set)\b|emitted_epoch|/v1/weights/raw' crates binsRepository: BaseIntelligence/base
Length of output: 50377
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
# Read the relevant orchestrator sections and design contract docs without modifying files.
sed -n '288,410p' crates/design-challenge/src/orchestrator.rs
printf '\n--- award_round scoring and emit section ---\n'
sed -n '404,570p' crates/design-challenge/src/orchestrator.rs
printf '\n--- emit_leaves section ---\n'
sed -n '1104,1157p' crates/design-challenge/src/orchestrator.rs
printf '\n--- design docs relevant excerpts ---\n'
fd -a . docs | sed 's#^\./##' | rg 'DESIGN_CHALLENGE|design_challenge|DESIGN.*CHALLENGE|challenge' | head -80
for f in $(fd -i 'design_challenge\.(md|json|yaml|yml)|DESIGN_CHALLENGE\.(md|json|yaml|yml)' docs); do
echo "%%%%%% $f %%%%%%"
wc -l "$f"
rg -n -C 6 '(d24|D24|leaf set|leaf sets|weights.raw|weights/-raw|expected set|expected set|submission|scoring|award|NotAttempted|NoScore|sealed|emitter|emitter_tick|emitted_epoch)' "$f" || true
doneRepository: BaseIntelligence/base
Length of output: 26702
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo '--- BUNDLE_SPEC relevant leaf seal excerpts ---'
sed -n '1,260p' docs/BUNDLE_SPEC.md
echo '--- raw weight store implementation excerpts ---'
fd -a '.*' crates/gateway crates/challenge-common | rg 'weights_store|raw|submit|leaf' | head -80 || true
rg -n -C 5 'struct RawWeightStore|RawWeightStore|RawWeightAccepted|RawWeightRequest|get\(|accept_raw_weight|expected_set|sealed|signature|weights_store' crates/gateway crates/challenge-common crates/gateway-core 2>/dev/null || true
echo '--- async-lock availability and design-challenge Cargo deps ---'
rg -n 'tokio::sync::Mutex|async-lock|futures::lock::Mutex|Mutex<T>' crates/design-challenge Cargo.toml crates -g '*.toml' | head -40Repository: BaseIntelligence/base
Length of output: 50380
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo '--- gateway raw weight acceptance ---'
sed -n '33,155p' crates/gateway/src/weights.rs
echo '--- raw weight store implementations ---'
rg -n -C 12 'impl RawWeightStore for MemoryRawWeightStore|AcceptError|raw_weight|Postgres.*raw|pgx|insert|Conflict|get_by_epoch|latest_epoch|get_by_epoch|epoch_end|epoch' crates/gateway crates/gateway-core crates -g 'weights_store.rs' -g 'sealer.rs' -g 'seal*.rs' | sed -n '1,240p'
echo '--- design store schema around scores_for_epoch ---'
rg -n -C 10 'scores_for_epoch|ratings|final_score|rating' crates/design-store crates/design-challenge src crates 2>/dev/null | sed -n '1,220p'
echo '--- Cargo dependencies Tokio/async lock ---'
rg -n 'tokio =|async-lock|futures|parking_lot' Cargo.toml devtools crates/design-challenge/Cargo.toml Cargo.lock || trueRepository: BaseIntelligence/base
Length of output: 34677
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo '--- all scores_for_epoch implementations ---'
rg -n -C 8 'async fn scores_for_epoch|fn scores_for_epoch|scores_for_epoch\(' crates/design-store crates -g '*.rs'
echo '--- gateway raw weight persistence implementations and list helper ---'
rg -n -C 12 'struct.*RawWeight|RawWeightStore for|impl RawWeightStore|insert\(|get\(|list_for_epoch|epochs_for_challenges|list_challenges|seal.*epoch|seal_epoch' crates/gateway crates -g '*.rs' | sed -n '1,260p'
echo '--- orchestrator award_round exact end ---'
sed -n '540,570p' crates/design-challenge/src/orchestrator.rsRepository: BaseIntelligence/base
Length of output: 27022
Serialize D24 leaf projection with award_round.
run_emitter can submit NotAttempted leaves before award_round persists scores and submits scored leaves. Since emitted_epoch only guards the emitter and award_round does not recheck it, emit_leaves can expose a stale D24 projection to sealing. Use one shared async lock, or a store-backed transaction, held across score persistence and leaf submission in award_round; acquire it before rechecking emitted_epoch in emitter_tick as well.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/design-challenge/src/orchestrator.rs` around lines 314 - 330,
Serialize D24 leaf projection updates between run_emitter and award_round using
one shared async lock or store transaction. Hold it across score persistence and
leaf submission in award_round, and acquire it in emitter_tick before rechecking
emitted_epoch and calling emit_leaves, so stale NotAttempted leaves cannot be
exposed before scored leaves.
Source: Coding guidelines
| if [[ '$ROLE' == 'validator' ]]; then | ||
| docker compose ${COMPOSE_FILES[*]} rm -sf \ | ||
| prism-challenge design-challenge design-egress-proxy socket-proxy \ | ||
| >/dev/null 2>&1 || true |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== files =="
git ls-files | rg '(^|/)deploy/scripts/remote-deploy\.sh$|(^|/)deploy/.*/role-.*\.ya?ml$|(^|/)deploy/[^/]+\.ya?ml$|(^|/)docker-compose.*ya?ml$' || true
echo "== remote-deploy relevant =="
if [ -f deploy/scripts/remote-deploy.sh ]; then
wc -l deploy/scripts/remote-deploy.sh
sed -n '350,460p' deploy/scripts/remote-deploy.sh | cat -n
fi
echo "== compose service/profile snippets =="
rg -n "gateway|updater|validator|role-validator|role-master|profile:" deploy --glob '*.{yml,yaml,sh,tf,hcl}' | head -n 200
echo "== behavioral probe: docker compose rm semantics from compose docs? no runtime =="
echo "Inspect compose files for profiles explicitly to validate transition hypothesis:"
python3 - <<'PY'
from pathlib import Path
for p in sorted(Path('deploy').rglob('*')):
if p.is_file() and p.suffix in {'.yml','.yaml'}:
text=p.read_text(errors='ignore')
if 'gateway' in text or 'updater' in text or 'profile:' in text:
print(f'--- {p} ---')
for i,line in enumerate(text.splitlines(),1):
if any(s in line for s in ['gateway','updater','profile:','profiles:', 'profiles']):
print(f'{i}: {line}')
PYRepository: BaseIntelligence/base
Length of output: 30485
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== role-validator.yml complete =="
cat -n deploy/compose/role-validator.yml
echo "== role-master.yml complete =="
cat -n deploy/compose/role-master.yml
echo "== base docker service profile definitions for gateway/updater =="
python3 - <<'PY'
from pathlib import Path
import re
for p in [Path('docker-compose.yml'), Path('docker-compose.e2e.yml')]:
if p.exists():
text=p.read_text(errors='ignore')
lines=text.splitlines()
in_svc=False
svc=None
for i,line in enumerate(lines,1):
if re.match(r'^[A-Za-z0-9_\-=/.]+\s*:$', line):
in_svc=True; svc=line.split(':')[0]
if svc in {'gateway','updater'}:
print(f'{p}:{i}: {line.strip()}')
while i < len(lines) and (not re.match(r'^[A-Za-z0-9_][A-Za-z0-9_./-]*:', lines[i]) or lines[i].startswith(' ') or lines[i].startswith('\t')):
print(f'{i+1}: {lines[i]}')
i+=1
break
PY
echo "== master profile arguments context =="
sed -n '60,85p;230,255p;396,440p' deploy/scripts/remote-deploy.sh | cat -n
echo "== read-only compose YAML service/profile model =="
python3 - <<'PY'
from pathlib import Path
import re
def parse_overrides(fname):
hits=[]
t=Path(fname).read_text(errors='ignore').splitlines()
for i,line in enumerate(t,1):
m=re.match(r'^\s+(gateway|updater|validator|prism-challenge|design-challenge|design-egress-proxy|socket-proxy):\s*(.*)?$', line)
if m:
svc=m[1]; tail=m[2]
profiles=[]
while i < len(t):
nxt=t[i]
if re.match(r'^\s{4}profiles:\s*\[([^\]]*)\]', nxt):
profiles=profiles + [x.strip().strip('"\'') for x in re.findall(r'[^,\]\s"]+', nxt)]
elif nxt.strip().startswith('profiles:'):
profiles=[]
elif re.match(r'^[A-Za-z0-9_][A-Za-z0-9_./-]*:', nxt):
break
i+=1
hits.append((svc, tail or '', profiles))
return hits
for f in ['deploy/compose/role-master.yml','deploy/compose/role-validator.yml']:
print(f'--- {f} ---')
for h in parse_overrides(f):
print(h)
PYRepository: BaseIntelligence/base
Length of output: 8422
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== docker compose files =="
git ls-files | rg '(^docker-compose\.ya?ml$|(^|/)compose/.*\.ya?ml$)'
echo "== gateway/updater definitions in base compose =="
for f in docker-compose.yml docker-compose.e2e.yml; do
[ -f "$f" ] || continue
echo "--- $f ---"
rg -n "gateway|updater|profiles:" "$f" -A 8 -B 2 || true
done
echo "== all gateway/updater occurrences =="
rg -n "^\s*gateway:\s*$|^\s*updater:\s*$|profiles:" deploy docker-compose.yml docker-compose.e2e.yml -A 5 -B 2 || trueRepository: BaseIntelligence/base
Length of output: 15895
Remove stale gateways during validator redeploy.
docker compose ... without --profile master leaves an existing gateway container running; deselecting the master profile does not remove it. The validator cleanup now only removes challenges, so a master-to-validator role switch can leave gateway serving on the validator host. Remove gateway from the validator cleanup path and fail if it remains afterward.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@deploy/scripts/remote-deploy.sh` around lines 422 - 425, Update the validator
cleanup block in the role check to include gateway alongside the existing
challenge and proxy services passed to docker compose rm. Ensure the cleanup
does not suppress failure when gateway remains, so the redeploy fails if the
stale gateway container cannot be removed.
Source: Coding guidelines
| docker compose ${COMPOSE_FILES[*]} ${PROFILE_ARGS[*]} rm -sf validator \ | ||
| >/dev/null 2>&1 || true |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -u
echo "== locate file =="
git ls-files | rg '(^|/)remote-deploy\.sh$' || true
echo "== file stats =="
wc -l deploy/scripts/remote-deploy.sh
echo "== relevant lines 380-460 =="
sed -n '380,460p' deploy/scripts/remote-deploy.sh | cat -n -v | sed 's/^/L/'
echo "== search validator/profile/compose refs =="
rg -n "validator|base-validator-1|base-prod-validator|profile|PROFILE_ARGS|COMPOSE_FILES|docker compose ps|profile:|profiles:" deploy -SRepository: BaseIntelligence/base
Length of output: 50377
🏁 Script executed:
#!/bin/bash
set -u
echo "== compose files and profiles =="
fd -e yml -e yaml . deploy | sort | xargs -r rg -n "name:|services:|profiles:|profile:" -S || true
echo "== validator occurrences in repo =="
rg -n "\bvalidator\b|base-validator-1|base-prod-validator|ROLE|role-master" deploy -SRepository: BaseIntelligence/base
Length of output: 50377
🏁 Script executed:
#!/bin/bash
set -u
echo "== remote-deploy master profile and validator check =="
sed -n '236,260p;416,448p' deploy/scripts/remote-deploy.sh | cat -n
echo "== master role compose =="
cat -n deploy/compose/role-master.yml
echo "== env compose validator sections =="
for f in deploy/compose/env-prod.yml deploy/compose/env-staging.yml deploy/compose/env-local.yml; do
echo "--- $f ---"
cat -n "$f" | sed -n '1,70p'
done
echo "== script error handling/exit behavior around absent validator message =="
rg -n "set -[ef]|trap|main|function main|\\}# exit|exit 0|echo \"remote-deploy: done|echo \"master: validator absent" deploy/scripts/remote-deploy.shRepository: BaseIntelligence/base
Length of output: 16582
🌐 Web query:
Docker Compose profiles ps --status running includes containers of inactive services
💡 Result:
The docker compose ps command is not currently profile-aware, meaning it does not automatically filter its output to match the profiles specified in the command [1][2]. Even if you execute docker compose --profile ps, the command will typically list all containers defined in your project, regardless of whether those containers belong to an active profile or an inactive one [1][2]. This behavior is a known limitation of the docker compose ps command, which was designed to maintain backward compatibility by listing all containers associated with the project [2]. While you can use the --status flag to filter by the state of the containers (e.g., --status running), this does not implicitly filter out containers based on their configured profiles [3][4]. If you need to limit the output to specific containers, you may need to rely on alternative methods such as using standard CLI tools (e.g., piping to grep) or specifically targeting the service names directly in the command (e.g., docker compose ps <service_name>), as the project continues to prioritize compatibility over adding profile-aware filtering to the ps command [2][5].
Citations:
- 1: [BUG] docker compose ps ignores profiles docker/compose#12361
- 2: [BUG] profiles does not affect ps docker/compose#11737
- 3: https://github.com/docker/compose/blob/main/docs/reference/compose_ps.md
- 4: https://docs.docker.com/reference/cli/docker/compose/ps/
- 5: Profile filter for docker-compose ps docker/compose#10312
Fail closed on validator removal and inspection.
rm -sf validator >/dev/null 2>&1 || true can hide a failed cleanup, and a later Compose/Docker failure can leave a stale dual-submit base-validator-1 container. Make the master remove the inactive validator container and exit loudly if removal or the subsequent absence check fails.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@deploy/scripts/remote-deploy.sh` around lines 427 - 428, Update the validator
cleanup flow around the docker compose rm command to fail closed: remove the
unconditional “|| true” and suppressed output, ensure removal errors are
reported and terminate the master, then perform an explicit absence check for
the inactive validator container and likewise exit loudly if it still exists.
Preserve the existing validator removal target and use the surrounding
deployment error-handling conventions.
Source: Coding guidelines
1 participant
Summary
base-validator-1alongsidebase-prod-validator, both with the5Gzi…wallet — dual CRV4 submitters fightingWeightsSetRateLimit.neverinrole-master.yml;env-local.ymlre-enables co-located validator for local-e2e only.remote-deploy.sh --role masterforce-removes any leftover validator and fails the deploy if it is still running.Test plan
./deploy/scripts/assert-compose-matrix.shbase-validator-1on prod master206.189.224.155; sole submitter remains192.81.218.11master: validator absentand nobase-validator-1[94,102,214](not 236 monopoly)Summary by CodeRabbit
New Features
Bug Fixes
Documentation