Skip to content

fix(selfhost): surface failed orb relay drains - #3785

Merged
JSONbored merged 1 commit into
mainfrom
codex/propose-fix-for-orb-drain-progress-issue
Jul 6, 2026
Merged

fix(selfhost): surface failed orb relay drains#3785
JSONbored merged 1 commit into
mainfrom
codex/propose-fix-for-orb-drain-progress-issue

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Motivation

  • The pull-mode drain path treated any resolved drain result (including []) as proof of broker progress, but the production drainOrbRelay collapsed non-OK HTTP responses and caught errors into [], making broker failures indistinguishable from successful empty polls.
  • This caused the no-progress monitoring gauge/alert to be falsely healthy during broker outages, delaying operator detection of a stuck pull-mode relay.

Description

  • Change drainOrbRelay in src/orb/broker-client.ts to throw on non-OK broker responses and propagate thrown/rejected errors instead of returning [], while preserving the [] no-op when not in broker mode.
  • Add unit tests in test/unit/orb-broker-client.test.ts to assert the new drainOrbRelay behavior for non-OK responses, thrown errors, and unsafe URLs.
  • Add a regression test in test/unit/selfhost-monitored-work.test.ts that wires the real drainOrbRelay through drainOrbRelayWithMonitor and verifies a 503 broker response does not stamp lastDrainAtMs or clear pending acks.

Testing

  • Ran npm run typecheck and it completed without type errors.
  • Ran the targeted unit tests with npm test -- --run test/unit/orb-broker-client.test.ts test/unit/selfhost-monitored-work.test.ts and all tests passed (55 tests across the two files).
  • Attempted a full unsharded coverage run with npm run test:coverage; the full coverage job was observed to take an extended time and was terminated to keep the submit cycle bounded, so a complete unsharded coverage report was not included in this PR submission.

Codex Task

@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored JSONbored self-assigned this Jul 6, 2026
@JSONbored JSONbored added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 6, 2026
@codecov

codecov Bot commented Jul 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.29%. Comparing base (de375d3) to head (a4d5633).
⚠️ Report is 7 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3785   +/-   ##
=======================================
  Coverage   93.29%   93.29%           
=======================================
  Files         317      317           
  Lines       32368    32369    +1     
  Branches    11872    11873    +1     
=======================================
+ Hits        30199    30200    +1     
  Misses       1537     1537           
  Partials      632      632           
Files with missing lines Coverage Δ
src/orb/broker-client.ts 99.11% <100.00%> (+<0.01%) ⬆️
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JSONbored
JSONbored merged commit c3bdb8b into main Jul 6, 2026
11 checks passed
@JSONbored
JSONbored deleted the codex/propose-fix-for-orb-drain-progress-issue branch July 6, 2026 09:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant