Skip to content

fix(release): correct two real bugs found in the just-shipped reconciliation job#7152

Merged
JSONbored merged 1 commit into
mainfrom
fix/reconcile-stale-run-detection
Jul 18, 2026
Merged

fix(release): correct two real bugs found in the just-shipped reconciliation job#7152
JSONbored merged 1 commit into
mainfrom
fix/reconcile-stale-run-detection

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

The reconcile-stale-releases job (#7149) shipped with two real bugs, both found by actually running it live against a real release cycle rather than trusting its "success" conclusion at face value.

Bug 1 -- stale run-ID detection. dispatch_and_wait() used gh run list --limit 1 immediately after gh workflow run to find the just-dispatched run's ID. Unreliable: the new run isn't always indexed by the API yet at that moment, so it can return a DIFFERENT, already-completed prior run instead. That's exactly what happened: it grabbed a stale run from an earlier manual dispatch, reported that run's old "success", and silently never published the new version at all (packages/loopover-engine committed 3.2.3 while npm still had 3.2.2, undetected until manually reading the actual log content).

Fixed by parsing the run ID directly from gh workflow run's own stdout (it prints the created run's URL), with a timestamp-filtered gh run list fallback for the rare case that URL isn't returned. Verified against several real dispatches before shipping this time.

Bug 2 -- npm registry propagation race. mcp/miner's own isolated pack/smoke-test resolves @loopover/engine from the real npm registry in a bare temp dir (no local workspace symlink there), and can transiently ETARGET for a short window after engine's own publish job reports success -- npm's registry is CDN-backed, not instantly consistent across every edge node. Confirmed live: mcp and miner dispatched at the same moment against the same already-published engine version, mcp's smoke-test hit an already-propagated edge node and succeeded, miner's hit one that hadn't and failed with the exact ETARGET this whole job exists to fix.

Added a settle delay after engine publishes, plus automatic retry (with backoff) for its dependents specifically -- this is registry propagation lag, not a genuine test regression (which would fail identically on every retry instead of resolving itself).

Test plan

  • npm run actionlint -- clean
  • Extracted both run: scripts, bash -n syntax-checked
  • Ran the corrected dispatch_and_wait run-ID-capture logic for real against a live (safely cancelled) dispatch -- confirmed it resolves the fresh run, not a stale one
  • Ran the full reconciliation script for real against this repo's current live state (all 4 packages now actually in sync at 3.2.3, after manually completing the release fix(release): self-heal the release-please "untagged, merged" abort #7149's bug got stuck on) -- correctly reports everything in sync and dispatches nothing
  • Ran the label-cleanup script for real -- correctly finds zero merged autorelease: pending PRs (already manually cleared)
  • Workflow-only change; not measured by Codecov.

…liation job

The reconcile-stale-releases job (#7149) shipped with two real bugs,
found by actually running it live rather than trusting its "success"
conclusion:

1. dispatch_and_wait() used `gh run list --limit 1` right after `gh
   workflow run` to find the just-dispatched run's ID -- unreliable, since
   the new run isn't always indexed by the API yet. It grabbed a STALE,
   already-completed prior run instead, reported that run's old "success",
   and silently never published the new version at all (confirmed live:
   packages/loopover-engine committed 3.2.3 while npm still had 3.2.2,
   undetected until manually checking the actual log content -- exactly
   the kind of unverified-success this whole session has been about not
   repeating). Fixed by parsing the run ID directly from `gh workflow
   run`'s own stdout (which prints the created run's URL), with a
   timestamp-filtered `gh run list` fallback for the rare case that URL
   isn't returned. Verified this against several real dispatches before
   shipping.

2. mcp/miner's own isolated pack/smoke-test resolves @loopover/engine
   from the real npm registry (no local workspace symlink in that bare
   temp dir), and can transiently ETARGET for a short window after
   engine's own publish reports success -- npm's registry is CDN-backed,
   not instantly consistent across every edge node (confirmed live:
   mcp and miner dispatched at the same moment against the same
   already-published engine version, mcp's smoke-test hit a
   already-propagated edge node and succeeded, miner's hit one that
   hadn't and failed with the exact ETARGET this job exists to fix).
   Added a settle delay after engine publishes plus automatic retry
   (with backoff) for its dependents -- this is registry propagation
   lag, not a genuine test regression, which would fail identically on
   every retry instead of resolving itself.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.75%. Comparing base (1bd3d5f) to head (b36ea6c).
⚠️ Report is 2 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #7152   +/-   ##
=======================================
  Coverage   93.75%   93.75%           
=======================================
  Files         692      692           
  Lines       68832    68832           
  Branches    18781    18781           
=======================================
  Hits        64535    64535           
  Misses       3302     3302           
  Partials      995      995           
Flag Coverage Δ
shard-1 43.75% <ø> (ø)
shard-2 36.91% <ø> (ø)
shard-3 33.14% <ø> (ø)
shard-4 34.44% <ø> (ø)
shard-5 32.27% <ø> (+<0.01%) ⬆️
shard-6 45.90% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

@JSONbored
JSONbored merged commit 82b39f0 into main Jul 18, 2026
17 checks passed
@JSONbored
JSONbored deleted the fix/reconcile-stale-run-detection branch July 18, 2026 01:40
@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant