fix(bot-kit): confirm mined replacements and fix premature stuck bumps - #98
Closed
cashd wants to merge 4 commits into
Closed
fix(bot-kit): confirm mined replacements and fix premature stuck bumps#98cashd wants to merge 4 commits into
cashd wants to merge 4 commits into
Conversation
The pending queue overwrote entry.txHash on fee replacement, so receipt checks and nonce reconciliation only ever saw the latest hash. When the ORIGINAL broadcast mined after a replacement was sent, the queue misreported tx.dropped: nonce_consumed instead of tx.confirmed (production, July 17, Base nonce 236). It also aged transactions from the chain head the tick captured BEFORE quoting, so a slow (~10s) quote made a fresh broadcast look >STUCK_BLOCKS old and triggered an immediate, pointless fee bump. - Track every hash broadcast for a nonce; scan them all (newest first) before replacing or classifying the nonce as consumed, and settle with the actually-mined hash and block. - Emit tx.dropped: nonce_consumed only when the nonce is consumed and none of our hashes has a receipt. - Measure stuck-age from the first onBlock that observes the broadcast; drop the pre-quote blockNumber from queue.submit (both bots updated). Fixes CRTR-2851
Collaborator
Author
|
Simplify pass (140002c):
|
Collaborator
Author
|
Review pass (244a2bd):
Two things surfaced but intentionally not changed: the reconciler settling a mined tx now applies the settled-cooldown (previously the nonce_consumed drop skipped it) — that's desired, a genuinely-mined tx should cool its label down; and the multi-hash sweep costs up to attempt+1 getReceipt reads per stuck entry per block, bounded by maxBumpAttempts — fine for now, can revisit if it shows up in RPC budgets. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Production incident (July 17, Base, midnight-liquidation, nonce 236):
0xaedfb547…73ccawas broadcast at 22:35:24.326 UTC and mined successfully in block 48,769,189.0xc95d215a…6dbee, which never landed.tx.droppedwithreason: nonce_consumedinstead oftx.confirmed.Two defects combined:
submittedAtBlock. Quoting took ~10 s (~5 Base blocks), so the tx was born already> STUCK_BLOCKSold and got fee-bumped immediately — the 286 ms replacement.pending-queue.tskept only the latest hash per nonce; receipt checks and the nonce-consumed reconciler never looked at earlier broadcasts. When the original hash mined, the queue had forgotten it and misclassified the settlement asnonce_consumed.Fix
Pending.txHash→txHashes[]: every hash broadcast for a nonce is retained; the last one remains the fee-replacement target (snapshot/logs still expose the latest hash).tx.confirmed/tx.revertedwith the actually-mined hash and block;tx.dropped: nonce_consumedfires only when the nonce is consumed and none of our hashes has a receipt.onBlockthat observes the broadcast (observedAtBlock), not from the tick's pre-quote head.queue.submitno longer takesblockNumber; both liquidation bots updated.Tests
packages/bot-kit/test/queue/pending-queue.test.ts(40 pass):nonce_consumedonly when no tracked hash has a receiptPer repo convention, one assertion was temporarily broken to confirm the suite fails, then restored.
Validation:
@repo/bot-kit+ both bots typecheck clean,bun lint0 warnings,bun formatclean,bun testgreen except pre-existing environment-dependent failures (midnight fork suite needsRPC_URL_8453+ anvil ≥1.5 — local anvil is 1.4.4, the version the CI comment documents as rejecting same-nonce replacements; crossed-books failures reproduce identically onorigin/main).Unresolved operational finding (documented, not fixed)
A second liquidation tx at nonce 237 (
0x0017b634…d86fe) from the same signer repeated the liquidation in the next block, but neither the Midnight nor Blue Better Stack source has a correspondingtx.sent. The code offers no unlogged send path: the signer performs exactly onesendTransactionpersend, and the queue logstx.sentsynchronously after every successful first-send. Two consistent-but-unproven explanations: (a)drop()deliberately skips the settle cooldown, so after the incorrectnonce_consumeddrop the position was immediately re-eligible and a normal re-submit could have fired whosetx.sentline was lost by the opt-in in-process BetterStack shipper; (b) an external send from the same key. No fix is included because the code provides no concrete evidence of the source.Fixes CRTR-2851 — https://linear.app/morpho-labs/issue/CRTR-2851/fixbot-kit-confirm-mined-replacements-and-fix-premature-stuck-bumps