Fix flaky loadgen failure in MissionDatabaseInplaceUpgrade#410
Merged
Conversation
The after-upgrade node is a watcher outside the quorum. When its catchup finishes applying buffered ledgers it flips to LM_SYNCED_STATE and reports "Synced!" even if the validators have since moved on (they close 1 ledger/sec with accelerated time, and the catchup replay takes long enough for them to get ~40 ledgers ahead). WaitUntilSynced only polls the state string, so it can return during that transient window; the mission then runs loadgen against a node whose local ledger is stalled until its next catchup round, and core's loadgen times out (~20s without local ledger closes) and reports the run failed. Add Peer.WaitUntilCaughtUpWith, which waits until a node reaches the ledger a reference peer has already closed, and use it in the mission to make sure the upgraded node is actually tracking the live network before generating load on it. MissionSorobanCatchupWithPrevAndCurr already uses the same WaitForLedgerNum idiom for its catchup node.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds synchronization safeguards before running loadgen on an upgraded watcher node.
Changes:
- Adds a peer catchup helper.
- Uses it after the database upgrade synchronization step.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
StellarCoreHTTP.fs |
Adds peer catchup waiting logic. |
MissionDatabaseInplaceUpgrade.fs |
Waits before starting loadgen. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Sampling the reference ledger once is not enough if the node needs more than two catchup rounds: a second round's catchup target necessarily lies beyond the sampled ledger, so the wait would pass even if that round also landed behind the live tip and the node stalled again. Re-reading the reference peer's ledger on each poll and requiring a small gap only passes once the node is genuinely tracking the network, regardless of how many rounds that takes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
MissionDatabaseInplaceUpgradeintermittently fails withLoadgen failed!on theafter-upgradenode.That node is a watcher outside the quorum, started mid-run from a DB snapshot, so it catches up while the validators keep closing 1 ledger/sec. When a catchup round finishes, core reports
Synced!even if the network has since moved on (in the failing runssc-2109z-39a050: node at ledger 167, validators at 206), and keeps reporting it while stalled waiting for the next round.WaitUntilSyncedonly polls the state string, so the mission starts loadgen against a node whose local ledger isn't advancing, and loadgen times out after ~20s. Whether the last catchup round lands on the live tip or behind it is timing luck — hence the flake.Fix
Add
Peer.WaitUntilCaughtUpWith(other)and call it afterWaitUntilSyncedon the after-upgrade node, with acorevalidator as reference. It waits until the node is within a couple of ledgers of the reference peer's latest, re-read on every poll so it holds across however many catchup rounds are needed. This generalizes theWaitForLedgerNum-against-a-live-peer idiomMissionSorobanCatchupWithPrevAndCurralready uses into a reusable helper.Testing
dotnet buildclean,dotnet test15/15,fantomas --checkclean.