io,ci: drain re-signals BLOCKED frames + auto-retry flaky transfer gate#244
Merged
Merged
Conversation
…fer gate (#229) Two related interop-robustness fixes in one PR. ## Lost-grant wedge + skipped delivery-budget test (#231) The fresh-send path emits STREAM_DATA_BLOCKED / DATA_BLOCKED once when a write first gates, but BLOCKED frames are not retransmitted on loss — only STREAM data has retransmit machinery. If the peer's window-GRANT datagram (MAX_STREAM_DATA / MAX_DATA) is lost after the payload was already accepted into `pending_stream_sends`, an idle app never calls the fresh-send path again, the drain silently skips the closed-window entries, and the transfer wedges forever. That kernel-loopback drop under CI load is exactly why the recv-delivery-budget socket-loopback test stalled intermittently on Linux and was skipped. Fix: the pending-send drain (server + client) now re-signals STREAM_DATA_BLOCKED / DATA_BLOCKED from its flow-control skip branches, rate-limited to one per 250 ms per conn (`blocked_signal_interval_ms`, `ConnState.blocked_signal_last_ms`). A lost grant self-heals: the peer answers the re-signal with a fresh MAX_(STREAM_)DATA, whose frame arm already re-drains. - Re-enabled the skipped test (`raw-app recv delivery budget` loopback). - New wedge-recovery test: payload placed directly into `pending_stream_sends` with conn credit exhausted (the exact lost-grant state — fresh-path emission bypassed); without the re-signal it wedges forever, with it the transfer completes via DATA_BLOCKED → MAX_DATA → re-drain. This closes the remaining item of #231 (recv round-robin + the credit-invariant test landed earlier in 57e1912). ## Flaky transfer gate auto-retry (#229) The bulk `transfer` testcases intermittently stall on loaded runners — a DIFFERENT transfer flakes each run on behavior-identical commits. CI now: - After the three run.py suites, re-runs ONLY failed `transfer` cases once per suite (into `results-<suite>-retry.json` / `logs-<suite>-retry`). cross-zquic-quinn is excluded — its transfer failure is already KNOWN_FAILING (#184). - The summary gate classifies a base-failed + retry-passed transfer as FLAKY-PASSED (reported, non-fatal). A real regression fails both attempts and still reds the gate; retry artifacts upload with the rest. Gate logic simulated both ways (flaky-passed → exit 0; double-fail → exit 1); retry detector probed on synthetic results; workflow YAML, step bash, and embedded python all syntax-checked. 301/301 unit tests pass (was 299 + 1 skip: the skip is re-enabled and one new test added), stable across 5 repeat runs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #231. Closes #229.
#231 — lost-grant wedge fix + re-enabled delivery-budget test
Root cause of the skipped test's Linux-CI stall: BLOCKED frames are emitted once by the fresh-send path and never retransmitted on loss (only STREAM data has retransmit machinery). When the peer's window-GRANT datagram (
MAX_STREAM_DATA/MAX_DATA) is dropped — kernel loopback drop under CI load — after the payload was already accepted intopending_stream_sends, an idle app never re-triggers the fresh-send path, the drain silently skips the closed-window entries, and the transfer wedges forever. This is a real production wedge class (fully-queued reqresp response + app stops writing), not just a harness artifact.Fix: the pending-send drain (server + client) re-signals
STREAM_DATA_BLOCKED/DATA_BLOCKEDfrom its flow-control skip branches, rate-limited to one per 250 ms per conn. The peer answers with a fresh grant; theMAX_DATA/MAX_STREAM_DATAframe arms already re-drain.raw-app recv delivery budgetsocket-loopback test (the last open item of recv delivery budget (v1.7.63): follow-ups — recv-side round-robin + re-enable skipped loopback test #231 — recv round-robin + the credit-invariant test landed in 57e1912).Build and Testjob on this very PR exercises the re-enabled test on Linux.#229 — flaky transfer auto-retry in the interop gate
transfercases once per suite intoresults-<suite>-retry.json/logs-<suite>-retry.cross-zquic-quinnexcluded (its transfer is alreadyKNOWN_FAILING, No per-stream send buffer; partial-frame retransmit pushed onto embedder #184).FLAKY-PASSED, non-fatal. A real regression fails both attempts and still reds the gate. Retry logs/json upload with the existing artifact.The #231 drain re-signal plausibly also reduces the #229 flake rate at the source — the same-impl transfer stalls look like exactly this wedge under runner load — but the retry keeps the gate honest either way.