Skip to content

Survive failed dependent/access batches instead of aborting the extract - #336

Merged
jp-spotter merged 1 commit into
thoughtspot:devfrom
jp-spotter:fix/dependent-fetch-resilience
Aug 4, 2026
Merged

Survive failed dependent/access batches instead of aborting the extract#336
jp-spotter merged 1 commit into
thoughtspot:devfrom
jp-spotter:fix/dependent-fetch-resilience

Conversation

@jp-spotter

@jp-spotter jp-spotter commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

A single dependent-object request can always blow past the read timeout -- dependents cannot currently be paginated (TS REST API limitation; enhancement on the roadmap). Today that one failure cancels every sibling request and throws away the whole extract. Rather than trying to prevent the unpreventable (the #324/#325 mistake, reverted in #326), this makes the extract survive it: finish with what succeeded, say exactly what is missing, never overwrite good data with an incomplete set.

  • The fetch and permissions workflows no longer fan out under the fail-fast TaskGroup; they gather with bounded concurrency and per-request outcomes, so a batch that fails after retries is skipped and reported, not fatal. Fetch already had partial-progress handling for exactly this -- the TaskGroup cancelled past it before it could run. The fetch_all workflow stays fail-fast on purpose, since it feeds everything downstream.
  • Retry exhaustion on 429/502/503/504 is unwrapped to the final response, so the recorded failure is a legible status error rather than an opaque RetryError.
  • Failures are loud without being noisy: a few detailed ERROR lines plus a running tally (a 7,000-failure storm no longer prints 7,000 lines), a per-phase warning with affected object counts, and an end-of-run summary grouped by type. Exit code is non-zero whenever gaps exist.
  • TRUNCATE loads are refused on gaps, so an incomplete extract never replaces good data. APPEND and UPSERT proceed, and the next run fills the gaps.
  • A run with no failures behaves exactly as before.

Tests drive the real workflows through a production client over an in-memory mock transport: sibling survival, collector accuracy, retry exhaustion, the console cap, and non-transport bugs still propagating. Validated live on a large multi-org cluster with fault injection -- partial extract completes, TRUNCATE refused with the target untouched, UPSERT lands the partial rows.

@jp-spotter
jp-spotter requested a review from a team as a code owner August 4, 2026 00:52
@jp-spotter
jp-spotter merged commit bc94d74 into thoughtspot:dev Aug 4, 2026
3 checks passed
@jp-spotter
jp-spotter deleted the fix/dependent-fetch-resilience branch August 4, 2026 00:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant