Skip to content

fix(snowflake): deduplicate outcome-unknown truncates - #960

Merged
farazdagi merged 1 commit into
mainfrom
fix/snowflake-sql-request-id
Aug 6, 2026
Merged

fix(snowflake): deduplicate outcome-unknown truncates#960
farazdagi merged 1 commit into
mainfrom
fix/snowflake-sql-request-id

Conversation

@farazdagi

Copy link
Copy Markdown
Contributor

Context

The truncate replay recovery from #957 can resubmit a Snowflake TRUNCATE after losing its SQL API outcome. Without a stable request ID, an earlier request could finish after replay restored later rows and erase them.

A concrete failing sequence is:

  1. A source transaction contains INSERT (1, 'before'), TRUNCATE, then INSERT (2, 'after').
  2. ETL submits the truncate as request R1. Snowflake accepts it, but ETL loses the response while R1 may still be executing, so the source checkpoint does not advance.
  3. On restart, the recovery from #957 replays the transaction. Without a stable requestId, ETL submits the same truncate as a separate request R2.
  4. R2 completes first. Replay inserts (2, 'after') and advances the channel offset past that insert.
  5. R1 then completes late and clears the table again.

The destination is now empty even though its channel offset records (2, 'after') as applied, so normal recovery will not restore the missing row.

This PR gives retries of the same fenced truncate attempt the same Snowflake requestId, causing the retry to reconcile R1 instead of scheduling R2. A genuinely new truncate attempt still receives a new ID.

Changes

  • Derive a deterministic UUIDv5 from the fenced physical truncate attempt.
  • Reuse its requestId for the same unknown attempt, while deriving a new ID after channel progress. This follows Snowflake’s documented SQL API resubmission protocol: resubmit the same outcome-unknown truncate with its original requestId and retry=true, while deriving a new requestId after channel progress proves that a genuinely new truncate attempt is required.
  • Process each source TRUNCATE separately, resolving an earlier request with an unknown outcome before moving to a later one.
  • Give ordinary DDL invocations UUIDv4 IDs reused by their internal HTTP retries. I will explore the possibility of some corner-cases failure modes for those other DDLs and add follow-up PR, if anything is discovered

Validation

Added identity and boundary unit tests, a credentialed cross-client idempotency test, and multi-cycle replay coverage proving post-truncate rows are restored exactly once.

@farazdagi farazdagi self-assigned this Aug 6, 2026
@farazdagi
farazdagi force-pushed the fix/snowflake-sql-request-id branch from 356d497 to ac04aae Compare August 6, 2026 08:57
@farazdagi
farazdagi marked this pull request as ready for review August 6, 2026 09:24
@farazdagi
farazdagi requested a review from a team as a code owner August 6, 2026 09:24

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No bugs found, but this PR introduces a new correctness-critical protocol (deterministic UUIDv5 request IDs tied to channel lineage/row-progress) for reconciling outcome-unknown Snowflake truncates, plus a behavioral change to process each truncate boundary separately instead of collapsing consecutive truncates to the latest offset. This is exactly the kind of subtle distributed-recovery design decision that benefits from a human sign-off before merging.

Extended reasoning...

Overview

Modifies the Snowflake destination truncate path: derives a deterministic UUIDv5 requestId from a physical truncate attempt's identity (pipeline/table/db/schema/table name/truncate offset/channel creation timestamp/rows-inserted baseline), reuses it across retries of the same physical attempt, and issues a fresh ID once channel progress proves a new attempt is required. Also changes apply_truncate_events to process each consecutive Truncate event as its own boundary (via take_truncate_operations) instead of collapsing to only the latest offset per table, and threads an explicit request_id: Uuid through SqlClient::truncate_table/execute_ddl. Adds unit tests for the identity encoding (stable hash, per-component sensitivity, length-delimited framing) and extends the credentialed integration test to cover a second replay cycle.

Security risks

None identified. No user input reaches SQL string interpolation beyond existing patterns (table/column identifiers are already quoted elsewhere); the new requestId is a UUID placed in a query string, not attacker-controlled data. No auth/permission changes.

Level of scrutiny

This touches a production-critical, correctness-sensitive recovery path (Snowflake truncate replay after a lost SQL API outcome) and encodes a new "durable protocol" that future changes must not silently break (per the code's own comment: changing the namespace or identity encoding would break resumption for in-flight recoveries). That combination — subtle idempotency/fencing logic, an implicit forward-compatibility contract, and a change to how consecutive truncates are batched — is more than a mechanical fix and merits a human confirming the reasoning (e.g., that rows_inserted/channel_created_on_ms observed via open_at before the SQL call are the right fencing signal, and that per-boundary truncate processing doesn't regress throughput unacceptably for tables with many consecutive truncates).

Other factors

The PR has solid test coverage (deterministic-encoding unit tests, a credentialed cross-client idempotency test, and a two-cycle replay integration test), and the author's PR description shows the reasoning was worked through carefully, tying the design to Snowflake's documented resubmission protocol. The bug-hunting pass found nothing, and my own reading of the identity derivation and channel-progress invariants didn't surface a correctness gap. The main reason to defer is the design/critical-path judgment call, not a specific defect.

@farazdagi
farazdagi merged commit dd2503c into main Aug 6, 2026
25 checks passed
@farazdagi
farazdagi deleted the fix/snowflake-sql-request-id branch August 6, 2026 09:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants