feat(core): Improve escaping in destinations and rework state structure#771
Merged
Conversation
farazdagi
approved these changes
May 27, 2026
farazdagi
left a comment
Contributor
There was a problem hiding this comment.
Snowflake destination definitely looks nicer with that sql.rs extracted.
bnjjj
approved these changes
May 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR improves destination SQL escaping/quoting, makes replica-identity failures easier to classify and remediate, and cleans up table state naming across core ETL and the Postgres-backed persistence layer.
Changes
Destination SQL quoting and escaping
bigquery/sql.rsfor GoogleSQL identifier/path quoting.clickhouse/sql.rsfor ClickHouse identifier quoting.ducklake/sql.rsfor DuckDB/DuckLake identifier quoting andlakecatalog qualification.snowflake/sql.rsfor Snowflake identifiers and string literals.sql.rsfor shared ANSI double-quoted identifier escaping.BigQueryClientand reuses it for fully qualified table paths,INFORMATION_SCHEMA.TABLESpaths, column DDL, and primary key clauses.quote_postgres_identifier, separating Postgres metadata SQL from DuckDB/DuckLake SQL.SHOW TERSE TABLES ... STARTS WITH ...plus exact row matching, with centralized string literal quoting.Replica identity error classification
ErrorKind::SourceReplicaIdentityError.REPLICA IDENTITY DEFAULTwith a primary key.REPLICA IDENTITY USING INDEXwhen supported.REPLICA IDENTITY FULLonly when full row images are needed.Table state structure and naming
etl::statemodule so table-specific state lives directly understate/:state/destination_table_metadata.rs.state/table_error.rs.state/table_retry_policy.rs.state/table_state.rs.TableState.TableStateType.TableError.TableRetryPolicy.etl::state::{TableState, TableStateType, TableError, TableRetryPolicy}instead of a nested table module.StateStoretable APIs to state-oriented names:get_table_state.get_table_states.load_table_states.update_table_state.update_table_states.rollback_table_state.TableReplicationStatestoTableStates.destination_metadata->destination_table_metadata.state->table_state.TableStateRowrepresents the database row.StoredTableStateTyperepresents the database enum value.TableState::to_storage_format()andTableState::from_state_row(...).get_table_state_rows,update_table_state_raw,rollback_table_state,reset_table_state,delete_table_state(s)) while exposing core ETL state values to the rest of the pipeline.TableErrorso it stores replication failure details and the source error without carrying a table id.Hashsupport and conversions forTableStateType.TableStateType-keyed counts, converting to the static metric label only at emission time.phasetostate.SimpleTableState,TableStatus,MissingTableState, andInvalidTableState.TableStateRowmetadata into coreTableStatevalues.Migration and persistence notes
origin/main; the table-state rename is code-level and does not require a database migration.type,lsn,reason,solution, andretry_policy).source_errmemory-only and skipped from persisted status JSON, so newly added or removed internalErrorKindvariants are not stored in table state.AGENTS.mdrule that migration files, including comments and non-executable text, must not be modified unless explicitly requested.