Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: introduce CDC write-side support for the Update operations #2486

Merged
merged 6 commits into from
Jun 4, 2024
Merged

feat: introduce CDC write-side support for the Update operations #2486

merged 6 commits into from
Jun 4, 2024

Commits on Jun 4, 2024

  1. feat: introduce CDC write-side support for the Update operations 

    This change introduces a `CDCTracker` which helps collect changes during
merges and update. This is admittedly rather inefficient, but my hope is
that this provides a place to start iterating and improving upon the
writer code

There is still additional work which needs to be done to handle table
features properly for other code paths (see the middleware discussion we
have had in Slack) but this produces CDC files for Update operations

Fixes delta-io#604
Fixes delta-io#2095
    @rtyler @ion-elgreco
    rtyler authored and ion-elgreco committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    3fad049 View commit details
    Browse the repository at this point in the history

  2. chore: add a Python integration test 

    This test has highlighted an apparent race condition when handling
structs or lists in how excerpt() is treated by the CDCObserver.
    @rtyler @ion-elgreco
    rtyler authored and ion-elgreco committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    6f52365 View commit details
    Browse the repository at this point in the history

  3. Ignore the some CDC related integration tests until upstream works 

    See apache/datafusion#10749
    @rtyler @ion-elgreco
    rtyler authored and ion-elgreco committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    03b7bb0 View commit details
    Browse the repository at this point in the history

  4. chore: provide a warning if the schema contains fields which break CDC 

    See delta-io#2568
    @rtyler @ion-elgreco
    rtyler authored and ion-elgreco committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    a3b2600 View commit details
    Browse the repository at this point in the history

  5. Allow writers lower than minWriterVersion 7 to be gracefully supported 

    Basically for older minWriterVersions we don't have to really worry
about generated columns unless an expression has been set, in which case
we must fail to write since we cannot honor generationExpression
    @rtyler @ion-elgreco
    rtyler authored and ion-elgreco committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    87c01cc View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into cdc-update-only

    @rtyler
    rtyler committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    cbcd819 View commit details
    Browse the repository at this point in the history