Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Data Synchronization/Matching] Delegate to Spark for checking existence of columns in the given dataframes #515

Merged
merged 1 commit into from
Oct 27, 2023
Merged

[Data Synchronization/Matching] Delegate to Spark for checking existence of columns in the given dataframes #515

merged 1 commit into from
Oct 27, 2023

Commits on Oct 26, 2023

  1. [Data Synchronization/Matching] Delegate to Spark for checking existe… 

    …nce of columns in the given dataframes

- Prior to this change, we were doing case sensitive equality checks of non-key columns.
- This makes the utility more restrictive, as Spark does not care about the casing of column names.
- With this change, we rely on Spark to check if a column exists in the given dataframe. If Spark can find the column, we can proceed with the rest of the check.
    @rdsharma26
    rdsharma26 committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    21be8d2 View commit details
    Browse the repository at this point in the history