Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove HiveConfig binding from Delta and Iceberg with Glue #19735

Merged
merged 4 commits into from
Nov 15, 2023
Merged

Remove HiveConfig binding from Delta and Iceberg with Glue #19735

merged 4 commits into from
Nov 15, 2023

Commits on Nov 14, 2023

  1. Fix typo

    @findepi
    findepi committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    c4fca0b View commit details
    Browse the repository at this point in the history

  2. Simplify MetastoreHiveStatisticsProvider code flow 

    Before the change `MetastoreHiveStatisticsProvider` was both abstract
and concrete class in one: one constructor allowed to plug
implementation of the most important method, while the other provided
the implementation in a form of a lambda. This made harder to follow the
code flow (e.g. finding call sites). The commit splits the class into
abstract class (shared logic) and a concrete implementation class.
    @findepi
    findepi committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    b5238a0 View commit details
    Browse the repository at this point in the history

  3. Respect Hive statistics_enabled session in Glue metastore 

    Before the change, the Hive Glue metastore wiring would respect the
`hive.table-statistics-enabled` configuration property, but not the
`statistics_enabled` session property.

After the change, handling of stats for Hive Glue metastore is same as
for other metastores. That means in particular that some of the
`DefaultGlueColumnStatisticsProvider` code will execute implicitly, as
not all stats-related operations are guarded by `statistics_enabled`
toggle today (see e.g. `SemiTransactionalHiveMetastore` and implicit
stats updates after table creation or alteration). This is considered
not a blocker.
    @findepi
    findepi committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    2fe1d20 View commit details
    Browse the repository at this point in the history

  4. Remove HiveConfig binding from Delta and Iceberg with Glue 

    Delta Lake and Iceberg connectors do not use configuration from
`HiveConfig` and so the `HiveConfig`-provided configuration should not
be accepted when configuring the connector.
    @findepi
    findepi committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    85049bc View commit details
    Browse the repository at this point in the history