Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new SnowflakeCatalog implementation to enable directly using Snowflake-managed Iceberg tables #6428

Merged
merged 23 commits into from
Jan 14, 2023

Commits on Dec 13, 2022

  1. Initial read-only Snowflake Catalog implementation by @sfc-gh-mparmar (

    …#1)
    
    Initial read-only Snowflake Catalog implementation built on top of the Snowflake JDBC driver,
    providing support for basic listing of namespaces, listing of tables, and loading/reads of tables.
    
    Auth options are passthrough to the JDBC driver.
    
    Co-authored-by: Maninder Parmar <maninder.parmar@snowflake.com>
    Co-authored-by: Maninder Parmar <maninder.parmar+oss@snowflake.com>
    Co-authored-by: Dennis Huo <dennis.huo+oss@snowflake.com>
    4 people authored Dec 13, 2022
    Configuration menu
    Copy the full SHA
    5ad32b0 View commit details
    Browse the repository at this point in the history

Commits on Dec 14, 2022

  1. Add JdbcSnowflakeClientTest using mocks (#2)

    Add JdbcSnowflakeClientTest using mocks; provides full coverage of JdbcSnowflakeClient
    and entities' ResultSetHandler logic.
    
    Also update target Spark runtime versions to be included.
    dennishuo authored Dec 14, 2022
    Configuration menu
    Copy the full SHA
    930a3f0 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    86b3d11 View commit details
    Browse the repository at this point in the history

Commits on Dec 16, 2022

  1. Add test { useJUnitPlatform() } tuple to iceberg-snowflake for

    consistency and future interoperability with inheriting from abstact
    unittest base classes.
    dennishuo committed Dec 16, 2022
    Configuration menu
    Copy the full SHA
    076a14a View commit details
    Browse the repository at this point in the history

Commits on Dec 17, 2022

  1. Configuration menu
    Copy the full SHA
    a7b5aa7 View commit details
    Browse the repository at this point in the history
  2. Misc test-related refactors per review suggestions

    -Convert unittests to all use assertj/Assertions for "fluent assertions"
    -Refactor test injection into overloaded initialize() method
    -Add test cases for close() propagation
    -Use CloseableGroup.
    dennishuo committed Dec 17, 2022
    Configuration menu
    Copy the full SHA
    dd5255c View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    500b36b View commit details
    Browse the repository at this point in the history
  4. Move TableIdentifier checks out of newTableOps into the

    SnowflakTableOperations class itself, add test case.
    dennishuo committed Dec 17, 2022
    Configuration menu
    Copy the full SHA
    ad2c55f View commit details
    Browse the repository at this point in the history
  5. Refactor out any Namespace-related business logic from the lower

    SnowflakeClient/JdbcSnowflakeClient layers and merge SnowflakeTable
    and SnowflakeSchema into a single SnowflakeIdentifier that also
    encompasses ROOT and DATABASE level identifiers.
    
    A SnowflakeIdentifier thus functions like a type-checked/constrained
    Iceberg TableIdentifier, and eliminates any tight coupling between
    a SnowflakeClient and Catalog business logic.
    
    Parsing of Namespace numerical levels into a SnowflakeIdentifier
    is now fully encapsulated in NamespaceHelpers so that callsites
    don't duplicate namespace-handling/validation logic.
    dennishuo committed Dec 17, 2022
    Configuration menu
    Copy the full SHA
    7f13674 View commit details
    Browse the repository at this point in the history
  6. Finish migrating JdbcSnowflakeClientTest off any usage of org.junit.A…

    …ssert
    
    in favor of assertj's Assertions.
    dennishuo committed Dec 17, 2022
    Configuration menu
    Copy the full SHA
    58d258e View commit details
    Browse the repository at this point in the history

Commits on Dec 20, 2022

  1. Style refactorings from review comments, expanded and moved InMemoryF…

    …ileIO into core
    
    with its own unittest.
    dennishuo committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    0183129 View commit details
    Browse the repository at this point in the history

Commits on Dec 21, 2022

  1. Fix behavior of getNamespaceMetadata to throw when the namespace doesn't

    exist.
    
    Refactor for naming conventions and consolidating identifier
    handling into NamespaceHandlers.
    
    Make FileIO instantiated fresh for each newTableOps call.
    dennishuo committed Dec 21, 2022
    Configuration menu
    Copy the full SHA
    ca6deab View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b3a2842 View commit details
    Browse the repository at this point in the history
  3. Define minimal ResultSetParser/QueryHarness classes to fully replace

    any use of commons-dbutils; refactor ResultSet handling fully into
    JdbcSnowflakeClient.java.
    dennishuo committed Dec 21, 2022
    Configuration menu
    Copy the full SHA
    676d024 View commit details
    Browse the repository at this point in the history

Commits on Dec 23, 2022

  1. Update snowflake/src/main/java/org/apache/iceberg/snowflake/Snowflake…

    …TableOperations.java
    
    Co-authored-by: Eduard Tudenhöfner <etudenhoefner@gmail.com>
    dennishuo and nastra authored Dec 23, 2022
    Configuration menu
    Copy the full SHA
    cc493d0 View commit details
    Browse the repository at this point in the history
  2. Refactor style suggestions; remove debug-level logging, arguments in …

    …exceptions,
    
    private members if not accessed outside, move precondition checks, add test for
    NamespaceHelpers.
    dennishuo committed Dec 23, 2022
    Configuration menu
    Copy the full SHA
    ce7e28c View commit details
    Browse the repository at this point in the history

Commits on Jan 6, 2023

  1. Configuration menu
    Copy the full SHA
    2729e64 View commit details
    Browse the repository at this point in the history

Commits on Jan 7, 2023

  1. Clean up varargs.

    dennishuo committed Jan 7, 2023
    Configuration menu
    Copy the full SHA
    4c1e79f View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    bc0c6ee View commit details
    Browse the repository at this point in the history

Commits on Jan 9, 2023

  1. Configuration menu
    Copy the full SHA
    8240d9e View commit details
    Browse the repository at this point in the history

Commits on Jan 10, 2023

  1. Configuration menu
    Copy the full SHA
    ebe5dd6 View commit details
    Browse the repository at this point in the history

Commits on Jan 13, 2023

  1. Configuration menu
    Copy the full SHA
    9e9b9e6 View commit details
    Browse the repository at this point in the history

Commits on Jan 14, 2023

  1. Configuration menu
    Copy the full SHA
    e8fee31 View commit details
    Browse the repository at this point in the history