Add new SnowflakeCatalog implementation to enable directly using Snowflake-managed Iceberg tables

[dennishuo]

This read-only implementation of the Catalog interface, initially built on top of the Snowflake JDBC driver for the connection layer, enables engines like Spark using the Iceberg Java SDK to be able to consume Snowflake-managed Iceberg Tables via Iceberg Catalog interfaces.

Example, assuming a Snowflake account with a database iot_data containing a schema public and a managed Iceberg table sensor_test_results:

spark-shell --conf spark.sql.catalog.snowlog=org.apache.iceberg.spark.SparkCatalog \
    --conf spark.sql.catalog.snowlog.catalog-impl=org.apache.iceberg.snowflake.SnowflakeCatalog \
    --conf spark.sql.catalog.snowlog.uri="jdbc:snowflake://$ACCOUNT.snowflakecomputing.com" \
    ....
scala> spark.sessionState.catalogManager.setCurrentCatalog("snowlog");
scala> spark.sql("show namespaces in iot_data").show(false);
scala> spark.sql("select * from iot_data.public.sensor_test_results limit 10").show(false);

Note that the involvement of a JDBC driver is only incidental, and functionality is different from the JdbcCatalog - here, Snowflake itself manages manifest/metadata files and table/snapshot metadata, and this catalog layer facilitates the coordination of metadata-file locations and discovery of the latest table snapshot versions without resorting to file-listing or "directory-name"-listing (for listTables or listNamespaces) like the HadoopCatalog.

[nastra]

Thanks for the PR @dennishuo, great to have this. I did a more thorough review and my comments are inlined.

[dennishuo]

@nastra Thanks for the thorough review and suggestions! Finished applying all your suggestions, including fully converting to assertj/Assertions and refactoring out the Namespace<->SnowflakeIdentifier parsing to better encapsulate all the argument-checking/parsing into one place.

[dennishuo]

Interesting, not sure why it removed @danielcweeks when I re-requested review from @nastra (definitely wasn't intentional -- as far as I can tell my repository permissions don't even provide the ability to remove reviewer requests). I wonder if this is possibly another manifestation of community/community#8939

[danielcweeks]

@dennishuo #6538 is merged, so you might want to rebase on top of that. I'm good, but I it would be great to have @nastra sign off as well since there are couple comments open.

[dennishuo]

Thanks for the note! Successfully merged main. I'll await @nastra 's review

[nastra]

thanks @dennishuo for working on this. I've commented on a few nits but overall this looks good to me.
It would also be good to have @rdblue review/approve this.

[jackye1995]

I see Azure and GCP paths converted, what about the S3 ones? Are all combinations like s3a, s3n, etc. all compatible natively in Snowflake?

[dennishuo]

Right, in situations where Snowflake handles paths coming from externally produced sources, Snowflake tracks a canonical form of it and indeed handles s3a and s3n.

Here, the conversion is for outbound paths produced by Snowflake, where the "s3://" prefix is already used natively, so only Azure and GCS need translation for "standard" scheme mappings accepted by e.g. HadoopFileSystem.

This does remind me though - I've been meaning to open a discussion to see if anyone has thought about maybe adding some ease-of-use hooks for last-mile automatic basic path translations right before they go to FileIO resolution. For example, someone might want s3://somebucket paths to be rewritten to a viewfs:// base path before letting HadoopFileIO automatically delegate to the right ViewFs impl. Or it could come up in cases where manifest files hold s3:// paths but someone wants everything to go through a corresponding dbfs:// mount point. Do you know of any prior discussions along those lines, and would it be worth opening an issue for broader input?

[jackye1995]

Thanks for the clarification, I approved the changes, very excited to see this from Snowflake.

I've been meaning to open a discussion to see if anyone has thought about maybe adding some ease-of-use hooks for last-mile automatic basic path translations right before they go to FileIO resolution.

For HadoopFileIO, i think this is not really needed because typically we see our users already configured HDFS settings to map schemes to whatever file system implementations they would like to use.

I think ResolvingFileIO to some extent already does this kind of translation to some extent, maybe we can extend its functionality at that front.

[rdblue]

This factory seems odd since it has no state that is passed through to the loadFileIO method. Couldn't this just call loadFileIO directly?

[dennishuo]

This was just extracted to preserve the flow of unittest setup to be able to return the pre-configured InMemoryFileIO fake instance rather than having dynamic classloading get an empty one. The alternative there would've been to introduce some static global state in InMemoryFileIO, but that seems prone to causing cross-test-case problems.

I could add a comment or annotation here to explain its existence if that'd make it cleaner.

[dennishuo]

Added code comment to clarify

[jackye1995]

Looks like some CI tests are failing? Could you check? Maybe need to rebase.

[dennishuo]

@jackye1995 Thanks for the heads up! Looks like merging to head fixed it.