Connect to S3 catalog

[IanVlasov]

Question

Hello!
I am rather new in using Iceberg, could you help me with the following question.

I have a catalog stored on S3 with the metadata stored nearby, the structure is simple:

.
└── iceberg/
    └── table_name/
        ├── data/
        │   └── *.parquet
        └── metadata/
            └── *.avro

I can successfully connect to it using Spark with the following parameters:

.config("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog")
.config("spark.sql.defaultCatalog", "iceberg")
.config("spark.sql.catalog.iceberg", "org.apache.iceberg.spark.SparkCatalog")
.config("spark.sql.catalog.iceberg.warehouse", "s3a://bucket_name/iceberg")
.config("spark.sql.catalog.iceberg.type", "hadoop")
.config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")

But PyIceberg insists on providing URI for metastore which I don't know where to find.
Is there any way to define a configuration that will work in this case?

[kevinjqliu]

Hey @IanVlasov
In the above example, you're using the hadoop catalog type in Spark. PyIceberg does not have the equivalent. And Iceberg discourages using the hadoop catalog. I would recommend interacting with a catalog implementation.

If its a one-time thing, you can try loading the metadata.json directly using StaticTable

[dingo4dev]

@IanVlasov
AWS has recently released S3 Table which supports iceberg. Check out the document https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-open-source.html

[kevinjqliu]

we also added a new example with glue catalog which is similar to s3 tables
https://github.com/apache/iceberg-python/blob/main/mkdocs/docs/configuration.md#aws-glue