Unable to read AWS Glue created iceberg tables from S3

[ganathan]

I am getting the following error when attempting to perform a select statement using trino from an iceberg table created in Glue.

"Query 20220414_160541_00007_h5c3z failed: Error reading tail from s3:///fund_perf_f/data/084faaf5/fund_perf_f/load_date=2022-01-31/1ccc2482-2274-46e2-a5c5-06b4a4c44a6b.gz.parquet with length 7757"

When I checked the physical location of the file.. there is an empty folder in the path.
s3:///fund_perf_f/data//084faaf5/fund_perf_f/load_date=2022-01-31/1ccc2482-2274-46e2-a5c5-06b4a4c44a6b.gz.parquet with length 7757

I have created the table using Athena console and also via a Glue pyspark job with the same end result. Could anyone help resolve this issue?

Not sure if this an Apache Iceberg issue, or AWS issue or Trino issue.

[amogh-jahagirdar]

I have created the table using Athena console and also via a Glue pyspark job with the same end result. Could anyone help resolve this issue?

Just to clarify, you mean you also tried creating the table through Glue Pyspark and after a subsequent read attempt it failed?

If so this seems to indicate an issue in Iceberg, since for Glue pyspark, you'd still be using the spark iceberg dependencies. I was also able to reproduce a similar empty "folder" issue through Athena when the table is created with a location where the locations is 1 bucket and 1 nested "folder" (in quotes because it's not really a folder).

Root Location 's3://insert-iceberg-test/f3' 

INSERT PATH
s3://insert-iceberg-test/f3/data/4e23cb0e//f3/id_bucket=6/4e6e49d0-98c2-4440-90de-76c0d7e6a440.gz.parquet

However, if there are 2 nested folders in the bucket, the insert path is as expected (Missing the extra / separator in the object key)

Root LOCATION
s3://insert-iceberg-test/f1/f2

INSERT PATH
s3://insert-iceberg-test/f1/f2/data/0eb31aa4/f1/f2/id_bucket=4/febda4a0-2ace-4447-a932-03fbeba36e48.gz.parquet

In the end I'm able to query the data since these are keys stored in the manifests, but it does cause issues for external systems which expect some physical layout.

But in your case the bucket itself is missing in the path, or is fund_perf_f your bucket?

[rdblue]

Looks like a malformed path since it starts with s3:///. There's no bucket.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'