Problems encountered in configuring the Delta connector for AWS Glue...

[overcoil]

I am trying to use 373's new Delta Lake connector with AWS Glue.

My configuration is as follows:

1. Using docker-compose with the canonical server tar.gz running a 2-node (1 coordinator + 1 worker) cluster.
2. Mounting a deltas3g.properties into all nodes
3. There's a Delta Lake table in one of my S3 bucket.

deltas3g.properties looks as follow (with secrets stubbed out):

connector.name=delta-lake

hive.metastore=glue
hive.metastore.glue.region=us-west-2

hive.metastore.glue.aws-access-key=MY-ACCESS-KEY
hive.metastore.glue.aws-secret-key=MY-SECRET-KEY

hive.metastore.glue.catalogid=MY-AWS-ACCOUNTID

hive.metastore.glue.iam-role=arn:aws:iam::MY-AWS-ACCOUNTID:role/dev_db_c3_role-sep

hive.s3.aws-access-key=MY-ACCESS-KEY
hive.s3.aws-secret-key=MY-SECRET-KEY

As I'm running from Docker, I'm using the same IAM user key pair for both hive.metastore.glue.aws-access-key/.aws-secret-key and hive.s3.aws-access-key/.aws-secret-key.

The IAM (user) role and policy were created according to the Starburst connector doc here:
https://docs.starburst.io/latest/aws/requirements.html

I'm following this:
https://docs.starburst.io/latest/connector/delta-lake.html

With the understanding that the schema is superfluous in this case, I use a simple query:

CREATE TABLE deltas3g.default.my_table (
  dummy bigint
)
WITH (
  location = 's3://weyland-yutani/delta/nyctaxi_2019'
)

to register my Delta table into Glue so as to use it.

But the query fails with

Query 20220318_234618_00021_66xdm failed: Failed checking path: s3://weyland-yutani/delta/nyctaxi_2019

The log on the web console:

io.trino.spi.TrinoException: Failed checking path: s3://weyland-yutani/delta/nyctaxi_2019
	at io.trino.plugin.hive.util.HiveWriteUtils.pathExists(HiveWriteUtils.java:474)
	at io.trino.plugin.deltalake.DeltaLakeMetadata.ensurePathExists(DeltaLakeMetadata.java:724)
	at io.trino.plugin.deltalake.DeltaLakeMetadata.createTable(DeltaLakeMetadata.java:576)
	at io.trino.plugin.base.classloader.ClassLoaderSafeConnectorMetadata.createTable(ClassLoaderSafeConnectorMetadata.java:366)
	at io.trino.metadata.MetadataManager.createTable(MetadataManager.java:703)
	at io.trino.execution.CreateTableTask.internalExecute(CreateTableTask.java:260)
	at io.trino.execution.CreateTableTask.execute(CreateTableTask.java:118)
	at io.trino.execution.CreateTableTask.execute(CreateTableTask.java:84)
	at io.trino.execution.DataDefinitionExecution.start(DataDefinitionExecution.java:145)
	at io.trino.execution.SqlQueryManager.createQuery(SqlQueryManager.java:243)
	at io.trino.dispatcher.LocalDispatchQuery.lambda$startExecution$7(LocalDispatchQuery.java:143)
	at io.trino.$gen.Trino_373____20220318_221010_2.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: io.trino.plugin.hive.s3.TrinoS3FileSystem.UnrecoverableS3OperationException: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: C5BXA4D2W15XWNB2; S3 Extended Request ID: 44RbPfXKFkOVckgy/aqnSmw7V4BNm73IpurEZtHRyWiNTJfpqbiZMBBNQC+yb+3oMYLdppX02/k=; Proxy: null), S3 Extended Request ID: 44RbPfXKFkOVckgy/aqnSmw7V4BNm73IpurEZtHRyWiNTJfpqbiZMBBNQC+yb+3oMYLdppX02/k= (Path: s3://weyland-yutani/delta/nyctaxi_2019)
	at io.trino.plugin.hive.s3.TrinoS3FileSystem.lambda$getS3ObjectMetadata$6(TrinoS3FileSystem.java:803)
	at io.trino.plugin.hive.util.RetryDriver.run(RetryDriver.java:130)
	at io.trino.plugin.hive.s3.TrinoS3FileSystem.getS3ObjectMetadata(TrinoS3FileSystem.java:791)
	at io.trino.plugin.hive.s3.TrinoS3FileSystem.getS3ObjectMetadata(TrinoS3FileSystem.java:775)
	at io.trino.plugin.hive.s3.TrinoS3FileSystem.getFileStatus(TrinoS3FileSystem.java:447)
	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454)
	at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1683)
	at io.trino.plugin.hive.util.HiveWriteUtils.pathExists(HiveWriteUtils.java:471)
	... 14 more
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: C5BXA4D2W15XWNB2; S3 Extended Request ID: 44RbPfXKFkOVckgy/aqnSmw7V4BNm73IpurEZtHRyWiNTJfpqbiZMBBNQC+yb+3oMYLdppX02/k=; Proxy: null)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1862)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1415)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1384)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1154)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:811)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5445)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5392)
	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1368)
	at io.trino.plugin.hive.s3.TrinoS3FileSystem.lambda$getS3ObjectMetadata$6(TrinoS3FileSystem.java:794)
	... 21 more

The S3 bucket has no ACL or policy applied; I expect the role's 3 S3 permissions (s3:GetObject, s3:ListBucket, s3:PutObject) to provide the required access.

Perhaps someone with experience with the connector when it was a Starburst component can advise on what's amiss? I'm read through the docs/code for the other configuration and nothing else jumps out at me as necessary.

FWIW, here are the config values I'm aware of that I have not set (relying on their default):

#hive.metastore.glue.external-id
#hive.metastore.glue.partitions-segments
#hive.metastore.glue.get-partition-threads
#hive.metastore.glue.read-statistics-threads
#hive.metastore.glue.write-statistics-threads
#delta.metadata.cache-ttl
#delta.metadata.live-files.cache-size
#delta.domain-compaction-threshold
#delta.max-outstanding-splits
#delta.max-splits-per-second
#delta.max-initial-splits
#delta.max-initial-split-size
#delta.max-split-size
#delta.max-partitions-per-writer
#delta.enable-non-concurrent-writes
#delta.default-checkpoint-writing-interval
#delta.experimental.ignore-checkpoint-write-failures
#delta.vacuum.min-retention
#delta.hive-catalog-name
#delta.checkpoint-row-statistics-writing.enabled
#delta.table-statistics-enabled
#delta.extended-statistics.enabled
#delta.compression-codec
#delta.hide-non-delta-lake-tables

George

[findepi]

The S3 bucket has no ACL or policy applied; I expect the role's 3 S3 permissions (s3:GetObject, s3:ListBucket, s3:PutObject) to provide the required access.

The failing operation is getObjectMetadata.

BTW we have #troubleshooting slack channel for this type of questions (https://trino.io/slack)

[overcoil]

Okay... I'll take this question to the #troubleshooting channel then.

(I didn't find a separate permission for getObjectMetadata in S3 so I don't see how to resolve this problem. I found one report of the same error in a different context/release in the #troubleshooting channel.)