Cannot create Hive+S3 managed table #9942
Comments
|
Ok, here I have some further information after a few experiments i performed. If i run {
"Version": "2012-10-17",
"Id": "PutObjPolicy",
"Statement": [
{
"Sid": "DenyUnEncryptedObjectUploads",
"Effect": "Deny",
"Principal": {
"AWS": "*"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::my-bucket/*",
"Condition": {
"StringNotEquals": {
"s3:x-amz-server-side-encryption": "AES256"
}
}
}
]
}Both create statements fail, I can workaround the schema one by creating the S3 "folder" but if I try to do the same with the table one it then fails with "folder already exists" error. What i think is happening here is that the policy in our prod-like buckets requires SSE enabled for EVERY S3 put action, and presto is only setting it on files-like S3 paths but not doing so for folder-like paths. Am I right? We really need this working to proceed with the integration of Presto into our data pipeline, i'd love to get this problem fixed, so I'm offering myself to help in whatever I can. Thanks. |
|
cc @sopel39 |
|
It seems to very similar issue to the one we recently experienced. Can you try to run |
|
It is similar to: #9916, but not quite. Apparently Metastore is not able to create table in destination directory. I would guess that some configuration on Hive/HDFS side is lacking. I don't know which Hadoop distro you use, but you could try enabling SSE in Hive/Hdfs similarly as here: |
|
Can you create table with Hive only? |
|
@kokosing as i was expecting |
|
@sopel39 same thing with Hive, I can only create external tables from existing files in S3, trying to create a writable table using hive directly failed with the same error. But I didn't bother to try to configure hive since I was counting on presto flag for SSE to override hive config, which happens only in the case of creating a non-empty table using |
|
Have you checked if you are able create empty s3 directory with other tools? |
|
@kokosing I can using |
|
As far I know Presto do not create any directory for table during Have you tried to run |
|
Did you figure out the problem? This error is coming from the Hive metastore ( |
|
Yeah, sorry. I started using <property>
<name>fs.s3a.server-side-encryption-algorithm</name>
<value>AES256</value>
</property>To Closing the issue since it's not presto's fault. Thank everyone for the support. |
I have a single node cluster with presto version 0.187 (version supported by EMR).
Hive connector has following configuration:
I'm trying to create a table in a S3 bucket which EC2 instance has r/w access to, it requires
--ssealways enabled, as you can see i've turned the flag on in the config.Presto seems to be picking up instance credentials correctly as i can create an external table with location
s3a://.....and query its content, however, when i try to create a managed table it fails with S3 Access Denied exception, the only details i have about the error is the stacktrace returned in presto-cli with --debug:As you can see no information about what S3 operation is being executed but the user should have all the required permissions.
As a side note,
testschema creation fails with similar error but i managed to sort it out by creating the S3 folder for its location in advance.How can I debug/solve this problem? I've tried many many things in the course of the last few days and nothing worked. Honestly i'm running out of ideas.
Thank you so much for such an awesome tool!
Cheers,
Gaston.
The text was updated successfully, but these errors were encountered: