[ISSUE] databricks_mount fails to mount AWS S3 bucket with Unable to execute HTTP request: Remote host terminated the handshake
#1500
Comments
|
relevant note to DBFS team: |
|
Are there any firewall/VPC settings possibly not whitelisting s3 endpoints, including regional ones? Regional endpoint whitelisting should be done for the region here the bucket exists from the top of my mind. |
|
This is the same issue that I ran into. In my case I was using S3 gateway endpoint. For some reason, newly created buckets can't be queried using the global endpoint I lodged the issue #1492 with the hope that I didn't have to create a dedicated cluster just for mounting S3 buckets in my workspace and I could just let the Terraform provider spin up a new cluster with a given instance profile and I could pass the However since issue was rejected, I had to work around that by doing something like below: |
|
Thanks for the feedback so far. I'm attempting @thaiphv's workaround now. There was some truth to @stormwindy's comment too. We had a firewall that was not whitelisting a regional endpoint. I'll update this issue after testing today. |
|
Ok so it's definitely related to our AWS Network Firewall. Here are some relevant packets that I captured:
So you can see that the Client Hello is not acknowledged. I'm still working out exactly what's wrong with our Network Firewall rules. I'll update this issue when I have a resolution. |
nfx
changed the title
databricks_mount fails to mount AWS S3 bucket with Unable to execute HTTP request: Remote host terminated the handshake
nfx
added
the
invalid
|
Ok fixed 😓 . Issue was that our Databricks cluster is deployed to the Traffic to Fix was to punch a hole through our Network Firewall allowing access to the Thanks to everyone for their comments! |
Configuration
Note that
{ "Version": "2012-10-17", "Statement": [ { "Sid": "AllowList", "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::test_databricks_data" }, { "Sid": "AllowObjectActions", "Effect": "Allow", "Action": [ "s3:PutObjectAcl", "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Resource": "arn:aws:s3::test_databricks_data/*" } ] }Expected Behavior
What should have happened?
databricks_mountresource should successfully reach the "created" status/mnt/datamount should be accessible on DatabricksActual Behavior
What actually happened?
databricks_mountresource is in "creating" status for 10 minutes and eventually times out/mnt/datamount actually is created but theterraform applycommand still fails due to thedatabricks_mountresource timing out in the "creating" stautsSteps to Reproduce
Please list the steps required to reproduce the issue, for example:
terraform applyTerraform and provider versions
Please paste the output of
terraform version. If version ofdatabricksprovider is not the latest (https://github.com/databricks/terraform-provider-databricks/releases), please make sure to use the latest one.Debug Output
Please add turn on logging, e.g.
TF_LOG=DEBUG terraform applyand run command again, paste it to gist & provide the link to gist. If you're still willing to paste in log output, make sure you provide only relevant log lines with requests.It would make it more readable, if you pipe the log through
| grep databricks | sed -E 's/^.* plugin[^:]+: (.*)$/\1/', e.g.:Relevant output as follows:
If Terraform produced a panic, please provide a link to a GitHub Gist containing the output of the
crash.log.Important Factoids
Are there anything atypical about your accounts that we should know?
The text was updated successfully, but these errors were encountered: