fix: get bucket region with authenticated request #242

tobikris · 2023-03-25T22:26:28Z

I discovered strange behavior with the new(ish) Wasabi region eu-central-2.
It seems that some of their servers in this region are stricter than the others.
This probably sounds very strange - it certainly was hard to debug and pinpoint.
I think that some servers deny HeadBucket requests without authentication, while others are allowing them. Even though I am totally confused by Wasabi's behavior and disappointed by their support on this issue, this PR solves the issue for me by adding authentication to the request.

The issue can be observed with this example:

provider "wasabi" {
  region                      = "eu-central-2"
  skip_requesting_account_id  = true
  skip_credentials_validation = true
  skip_metadata_api_check     = true

  endpoints {
    s3  = "https://s3.eu-central-2.wasabisys.com"
    iam = "https://iam.wasabisys.com"
    sts = "https://sts.wasabisys.com"
  }
}

resource "wasabi_bucket" "test" {
  bucket   = "test-eu-central-2-a0dbawpwd"
}

After the initial apply I observed the following two cases - in seemingly random fashion:

Success:

2023-03-25T18:53:33.259+0100 [INFO]  provider.terraform-provider-wasabi_v4.1.3: 2023/03/25 18:53:33 [DEBUG] [aws-sdk-go] DEBUG: Validate Response s3/GetBucketLifecycleConfiguration failed, attempt 0/25, error NoSuchLifecycleConfiguration: The lifecycle configuration does not exist
        status code: 404, request id: 761B7D272F17CF52:A, host id: iygoWM5ByNFJuDqhQrKnEDEYRc3rFjFWMgBQUgJqJ2oRfIgmcQbc9EUCFk6HqL77EsQtsUm8KzVO: timestamp=2023-03-25T18:53:33.259+0100
2023-03-25T18:53:33.259+0100 [INFO]  provider.terraform-provider-wasabi_v4.1.3: 2023/03/25 18:53:33 [DEBUG] Waiting for state to become: [success]: timestamp=2023-03-25T18:53:33.259+0100
2023-03-25T18:53:33.260+0100 [INFO]  provider.terraform-provider-wasabi_v4.1.3: 2023/03/25 18:53:33 [DEBUG] [aws-sdk-go] DEBUG: Request s3/HeadBucket Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /test-eu-central-2-a0dbawpwd HTTP/1.1
Host: s3.eu-central-2.wasabisys.com
User-Agent: aws-sdk-go/1.34.26 (go1.16.15; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/1.4.1 (+https://www.terraform.io)


-----------------------------------------------------: timestamp=2023-03-25T18:53:33.259+0100
2023-03-25T18:53:34.308+0100 [INFO]  provider.terraform-provider-wasabi_v4.1.3: 2023/03/25 18:53:34 [DEBUG] [aws-sdk-go] DEBUG: Response s3/HeadBucket Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Connection: close
Content-Type: application/xml
Date: Sat, 25 Mar 2023 17:53:34 GMT
Server: WasabiS3/7.12.1004-2023-02-17-7ff2f5bdd9 (head3)
X-Amz-Bucket-Region: eu-central-2
X-Amz-Id-2: DTImrGgxkz+NDzQGZ2Gn05iR5TtYCSt0mNx5BuI/piAdckE3FAuLwUZ9B0SLSf4CG2pzPDdN2ZEG
X-Amz-Request-Id: A8C5786BC2AAAC41


-----------------------------------------------------: timestamp=2023-03-25T18:53:34.307+0100
2023-03-25T18:53:34.308+0100 [INFO]  provider.terraform-provider-wasabi_v4.1.3: 2023/03/25 18:53:34 [DEBUG] [aws-sdk-go]: timestamp=2023-03-25T18:53:34.308+0100
2023-03-25T18:53:34.313+0100 [WARN]  Provider "registry.terraform.io/terrabitz/wasabi" produced an invalid plan for wasabi_bucket.test, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .acl: planned value cty.StringVal("private") for a non-computed attribute
      - .force_destroy: planned value cty.False for a non-computed attribute
      - .versioning: block count in plan (1) disagrees with count in config (0)
2023-03-25T18:53:34.314+0100 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2023-03-25T18:53:34.317+0100 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/terrabitz/wasabi/4.1.3/linux_amd64/terraform-provider-wasabi_v4.1.3 pid=2731245
2023-03-25T18:53:34.332+0100 [DEBUG] provider: plugin exited
2023-03-25T18:53:34.333+0100 [DEBUG] no planned changes, skipping apply graph check

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

Error:

2023-03-25T18:54:12.463+0100 [INFO]  provider.terraform-provider-wasabi_v4.1.3: 2023/03/25 18:54:12 [DEBUG] [aws-sdk-go] DEBUG: Validate Response s3/GetBucketLifecycleConfiguration failed, attempt 0/25, error NoSuchLifecycleConfiguration: The lifecycle configuration does not exist
        status code: 404, request id: 005ABEF4F864FF73:B, host id: VP6OJ46u1aCLml9CNSumrZ7zXwtQC1Fnz8174F3w88KPZTeKBoWW1hKjvCQesAEIXdIdvtmabdSx: timestamp=2023-03-25T18:54:12.462+0100
2023-03-25T18:54:12.463+0100 [INFO]  provider.terraform-provider-wasabi_v4.1.3: 2023/03/25 18:54:12 [DEBUG] Waiting for state to become: [success]: timestamp=2023-03-25T18:54:12.462+0100
2023-03-25T18:54:12.463+0100 [INFO]  provider.terraform-provider-wasabi_v4.1.3: 2023/03/25 18:54:12 [DEBUG] [aws-sdk-go] DEBUG: Request s3/HeadBucket Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /test-eu-central-2-a0dbawpwd HTTP/1.1
Host: s3.eu-central-2.wasabisys.com
User-Agent: aws-sdk-go/1.34.26 (go1.16.15; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/1.4.1 (+https://www.terraform.io)


-----------------------------------------------------: timestamp=2023-03-25T18:54:12.463+0100
2023-03-25T18:54:12.537+0100 [INFO]  provider.terraform-provider-wasabi_v4.1.3: 2023/03/25 18:54:12 [DEBUG] [aws-sdk-go] DEBUG: Response s3/HeadBucket Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 403 Forbidden
Connection: close
Content-Type: application/xml
Date: Sat, 25 Mar 2023 17:54:12 GMT
Server: WasabiS3/7.12.1004-2023-02-17-7ff2f5bdd9 (R204-U11)
X-Amz-Id-2: xXag44oeqipeke6ZDlGjztH8FEod/h32lAgDdefTOYQCBCKQyxRdtZArEtRDSXk6dkeXCxOfFedi
X-Amz-Request-Id: A784DC286A189ABD:A


-----------------------------------------------------: timestamp=2023-03-25T18:54:12.537+0100
2023-03-25T18:54:12.537+0100 [INFO]  provider.terraform-provider-wasabi_v4.1.3: 2023/03/25 18:54:12 [DEBUG] [aws-sdk-go]: timestamp=2023-03-25T18:54:12.537+0100
2023-03-25T18:54:12.537+0100 [INFO]  provider.terraform-provider-wasabi_v4.1.3: 2023/03/25 18:54:12 [DEBUG] [aws-sdk-go] DEBUG: Validate Response s3/HeadBucket failed, attempt 0/25, error Forbidden: Forbidden
        status code: 403, request id: A784DC286A189ABD:A, host id: xXag44oeqipeke6ZDlGjztH8FEod/h32lAgDdefTOYQCBCKQyxRdtZArEtRDSXk6dkeXCxOfFedi: timestamp=2023-03-25T18:54:12.537+0100
2023-03-25T18:54:12.538+0100 [ERROR] vertex "wasabi_bucket.test" error: error getting S3 Bucket location: Forbidden: Forbidden
        status code: 403, request id: A784DC286A189ABD:A, host id: xXag44oeqipeke6ZDlGjztH8FEod/h32lAgDdefTOYQCBCKQyxRdtZArEtRDSXk6dkeXCxOfFedi
2023-03-25T18:54:12.538+0100 [ERROR] vertex "wasabi_bucket.test (expand)" error: error getting S3 Bucket location: Forbidden: Forbidden
        status code: 403, request id: A784DC286A189ABD:A, host id: xXag44oeqipeke6ZDlGjztH8FEod/h32lAgDdefTOYQCBCKQyxRdtZArEtRDSXk6dkeXCxOfFedi
╷
│ Error: error getting S3 Bucket location: Forbidden: Forbidden
│       status code: 403, request id: A784DC286A189ABD:A, host id: xXag44oeqipeke6ZDlGjztH8FEod/h32lAgDdefTOYQCBCKQyxRdtZArEtRDSXk6dkeXCxOfFedi
│ 
│   with wasabi_bucket.test,
│   on tfbackend.tf line 25, in resource "wasabi_bucket" "test":
│   25: resource "wasabi_bucket" "test" {

dcardellino · 2023-05-05T08:46:25Z

@terrabitz @tobikris Any chance to get this merged? We have the same problem in our environment.

tobikris · 2023-05-05T11:53:07Z

Not much I can do here without a response of @terrabitz. You probably know how to use my fork in the meantime?

isshwar · 2023-05-11T10:55:20Z

could confirm that the fix helped in bypassing the sporadic errors. could have been better if wasabi fixes it from their end but nonetheless thanks.
@terrabitz would be really helpful, if this is merged to master.

tobikris · 2023-05-11T11:55:32Z

Yes, unfortunately, Wasabi does not seem to see this as an error on their side. They are not responding to my Emails anymore where I sent them all the debugging output I could get. I am very disappointed by this behavior 😞 .

dcardellino · 2023-05-17T12:25:21Z

@tobikris Seems to work for me now with your fork. Will build this using our Gitlab CI until this MR eventually get merged..

sammarks · 2023-05-17T20:06:58Z

Running into these same errors on my end sporadically during deployments now too. Has anyone released a public version of this PR on the registry yet?

sammarks · 2023-05-18T13:37:49Z

Published here: https://registry.terraform.io/providers/thesisedu/wasabi/latest

sammarks · 2023-05-18T19:41:03Z

FYI there's a bug with this code for buckets in the us-east-1 region. GetBucketLocation returns nil for LocationConstraint if the bucket is in us-east-1. The following code worked for me (and has been published as 4.1.9 to the registry):

res, err := s3conn.GetBucketLocation(&s3.GetBucketLocationInput{
  Bucket: aws.String(d.Id()),
})
if err != nil {
  return nil, err
}
location := "us-east-1"
if res.LocationConstraint != nil {
  location = *res.LocationConstraint
}

return s3.NormalizeBucketLocation(location), nil

fix: get bucket region with authenticated request

37e4d04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: get bucket region with authenticated request #242

fix: get bucket region with authenticated request #242

Uh oh!

tobikris commented Mar 25, 2023

Uh oh!

dcardellino commented May 5, 2023

Uh oh!

tobikris commented May 5, 2023

Uh oh!

isshwar commented May 11, 2023 •

edited

Loading

Uh oh!

tobikris commented May 11, 2023

Uh oh!

dcardellino commented May 17, 2023

Uh oh!

sammarks commented May 17, 2023

Uh oh!

sammarks commented May 18, 2023

Uh oh!

sammarks commented May 18, 2023

Uh oh!

Uh oh!

fix: get bucket region with authenticated request #242

Are you sure you want to change the base?

fix: get bucket region with authenticated request #242

Uh oh!

Conversation

tobikris commented Mar 25, 2023

Uh oh!

dcardellino commented May 5, 2023

Uh oh!

tobikris commented May 5, 2023

Uh oh!

isshwar commented May 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tobikris commented May 11, 2023

Uh oh!

dcardellino commented May 17, 2023

Uh oh!

sammarks commented May 17, 2023

Uh oh!

sammarks commented May 18, 2023

Uh oh!

sammarks commented May 18, 2023

Uh oh!

Uh oh!

isshwar commented May 11, 2023 •

edited

Loading