Failing backup with miss-leading error message.

[lalyos]

I Was trying to create a backup to an AWS s3 bucket. The backup stuck in retry state. Iwas checking for error messages in the pod log"

handler backups: failed to check s3 bucket:rancher-backups-xxx, err:Head https://rancher-backups-xxx.s3.dualstack.eu-west-1.amazonaws.com/: 301 response missing Location header, requeuing 

My backup CRD did contain the loaction:

apiVersion: resources.cattle.io/v1
kind: Backup
metadata:
  name: s3-backup-demo
spec:
  storageLocation:
    s3:
      ...
      region: eu-west-1
      endpoint: s3.eu-west-1.amazonaws.com

After a while I relized the bucket was in a different region, so it was clearly miss-configured, but it wasn't obvious from the error message above.

Solution:

• just use this issue as refence, so if somebody is searching for the error text can find the solution
• provide a hint in the error message to check for the bucket location

Background

Unfortunately minio-go uses string based errors, so it can only be string-matched ... Furthermore seems like the error message is not coming from minio directly, but from the aws-sdk-go library. Actually the s3 api sends a redirect (hence the 301) but that is ignore by the official sdk:
aws/aws-sdk-go#356

The SDK should not be retrying request when they error with this request. The 301 error returned is due to the request being made to the wrong region. The Go SDK does require that the request are made to the correct region.
The redirect provided by S3 is not intended to actually be followed in this case.

[lalyos]

I was thinking about making the region optional, and use the GetBucketLocation() minio client call to figure it out.

But it seems that isn't working either. I was trying to get the bucket location with:

• aws cli
• a small go program using mini-go

aws cli returns the correct location, even when talking to the wrong endpoint, while the mini-go client
responds wrongly with the same region as the endpoint is:

$ AWS_DEFAULT_REGION=eu-west-1 aws s3api get-bucket-location --bucket $BUCKET
{
    "LocationConstraint": "eu-central-1"
}

$ go run main.go
location: eu-west-1 

Here is my minio-go tester code:

package main

import (
        "fmt"
        "os"

        "github.com/minio/minio-go/v6"
        "github.com/minio/minio-go/v6/pkg/credentials"
)

func main() {
        client, _ := minio.NewWithOptions("s3.eu-west-1.amazonaws.com", &minio.Options{
                Creds: credentials.NewEnvAWS(),
        })
        bl, _ := client.GetBucketLocation(os.Getenv("BUCKET"))
        fmt.Println("location:", bl)
}

[mrajashree]

Hi @lalyos thanks for taking a look into the cause of the bug. We decided to use the minio client because it seems to work for other s3 compatible stores too, whereas there were some problems with using the s3 go client.

[github-actions]

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.