Commit 2dad7e1f810369a2c3cff5453a309d2eefef3e68 makes s5cmd not working with S3-compatible services

[somsak]

The code added for cross region support makes s5cmd not working anymore with S3-compatible services other than AWS.

Step to reproduce.

1. Apply for any S3-compatible services, such as Digital Ocean Spaces (https://www.digitalocean.com/products/spaces/)
2. Try to run s5cmd with --endpoint-url set to the correct endpoint (for my case, it's s5cmd --endpoint-url https://sgp1.digitaloceanspaces.com).
   The simple "ls" command works fine. But trying to do "ls s3://bucket-name-here" will always result in 403 access denied.

The commit before this worked just as expected.

I have tested this with my own S3-compatible services and failed. Since it also occurred to DO Space I think the problem could be applied to any S3-compatible platform out there.

I also tried setting environment AWS_REGION and AWS_DEFAULT_REGION to "sgp1" but still not working.

[aceeric]

We are experiencing the same thing, though we are using the docker image peakcom/s5cmd:v1.2.0. Prior to this we used s5cmd v1.1.0-4c36b52 without issues. Newer version v1.2.0-24afd09 in the docker image exhibits the error described by @somsak. In our case we use Rook Ceph to provision S3-compatible buckets in Kubernetes and can no longer list these buckets. The error is:

ERROR "ls s3://my-bucket-<uid omitted>": Forbidden: Forbidden status code: 403, request id: tx00000000000000000052d-005fb6bdc9-352be-object-store, host id:

[ddpolyakov]

I believe my issue is all about the same #252
As a workaround - commenting out GetBucketRegion section at storage/s3.go works pretty well

[somsak]

For the record my own S3-compatible service is also a Ceph+RadosGW.

[andryan]

diff --git a/storage/s3.go b/storage/s3.go
index b449d7c..eb869d7 100644
--- a/storage/s3.go
+++ b/storage/s3.go
@@ -678,16 +678,18 @@ func (s *s3Session) newSession(ctx context.Context, opts Options) (*session.Sess
                sess.Config.Region = aws.String(endpoints.UsEast1RegionID)
        }
 
-       // get region of the bucket and create session accordingly
-       // if the region is not provided, it means we want region-independent session
-       // for operations such as listing buckets, making a new bucket, ...
-       if opts.bucket != "" {
-               region, err := s3manager.GetBucketRegion(ctx, sess, opts.bucket, "")
-               if err != nil {
-                       return nil, err
-               }
+       if opts.Endpoint == "" {
+               // get region of the bucket and create session accordingly
+               // if the region is not provided, it means we want region-independent session
+               // for operations such as listing buckets, making a new bucket, ...
+               if opts.bucket != "" {
+                       region, err := s3manager.GetBucketRegion(ctx, sess, opts.bucket, "")
+                       if err != nil {
+                               return nil, err
+                       }
 
-               sess.Config.Region = aws.String(region)
+                       sess.Config.Region = aws.String(region)
+               }
        }
 
        s.sessions[opts] = sess

this seems to fix it