Fix: Policy action name for Get/Put BucketLifeCycle #9893

kannappanr · 2020-06-22T09:20:59Z

Description

S3:GetBucketLifeCycle should be changed to S3:GetLifeCycleConfiguration
S3:PutBucketLifeCycle should be changed to S3:PutLifeCycleConfiguration

Motivation and Context

minio/mc#3265

How to test this PR?

package main

import (
	"bytes"
	"log"
	iampolicy "github.com/minio/minio/pkg/iam/policy"
)

func main() {
	policy := "{\"Version\": \"2012-10-17\",\"Statement\": [{\"Action\": [\"s3:GetLifecycleConfiguration\", \"s3:PutLifecycleConfiguration\"],\"Effect\": \"Allow\",\"Resource\": \"arn:aws:s3:::*\" }  ]}"

	_, e := iampolicy.ParseConfig(bytes.NewReader([]byte(policy)))
	log.Println(e)
}

The above code can be used to reproduce the error

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

Fixes a regression (If yes, please add commit-id or PR # here)
Documentation needed
Unit tests needed
Functional tests needed (If yes, add mint PR # here: )

klauspost

LGTM

minio-trusted · 2020-06-22T11:20:34Z

Mint Automation

Test	Result
mint-fs.sh	✔️
mint-gateway-s3.sh	✔️
mint-gateway-azure.sh	✔️
mint-gateway-nas.sh	✔️
mint-xl.sh	❌ more...
mint-large-bucket.sh	❌ more...
mint-dist-xl.sh	❌ more...

9893-e44167d/mint-large-bucket.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev7.minio.io:31056
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 98538c22ead5:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 30 seconds
(2/15) Running aws-sdk-java tests ... done in 4 seconds
(3/15) Running aws-sdk-php tests ... done in 1 minutes and 10 seconds
(4/15) Running aws-sdk-ruby tests ... done in 1 minutes and 15 seconds
(5/15) Running awscli tests ... done in 4 minutes and 21 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 1 minutes and 1 seconds
(8/15) Running minio-dotnet tests ... done in 3 minutes and 51 seconds
(9/15) Running minio-go tests ... FAILED in 4 minutes and 44 seconds
{
  "args": {
    "bucketName": "minio-go-test-gtb0kj21juzssyne",
    "objectName": "ip4pqz2mmfnmtbssmap1wsv6kifbsl",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 7326,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... FAILED in 9 minutes and 48 seconds
{
  "name": "minio-java",
  "function": "enableObjectLegalHold(String bucketName, String objectName, String versionId)",
  "duration": 3606,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = BucketNotEmpty, message = The bucket you tried to delete is not empty. You must delete all versions in the bucket., bucketName = minio-java-test-1omfrv1, objectName = null, resource = /minio-java-test-1omfrv1, requestId = 161AD80895FEC759, hostId = 381d2ac1-13a7-4ac1-825f-7a54cd122031)\nrequest={method=DELETE, url=http://minio-dev7.minio.io:31056/minio-java-test-1omfrv1, headers=Host: minio-dev7.minio.io:31056\nAccept-Encoding: identity\nUser-Agent: MinIO (amd64; amd64) minio-java/dev\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20200622T104909Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200622/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=409, headers=Accept-Ranges: bytes\nContent-Length: 373\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO/DEVELOPMENT.2020-06-22T10-16-36Z\nVary: Origin\nX-Amz-Request-Id: 161AD80895FEC759\nX-Xss-Protection: 1; mode=block\nDate: Mon, 22 Jun 2020 10:49:09 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:1211), io.minio.MinioClient.execute(MinioClient.java:1236), io.minio.MinioClient.executeDelete(MinioClient.java:1366), io.minio.MinioClient.removeBucket(MinioClient.java:3686), FunctionalTest.enableObjectLegalHold_test(FunctionalTest.java:3314), FunctionalTest.runTests(FunctionalTest.java:4079), FunctionalTest.main(FunctionalTest.java:4192)]"
}
(9/15) Running minio-js tests ... done in 1 minutes and 14 seconds
(10/15) Running minio-py tests ... done in 13 minutes and 17 seconds
(11/15) Running s3cmd tests ... done in 38 seconds
(12/15) Running s3select tests ... done in 51 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

9893-e44167d/mint-dist-xl.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev7.minio.io:30206
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 5f73d483e73c:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 10 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 43 seconds
(4/15) Running aws-sdk-ruby tests ... done in 4 seconds
(5/15) Running awscli tests ... done in 3 minutes and 13 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 1 minutes and 3 seconds
(8/15) Running minio-dotnet tests ... done in 1 minutes and 53 seconds
(9/15) Running minio-go tests ... FAILED in 1 minutes and 49 seconds
{
  "args": {
    "bucketName": "minio-go-test-c6ekxzzgks0sln5r",
    "objectName": "d1pgtskozaxj5364fz52gugjr39xfp",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 3675,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... FAILED in 3 minutes and 50 seconds
{
  "name": "minio-java",
  "function": "enableObjectLegalHold(String bucketName, String objectName, String versionId)",
  "duration": 337,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = BucketNotEmpty, message = The bucket you tried to delete is not empty. You must delete all versions in the bucket., bucketName = minio-java-test-2uh8bi5, objectName = null, resource = /minio-java-test-2uh8bi5, requestId = 161AD745C24EF72C, hostId = 1aa5279a-6821-43da-b410-cde18a193a8f)\nrequest={method=DELETE, url=http://minio-dev7.minio.io:30206/minio-java-test-2uh8bi5, headers=Host: minio-dev7.minio.io:30206\nAccept-Encoding: identity\nUser-Agent: MinIO (amd64; amd64) minio-java/dev\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20200622T103512Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200622/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=409, headers=Accept-Ranges: bytes\nContent-Length: 373\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO/DEVELOPMENT.2020-06-22T10-16-36Z\nVary: Origin\nX-Amz-Request-Id: 161AD745C24EF72C\nX-Xss-Protection: 1; mode=block\nDate: Mon, 22 Jun 2020 10:35:12 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:1211), io.minio.MinioClient.execute(MinioClient.java:1236), io.minio.MinioClient.executeDelete(MinioClient.java:1366), io.minio.MinioClient.removeBucket(MinioClient.java:3686), FunctionalTest.enableObjectLegalHold_test(FunctionalTest.java:3314), FunctionalTest.runTests(FunctionalTest.java:4079), FunctionalTest.main(FunctionalTest.java:4192)]"
}
(9/15) Running minio-js tests ... done in 49 seconds
(10/15) Running minio-py tests ... done in 2 minutes and 25 seconds
(11/15) Running s3cmd tests ... done in 20 seconds
(12/15) Running s3select tests ... done in 12 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

9893-e44167d/mint-xl.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev3.minio.io:30039
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp a6760a175705:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 5 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 44 seconds
(4/15) Running aws-sdk-ruby tests ... done in 3 seconds
(5/15) Running awscli tests ... done in 1 minutes and 19 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 30 seconds
(8/15) Running minio-dotnet tests ... done in 37 seconds
(9/15) Running minio-go tests ... FAILED in 1 minutes and 21 seconds
{
  "args": {
    "bucketName": "minio-go-test-w2x9sb3vcevwceki",
    "objectName": "wjni3xx5edhbkujur4wdt3n35n13qa",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 3119,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... FAILED in 1 minutes and 55 seconds
{
  "name": "minio-java",
  "function": "enableObjectLegalHold(String bucketName, String objectName, String versionId)",
  "duration": 168,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = BucketNotEmpty, message = The bucket you tried to delete is not empty. You must delete all versions in the bucket., bucketName = minio-java-test-8mit1k, objectName = null, resource = /minio-java-test-8mit1k, requestId = 161AD6EF2074BA2C, hostId = 337e09ef-a18b-49e6-afd1-32ce27d22c1c)\nrequest={method=DELETE, url=http://minio-dev3.minio.io:30039/minio-java-test-8mit1k, headers=Host: minio-dev3.minio.io:30039\nAccept-Encoding: identity\nUser-Agent: MinIO (amd64; amd64) minio-java/dev\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20200622T102900Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200622/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=409, headers=Accept-Ranges: bytes\nContent-Length: 371\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO/DEVELOPMENT.2020-06-22T10-16-36Z\nVary: Origin\nX-Amz-Request-Id: 161AD6EF2074BA2C\nX-Xss-Protection: 1; mode=block\nDate: Mon, 22 Jun 2020 10:29:00 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:1211), io.minio.MinioClient.execute(MinioClient.java:1236), io.minio.MinioClient.executeDelete(MinioClient.java:1366), io.minio.MinioClient.removeBucket(MinioClient.java:3686), FunctionalTest.enableObjectLegalHold_test(FunctionalTest.java:3314), FunctionalTest.runTests(FunctionalTest.java:4079), FunctionalTest.main(FunctionalTest.java:4192)]"
}
(9/15) Running minio-js tests ... done in 48 seconds
(10/15) Running minio-py tests ... done in 4 minutes and 21 seconds
(11/15) Running s3cmd tests ... done in 22 seconds
(12/15) Running s3select tests ... done in 9 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

harshavardhana · 2020-06-22T15:33:21Z

@kannappanr both are valid in AWS S3

harshavardhana · 2020-06-22T16:12:46Z

@kannappanr both are valid in AWS S3

AFAIK @vadmeste we need to support both because we have users with s3:PutBucketLifecycle value already in their policies.

vadmeste · 2020-06-22T17:17:40Z

AFAIK @vadmeste we need to support both because we have users with s3:PutBucketLifecycle value already in their policies.

oups yes.. I think I can add a custom parser to replace s3:PutBucketLifecycle with s3:PutLifecycleConfiguration when loading the policy document. This seems a stable way to fix this with old deployments.

kannappanr · 2020-06-23T04:56:26Z

AFAIK @vadmeste we need to support both because we have users with s3:PutBucketLifecycle value already in their policies.

oups yes.. I think I can add a custom parser to replace s3:PutBucketLifecycle with s3:PutLifecycleConfiguration when loading the policy document. This seems a stable way to fix this with old deployments.

@vadmeste let me know if you want me to close this PR, if you will take care of both the issues?

vadmeste · 2020-06-23T15:06:15Z

@kannappanr, well, it seems that Get/PutBucketLifecycle permissions were never working for users. Meaning that if a user has a policy with an allowed GetBucketLifecycle or PutBucketLifecycle, he will still not able to set a new lifecycle document to a given bucket. I don't think we need to do anything other than keeping the old wrong permissions and add new ones.

harshavardhana · 2020-06-23T17:17:57Z

@kannappanr, well, it seems that Get/PutBucketLifecycle permissions were never working for users. Meaning that if a user has a policy with an allowed GetBucketLifecycle or PutBucketLifecycle, he will still not able to set a new lifecycle document to a given bucket. I don't think we need to do anything other than keeping the old wrong permissions and add new ones.

@vadmeste why is that?

LarsBrenker · 2020-06-24T08:06:44Z

@kannappanr, well, it seems that Get/PutBucketLifecycle permissions were never working for users. Meaning that if a user has a policy with an allowed GetBucketLifecycle or PutBucketLifecycle, he will still not able to set a new lifecycle document to a given bucket. I don't think we need to do anything other than keeping the old wrong permissions and add new ones.

I can confirm that. When I read that you want to include a mapping and both are supported, I wanted to try it out. But S3: GetBucketLifeCycle and S3: PutBucketLifeCycle didn't work either. I'm waiting for the patch. Thank you ;)

LarsBrenker · 2020-07-07T10:09:49Z

BugFix didn't work for me.

mc version RELEASE.2020-06-26T19-56-55Z
Server Version 2020-07-02T00:15:09Z

Message: Unable to parse the input policy: unsupported action 's3:GetLifecycleConfiguration'.

Maybe something needs to be adjusted in the client?

harshavardhana · 2020-07-07T18:53:42Z

BugFix didn't work for me.

mc version RELEASE.2020-06-26T19-56-55Z
Server Version 2020-07-02T00:15:09Z

Message: Unable to parse the input policy: unsupported action 's3:GetLifecycleConfiguration'.

Maybe something needs to be adjusted in the client?

Its a client issue @LarsBrenker

Fix: Policy action name for Get/Put BucketLifeCycle

e44167d

kannappanr requested review from vadmeste and harshavardhana June 22, 2020 09:21

klauspost approved these changes Jun 22, 2020

View reviewed changes

harshavardhana approved these changes Jun 23, 2020

View reviewed changes

harshavardhana merged commit b460b59 into minio:master Jun 23, 2020

harshavardhana mentioned this pull request Jun 23, 2020

Unable to parse the input policy: unsupported action 's3:GetLifecycleConfiguration'. minio/mc#3265

Closed

harshavardhana pushed a commit that referenced this pull request Jun 23, 2020

fix IAM policy action name for Get/PutBucketLifeCycle (#9893)

152090b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix: Policy action name for Get/Put BucketLifeCycle #9893

Fix: Policy action name for Get/Put BucketLifeCycle #9893

kannappanr commented Jun 22, 2020

klauspost left a comment

minio-trusted commented Jun 22, 2020

harshavardhana commented Jun 22, 2020

harshavardhana commented Jun 22, 2020

vadmeste commented Jun 22, 2020

kannappanr commented Jun 23, 2020

vadmeste commented Jun 23, 2020

harshavardhana commented Jun 23, 2020

LarsBrenker commented Jun 24, 2020

LarsBrenker commented Jul 7, 2020

harshavardhana commented Jul 7, 2020

Fix: Policy action name for Get/Put BucketLifeCycle #9893

Fix: Policy action name for Get/Put BucketLifeCycle #9893

Conversation

kannappanr commented Jun 22, 2020

Description

Motivation and Context

How to test this PR?

Types of changes

Checklist:

klauspost left a comment

Choose a reason for hiding this comment

minio-trusted commented Jun 22, 2020

Mint Automation

9893-e44167d/mint-large-bucket.sh.log:

9893-e44167d/mint-dist-xl.sh.log:

9893-e44167d/mint-xl.sh.log:

harshavardhana commented Jun 22, 2020

harshavardhana commented Jun 22, 2020

vadmeste commented Jun 22, 2020

kannappanr commented Jun 23, 2020

vadmeste commented Jun 23, 2020

harshavardhana commented Jun 23, 2020

LarsBrenker commented Jun 24, 2020

LarsBrenker commented Jul 7, 2020

harshavardhana commented Jul 7, 2020