Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more friendly error message for policy object #7412

Merged
merged 2 commits into from Apr 22, 2019

Conversation

lunixoid
Copy link
Contributor

In our projects we use minio as the storage sever. Each bucket equals - git project. After building project CI uploads artefact into minio and sets mc policy download for subdirectory. Name of subdirectory in bucket depends on name of current brunch (develop, master).
So after some time of working our bucket policy file reach limit 20K and send us error:
Your proposed upload exceeds the maximum allowed object size.

We spent a lot of time to understand and resolve this issue, so I suggest to extend error message for this case.

@lunixoid lunixoid force-pushed the friendly_policy_error branch 2 times, most recently from 9c416c3 to 820626e Compare March 22, 2019 08:54
@codecov
Copy link

codecov bot commented Mar 22, 2019

Codecov Report

Merging #7412 into master will increase coverage by 0.02%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7412      +/-   ##
==========================================
+ Coverage    47.4%   47.43%   +0.02%     
==========================================
  Files         294      294              
  Lines       46740    46740              
==========================================
+ Hits        22158    22172      +14     
+ Misses      22528    22517      -11     
+ Partials     2054     2051       -3
Impacted Files Coverage Δ
cmd/api-errors.go 50.62% <ø> (ø) ⬆️
cmd/bucket-policy-handlers.go 88.99% <100%> (ø) ⬆️
cmd/posix.go 63.84% <0%> (+0.1%) ⬆️
cmd/fs-v1.go 62.18% <0%> (+0.37%) ⬆️
cmd/fs-v1-helpers.go 68.8% <0%> (+0.61%) ⬆️
cmd/bitrot-streaming.go 83.17% <0%> (+0.93%) ⬆️
cmd/posix-list-dir_unix.go 75% <0%> (+2.63%) ⬆️
cmd/retry.go 87.5% <0%> (+5.35%) ⬆️
cmd/disk-usage.go 78.94% <0%> (+10.52%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d42496c...b7e1f5e. Read the comment docs.

@poornas
Copy link
Contributor

poornas commented Mar 22, 2019

@lunixoid, trying to understand why you need to give download access on each individual subdirectory prefix when you might want to just give download access on bucket/* which would in essence accomplish the same without policy bloat.

Copy link
Member

@harshavardhana harshavardhana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, some review comments..

cmd/api-errors.go Outdated Show resolved Hide resolved
cmd/api-errors.go Outdated Show resolved Hide resolved
cmd/typed-errors.go Outdated Show resolved Hide resolved
cmd/api-errors.go Outdated Show resolved Hide resolved
cmd/object-api-errors.go Outdated Show resolved Hide resolved
@lunixoid
Copy link
Contributor Author

lunixoid commented Apr 3, 2019

@lunixoid, trying to understand why you need to give download access on each individual subdirectory prefix when you might want to just give download access on bucket/* which would in essence accomplish the same without policy bloat.

It was my mistake, but in any case current error message don't help to understand the problem.
I did it because mc prints links to the each file in bucket only if I set policy download for each subdirectory. I use those links for auto downloading files while CI stages.

Copy link
Member

@harshavardhana harshavardhana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you check why CI failed ?

Copy link
Contributor

@poornas poornas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@minio minio deleted a comment from minio-ops Apr 18, 2019
@minio-ops
Copy link

Mint Automation

Test Result
mint-compression-xl.sh ✔️
mint-xl.sh ✔️
mint-compression-dist-xl.sh ✔️
mint-compression-fs.sh ✔️
mint-worm.sh ✔️
mint-dist-xl.sh ✔️
mint-gateway-nas.sh ✔️
mint-large-bucket.sh more...
mint-fs.sh more...

7412-b7e1f5e/mint-fs.sh.log:

Running with
SERVER_ENDPOINT:      72.28.97.60:31953
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp d222bbee8411:/mint/log /tmp/mint-logs'

(1/14) Running aws-sdk-go tests ... FAILED in 1 seconds
{
  "alert": "",
  "args": {
    "bucketName": "aws-sdk-go-test-oxvib3i3jp0e69",
    "expiry": 60000000000,
    "objectName": "presignedTest"
  },
  "duration": 214,
  "error": "RequestError: send request failed\ncaused by: Put http://72.28.97.60:31953/aws-sdk-go-test-oxvib3i3jp0e69: dial tcp 72.28.97.60:31953: connect: connection refused",
  "function": "PresignedPut",
  "message": "AWS SDK Go CreateBucket Failed",
  "name": "aws-sdk-go",
  "status": "FAIL"
}

Executed 0 out of 14 tests successfully.

7412-b7e1f5e/mint-large-bucket.sh.log:

Running with
SERVER_ENDPOINT:      72.28.97.55:32177
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp d80237274fd0:/mint/log /tmp/mint-logs'

(1/14) Running aws-sdk-go tests ... FAILED in 0 seconds
{
  "alert": "",
  "args": {
    "bucketName": "aws-sdk-go-test-t3wehwifls2f0w",
    "expiry": 60000000000,
    "objectName": "presignedTest"
  },
  "duration": 261,
  "error": "RequestError: send request failed\ncaused by: Put http://72.28.97.55:32177/aws-sdk-go-test-t3wehwifls2f0w: dial tcp 72.28.97.55:32177: connect: connection refused",
  "function": "PresignedPut",
  "message": "AWS SDK Go CreateBucket Failed",
  "name": "aws-sdk-go",
  "status": "FAIL"
}

Executed 0 out of 14 tests successfully.

@harshavardhana
Copy link
Member

ping @poornas

@harshavardhana harshavardhana merged commit 188cf1d into minio:master Apr 22, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants