Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update on-disk storage format for users system #7949

Merged
merged 3 commits into from Jul 25, 2019

Conversation

donatello
Copy link
Member

Description

  • Policy mapping is now at config/iam/policydb/users/myuser1.json
    and includes version.

  • User identity file is now versioned.

  • Migrate old data to the new format.

Motivation and Context

This change is in preparation to add the user-groups feature to MinIO server, so that server admins can create and manage user groups.

In this change, the policy mapped to a user is being moved and on disk json files are being versioned.

Migration from old format to new in included in this change.

How to test this PR?

Two things to test:

  1. Start server on this PR, and check if users system works as before.

  2. Start server on master, create some users. Then build this PR and start the server, and make sure users created on master are present in new version on this PR and works as expected.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • Fixes a regression (If yes, please add commit-id or PR # here)
  • Documentation needed
  • Unit tests needed
  • Functional tests needed (If yes, add mint PR # here: )

@donatello
Copy link
Member Author

Remaining items before making this PR ready for merge:

  1. Implement migration of etcd data.
  2. Testing.

@donatello donatello force-pushed the user-groups branch 6 times, most recently from 5f21df9 to 847c8d0 Compare July 22, 2019 18:52
@donatello donatello marked this pull request as ready for review July 22, 2019 19:11
@minio minio deleted a comment from minio-ops Jul 23, 2019
@minio minio deleted a comment from minio-ops Jul 23, 2019
@codecov-io
Copy link

codecov-io commented Jul 23, 2019

Codecov Report

Merging #7949 into master will increase coverage by 0.34%.
The diff coverage is 16.75%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7949      +/-   ##
==========================================
+ Coverage   44.78%   45.13%   +0.34%     
==========================================
  Files         302      314      +12     
  Lines       38947    50659   +11712     
==========================================
+ Hits        17442    22863    +5421     
- Misses      19399    25674    +6275     
- Partials     2106     2122      +16
Impacted Files Coverage Δ
cmd/admin-handlers.go 10.51% <0%> (-0.18%) ⬇️
cmd/sts-handlers.go 10.82% <0%> (+0.63%) ⬆️
cmd/iam.go 15% <16.81%> (+2.82%) ⬆️
cmd/disk-usage.go 78.94% <0%> (-21.06%) ⬇️
cmd/service.go 27.27% <0%> (-10.23%) ⬇️
pkg/madmin/utils.go 16.66% <0%> (-7.15%) ⬇️
cmd/auth-handler.go 62.19% <0%> (-7.1%) ⬇️
cmd/config-dir.go 40.74% <0%> (-7.09%) ⬇️
cmd/globals.go 55.88% <0%> (-7.09%) ⬇️
cmd/gateway-main.go 14.22% <0%> (-7.04%) ⬇️
... and 278 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 55d4eee...128af90. Read the comment docs.

@donatello donatello force-pushed the user-groups branch 3 times, most recently from 76e52bf to 3953f9c Compare July 24, 2019 00:44
- Policy mapping is now at `config/iam/policydb/users/myuser1.json`
  and includes version.

- User identity file is now versioned.

- Migrate old data to the new format.
Copy link
Contributor

@kannappanr kannappanr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Tested with both etcd and non-etcd scenarios

@minio-ops
Copy link

Mint Automation

Test Result
mint-xl.sh ✔️
mint-compression-dist-xl.sh ✔️
mint-compression-fs.sh ✔️
mint-worm.sh ✔️
mint-fs.sh ✔️
mint-gateway-nas.sh ✔️
mint-compression-xl.sh more...
mint-large-bucket.sh more...
mint-dist-xl.sh more...

7949-906e5b5/mint-large-bucket.sh.log:

Running with
SERVER_ENDPOINT:      72.28.97.52:31759
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 234752a10b86:/mint/log /tmp/mint-logs'

(1/14) Running aws-sdk-go tests ... done in 2 seconds
(2/14) Running aws-sdk-java tests ... done in 2 seconds
(3/14) Running aws-sdk-php tests ... done in 49 seconds
(4/14) Running aws-sdk-ruby tests ... done in 13 seconds
(5/14) Running awscli tests ... done in 1 minutes and 23 seconds
(6/14) Running healthcheck tests ... done in 0 seconds
(7/14) Running mc tests ... FAILED in 2 minutes and 5 seconds
{
  "name": "mc",
  "duration": "85554",
  "function": "test_mirror_list_objects_storage_class",
  "status": "FAIL",
  "error": "/mint/run/core/mc/mc --config-dir /tmp/.mc-9416 --quiet --no-color mirror --storage-class REDUCED_REDUNDANCY /mint/data myminio/mc-test-bucket-28395  >>> \n`/mint/data/datafile-1-MB` -> `myminio/mc-test-bucket-28395/datafile-1-MB`\n`/mint/data/datafile-0-b` -> `myminio/mc-test-bucket-28395/datafile-0-b`\n`/mint/data/datafile-1.03-MB` -> `myminio/mc-test-bucket-28395/datafile-1.03-MB`\n`/mint/data/datafile-1-kB` -> `myminio/mc-test-bucket-28395/datafile-1-kB`\n`/mint/data/datafile-10-MB` -> `myminio/mc-test-bucket-28395/datafile-10-MB`\n`/mint/data/datafile-10-kB` -> `myminio/mc-test-bucket-28395/datafile-10-kB`\n`/mint/data/datafile-1-b` -> `myminio/mc-test-bucket-28395/datafile-1-b`\n`/mint/data/datafile-129-MB` -> `myminio/mc-test-bucket-28395/datafile-129-MB`\n`/mint/data/datafile-100-kB` -> `myminio/mc-test-bucket-28395/datafile-100-kB`\n`/mint/data/datafile-11-MB` -> `myminio/mc-test-bucket-28395/datafile-11-MB`\n`/mint/data/datafile-5-MB` -> `myminio/mc-test-bucket-28395/datafile-5-MB`\n`/mint/data/datafile-5243880-b` -> `myminio/mc-test-bucket-28395/datafile-5243880-b`\n`/mint/data/datafile-33-kB` -> `myminio/mc-test-bucket-28395/datafile-33-kB`\n`/mint/data/datafile-6-MB` -> `myminio/mc-test-bucket-28395/datafile-6-MB`\n`/mint/data/datafile-65-MB` -> `myminio/mc-test-bucket-28395/datafile-65-MB`\nmc: <ERROR> Failed to copy `/mint/data/datafile-65-MB`. Please reduce your request\nmc: <ERROR> Failed to copy `/mint/data/datafile-11-MB`. Please reduce your request\nmc: <ERROR> Failed to copy `/mint/data/datafile-10-MB`. Please reduce your request\nmc: <ERROR> Failed to copy `/mint/data/datafile-129-MB`. Please reduce your request\nTotal: 233.17 MiB, Transferred: 459.30 MiB, Speed: 5.37 MiB/s"
}

Executed 6 out of 14 tests successfully.

7949-906e5b5/mint-dist-xl.sh.log:

Running with
SERVER_ENDPOINT:      72.28.97.61:31717
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp b6f4e4f87153:/mint/log /tmp/mint-logs'

(1/14) Running aws-sdk-go tests ... done in 0 seconds
(2/14) Running aws-sdk-java tests ... done in 1 seconds
(3/14) Running aws-sdk-php tests ... done in 45 seconds
(4/14) Running aws-sdk-ruby tests ... done in 3 seconds
(5/14) Running awscli tests ... done in 1 minutes and 10 seconds
(6/14) Running healthcheck tests ... done in 0 seconds
(7/14) Running mc tests ... done in 56 seconds
(8/14) Running minio-dotnet tests ... done in 39 seconds
(9/14) Running minio-go tests ... done in 1 minutes and 8 seconds
(10/14) Running minio-java tests ... FAILED in 1 minutes and 15 seconds
{
  "name": "minio-java",
  "function": "listIncompleteUploads(final String bucketName, final String prefix, final boolean recursive)",
  "args": "prefix: minio, recursive: true",
  "duration": 298,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code=SlowDown, message=Please reduce your request, bucketName=minio-java-test-b7fnm9, objectName=minio-java-test-p4b6da, resource=/minio-java-test-b7fnm9/minio-java-test-p4b6da, requestId=15B477F0CC91167D, hostId=83403594-619c-4cb5-8f67-417e75f4f48d)\nrequest={method=DELETE, url=http://72.28.97.61:31717/minio-java-test-b7fnm9/minio-java-test-p4b6da?uploadId=df5026cc-3054-48ef-900e-30a733cad46f, headers=Host: 72.28.97.61:31717\nUser-Agent: MinIO (amd64; amd64) minio-java/dev\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20190724T222046Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20190724/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=503, headers=Accept-Ranges: bytes\nContent-Length: 359\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nRetry-After: 120\nServer: MinIO/DEVELOPMENT.2019-07-24T22-01-53Z\nVary: Origin\nX-Amz-Request-Id: 15B477F0CC91167D\nX-Xss-Protection: 1; mode=block\nDate: Wed, 24 Jul 2019 22:20:46 GMT\n}\n >>> [io.minio.MinioClient.executeReq(MinioClient.java:1204), io.minio.MinioClient.execute(MinioClient.java:1070), io.minio.MinioClient.executeDelete(MinioClient.java:1347), io.minio.MinioClient.abortMultipartUpload(MinioClient.java:5149), io.minio.MinioClient.putObject(MinioClient.java:4337), io.minio.MinioClient.putObject(MinioClient.java:4183), FunctionalTest.listIncompleteUploads_test3(FunctionalTest.java:1679), FunctionalTest.runTests(FunctionalTest.java:2990), FunctionalTest.main(FunctionalTest.java:3118)]"
}

Executed 9 out of 14 tests successfully.

7949-906e5b5/mint-compression-xl.sh.log:

Running with
SERVER_ENDPOINT:      72.28.97.57:30682
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp efb408227bc4:/mint/log /tmp/mint-logs'

(1/14) Running aws-sdk-go tests ... FAILED in 12 seconds
{
  "alert": "",
  "args": {
    "bucketName": "aws-sdk-go-test-ej06hro4wuj2pm",
    "expiry": 60000000000,
    "objectName": "presignedTest"
  },
  "duration": 12530,
  "error": "RequestError: send request failed\ncaused by: Put http://72.28.97.57:30682/aws-sdk-go-test-ej06hro4wuj2pm: dial tcp 72.28.97.57:30682: connect: no route to host",
  "function": "PresignedPut",
  "message": "AWS SDK Go CreateBucket Failed",
  "name": "aws-sdk-go",
  "status": "FAIL"
}

Executed 0 out of 14 tests successfully.

cmd/iam.go Outdated Show resolved Hide resolved
@kannappanr kannappanr merged commit 7bdaf9b into minio:master Jul 25, 2019
harshavardhana added a commit to harshavardhana/minio that referenced this pull request Aug 4, 2022
```
commit 7bdaf9b
Author: Aditya Manthramurthy <donatello@users.noreply.github.com>
Date:   Wed Jul 24 17:34:23 2019 -0700

    Update on-disk storage format for users system (minio#7949)
```
harshavardhana added a commit to harshavardhana/minio that referenced this pull request Aug 4, 2022
```
commit 7bdaf9b
Author: Aditya Manthramurthy <donatello@users.noreply.github.com>
Date:   Wed Jul 24 17:34:23 2019 -0700

    Update on-disk storage format for users system (minio#7949)
```

Bonus: fixes a bug when etcd keys were being re-encrypted.
harshavardhana added a commit to harshavardhana/minio that referenced this pull request Aug 4, 2022
```
commit 7bdaf9b
Author: Aditya Manthramurthy <donatello@users.noreply.github.com>
Date:   Wed Jul 24 17:34:23 2019 -0700

    Update on-disk storage format for users system (minio#7949)
```

Bonus: fixes a bug when etcd keys were being re-encrypted.
harshavardhana added a commit to harshavardhana/minio that referenced this pull request Aug 4, 2022
```
commit 7bdaf9b
Author: Aditya Manthramurthy <donatello@users.noreply.github.com>
Date:   Wed Jul 24 17:34:23 2019 -0700

    Update on-disk storage format for users system (minio#7949)
```

Bonus: fixes a bug when etcd keys were being re-encrypted.
harshavardhana added a commit to harshavardhana/minio that referenced this pull request Aug 4, 2022
```
commit 7bdaf9b
Author: Aditya Manthramurthy <donatello@users.noreply.github.com>
Date:   Wed Jul 24 17:34:23 2019 -0700

    Update on-disk storage format for users system (minio#7949)
```

Bonus: fixes a bug when etcd keys were being re-encrypted.
harshavardhana added a commit to harshavardhana/minio that referenced this pull request Aug 5, 2022
```
commit 7bdaf9b
Author: Aditya Manthramurthy <donatello@users.noreply.github.com>
Date:   Wed Jul 24 17:34:23 2019 -0700

    Update on-disk storage format for users system (minio#7949)
```

Bonus: fixes a bug when etcd keys were being re-encrypted.
minio-trusted pushed a commit that referenced this pull request Aug 6, 2022
```
commit 7bdaf9b
Author: Aditya Manthramurthy <donatello@users.noreply.github.com>
Date:   Wed Jul 24 17:34:23 2019 -0700

    Update on-disk storage format for users system (#7949)
```

Bonus: fixes a bug when etcd keys were being re-encrypted.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants