Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add import/export APIs to migrate bucket metadata #14929

Merged
merged 2 commits into from
Jun 18, 2022

Conversation

poornas
Copy link
Contributor

@poornas poornas commented May 16, 2022

Description

Motivation and Context

Allow ability to migrate bucket metadata for ease of administration

How to test this PR?

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Optimization (provides speedup with no functional changes)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • Fixes a regression (If yes, please add commit-id or PR # here)
  • Documentation updated
  • Unit tests added/updated

@klauspost
Copy link
Contributor

Running lint check
cmd/admin-bucket-handlers.go:453: File is not `gofumpt`-ed (gofumpt)

cmd/admin-router.go Outdated Show resolved Hide resolved
cmd/admin-bucket-handlers.go Outdated Show resolved Hide resolved
cmd/admin-bucket-handlers.go Outdated Show resolved Hide resolved
cmd/admin-bucket-handlers.go Show resolved Hide resolved
cmd/admin-bucket-handlers.go Outdated Show resolved Hide resolved
@harshavardhana
Copy link
Member

@poornas we made mc release for this but server-side change has not been merged - can this be rebased and made available for the next MinIO release?

@minio-trusted
Copy link
Contributor

Mint Automation

Test Result
mint-large-bucket.sh ✔️
mint-gateway-s3.sh ✔️
mint-dist-erasure.sh ✔️
mint-compress-encrypt-dist-erasure.sh ✔️
mint-pools.sh ✔️
mint-fs.sh more...
mint-erasure.sh more...

14929-234d717/mint-erasure.sh.log:

Running with
SERVER_ENDPOINT:      minio-k8s5:31169
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0
RUN_ON_FAIL:          0

To get logs, run 'docker cp :/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 7 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 43 seconds
(4/15) Running aws-sdk-ruby tests ... done in 5 seconds
(5/15) Running awscli tests ... done in 56 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 34 seconds
FAILED in 7 minutes and 34 seconds
Unhandled exception. System.AggregateException: One or more errors occurred. (A task was canceled.)
 ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Threading.Tasks.Task.GetExceptions(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at Minio.Functional.Tests.Program.Main(String[] args) in /mint/run/core/minio-dotnet/temp/Minio.Functional.Tests/Program.cs:line 194
--- End of stack trace from previous location ---

   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at Minio.Functional.Tests.Program.Main(String[] args) in /mint/run/core/minio-dotnet/temp/Minio.Functional.Tests/Program.cs:line 194
(8/15) Running minio-go tests ... done in 55 seconds
(9/15) Running minio-java tests ... done in 19 seconds
(10/15) Running minio-js tests ... done in 59 seconds
(11/15) Running minio-py tests ... done in 50 seconds
(12/15) Running s3cmd tests ... done in 24 seconds
(13/15) Running s3select tests ... done in 1 seconds
(14/15) Running versioning tests ... done in 3 minutes and 2 seconds

Executed 14 out of 15 tests successfully.

14929-234d717/mint-fs.sh.log:

Running with
SERVER_ENDPOINT:      minio-k8s1:30961
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0
RUN_ON_FAIL:          0

To get logs, run 'docker cp :/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 8 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 42 seconds
(4/15) Running aws-sdk-ruby tests ... done in 5 seconds
(5/15) Running awscli tests ... done in 53 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 23 seconds
FAILED in 7 minutes and 47 seconds
Unhandled exception. System.AggregateException: One or more errors occurred. (A task was canceled.)
 ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Threading.Tasks.Task.GetExceptions(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at Minio.Functional.Tests.Program.Main(String[] args) in /mint/run/core/minio-dotnet/temp/Minio.Functional.Tests/Program.cs:line 201
--- End of stack trace from previous location ---

   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at Minio.Functional.Tests.Program.Main(String[] args) in /mint/run/core/minio-dotnet/temp/Minio.Functional.Tests/Program.cs:line 201
(8/15) Running minio-go tests ... done in 1 minutes and 5 seconds
(9/15) Running minio-java tests ... done in 18 seconds
(10/15) Running minio-js tests ... done in 59 seconds
(11/15) Running minio-py tests ... done in 47 seconds
(12/15) Running s3cmd tests ... done in 23 seconds
(13/15) Running s3select tests ... done in 1 seconds
(14/15) Running versioning tests ... done in 3 minutes and 2 seconds

Executed 14 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

@harshavardhana harshavardhana merged commit 2fa1d8a into minio:master Jun 18, 2022
@harshavardhana harshavardhana deleted the migratemeta branch June 18, 2022 13:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants