Implement Alibaba Cloud OSS gateway support #5103

timonwong · 2017-10-23T08:03:54Z

Motivation and Context

Implement Alibaba Cloud (Aliyun) OSS gateway support.

API Documentation (very similar to AWS S3): https://www.alibabacloud.com/help/doc-detail/31947.htm

How Has This Been Tested?

Manually & Mint

Notes about failed mint tests:

Since OSS doesn't support underscores("_") in custom metadata headers, so aws-sdk-php will fail. (Workaround by sed -i 's/Param_1/Param-1/' /mint/run/core/aws-sdk-php/quick-tests.php)
minio-dotnet failed during put objects (tons of connect failed errors, may be caused by too many file descriptors)

Besides, in order to get mint tests pass, there is a check in CompleteMultipartUpload() which iterate uploaded parts, and error out if uploadedParts except last part sizing < 5MiB.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have added tests to cover my changes.
All new and existing tests passed.

codecov · 2017-10-23T13:45:21Z

Codecov Report

Merging #5103 into master will decrease coverage by 0.83%.
The diff coverage is 19.78%.

@@            Coverage Diff             @@
##           master    #5103      +/-   ##
==========================================
- Coverage    62.1%   61.26%   -0.84%     
==========================================
  Files         197      199       +2     
  Lines       28329    28900     +571     
==========================================
+ Hits        17593    17705     +112     
- Misses       9407     9863     +456     
- Partials     1329     1332       +3

Impacted Files	Coverage Δ
cmd/gateway/oss/gateway-oss-anonymous.go	`0% <0%> (ø)`
cmd/gateway/oss/gateway-oss.go	`20.21% <20.21%> (ø)`
cmd/fs-v1-rwpool.go	`64.34% <0%> (-2.61%)`	⬇️
cmd/fs-v1-background-append.go	`79.86% <0%> (+1.34%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8c08571...8611d3b. Read the comment docs.

houndci-bot · 2017-10-28T01:43:22Z

cmd/gateway-oss.go

+	return ossBackend
+}
+
+func (g *OSSGateway) NewGatewayLayer() (GatewayLayer, error) {


exported method OSSGateway.NewGatewayLayer should have comment or be unexported

houndci-bot · 2017-10-28T01:43:22Z

cmd/gateway-oss.go

+	host string
+}
+
+func (g *OSSGateway) Name() string {


exported method OSSGateway.Name should have comment or be unexported

houndci-bot · 2017-10-28T01:43:22Z

cmd/gateway-oss.go

+	startGateway(ctx, &OSSGateway{host})
+}
+
+type OSSGateway struct {


exported type OSSGateway should have comment or be unexported

houndci-bot · 2017-10-28T08:37:00Z

cmd/gateway-oss.go

+	host string
+}
+
+// OSSGateway implements Gateway interface.


comment on exported method OSSGateway.Name should be of the form "Name ..."

harshavardhana · 2017-12-06T03:01:44Z

@timonwong i have rebased upon your work timonwong#1 to point to latest master, made relevant changes. Please merge the PR, we can then plan on merging this PR.

timonwong · 2017-12-06T05:45:07Z

@harshavardhana Thanks :) And besides, I just did a rebase due to CI failure after merging...

harshavardhana · 2017-12-06T07:55:12Z

cmd/gateway/oss/gateway-oss_test.go

@@ -0,0 +1,177 @@
+package oss


can you add license header here @timonwong ? just like other gateways?

@harshavardhana Fixed

harshavardhana · 2017-12-06T07:55:30Z

cmd/gateway/oss/gateway-oss.go

@@ -0,0 +1,847 @@
+package oss


license header.

timonwong · 2017-12-12T03:12:10Z

After running sed: sed -i 's/Param_1/Param-1/' /mint/run/core/aws-sdk-php/quick-tests.php, now I can pass aws-sdk-php tests, but run in trouble in minio-dotnet tests.

It throws tons of "Couldn't connect to server" error messages in error.json, may be I'm running out of file descriptors (yeah I'm using macOS...)?

To get logs, run 'sudo docker cp 7ea6ee893b75:/mint/log /tmp/mint-logs'
(1/11) Running aws-sdk-go tests ... done in 1 seconds
(2/11) Running aws-sdk-php tests ... done in 1 minutes and 17 seconds
(3/11) Running aws-sdk-ruby tests ... done in 52 seconds
(4/11) Running awscli tests ... done in 3 minutes and 29 seconds
(5/11) Running mc tests ... done in 8 minutes and 37 seconds
(6/11) Running minio-dotnet tests ... ./run.sh: line 26:  1041 Aborted                 /mint/run/core/minio-dotnet/out/Minio.Functional.Tests >> "$output_log_file" 2> "$error_log_file"
FAILED in 4 minutes and 37 seconds
parse error: Invalid numeric literal at line 1, column 6
   at Minio.Functional.Tests.FunctionalTest.<PutObject_Task>d__28.MoveNext() in /mint/run/core/minio-dotnet/FunctionalTest.cs:line 630

Unhandled Exception: System.AggregateException: One or more errors occurred. (One or more errors occurred. (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. )) ---> System.AggregateException: One or more errors occurred. (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) (Assert.Fail failed. ) ---> Microsoft.VisualStudio.TestTools.UnitTesting.AssertFailedException: Assert.Fail failed.
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.HandleFail(String assertionName, String message, Object[] parameters)
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.Fail()
   at Minio.Functional.Tests.FunctionalTest.<PutObject_Task>d__28.MoveNext() in /mint/run/core/minio-dotnet/FunctionalTest.cs:line 642
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at Minio.Functional.Tests.FunctionalTest.<RemoveObjects_Test2>d__51.MoveNext() in /mint/run/core/minio-dotnet/FunctionalTest.cs:line 1493
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at Minio.Functional.Tests.FunctionalTest.Main(String[] args) in /mint/run/core/minio-dotnet/FunctionalTest.cs:line 196

Executed 5 out of 11 tests successfully.

[Bucket]  Exception: Unsuccessful response from server without XML error: An error occurred while sending the request. Couldn't connect to server: Minio.Exceptions.InternalClientException: Minio API responded with message=Unsuccessful response from server without XML error: An error occurred while sending the request. Couldn't connect to server
   at Minio.MinioClient.ParseError(IRestResponse response)
   at Minio.MinioClient.<>c.<.ctor>b__78_0(IRestResponse response)
   at Minio.MinioClient.HandleIfErrorResponse(IRestResponse response, IEnumerable`1 handlers, DateTime startTime)
   at Minio.MinioClient.<ExecuteTaskAsync>d__81.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Minio.MinioClient.<PutObjectAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Minio.MinioClient.<PutObjectAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Minio.Functional.Tests.FunctionalTest.<PutObject_Task>d__28.MoveNext() in /mint/run/core/minio-dotnet/FunctionalTest.cs:line 630
[Bucket]  Exception: Unsuccessful response from server without XML error: An error occurred while sending the request. Couldn't connect to server: Minio.Exceptions.InternalClientException: Minio API responded with message=Unsuccessful response from server without XML error: An error occurred while sending the request. Couldn't connect to server
   at Minio.MinioClient.ParseError(IRestResponse response)
   at Minio.MinioClient.<>c.<.ctor>b__78_0(IRestResponse response)
   at Minio.MinioClient.HandleIfErrorResponse(IRestResponse response, IEnumerable`1 handlers, DateTime startTime)
   at Minio.MinioClient.<ExecuteTaskAsync>d__81.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Minio.MinioClient.<PutObjectAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Minio.MinioClient.<PutObjectAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Minio.Functional.Tests.FunctionalTest.<PutObject_Task>d__28.MoveNext() in /mint/run/core/minio-dotnet/FunctionalTest.cs:line 630
[Bucket]  Exception: Unsuccessful response from server without XML error: An error occurred while sending the request. Couldn't connect to server: Minio.Exceptions.InternalClientException: Minio API responded with message=Unsuccessful response from server without XML error: An error occurred while sending the request. Couldn't connect to server
   at Minio.MinioClient.ParseError(IRestResponse response)
   at Minio.MinioClient.<>c.<.ctor>b__78_0(IRestResponse response)
   at Minio.MinioClient.HandleIfErrorResponse(IRestResponse response, IEnumerable`1 handlers, DateTime startTime)
   at Minio.MinioClient.<ExecuteTaskAsync>d__81.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Minio.MinioClient.<PutObjectAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Minio.MinioClient.<PutObjectAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
...

harshavardhana · 2017-12-12T03:23:09Z

@timonwong you can run only specific tests by docker run minio/mint:edge minio-go minio-py mc minio-java aws-sdk-go aws-sdk-ruby

harshavardhana · 2017-12-12T03:24:09Z

it is okay that these fail and we can catch up in perhaps future versions, just wanted to see how ready is this support . If things are failing perhaps you can write down which all tests fail we can review and deal with this in case by case basis.

timonwong · 2017-12-12T05:17:58Z

Current condition:

aws-sdk-php failed with custom metadata header with underscores (after patching without underscore, it passes)
minio-dotnet failed
s3cmd Failed in test_put_object, throws "MD5 Sums don't match" warnings (etag mismatch, OSS returns uppercase string, however, md5.hexdigest() in Python returns lowercase string).

All other tests passes.

(1/5) Running minio-go tests ... done in 19 minutes and 23 seconds
(2/5) Running minio-java tests ... done in 43 seconds
(3/5) Running minio-js tests ... done in 1 minutes and 28 seconds
(4/5) Running minio-py tests ... done in 2 minutes and 31 seconds
(5/5) Running s3cmd tests ... FAILED in 10 seconds
{
  "name": "s3cmd",
  "duration": "7431",
  "function": "test_put_object",
  "status": "FAIL",
  "error": "WARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-1-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-1-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-1-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-1-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-1-MB\nWARNING: MD5 Sums don't match!\nWARNING: Too many failures. Giving up on '/mint/data/datafile-1-MB'\nERROR: Upload of '/mint/data/datafile-1-MB' failed too many times (Last reason: )"
}

harshavardhana · 2017-12-12T05:27:22Z

For md5 you can simply add -1 at the end here . We do that with all gateways which do not support direct md5sum.

timonwong · 2017-12-12T06:13:35Z

For md5 you can simply add -1 at the end here . We do that with all gateways which do not support direct md5sum.

Actually OSS did have MD5 Etag, but it's upper-cased.
In s3cmd, it calculate md5 hash of uploaded content itself (https://github.com/s3tools/s3cmd/blob/6d8e621738f4b37a4bbc0b5b28d1c0717264f7d8/S3/S3.py#L1548).

So I should just use minio.ToS3ETag?

harshavardhana · 2017-12-12T06:27:31Z

Yes @timonwong just use that .it should be fine

timonwong · 2017-12-12T06:43:51Z

Ok, so now s3cmd passes :)

harshavardhana · 2017-12-12T08:46:13Z

Ok, so now s3cmd passes :)

So other than aws-sdk-php all tests pass?? @timonwong

timonwong · 2017-12-12T10:10:07Z

So other than aws-sdk-php all tests pass?? @timonwong

@harshavardhana And minio-dotnet failed due to tons of "Couldn't connect to server" errors.

harshavardhana · 2017-12-12T17:15:34Z

@harshavardhana And minio-dotnet failed due to tons of "Couldn't connect to server" errors.

That is nice to know @timonwong LGTM we can take this change in.

timonwong · 2017-12-13T03:11:45Z

Please just wait for a moment, I need to investigate some weird behavior in CompleteMultipartUpload

timonwong · 2017-12-13T06:33:16Z

Mint tests verified:

aws-sdk-go
aws-sdk-php
aws-sdk-ruby

docker run --rm -e SERVER_ENDPOINT=docker.for.mac.localhost:9000 -e ACCESS_KEY=$MINIO_ACCESS_KEY -e SECRET_KEY=$MINIO_SECRET_KEY -e ENABLE_HTTPS=0 mint2 aws-sdk-go aws-sdk-php aws-sdk-ruby
Running with
SERVER_ENDPOINT: docker.for.mac.localhost:9000
ACCESS_KEY:      ***REDACTED***
SECRET_KEY:      ***REDACTED***
ENABLE_HTTPS:    0
SERVER_REGION:   us-east-1
MINT_DATA_DIR:   /mint/data
MINT_MODE:       core

To get logs, run 'sudo docker cp f16f3755d3d3:/mint/log /tmp/mint-logs'
(1/3) Running aws-sdk-go tests ... done in 5 seconds
(2/3) Running aws-sdk-php tests ... done in 1 minutes and 9 seconds
(3/3) Running aws-sdk-ruby tests ... done in 48 seconds

All tests ran successfully

minio-java
minio-js
minio-py

docker run --rm -e SERVER_ENDPOINT=docker.for.mac.localhost:9000 -e ACCESS_KEY=$MINIO_ACCESS_KEY -e SECRET_KEY=$MINIO_SECRET_KEY -e ENABLE_HTTPS=0 mint2 minio-java minio-js minio-py
Running with
SERVER_ENDPOINT: docker.for.mac.localhost:9000
ACCESS_KEY:      ***REDACTED***
SECRET_KEY:      ***REDACTED***
ENABLE_HTTPS:    0
SERVER_REGION:   us-east-1
MINT_DATA_DIR:   /mint/data
MINT_MODE:       core

To get logs, run 'sudo docker cp 0552eda8334a:/mint/log /tmp/mint-logs'
(1/3) Running minio-java tests ... done in 56 seconds
(2/3) Running minio-js tests ... done in 2 minutes and 16 seconds
(3/3) Running minio-py tests ... done in 3 minutes and 52 seconds

All tests ran successfully

s3cmd

docker run --rm -e SERVER_ENDPOINT=docker.for.mac.localhost:9000 -e ACCESS_KEY=$MINIO_ACCESS_KEY -e SECRET_KEY=$MINIO_SECRET_KEY -e ENABLE_HTTPS=0 mint2 s3cmd
Running with
SERVER_ENDPOINT: docker.for.mac.localhost:9000
ACCESS_KEY:      ***REDACTED***
SECRET_KEY:      ***REDACTED***
ENABLE_HTTPS:    0
SERVER_REGION:   us-east-1
MINT_DATA_DIR:   /mint/data
MINT_MODE:       core

To get logs, run 'sudo docker cp 28a6e865a68b:/mint/log /tmp/mint-logs'
(1/1) Running s3cmd tests ... done in 7 minutes and 33 seconds

All tests ran successfully

awscli
mc

docker run --rm -e SERVER_ENDPOINT=docker.for.mac.localhost:9000 -e ACCESS_KEY=$MINIO_ACCESS_KEY -e SECRET_KEY=$MINIO_SECRET_KEY -e ENABLE_HTTPS=0 mint2 awscli mc
Running with
SERVER_ENDPOINT: docker.for.mac.localhost:9000
ACCESS_KEY:      ***REDACTED***
SECRET_KEY:      ***REDACTED***
ENABLE_HTTPS:    0
SERVER_REGION:   us-east-1
MINT_DATA_DIR:   /mint/data
MINT_MODE:       core

To get logs, run 'sudo docker cp fbe4a313d265:/mint/log /tmp/mint-logs'
(1/2) Running awscli tests ... done in 4 minutes and 49 seconds
(2/2) Running mc tests ... done in 11 minutes and 3 seconds

All tests ran successfully

krishnasrinivas

Did not test it as I don't have alibaba account details. I had reviewed the code which looks good. If the mint is passing then it should be good to go.

timonwong force-pushed the add-aliyun-oss branch 4 times, most recently from 4961a7f to 0398226 Compare October 23, 2017 08:44

timonwong changed the title ~~WIP: Implement Aliyun OSS gateway support~~ [WIP] Implement Aliyun OSS gateway support Oct 23, 2017

timonwong force-pushed the add-aliyun-oss branch from 0398226 to 181002a Compare October 23, 2017 13:45

timonwong force-pushed the add-aliyun-oss branch 2 times, most recently from 0df199c to bc6c3a4 Compare October 24, 2017 02:52

timonwong mentioned this pull request Oct 24, 2017

Simplify gateway backend registration #5111

Merged

8 tasks

timonwong force-pushed the add-aliyun-oss branch from bc6c3a4 to 2ce9e88 Compare October 28, 2017 01:43

houndci-bot reviewed Oct 28, 2017

View reviewed changes

timonwong force-pushed the add-aliyun-oss branch from 2ce9e88 to 1a3468d Compare October 28, 2017 08:36

houndci-bot reviewed Oct 28, 2017

View reviewed changes

timonwong force-pushed the add-aliyun-oss branch from 1a3468d to 6d05cdc Compare October 28, 2017 08:38

timonwong changed the title ~~[WIP] Implement Aliyun OSS gateway support~~ [WIP] Implement Alibaba Cloud OSS gateway support Oct 28, 2017

timonwong force-pushed the add-aliyun-oss branch 5 times, most recently from 5674a21 to 73be360 Compare October 28, 2017 11:22

timonwong changed the title ~~[WIP] Implement Alibaba Cloud OSS gateway support~~ Implement Alibaba Cloud OSS gateway support Oct 28, 2017

deekoder requested review from harshavardhana and krishnasrinivas November 1, 2017 17:50

timonwong and others added 2 commits December 6, 2017 13:36

Implement Aliyun OSS gateway support

d091743

Migrate to new style

19f2885

timonwong force-pushed the add-aliyun-oss branch from e5f3661 to 19f2885 Compare December 6, 2017 05:37

harshavardhana requested changes Dec 6, 2017

View reviewed changes

timonwong force-pushed the add-aliyun-oss branch from d9c2f5c to 302db0d Compare December 11, 2017 13:01

timonwong force-pushed the add-aliyun-oss branch 4 times, most recently from e2565a9 to 99932bd Compare December 12, 2017 06:10

timonwong force-pushed the add-aliyun-oss branch from 99932bd to 29086b3 Compare December 12, 2017 06:17

timonwong force-pushed the add-aliyun-oss branch from 29086b3 to e150b2a Compare December 12, 2017 06:52

harshavardhana previously approved these changes Dec 12, 2017

View reviewed changes

Aliyun OSS: Fix mint errors

8611d3b

timonwong dismissed harshavardhana’s stale review via 8611d3b December 13, 2017 03:46

timonwong force-pushed the add-aliyun-oss branch from e150b2a to 8611d3b Compare December 13, 2017 03:46

harshavardhana approved these changes Dec 13, 2017

View reviewed changes

krishnasrinivas approved these changes Dec 19, 2017

View reviewed changes

nitisht merged commit 84fc78d into minio:master Dec 19, 2017

timonwong deleted the add-aliyun-oss branch December 19, 2017 09:05

jkatz mentioned this pull request Aug 21, 2020

Add alicloud OSS repository of backup CrunchyData/postgres-operator#1803

Closed

deanwong mentioned this pull request Sep 29, 2020

Recovery gateway of Alibaba Cloud OSS with lastest oss-go-sdk #10592

Closed

6 tasks

Implement Alibaba Cloud OSS gateway support #5103

Implement Alibaba Cloud OSS gateway support #5103

timonwong commented Oct 23, 2017 •

edited

codecov bot commented Oct 23, 2017 •

edited

houndci-bot Oct 28, 2017

houndci-bot Oct 28, 2017

houndci-bot Oct 28, 2017

houndci-bot Oct 28, 2017

harshavardhana commented Dec 6, 2017

timonwong commented Dec 6, 2017

harshavardhana Dec 6, 2017

timonwong Dec 6, 2017

harshavardhana Dec 6, 2017

timonwong commented Dec 12, 2017 •

edited

harshavardhana commented Dec 12, 2017

harshavardhana commented Dec 12, 2017

timonwong commented Dec 12, 2017 •

edited

harshavardhana commented Dec 12, 2017 •

edited

timonwong commented Dec 12, 2017 •

edited

harshavardhana commented Dec 12, 2017 •

edited

timonwong commented Dec 12, 2017

harshavardhana commented Dec 12, 2017

timonwong commented Dec 12, 2017 •

edited

harshavardhana commented Dec 12, 2017

timonwong commented Dec 13, 2017

timonwong commented Dec 13, 2017

krishnasrinivas left a comment

Implement Alibaba Cloud OSS gateway support #5103

Implement Alibaba Cloud OSS gateway support #5103

Conversation

timonwong commented Oct 23, 2017 • edited

Motivation and Context

How Has This Been Tested?

Types of changes

Checklist:

codecov bot commented Oct 23, 2017 • edited

Codecov Report

houndci-bot Oct 28, 2017

Choose a reason for hiding this comment

houndci-bot Oct 28, 2017

Choose a reason for hiding this comment

houndci-bot Oct 28, 2017

Choose a reason for hiding this comment

houndci-bot Oct 28, 2017

Choose a reason for hiding this comment

harshavardhana commented Dec 6, 2017

timonwong commented Dec 6, 2017

harshavardhana Dec 6, 2017

Choose a reason for hiding this comment

timonwong Dec 6, 2017

Choose a reason for hiding this comment

harshavardhana Dec 6, 2017

Choose a reason for hiding this comment

timonwong commented Dec 12, 2017 • edited

harshavardhana commented Dec 12, 2017

harshavardhana commented Dec 12, 2017

timonwong commented Dec 12, 2017 • edited

harshavardhana commented Dec 12, 2017 • edited

timonwong commented Dec 12, 2017 • edited

harshavardhana commented Dec 12, 2017 • edited

timonwong commented Dec 12, 2017

harshavardhana commented Dec 12, 2017

timonwong commented Dec 12, 2017 • edited

harshavardhana commented Dec 12, 2017

timonwong commented Dec 13, 2017

timonwong commented Dec 13, 2017

krishnasrinivas left a comment

Choose a reason for hiding this comment

timonwong commented Oct 23, 2017 •

edited

codecov bot commented Oct 23, 2017 •

edited

timonwong commented Dec 12, 2017 •

edited

timonwong commented Dec 12, 2017 •

edited

harshavardhana commented Dec 12, 2017 •

edited

timonwong commented Dec 12, 2017 •

edited

harshavardhana commented Dec 12, 2017 •

edited

timonwong commented Dec 12, 2017 •

edited