Azure: Check if container is empty before issuing DeleteContainer #9733

kannappanr · 2020-05-29T10:32:30Z

Description

Containers in Azure can be deleted even if there are objects within them. Azure gateway should behave like S3 wherever possible. So, it should not delete a bucket which is not empty.

Motivation and Context

Motivation is to make Azure Gateway behave like S3

How to test this PR?

Create a bucket using Azure Gatewat and upload an object and issue Delete Bucket S3 API and see the bucket getting deleted.
With this PR, that will not hapen.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

Fixes a regression (If yes, please add commit-id or PR # here)
Documentation needed
Unit tests needed
Functional tests needed (If yes, add mint PR # here: )

cmd/gateway/azure/gateway-azure.go

vadmeste · 2020-05-29T18:00:22Z

By the way, I guess it is okay to ignore incomplete multipart uploads ? I don't see why we need to consider it but just wanted to share with you.

harshavardhana · 2020-05-29T18:08:30Z

By the way, I guess it is okay to ignore incomplete multipart uploads ? I don't see why we need to consider it but just wanted to share with you.

No we don't care about them @vadmeste S3 delete's bucket with incomplete uploads

vadmeste

LGTM & tested

minio-trusted · 2020-05-29T21:23:10Z

Mint Automation

Test	Result
mint-large-bucket.sh	✔️
mint-fs.sh	✔️
mint-dist-xl.sh	✔️
mint-gateway-s3.sh	✔️
mint-gateway-nas.sh	✔️
mint-xl.sh	❌ more...
mint-gateway-azure.sh	❌ more...

9733-af95ae8/mint-xl.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev7.minio.io:30397
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 245988277dcd:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 1 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 42 seconds
(4/15) Running aws-sdk-ruby tests ... done in 1 seconds
(5/15) Running awscli tests ... done in 2 minutes and 13 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 33 seconds
(8/15) Running minio-dotnet tests ... done in 31 seconds
(9/15) Running minio-go tests ... done in 47 seconds
(10/15) Running minio-java tests ... done in 54 seconds
(11/15) Running minio-js tests ... FAILED in 18 seconds
{
  "name": "minio-js",
  "function": "\"after all\" hook in \"functional tests\"",
  "duration": 20,
  "status": "FAIL",
  "error": "S3Error: The bucket you tried to delete is not empty at Object.parseError (node_modules/minio/dist/main/xml-parsers.js:86:11) at /mint/run/core/minio-js/node_modules/minio/dist/main/transformers.js:156:22 at DestroyableTransform._flush (node_modules/minio/dist/main/transformers.js:80:10) at DestroyableTransform.prefinish (node_modules/readable-stream/lib/_stream_transform.js:129:10) at prefinish (node_modules/readable-stream/lib/_stream_writable.js:611:14) at finishMaybe (node_modules/readable-stream/lib/_stream_writable.js:620:5) at endWritable (node_modules/readable-stream/lib/_stream_writable.js:643:3) at DestroyableTransform.Writable.end (node_modules/readable-stream/lib/_stream_writable.js:571:22) at IncomingMessage.onend (_stream_readable.js:673:10) at endReadableNT (_stream_readable.js:1201:12) at processTicksAndRejections (internal/process/task_queues.js:84:21)"
}
(11/15) Running minio-py tests ... done in 2 minutes and 26 seconds
(12/15) Running s3cmd tests ... done in 20 seconds
(13/15) Running s3select tests ... done in 2 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

9733-af95ae8/mint-gateway-azure.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev7.minio.io:32003
ACCESS_KEY:           minioazure
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 6d4976f38ad7:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 8 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 27 seconds
(4/15) Running aws-sdk-ruby tests ... done in 15 seconds
(5/15) Running awscli tests ... done in 2 minutes and 53 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 3 minutes and 7 seconds
(8/15) Running minio-dotnet tests ... done in 1 minutes and 24 seconds
(9/15) Running minio-go tests ... done in 3 minutes and 44 seconds
(10/15) Running minio-java tests ... FAILED in 23 minutes and 45 seconds
[minio-java-test-3ed6fg9]: threaded put object
java.io.EOFException
	at okio.RealBufferedSink.write(RealBufferedSink.java:115)
	at io.minio.HttpRequestBody.writeTo(HttpRequestBody.java:76)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:83)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:125)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:264)
	at okhttp3.RealCall.execute(RealCall.java:93)
	at io.minio.MinioClient.execute(MinioClient.java:1097)
	at io.minio.MinioClient.execute(MinioClient.java:1236)
	at io.minio.MinioClient.executePut(MinioClient.java:1410)
	at io.minio.MinioClient.executePut(MinioClient.java:1424)
	at io.minio.MinioClient.uploadPart(MinioClient.java:5492)
	at io.minio.MinioClient.putObject(MinioClient.java:3740)
	at io.minio.MinioClient.putObject(MinioClient.java:3822)
	at PutObjectRunnable.run(PutObjectRunnable.java:42)
	at java.base/java.lang.Thread.run(Thread.java:834)
error occurred
ErrorResponse(code = BucketNotEmpty, message = The bucket you tried to delete is not empty, bucketName = minio-java-test-14pnt7u, objectName = null, resource = /minio-java-test-14pnt7u, requestId = 16139BCE933A3DFC, hostId = 1f7d38d3-4b74-46d0-9950-b7b264abbbcd)
request={method=DELETE, url=http://minio-dev7.minio.io:32003/minio-java-test-14pnt7u, headers=Host: minio-dev7.minio.io:32003
Accept-Encoding: identity
User-Agent: MinIO (amd64; amd64) minio-java/dev
Content-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20200529T210644Z
Authorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200529/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*
}
response={code=409, headers=Accept-Ranges: bytes
Content-Length: 328
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Server: MinIO/DEVELOPMENT.GOGET
Vary: Origin
X-Amz-Request-Id: 16139BCE933A3DFC
X-Xss-Protection: 1; mode=block
Date: Fri, 29 May 2020 21:06:45 GMT
}

	at io.minio.MinioClient.execute(MinioClient.java:1211)
	at io.minio.MinioClient.execute(MinioClient.java:1236)
	at io.minio.MinioClient.executeDelete(MinioClient.java:1366)
	at io.minio.MinioClient.removeBucket(MinioClient.java:3686)
	at FunctionalTest.teardown(FunctionalTest.java:414)
	at FunctionalTest.runTests(FunctionalTest.java:4122)
	at FunctionalTest.main(FunctionalTest.java:4192)
(10/15) Running minio-js tests ... done in 2 minutes and 39 seconds
(11/15) Running minio-py tests ... done in 9 minutes and 41 seconds
(12/15) Running s3cmd tests ... done in 2 minutes and 24 seconds
(13/15) Running s3select tests ... done in 1 minutes and 1 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

kannappanr requested review from harshavardhana and vadmeste May 29, 2020 10:32

vadmeste requested changes May 29, 2020

View reviewed changes

cmd/gateway/azure/gateway-azure.go Outdated Show resolved Hide resolved

cmd/gateway/azure/gateway-azure.go Outdated Show resolved Hide resolved

kannappanr force-pushed the azure_delete branch 3 times, most recently from 363cf23 to e7a5cde Compare May 29, 2020 17:33

Azure: Check if container is empty before issuing DeleteContainer

af95ae8

kannappanr force-pushed the azure_delete branch from e7a5cde to af95ae8 Compare May 29, 2020 17:34

harshavardhana approved these changes May 29, 2020

View reviewed changes

vadmeste approved these changes May 29, 2020

View reviewed changes

harshavardhana merged commit d583f1a into minio:master May 29, 2020

blaenk pushed a commit to blaenk/minio that referenced this pull request Aug 26, 2020

check if container is empty before invoking DeleteContainer (minio#9733)

60e619d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Azure: Check if container is empty before issuing DeleteContainer #9733

Azure: Check if container is empty before issuing DeleteContainer #9733

kannappanr commented May 29, 2020

vadmeste commented May 29, 2020

harshavardhana commented May 29, 2020

vadmeste left a comment

minio-trusted commented May 29, 2020

Azure: Check if container is empty before issuing DeleteContainer #9733

Azure: Check if container is empty before issuing DeleteContainer #9733

Conversation

kannappanr commented May 29, 2020

Description

Motivation and Context

How to test this PR?

Types of changes

Checklist:

vadmeste commented May 29, 2020

harshavardhana commented May 29, 2020

vadmeste left a comment

Choose a reason for hiding this comment

minio-trusted commented May 29, 2020

Mint Automation

9733-af95ae8/mint-xl.sh.log:

9733-af95ae8/mint-gateway-azure.sh.log: