gateway object tagging

[BigUstad]

Description

Object Tagging for Gateway.
PR depends on minio/minio-go#1294 for PutObject.

Motivation and Context

Tags for objects using minio gateway.

How to test this PR?

Run minio in gateway mode. Check results for commands get-object-tagging, put-object-tagging, delete-object-tagging, put-object with tagging, copy-object with tagging & create-multipart-upload with tagging (upload-part/complete-multipart-upload to complete the transfer).

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• [√ ] New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• Fixes a regression (If yes, please add commit-id or PR # here)
• Documentation needed
• [√ ] Unit tests needed
• Functional tests needed (If yes, add mint PR # here: )

[nitisht]

Generally looking good, can you mark it as ready for review @BigUstad

[kannappanr]

LGTM

[kannappanr]

@BigUstad Please resolve the conflicts

[BigUstad]

@BigUstad Is below a real failure?

(1/15) Running aws-sdk-go tests ... FAILED in 0 seconds
{
  "alert": "",
  "args": {
    "bucketName": "aws-sdk-go-test-9palgt6qjavlz9",
    "objectName": "nqzxrxwukm1p1kzsvpgx3mfyrsjxz2"
  },
  "duration": 7,
  "function": "testObjectTagging",
  "message": "AWS SDK Go PUTObject Tag input did not match with GetObjectTagging output <nil>",
  "name": "aws-sdk-go",
  "status": "FAIL"
}

Taking a look.

@balamurugana The failure is related to GetBucketObjectLockConfig(). PTAL.

Time: 12:25:00 PDT 05/18/2020
DeploymentID: 9342d887-3b78-4e81-b66c-50037dd174ea
Error: Not Implemented
       4: github.com/2-minio-gateway-object-tagging/cmd/gateway-unsupported.go:210:cmd.GatewayUnsupported.GetBucketObjectLockConfig()
       3: github.com/2-minio-gateway-object-tagging/cmd/bucket-object-lock.go:63:cmd.(*BucketObjectLockSys).Get()
       2: github.com/2-minio-gateway-object-tagging/cmd/bucket-handlers.go:911:cmd.objectAPIHandlers.DeleteBucketHandler()
       1: net/http/server.go:2007:http.HandlerFunc.ServeHTTP()

API: SYSTEM()
Time: 12:25:00 PDT 05/18/2020
DeploymentID: 9342d887-3b78-4e81-b66c-50037dd174ea
Error: Not Implemented
       5: github.com/2-minio-gateway-object-tagging/cmd/gateway-unsupported.go:210:cmd.GatewayUnsupported.GetBucketObjectLockConfig()
       4: github.com/2-minio-gateway-object-tagging/cmd/bucket-object-lock.go:63:cmd.(*BucketObjectLockSys).Get()
       3: github.com/2-minio-gateway-object-tagging/cmd/bucket-object-lock.go:348:cmd.checkPutObjectLockAllowed()
       2: github.com/2-minio-gateway-object-tagging/cmd/object-handlers.go:1383:cmd.objectAPIHandlers.PutObjectHandler()
       1: net/http/server.go:2007:http.HandlerFunc.ServeHTTP()```

[kannappanr]

@BigUstad Is below a real failure?

(1/15) Running aws-sdk-go tests ... FAILED in 0 seconds
{
  "alert": "",
  "args": {
    "bucketName": "aws-sdk-go-test-9palgt6qjavlz9",
    "objectName": "nqzxrxwukm1p1kzsvpgx3mfyrsjxz2"
  },
  "duration": 7,
  "function": "testObjectTagging",
  "message": "AWS SDK Go PUTObject Tag input did not match with GetObjectTagging output <nil>",
  "name": "aws-sdk-go",
  "status": "FAIL"
}

Taking a look.

@balamurugana The failure is related to GetBucketObjectLockConfig(). PTAL.

Time: 12:25:00 PDT 05/18/2020
DeploymentID: 9342d887-3b78-4e81-b66c-50037dd174ea
Error: Not Implemented
       4: github.com/2-minio-gateway-object-tagging/cmd/gateway-unsupported.go:210:cmd.GatewayUnsupported.GetBucketObjectLockConfig()
       3: github.com/2-minio-gateway-object-tagging/cmd/bucket-object-lock.go:63:cmd.(*BucketObjectLockSys).Get()
       2: github.com/2-minio-gateway-object-tagging/cmd/bucket-handlers.go:911:cmd.objectAPIHandlers.DeleteBucketHandler()
       1: net/http/server.go:2007:http.HandlerFunc.ServeHTTP()

API: SYSTEM()
Time: 12:25:00 PDT 05/18/2020
DeploymentID: 9342d887-3b78-4e81-b66c-50037dd174ea
Error: Not Implemented
       5: github.com/2-minio-gateway-object-tagging/cmd/gateway-unsupported.go:210:cmd.GatewayUnsupported.GetBucketObjectLockConfig()
       4: github.com/2-minio-gateway-object-tagging/cmd/bucket-object-lock.go:63:cmd.(*BucketObjectLockSys).Get()
       3: github.com/2-minio-gateway-object-tagging/cmd/bucket-object-lock.go:348:cmd.checkPutObjectLockAllowed()
       2: github.com/2-minio-gateway-object-tagging/cmd/object-handlers.go:1383:cmd.objectAPIHandlers.PutObjectHandler()
       1: net/http/server.go:2007:http.HandlerFunc.ServeHTTP()```

The errors do not have anything to do with this. PutObject with tagging is not working properly

[harshavardhana]

@BigUstad it is better to rebase and fix the build issues locally

[balamurugana]

LGTM

[kannappanr]

None of the tests that I did PutObejct with tagging, HeadObject is working correctly

[kannappanr]

$ git diff
diff --git a/cmd/gateway/s3/gateway-s3.go b/cmd/gateway/s3/gateway-s3.go
index 8fed5365..d124b876 100644
--- a/cmd/gateway/s3/gateway-s3.go
+++ b/cmd/gateway/s3/gateway-s3.go
@@ -454,7 +454,6 @@ func (l *s3Objects) GetObjectInfo(ctx context.Context, bucket string, object str
        if err != nil {
                return minio.ObjectInfo{}, minio.ErrorRespToObjectError(err, bucket, object)
        }
-
        return minio.FromMinioClientObjectInfo(bucket, oi), nil
 }
 
@@ -468,6 +467,7 @@ func (l *s3Objects) PutObject(ctx context.Context, bucket string, object string,
                        return objInfo, minio.ErrorRespToObjectError(err, bucket, object)
                }
                tagMap = tagObj.ToMap()
+               delete(opts.UserDefined, xhttp.AmzObjectTagging)
        }
        putOpts := miniogo.PutObjectOptions{
                UserMetadata:         opts.UserDefined,
diff --git a/cmd/http/headers.go b/cmd/http/headers.go
index 982a9ae4..261ac591 100644
--- a/cmd/http/headers.go
+++ b/cmd/http/headers.go
@@ -58,7 +58,7 @@ const (
 
        // S3 object tagging
        AmzObjectTagging = "X-Amz-Tagging"
-       AmzTagCount      = "X-Amz-Tag-Count"
+       AmzTagCount      = "X-Amz-Tagging-Count"
        AmzTagDirective  = "X-Amz-Tagging-Directive"
 

Along with changes in minio/minio-go#1291 will fix issues I saw with PutObject and HeadObejct. I have not tested copyobject, get and put obejct tagging commands

[kannappanr]

LGTM

[minio-trusted]

Mint Automation

Test	Result
mint-xl.sh	✔️
mint-large-bucket.sh	✔️
mint-fs.sh	✔️
mint-dist-xl.sh	✔️
mint-gateway-s3.sh	✔️
mint-gateway-azure.sh	✔️
mint-gateway-nas.sh	❌ more...

9124-d4ceed9/mint-gateway-nas.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev6.minio.io:31068
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp ce5dc9d5f098:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 1 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 1 seconds
(4/15) Running aws-sdk-ruby tests ... done in 2 seconds
(5/15) Running awscli tests ... done in 2 minutes and 5 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 35 seconds
(8/15) Running minio-dotnet tests ... done in 29 seconds
(9/15) Running minio-go tests ... done in 34 seconds
(10/15) Running minio-java tests ... done in 35 seconds
(11/15) Running minio-js tests ... FAILED in 14 seconds
{
  "name": "minio-js",
  "function": "getBucketPolicy(bucketName, cb)",
  "args": "bucketName:minio-js-test-f9c6bae2-6616-48de-8bb0-a310078ff37e",
  "duration": 6,
  "status": "FAIL",
  "error": "S3Error: The bucket policy does not exist at Object.parseError (node_modules/minio/dist/main/xml-parsers.js:86:11) at /mint/run/core/minio-js/node_modules/minio/dist/main/transformers.js:156:22 at DestroyableTransform._flush (node_modules/minio/dist/main/transformers.js:80:10) at DestroyableTransform.prefinish (node_modules/readable-stream/lib/_stream_transform.js:129:10) at prefinish (node_modules/readable-stream/lib/_stream_writable.js:611:14) at finishMaybe (node_modules/readable-stream/lib/_stream_writable.js:620:5) at endWritable (node_modules/readable-stream/lib/_stream_writable.js:643:3) at DestroyableTransform.Writable.end (node_modules/readable-stream/lib/_stream_writable.js:571:22) at IncomingMessage.onend (_stream_readable.js:673:10) at endReadableNT (_stream_readable.js:1201:12) at processTicksAndRejections (internal/process/task_queues.js:84:21)"
}
(11/15) Running minio-py tests ... done in 54 seconds
(12/15) Running s3cmd tests ... done in 15 seconds
(13/15) Running s3select tests ... done in 2 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally