Fix replication regression

[poornas]

From PR #11165 due to incorrect
proxying for 2 way replication even
when the object was not yet replicated

Also fixing metadata comparisons when
deciding to do full replciation vs metadata copy.

Also Fixes #11340

Description

Motivation and Context

How to test this PR?

Turn on 2 way replication and do some uploads to source. mc admin trace should show a PUT operation on the dest, not a COPY in master due to incorrect proxying.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Optimization (provides speedup with no functional changes)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• Fixes a regression (If yes, please add commit-id or PR # here)
• Documentation updated
• Unit tests added/updated

[vadmeste]

Some questions, but tested and it works

[vadmeste]

LGTM & tested

[minio-trusted]

Mint Automation

Test	Result
mint-large-bucket.sh	✔️
mint-fs.sh	✔️
mint-gateway-s3.sh	✔️
mint-erasure.sh	✔️
mint-dist-erasure.sh	✔️
mint-zoned.sh	✔️
mint-gateway-nas.sh	✔️
mint-gateway-azure.sh	❌ more...

11356-941f750/mint-gateway-azure.sh.log:

Running with
SERVER_ENDPOINT:      minio-c2.minio.io:31440
ACCESS_KEY:           minioazure
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp dbab526f51b4:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 11 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 3 minutes and 49 seconds
(4/15) Running aws-sdk-ruby tests ... done in 21 seconds
(5/15) Running awscli tests ... done in 3 minutes and 3 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 39 minutes and 37 seconds
(8/15) Running minio-dotnet tests ... done in 4 minutes and 12 seconds
(9/15) Running minio-go tests ... done in 26 minutes and 54 seconds
(10/15) Running minio-java tests ... FAILED in 9 minutes and 18 seconds
{
  "name": "minio-java",
  "function": "putObject()",
  "args": "[user metadata]",
  "duration": 169,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = AuthenticationFailed, message = -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, github.com/Azure/azure-storage-blob-go@v0.10.0/azblob/zc_storage_error.go:42\n===== RESPONSE ERROR (ServiceCode=AuthenticationFailed) =====\nDescription=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:28a157f7-e01e-0050-65e3-f4f2e6000000\nTime:2021-01-27T19:35:15.5158279Z, Details: \n   AuthenticationErrorDetail: The MAC signature found in the HTTP request 'Oy5KNJJ3h8jCwOLOofTNqfU5be1e/Lg/u6oOCJkEtHg=' is not the same as any computed signature. Server used following string to sign: 'PUT\n\n\n128\n\napplication/xml\n\n\n\n\n\n\nx-ms-blob-cache-control:\nx-ms-blob-content-disposition:\nx-ms-blob-content-encoding:\nx-ms-blob-content-language:\nx-ms-blob-content-type:application/octet-stream\nx-ms-client-request-id:8beaa75b-2a04-467a-45d5-bb92a10c1f1d\nx-ms-date:Wed, 27 Jan 2021 19:35:15 GMT\nx-ms-meta-my_header1:a   b   c\nx-ms-meta-my_header2:\"a   b   c\"\nx-ms-meta-my_project:Project One\nx-ms-meta-my_unicode_tag:商å“�\nx-ms-version:2019-02-02\n/minioazure/minio-java-test-1mil7kh/minio-java-test-1hilfrb\ncomp:blocklist\ntimeout:1501'.\n   Code: AuthenticationFailed\n   PUT https://minioazure.blob.core.windows.net/minio-java-test-1mil7kh/minio-java-test-1hilfrb?comp=blocklist&timeout=1501\n   Authorization: REDACTED\n   Content-Length: [128]\n   Content-Type: [application/xml]\n   User-Agent: [APN/1.0 MinIO/1.0 MinIO/2021-01-27T18:01:26Z]\n   X-Ms-Blob-Cache-Control: []\n   X-Ms-Blob-Content-Disposition: []\n   X-Ms-Blob-Content-Encoding: []\n   X-Ms-Blob-Content-Language: []\n   X-Ms-Blob-Content-Type: [application/octet-stream]\n   X-Ms-Client-Request-Id: [8beaa75b-2a04-467a-45d5-bb92a10c1f1d]\n   X-Ms-Date: [Wed, 27 Jan 2021 19:35:15 GMT]\n   X-Ms-Meta-My_header1: [a   b   c]\n   X-Ms-Meta-My_header2: [\"a   b   c\"]\n   X-Ms-Meta-My_project: [Project One]\n   X-Ms-Meta-My_unicode_tag: [商品]\n   X-Ms-Version: [2019-02-02]\n   --------------------------------------------------------------------------------\n   RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\n   Content-Length: [1092]\n   Content-Type: [application/xml]\n   Date: [Wed, 27 Jan 2021 19:35:15 GMT]\n   Server: [Microsoft-HTTPAPI/2.0]\n   X-Ms-Error-Code: [AuthenticationFailed]\n   X-Ms-Request-Id: [28a157f7-e01e-0050-65e3-f4f2e6000000]\n\n\n, bucketName = minio-java-test-1mil7kh, objectName = minio-java-test-1hilfrb, resource = /minio-java-test-1mil7kh/minio-java-test-1hilfrb, requestId = 165E2DD660417345, hostId = c4d4245d-28bd-4c36-a10d-58974a9be285)\nrequest={method=PUT, url=http://minio-c2.minio.io:31440/minio-java-test-1mil7kh/minio-java-test-1hilfrb, headers=x-amz-meta-My-Unicode-Tag: 商品\nx-amz-meta-My-Project: Project One\nx-amz-meta-My-header1: a   b   c\nx-amz-meta-My-Header2: \"a   b   c\"\nContent-Type: application/octet-stream\nHost: minio-c2.minio.io:31440\nAccept-Encoding: identity\nUser-Agent: MinIO (Linux; amd64) minio-java/8.0.3\nContent-MD5: A9oFTxee7YVcJ9fWsgQeKg==\nx-amz-content-sha256: 1ff7959f86334ddc5c188a5083268f600146328b2b6c5185e75bf7d9387d6b74\nx-amz-date: 20210127T193515Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20210127/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date;x-amz-meta-my-header1;x-amz-meta-my-header2;x-amz-meta-my-project;x-amz-meta-my-unicode-tag, Signature=*REDACTED*\n}\nresponse={code=403, headers=Accept-Ranges: bytes\nContent-Length: 3086\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO\nVary: Origin\nX-Amz-Request-Id: 165E2DD660417345\nX-Xss-Protection: 1; mode=block\nDate: Wed, 27 Jan 2021 19:35:15 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:775), io.minio.MinioClient.putObject(MinioClient.java:4547), io.minio.MinioClient.putObject(MinioClient.java:2713), io.minio.MinioClient.putObject(MinioClient.java:2830), FunctionalTest.testPutObject(FunctionalTest.java:763), FunctionalTest.putObject(FunctionalTest.java:890), FunctionalTest.runObjectTests(FunctionalTest.java:3751), FunctionalTest.runTests(FunctionalTest.java:3783), FunctionalTest.main(FunctionalTest.java:3927)]"
}
(10/15) Running minio-js tests ... done in 3 minutes and 56 seconds
(11/15) Running minio-py tests ... done in 22 minutes and 43 seconds
(12/15) Running s3cmd tests ... done in 9 minutes and 53 seconds
(13/15) Running s3select tests ... done in 1 minutes and 7 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally