Trace requests rejected

[kerneltime]

Description

The hierarchy of func calls needs to re juggled to allow for tracing requests rejected due to busy server.

Motivation and Context

Requests rejected by 503 were not traced.
Latency measurements were in correct.

How to test this PR?

Change the maxClients code to reject all and run the server.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• Fixes a regression (If yes, please add commit-id or PR # here)
• Documentation needed
• Unit tests needed

[kannappanr]

LGTM

[minio-trusted]

Mint Automation

Test	Result
mint-large-bucket.sh	✔️
mint-fs.sh	✔️
mint-gateway-s3.sh	✔️
mint-erasure.sh	✔️
mint-dist-erasure.sh	✔️
mint-zoned.sh	✔️
mint-gateway-nas.sh	✔️
mint-gateway-azure.sh	❌ more...

11192-551ab85/mint-gateway-azure.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev7.minio.io:32415
ACCESS_KEY:           minioazure
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 29b53736d05a:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 8 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 1 minutes and 22 seconds
(4/15) Running aws-sdk-ruby tests ... done in 20 seconds
(5/15) Running awscli tests ... done in 2 minutes and 58 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 3 minutes and 37 seconds
(8/15) Running minio-dotnet tests ... done in 1 minutes and 41 seconds
(9/15) Running minio-go tests ... done in 6 minutes and 20 seconds
(10/15) Running minio-java tests ... FAILED in 9 minutes and 0 seconds
{
  "name": "minio-java",
  "function": "putObject()",
  "args": "[user metadata]",
  "duration": 173,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = AuthenticationFailed, message = -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, github.com/Azure/azure-storage-blob-go@v0.10.0/azblob/zc_storage_error.go:42\n===== RESPONSE ERROR (ServiceCode=AuthenticationFailed) =====\nDescription=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:033d0b2e-d01e-00f9-382d-df270e000000\nTime:2020-12-31T04:28:58.2870701Z, Details: \n   AuthenticationErrorDetail: The MAC signature found in the HTTP request 'cErg7m8UKLrrRr+qTbvyl5ZQ5QUXNTvf3n6qYTEAYPY=' is not the same as any computed signature. Server used following string to sign: 'PUT\n\n\n128\n\napplication/xml\n\n\n\n\n\n\nx-ms-blob-cache-control:\nx-ms-blob-content-disposition:\nx-ms-blob-content-encoding:\nx-ms-blob-content-language:\nx-ms-blob-content-type:application/octet-stream\nx-ms-client-request-id:7542ba44-8146-462f-46e0-ca56bec4bb1f\nx-ms-date:Thu, 31 Dec 2020 04:28:58 GMT\nx-ms-meta-my_header1:a   b   c\nx-ms-meta-my_header2:\"a   b   c\"\nx-ms-meta-my_project:Project One\nx-ms-meta-my_unicode_tag:商å“�\nx-ms-version:2019-02-02\n/minioazure/minio-java-test-2ojrbac/minio-java-test-3fsjm0b\ncomp:blocklist\ntimeout:1501'.\n   Code: AuthenticationFailed\n   PUT https://minioazure.blob.core.windows.net/minio-java-test-2ojrbac/minio-java-test-3fsjm0b?comp=blocklist&timeout=1501\n   Authorization: REDACTED\n   Content-Length: [128]\n   Content-Type: [application/xml]\n   User-Agent: [APN/1.0 MinIO/1.0 MinIO/2020-12-31T03:57:14Z]\n   X-Ms-Blob-Cache-Control: []\n   X-Ms-Blob-Content-Disposition: []\n   X-Ms-Blob-Content-Encoding: []\n   X-Ms-Blob-Content-Language: []\n   X-Ms-Blob-Content-Type: [application/octet-stream]\n   X-Ms-Client-Request-Id: [7542ba44-8146-462f-46e0-ca56bec4bb1f]\n   X-Ms-Date: [Thu, 31 Dec 2020 04:28:58 GMT]\n   X-Ms-Meta-My_header1: [a   b   c]\n   X-Ms-Meta-My_header2: [\"a   b   c\"]\n   X-Ms-Meta-My_project: [Project One]\n   X-Ms-Meta-My_unicode_tag: [商品]\n   X-Ms-Version: [2019-02-02]\n   --------------------------------------------------------------------------------\n   RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\n   Content-Length: [1092]\n   Content-Type: [application/xml]\n   Date: [Thu, 31 Dec 2020 04:28:58 GMT]\n   Server: [Microsoft-HTTPAPI/2.0]\n   X-Ms-Error-Code: [AuthenticationFailed]\n   X-Ms-Request-Id: [033d0b2e-d01e-00f9-382d-df270e000000]\n\n\n, bucketName = minio-java-test-2ojrbac, objectName = minio-java-test-3fsjm0b, resource = /minio-java-test-2ojrbac/minio-java-test-3fsjm0b, requestId = 1655B2B65D36FABF, hostId = 53a6b2ed-f838-4277-b238-3eff29d0aaf5)\nrequest={method=PUT, url=http://minio-dev7.minio.io:32415/minio-java-test-2ojrbac/minio-java-test-3fsjm0b, headers=x-amz-meta-My-Unicode-Tag: 商品\nx-amz-meta-My-Project: Project One\nx-amz-meta-My-header1: a   b   c\nx-amz-meta-My-Header2: \"a   b   c\"\nContent-Type: application/octet-stream\nHost: minio-dev7.minio.io:32415\nAccept-Encoding: identity\nUser-Agent: MinIO (Linux; amd64) minio-java/8.0.3\nContent-MD5: A9oFTxee7YVcJ9fWsgQeKg==\nx-amz-content-sha256: 1ff7959f86334ddc5c188a5083268f600146328b2b6c5185e75bf7d9387d6b74\nx-amz-date: 20201231T042858Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20201231/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date;x-amz-meta-my-header1;x-amz-meta-my-header2;x-amz-meta-my-project;x-amz-meta-my-unicode-tag, Signature=*REDACTED*\n}\nresponse={code=403, headers=Accept-Ranges: bytes\nContent-Length: 3086\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO\nVary: Origin\nX-Amz-Request-Id: 1655B2B65D36FABF\nX-Xss-Protection: 1; mode=block\nDate: Thu, 31 Dec 2020 04:28:58 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:775), io.minio.MinioClient.putObject(MinioClient.java:4547), io.minio.MinioClient.putObject(MinioClient.java:2713), io.minio.MinioClient.putObject(MinioClient.java:2830), FunctionalTest.testPutObject(FunctionalTest.java:763), FunctionalTest.putObject(FunctionalTest.java:890), FunctionalTest.runObjectTests(FunctionalTest.java:3751), FunctionalTest.runTests(FunctionalTest.java:3783), FunctionalTest.main(FunctionalTest.java:3927)]"
}
(10/15) Running minio-js tests ... done in 2 minutes and 37 seconds
(11/15) Running minio-py tests ... done in 18 minutes and 34 seconds
(12/15) Running s3cmd tests ... done in 2 minutes and 13 seconds
(13/15) Running s3select tests ... done in 58 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally