fix: export prometheus metrics for cache GC triggers #9815

harshavardhana · 2020-06-12T05:55:24Z

Description

fix: export prometheus metrics for cache GC triggers

Motivation and Context

Bonus change to use channel to serialize triggers,
instead of using atomic variables. A more efficient
mechanism for synchronization.

How to test this PR?

Nothing special fill the cache and check for prometheus metrics
to display cache specific gcCounter values

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

Fixes a regression (If yes, please add commit-id or PR # here)
Documentation needed
Unit tests needed
Functional tests needed (If yes, add mint PR # here: )

cmd/disk-cache-backend.go

cmd/metrics.go

cmd/disk-cache.go

cmd/disk-cache-backend.go

poornas

tested,LGTM

harshavardhana · 2020-06-15T00:44:21Z

PTAL @nitisht @kannappanr

Bonus change to use channel to serialize triggers, instead of using atomic variables. More efficient mechanism for synchronization.

nitisht

LGTM!

minio-trusted · 2020-06-15T09:01:46Z

Mint Automation

Test	Result
mint-fs.sh	✔️
mint-gateway-s3.sh	✔️
mint-gateway-azure.sh	✔️
mint-gateway-nas.sh	✔️
mint-xl.sh	❌ more...
mint-large-bucket.sh	❌ more...
mint-dist-xl.sh	❌ more...

9815-5362ee7/mint-large-bucket.sh.log:

Running with
SERVER_ENDPOINT:      minio-c3.minio.io:32647
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 762131ccb754:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 5 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 47 seconds
(4/15) Running aws-sdk-ruby tests ... done in 9 seconds
(5/15) Running awscli tests ... done in 2 minutes and 41 seconds
(6/15) Running healthcheck tests ... done in 1 seconds
(7/15) Running mc tests ... done in 1 minutes and 1 seconds
(8/15) Running minio-dotnet tests ... done in 1 minutes and 26 seconds
(9/15) Running minio-go tests ... FAILED in 1 minutes and 55 seconds
{
  "args": {
    "bucketName": "minio-go-test-hsevnyvbs1ah4f1n",
    "objectName": "ayghiq3wwyzux3xgmks63vd12zvjj5",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 4808,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... FAILED in 6 minutes and 40 seconds
{
  "name": "minio-java",
  "function": "enableObjectLegalHold(String bucketName, String objectName, String versionId)",
  "duration": 884,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = BucketNotEmpty, message = The bucket you tried to delete is not empty. You must delete all versions in the bucket., bucketName = minio-java-test-3utundb, objectName = null, resource = /minio-java-test-3utundb, requestId = 1618A9E8AD463281, hostId = abe7d880-80cd-4595-a47e-8db5fe22ec04)\nrequest={method=DELETE, url=http://minio-c3.minio.io:32647/minio-java-test-3utundb, headers=Host: minio-c3.minio.io:32647\nAccept-Encoding: identity\nUser-Agent: MinIO (amd64; amd64) minio-java/dev\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20200615T082125Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200615/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=409, headers=Accept-Ranges: bytes\nContent-Length: 373\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO/DEVELOPMENT.2020-06-15T08-00-50Z\nVary: Origin\nX-Amz-Request-Id: 1618A9E8AD463281\nX-Xss-Protection: 1; mode=block\nDate: Mon, 15 Jun 2020 08:21:25 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:1211), io.minio.MinioClient.execute(MinioClient.java:1236), io.minio.MinioClient.executeDelete(MinioClient.java:1366), io.minio.MinioClient.removeBucket(MinioClient.java:3686), FunctionalTest.enableObjectLegalHold_test(FunctionalTest.java:3314), FunctionalTest.runTests(FunctionalTest.java:4079), FunctionalTest.main(FunctionalTest.java:4192)]"
}
(9/15) Running minio-js tests ... done in 55 seconds
(10/15) Running minio-py tests ... done in 5 minutes and 18 seconds
(11/15) Running s3cmd tests ... done in 26 seconds
(12/15) Running s3select tests ... done in 16 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

9815-5362ee7/mint-dist-xl.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev7.minio.io:30736
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 5e71deb95df9:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 2 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 43 seconds
(4/15) Running aws-sdk-ruby tests ... done in 4 seconds
(5/15) Running awscli tests ... done in 2 minutes and 22 seconds
(6/15) Running healthcheck tests ... done in 1 seconds
(7/15) Running mc tests ... done in 55 seconds
(8/15) Running minio-dotnet tests ... done in 47 seconds
(9/15) Running minio-go tests ... FAILED in 1 minutes and 21 seconds
{
  "args": {
    "bucketName": "minio-go-test-ediz0uk23kl9vgbd",
    "objectName": "3doemjycwgsa55jnsknqzctpy6wj4v",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 5701,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... FAILED in 2 minutes and 19 seconds
{
  "name": "minio-java",
  "function": "enableObjectLegalHold(String bucketName, String objectName, String versionId)",
  "duration": 316,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = BucketNotEmpty, message = The bucket you tried to delete is not empty. You must delete all versions in the bucket., bucketName = minio-java-test-c7slg0, objectName = null, resource = /minio-java-test-c7slg0, requestId = 1618A992090218F0, hostId = 182fc948-5ecf-4f45-a2f6-8c809fad2dff)\nrequest={method=DELETE, url=http://minio-dev7.minio.io:30736/minio-java-test-c7slg0, headers=Host: minio-dev7.minio.io:30736\nAccept-Encoding: identity\nUser-Agent: MinIO (amd64; amd64) minio-java/dev\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20200615T081512Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200615/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=409, headers=Accept-Ranges: bytes\nContent-Length: 371\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO/DEVELOPMENT.2020-06-15T08-00-50Z\nVary: Origin\nX-Amz-Request-Id: 1618A992090218F0\nX-Xss-Protection: 1; mode=block\nDate: Mon, 15 Jun 2020 08:15:12 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:1211), io.minio.MinioClient.execute(MinioClient.java:1236), io.minio.MinioClient.executeDelete(MinioClient.java:1366), io.minio.MinioClient.removeBucket(MinioClient.java:3686), FunctionalTest.enableObjectLegalHold_test(FunctionalTest.java:3314), FunctionalTest.runTests(FunctionalTest.java:4079), FunctionalTest.main(FunctionalTest.java:4192)]"
}
(9/15) Running minio-js tests ... done in 49 seconds
(10/15) Running minio-py tests ... done in 1 minutes and 46 seconds
(11/15) Running s3cmd tests ... done in 20 seconds
(12/15) Running s3select tests ... done in 7 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

9815-5362ee7/mint-xl.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev6.minio.io:30984
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 154691632b58:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 1 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 42 seconds
(4/15) Running aws-sdk-ruby tests ... done in 2 seconds
(5/15) Running awscli tests ... done in 1 minutes and 22 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 27 seconds
(8/15) Running minio-dotnet tests ... done in 35 seconds
(9/15) Running minio-go tests ... FAILED in 1 minutes and 32 seconds
{
  "args": {
    "bucketName": "minio-go-test-cpoccw1ahcicazgt",
    "objectName": "ckuk923ok2aa3sb06utxc4kn5utwbw",
    "objectPrefix": "",
    "recursive": "true"
  },
  "duration": 5996,
  "error": "The bucket you tried to delete is not empty. You must delete all versions in the bucket.",
  "function": "RemoveObjectsWithOptions(bucketName, objectsCh, opts)",
  "message": "Cleanup failed",
  "name": "minio-go: testRemoveObjectsWithOptions",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... FAILED in 1 minutes and 11 seconds
{
  "name": "minio-java",
  "function": "enableObjectLegalHold(String bucketName, String objectName, String versionId)",
  "duration": 136,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = BucketNotEmpty, message = The bucket you tried to delete is not empty. You must delete all versions in the bucket., bucketName = minio-java-test-1d2p3mq, objectName = null, resource = /minio-java-test-1d2p3mq, requestId = 1618A96C60633E52, hostId = af912427-5812-4299-8c11-e1e2cdd438b5)\nrequest={method=DELETE, url=http://minio-dev6.minio.io:30984/minio-java-test-1d2p3mq, headers=Host: minio-dev6.minio.io:30984\nAccept-Encoding: identity\nUser-Agent: MinIO (amd64; amd64) minio-java/dev\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20200615T081231Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200615/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=409, headers=Accept-Ranges: bytes\nContent-Length: 373\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO/DEVELOPMENT.2020-06-15T08-00-50Z\nVary: Origin\nX-Amz-Request-Id: 1618A96C60633E52\nX-Xss-Protection: 1; mode=block\nDate: Mon, 15 Jun 2020 08:12:31 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:1211), io.minio.MinioClient.execute(MinioClient.java:1236), io.minio.MinioClient.executeDelete(MinioClient.java:1366), io.minio.MinioClient.removeBucket(MinioClient.java:3686), FunctionalTest.enableObjectLegalHold_test(FunctionalTest.java:3314), FunctionalTest.runTests(FunctionalTest.java:4079), FunctionalTest.main(FunctionalTest.java:4192)]"
}
(9/15) Running minio-js tests ... done in 1 minutes and 5 seconds
(10/15) Running minio-py tests ... done in 1 minutes and 18 seconds
(11/15) Running s3cmd tests ... done in 18 seconds
(12/15) Running s3select tests ... done in 5 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

Bonus change to use channel to serialize triggers, instead of using atomic variables. More efficient mechanism for synchronization.

harshavardhana requested review from poornas and nitisht June 12, 2020 05:55

poornas requested changes Jun 12, 2020

View reviewed changes

cmd/disk-cache-backend.go Outdated Show resolved Hide resolved

cmd/metrics.go Outdated Show resolved Hide resolved

cmd/disk-cache.go Outdated Show resolved Hide resolved

harshavardhana force-pushed the add-prometheus branch 4 times, most recently from 3d57802 to 1b32e82 Compare June 12, 2020 21:42

poornas requested changes Jun 12, 2020

View reviewed changes

cmd/disk-cache-backend.go Outdated Show resolved Hide resolved

minio deleted a comment from minio-trusted Jun 12, 2020

poornas approved these changes Jun 12, 2020

View reviewed changes

harshavardhana requested a review from kannappanr June 12, 2020 22:43

harshavardhana force-pushed the add-prometheus branch 2 times, most recently from 30712cc to 65cee54 Compare June 13, 2020 05:15

harshavardhana added the do-not-close label Jun 13, 2020

harshavardhana force-pushed the add-prometheus branch from 65cee54 to f82a1ef Compare June 13, 2020 05:48

minio deleted a comment from minio-trusted Jun 13, 2020

harshavardhana force-pushed the add-prometheus branch from f82a1ef to 4e46765 Compare June 13, 2020 05:50

minio deleted a comment from minio-trusted Jun 13, 2020

harshavardhana force-pushed the add-prometheus branch from 4e46765 to de502f1 Compare June 15, 2020 00:38

harshavardhana removed the do-not-close label Jun 15, 2020

fix: export prometheus metrics for cache GC triggers

2fb2862

Bonus change to use channel to serialize triggers, instead of using atomic variables. More efficient mechanism for synchronization.

harshavardhana force-pushed the add-prometheus branch from de502f1 to 2fb2862 Compare June 15, 2020 06:58

harshavardhana mentioned this pull request Jun 15, 2020

Add cache garbage collection to Prometheus exposed metrics #9072

Closed

nitisht approved these changes Jun 15, 2020

View reviewed changes

Merge branch 'master' into add-prometheus

5362ee7

harshavardhana merged commit f9aa239 into minio:master Jun 15, 2020

harshavardhana deleted the add-prometheus branch June 15, 2020 16:05

harshavardhana added a commit that referenced this pull request Jun 16, 2020

fix: export prometheus metrics for cache GC triggers (#9815)

49c8d36

Bonus change to use channel to serialize triggers, instead of using atomic variables. More efficient mechanism for synchronization.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: export prometheus metrics for cache GC triggers #9815

fix: export prometheus metrics for cache GC triggers #9815

harshavardhana commented Jun 12, 2020

poornas left a comment

harshavardhana commented Jun 15, 2020

nitisht left a comment

minio-trusted commented Jun 15, 2020

fix: export prometheus metrics for cache GC triggers #9815

fix: export prometheus metrics for cache GC triggers #9815

Conversation

harshavardhana commented Jun 12, 2020

Description

Motivation and Context

How to test this PR?

Types of changes

Checklist:

poornas left a comment

Choose a reason for hiding this comment

harshavardhana commented Jun 15, 2020

nitisht left a comment

Choose a reason for hiding this comment

minio-trusted commented Jun 15, 2020

Mint Automation

9815-5362ee7/mint-large-bucket.sh.log:

9815-5362ee7/mint-dist-xl.sh.log:

9815-5362ee7/mint-xl.sh.log: