Onboard Diagnostics #9024

wlan0 · 2020-02-21T01:15:01Z

on board diagnostics

@krishnasrinivas @poornas @abperiasamy

The output from the ascii video is here: https://gist.github.com/wlan0/670db2881b4163e3af35a94f83bf642a

Some interesting observations to not in the json above:

The speed diff between minio-4, which has nvme, and others, which only have hdds.
The 100 Gbit network link is saturated
Differences between the CPUs of different machines

Here's another output JSON of a fs setup with 1 drive on my laptop. It's smaller, so its easier to read

https://gist.github.com/wlan0/aa892c51c126dfed6a47305eddeec967

poornas

admin docs will also need an update

cmd/admin-handlers.go

cmd/admin-router.go

cmd/admin-server-info.go

cmd/notification.go

pkg/disk/obd.go

cmd/utils.go

kannappanr · 2020-03-19T21:47:29Z

ping @aead @poornas

kannappanr · 2020-03-22T07:38:57Z

ping @aead @poornas

cmd/api-errors.go

cmd/fs-v1.go

cmd/object-api-interface.go

cmd/obdinfo.go

cmd/peer-rest-client.go

pkg/disk/obd.go

pkg/net/obd.go

cmd/gateway/azure/gateway-azure.go

cmd/gateway/b2/gateway-b2.go

cmd/gateway/hdfs/gateway-hdfs.go

cmd/gateway/nas/gateway-nas.go

cmd/gateway/oss/gateway-oss.go

cmd/gateway/s3/gateway-s3.go

cmd/gateway/gcs/gateway-gcs.go

wlan0 · 2020-03-25T03:02:46Z

@poornas @klauspost I've addressed your comments and pushed changes. PTAL

klauspost

Awesome 👍

poornas

LGTM

harshavardhana

Testing right now using mc everything looks good - just some idiomatic changes needed.

cmd/admin-handlers.go

cmd/fs-v1.go

cmd/notification.go

cmd/object-api-interface.go

cmd/peer-rest-common.go

cmd/rest/client.go

cmd/server-main.go

cmd/xl-zones.go

disk diagnostics: preserve diagnostics in memory after 1st run config format: return map of values instead of bytestream admin obd command: fix logic for setting query params to ensure all obd types are either true or false

1. add admin info to obd api 2. add network OBD info 3. lock obd commands to allow only one at a time

- Implement graph algorithm to test network bandwidth from every node to every other node - Saturate any network bandwidth adaptively, accounting for slow and fast network capacity - Implement parallel drive OBD tests - Implement a paging mechanism for OBD test to provide periodic updates to client - Implement mechanism to dynamically cancel long running OBD tests

- Sys: Disk (Partitions, DiskInfo) , CPU (HW, Timers) - Host: Platform (OS, Kernel), Sensors (Temperatures) - Mem: Swap Mem, Virtual Mem - Process: Everything about the MinIO Process

minio-trusted · 2020-03-27T04:18:08Z

Mint Automation

Test	Result
mint-xl.sh	✔️
mint-large-bucket.sh	✔️
mint-worm.sh	✔️
mint-fs.sh	✔️
mint-dist-xl.sh	✔️
mint-gateway-nas.sh	✔️
mint-gateway-azure.sh	❌ more...

9024-e8b0e61/mint-gateway-azure.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev7.minio.io:31561
ACCESS_KEY:           minioazure
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 5b9dea141f2e:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 8 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 8 minutes and 14 seconds
(4/15) Running aws-sdk-ruby tests ... done in 19 seconds
(5/15) Running awscli tests ... done in 2 minutes and 47 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 7 minutes and 27 seconds
(8/15) Running minio-dotnet tests ... done in 1 minutes and 46 seconds
(9/15) Running minio-go tests ... done in 21 minutes and 41 seconds
(10/15) Running minio-java tests ... FAILED in 8 minutes and 53 seconds
{
  "name": "minio-java",
  "function": "composeObject(String bucketName, String objectName,List<ComposeSource> composeSources, Map <String,String > headerMap, ServerSideEncryption sseTarget)",
  "args": "size: 6 MB & 6 MB",
  "duration": 11373,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code=NotImplemented, message=A header you provided implies functionality that is not implemented, bucketName=minio-java-test-h4t6tv, objectName=minio-java-test-3cdblht, resource=/minio-java-test-h4t6tv/minio-java-test-3cdblht, requestId=16000E2880C71324, hostId=aba24c6f-85d5-47e9-ac45-d5e2fed80939)\nrequest={method=PUT, url=http://minio-dev7.minio.io:31561/minio-java-test-h4t6tv/minio-java-test-3cdblht?uploadId=8b775148bbab49c7&partNumber=1, headers=x-amz-copy-source-if-match: dda202f43c2b2df39d130f43ab1fdac7-2\nx-amz-copy-source: minio-java-test-h4t6tv/minio-java-test-30ni5eo\nAccept-Encoding: identity\nHost: minio-dev7.minio.io:31561\nUser-Agent: MinIO (amd64; amd64) minio-java/dev\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20200327T041715Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20200327/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;host;x-amz-content-sha256;x-amz-copy-source;x-amz-copy-source-if-match;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=501, headers=Accept-Ranges: bytes\nContent-Length: 408\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO/DEVELOPMENT.GOGET\nVary: Origin\nX-Amz-Request-Id: 16000E2880C71324\nX-Xss-Protection: 1; mode=block\nDate: Fri, 27 Mar 2020 04:17:16 GMT\n}\n >>> [io.minio.MinioClient.executeReq(MinioClient.java:1220), io.minio.MinioClient.execute(MinioClient.java:1082), io.minio.MinioClient.executePut(MinioClient.java:1446), io.minio.MinioClient.executePut(MinioClient.java:1468), io.minio.MinioClient.uploadPartCopy(MinioClient.java:2619), io.minio.MinioClient.composeObject(MinioClient.java:2572), FunctionalTest.composeObject_test1(FunctionalTest.java:2587), FunctionalTest.runTests(FunctionalTest.java:3396), FunctionalTest.main(FunctionalTest.java:3514)]"
}

Executed 9 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

wlan0 force-pushed the diagnose branch 3 times, most recently from 1f78e9a to feedb3c Compare February 21, 2020 02:23

minio deleted a comment from minio-ops Feb 21, 2020

poornas requested changes Feb 21, 2020

View reviewed changes

cmd/admin-handlers.go Show resolved Hide resolved

cmd/admin-router.go Outdated Show resolved Hide resolved

cmd/admin-server-info.go Outdated Show resolved Hide resolved

cmd/notification.go Show resolved Hide resolved

pkg/disk/obd.go Show resolved Hide resolved

This comment has been minimized.

Sign in to view

wlan0 force-pushed the diagnose branch from 5884474 to ea72eda Compare March 12, 2020 20:30

wlan0 mentioned this pull request Mar 12, 2020

Onboard Diagnostics minio/mc#3114

Merged

wlan0 commented Mar 12, 2020

View reviewed changes

cmd/utils.go Outdated Show resolved Hide resolved

wlan0 force-pushed the diagnose branch 2 times, most recently from febad3a to 775728d Compare March 13, 2020 04:51

kannappanr requested review from krishnasrinivas and aead March 14, 2020 22:01

This comment has been minimized.

Sign in to view

wlan0 force-pushed the diagnose branch 3 times, most recently from 6e0ec44 to 6a10248 Compare March 21, 2020 05:39

minio deleted a comment from minio-trusted Mar 21, 2020

wlan0 force-pushed the diagnose branch from 6a10248 to 0bb0eec Compare March 22, 2020 07:03

klauspost reviewed Mar 22, 2020

View reviewed changes

This comment has been minimized.

Sign in to view

poornas requested changes Mar 24, 2020

View reviewed changes

wlan0 force-pushed the diagnose branch from 0bb0eec to f872786 Compare March 25, 2020 03:01

wlan0 force-pushed the diagnose branch from f872786 to a329e4b Compare March 25, 2020 03:23

klauspost approved these changes Mar 25, 2020

View reviewed changes

poornas approved these changes Mar 25, 2020

View reviewed changes

harshavardhana requested changes Mar 25, 2020

View reviewed changes

wlan0 force-pushed the diagnose branch 3 times, most recently from e83b024 to 992b934 Compare March 27, 2020 01:56

wlan0 added 4 commits March 26, 2020 18:58

add obd backend support

e73eaed

disk diagnostic and config format updates

b99a84c

disk diagnostics: preserve diagnostics in memory after 1st run config format: return map of values instead of bytestream admin obd command: fix logic for setting query params to ensure all obd types are either true or false

NetOBD info and Admin Info

3b307c5

1. add admin info to obd api 2. add network OBD info 3. lock obd commands to allow only one at a time

wlan0 force-pushed the diagnose branch from 992b934 to 9e78dee Compare March 27, 2020 01:58

Implement Sys, Process, Host, Mem OBD Infos

e8b0e61

- Sys: Disk (Partitions, DiskInfo) , CPU (HW, Timers) - Host: Platform (OS, Kernel), Sensors (Temperatures) - Mem: Swap Mem, Virtual Mem - Process: Everything about the MinIO Process

wlan0 force-pushed the diagnose branch from 9e78dee to e8b0e61 Compare March 27, 2020 02:00

harshavardhana approved these changes Mar 27, 2020

View reviewed changes

harshavardhana merged commit 0c80bf4 into minio:master Mar 27, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Onboard Diagnostics #9024

Onboard Diagnostics #9024

wlan0 commented Feb 21, 2020 •

edited

Loading

poornas left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

kannappanr commented Mar 19, 2020

This comment has been minimized.

kannappanr commented Mar 22, 2020

This comment has been minimized.

wlan0 commented Mar 25, 2020

klauspost left a comment

poornas left a comment

harshavardhana left a comment

minio-trusted commented Mar 27, 2020

Onboard Diagnostics #9024

Onboard Diagnostics #9024

Conversation

wlan0 commented Feb 21, 2020 • edited Loading

poornas left a comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

kannappanr commented Mar 19, 2020

This comment has been minimized.

kannappanr commented Mar 22, 2020

This comment has been minimized.

wlan0 commented Mar 25, 2020

klauspost left a comment

Choose a reason for hiding this comment

poornas left a comment

Choose a reason for hiding this comment

harshavardhana left a comment

Choose a reason for hiding this comment

minio-trusted commented Mar 27, 2020

Mint Automation

9024-e8b0e61/mint-gateway-azure.sh.log:

wlan0 commented Feb 21, 2020 •

edited

Loading