crawler: Remove waitForLowActiveIO

[klauspost]

Description

Only use dynamic delays for the crawler. Even though max wait was 1 second the number of waits could severely impact crawler speed.

Instead of relying on a global metric we use the stateless local delays to keep the crawler running at a speed more adjusted to current conditions.

The only case we keep it is before bitrot checks, when enabled.

How to test this PR?

Observe crawler speed with a server under load.

Types of changes

• Performance adjustment

[minio-trusted]

Mint Automation

Test	Result
mint-large-bucket.sh	✔️
mint-fs.sh	✔️
mint-gateway-s3.sh	✔️
mint-erasure.sh	✔️
mint-dist-erasure.sh	✔️
mint-zoned.sh	✔️
mint-gateway-nas.sh	✔️
mint-gateway-azure.sh	❌ more...

10667-4d8eda2/mint-gateway-azure.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev4.minio.io:31015
ACCESS_KEY:           minioazure
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 5f5deea76556:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 8 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 1 minutes and 5 seconds
(4/15) Running aws-sdk-ruby tests ... done in 19 seconds
(5/15) Running awscli tests ... done in 3 minutes and 8 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 3 minutes and 20 seconds
(8/15) Running minio-dotnet tests ... done in 1 minutes and 43 seconds
(9/15) Running minio-go tests ... FAILED in 1 minutes and 57 seconds
{
  "args": {
    "bucketName": "minio-go-test-9ee1epb5as3xathq",
    "objectName": "test-object",
    "opts": "",
    "size": -1
  },
  "duration": 1027,
  "function": "PutObject(bucketName, objectName, reader,size,opts)",
  "message": "Unexpected size",
  "name": "minio-go: testPutObjectStreaming",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... done in 10 minutes and 20 seconds
(10/15) Running minio-js tests ... FAILED in 3 minutes and 3 seconds
{
  "name": "minio-js",
  "function": "\"after all\" hook in \"functional tests\"",
  "duration": 72536,
  "status": "FAIL",
  "error": "S3Error: We encountered an internal error, please try again. at Object.parseError (node_modules/minio/dist/main/xml-parsers.js:86:11) at /mint/run/core/minio-js/node_modules/minio/dist/main/transformers.js:156:22 at DestroyableTransform._flush (node_modules/minio/dist/main/transformers.js:80:10) at DestroyableTransform.prefinish (node_modules/readable-stream/lib/_stream_transform.js:129:10) at prefinish (node_modules/readable-stream/lib/_stream_writable.js:611:14) at finishMaybe (node_modules/readable-stream/lib/_stream_writable.js:620:5) at endWritable (node_modules/readable-stream/lib/_stream_writable.js:643:3) at DestroyableTransform.Writable.end (node_modules/readable-stream/lib/_stream_writable.js:571:22) at IncomingMessage.onend (_stream_readable.js:682:10) at endReadableNT (_stream_readable.js:1252:12) at processTicksAndRejections (internal/process/task_queues.js:80:21)"
}
(10/15) Running minio-py tests ... done in 19 minutes and 12 seconds
(11/15) Running s3cmd tests ... done in 4 minutes and 51 seconds
(12/15) Running s3select tests ... done in 38 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally