crawler: Allow deeper sleep

[klauspost]

Description

Allow each crawler operation to sleep up to 10 seconds on very heavily loaded systems.

This will of course make minimum crawler speed less, but should be more effective at stopping.

How to test this PR?

Run crawler in heavily (over)loaded scenario.

Types of changes

• New feature (non-breaking change which adds functionality)

[minio-trusted]

Mint Automation

Test	Result
mint-large-bucket.sh	✔️
mint-fs.sh	✔️
mint-gateway-s3.sh	✔️
mint-erasure.sh	✔️
mint-dist-erasure.sh	✔️
mint-zoned.sh	✔️
mint-gateway-nas.sh	✔️
mint-gateway-azure.sh	❌ more...

10883-93db48e/mint-gateway-azure.sh.log:

Running with
SERVER_ENDPOINT:      minio-c2.minio.io:31960
ACCESS_KEY:           minioazure
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 4692b6100f37:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 9 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 1 minutes and 46 seconds
(4/15) Running aws-sdk-ruby tests ... done in 21 seconds
(5/15) Running awscli tests ... done in 2 minutes and 53 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 3 minutes and 32 seconds
(8/15) Running minio-dotnet tests ... done in 1 minutes and 38 seconds
(9/15) Running minio-go tests ... FAILED in 1 minutes and 53 seconds
{
  "args": {
    "bucketName": "minio-go-test-fhpw5ozh5ji2g6c0",
    "objectName": "test-object",
    "opts": "",
    "size": -1
  },
  "duration": 1028,
  "function": "PutObject(bucketName, objectName, reader,size,opts)",
  "message": "Unexpected size",
  "name": "minio-go: testPutObjectStreaming",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... done in 10 minutes and 38 seconds
(10/15) Running minio-js tests ... FAILED in 46 seconds
{
  "name": "minio-js",
  "function": "\"after all\" hook in \"functional tests\"",
  "duration": 79,
  "status": "FAIL",
  "error": "S3Error: The bucket you tried to delete is not empty at Object.parseError (node_modules/minio/dist/main/xml-parsers.js:86:11) at /mint/run/core/minio-js/node_modules/minio/dist/main/transformers.js:156:22 at DestroyableTransform._flush (node_modules/minio/dist/main/transformers.js:80:10) at DestroyableTransform.prefinish (node_modules/readable-stream/lib/_stream_transform.js:129:10) at prefinish (node_modules/readable-stream/lib/_stream_writable.js:611:14) at finishMaybe (node_modules/readable-stream/lib/_stream_writable.js:620:5) at endWritable (node_modules/readable-stream/lib/_stream_writable.js:643:3) at DestroyableTransform.Writable.end (node_modules/readable-stream/lib/_stream_writable.js:571:22) at IncomingMessage.onend (_stream_readable.js:682:10) at endReadableNT (_stream_readable.js:1252:12) at processTicksAndRejections (internal/process/task_queues.js:80:21)"
}
(10/15) Running minio-py tests ... done in 18 minutes and 44 seconds
(11/15) Running s3cmd tests ... done in 2 minutes and 45 seconds
(12/15) Running s3select tests ... done in 39 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

[harshavardhana]

Wouldn't this make crawler very slow? @klauspost effectively making it non effective?

[klauspost]

@harshavardhana For normal operation I don't think so, since it would never get close to the current 1 second limit.

But for heavily (like really heavy) loaded systems, yes.