fix: bring back delayed leaf detection in listing

[harshavardhana]

Description

fix: bring back delayed leaf detection in listing

Motivation and Context

improve List performance

How to test this PR?

No functionality should change

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• Fixes a regression (If yes, please add commit-id or PR # here)
• Documentation needed
• Unit tests needed

[vadmeste]

Some comments

[harshavardhana]

NOTE: The entire code is actually from previous implementation of Walkers() when they used
to be at the object layer - the assumptions underneath satisfy all the unit tests we have especially
regarding how the directories v/s objects

These unit tests have been captured over the years in checking various corner cases which
comes under S3 compatibility.

[klauspost]

Looks good. I think I understand it but I assume you vouch for correctness (along with existing tests)

Few minor things.

[minio-trusted]

Mint Automation

Test	Result
mint-large-bucket.sh	✔️
mint-fs.sh	✔️
mint-gateway-s3.sh	✔️
mint-erasure.sh	✔️
mint-dist-erasure.sh	✔️
mint-zoned.sh	✔️
mint-gateway-nas.sh	✔️
mint-gateway-azure.sh	❌ more...

10346-9dc4b9d/mint-gateway-azure.sh.log:

Running with
SERVER_ENDPOINT:      minio-c2.minio.io:32567
ACCESS_KEY:           minioazure
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 99b60516271b:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 7 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... FAILED in 5 minutes and 39 seconds
{
  "name": "aws-sdk-php",
  "function": "getBucketPolicy ( array $params = [] )",
  "args": {
    "Bucket": "aws-sdk-php-5f4552dc3f1e2"
  },
  "duration": "41529",
  "status": "FAIL",
  "error": "Error executing \"CreateBucket\" on \"http://minio-c2.minio.io:32567/aws-sdk-php-5f4552dc3f1e2\"; AWS HTTP error: Client error: `PUT http://minio-c2.minio.io:32567/aws-sdk-php-5f4552dc3f1e2` resulted in a `404 Not Found` response:\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Mes (truncated...)\n NoSuchBucket (client): The specified bucket does not exist - <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Message><BucketName>aws-sdk-php-5f4552dc3f1e2</BucketName><Resource>/aws-sdk-php-5f4552dc3f1e2</Resource><RequestId>162E95485F609AD5</RequestId><HostId>81966869-52fe-4c4a-8efc-643f19ce8ca0</HostId></Error>"
}
(3/15) Running aws-sdk-ruby tests ... done in 17 seconds
(4/15) Running awscli tests ... done in 2 minutes and 44 seconds
(5/15) Running healthcheck tests ... done in 0 seconds
(6/15) Running mc tests ... done in 3 minutes and 16 seconds
(7/15) Running minio-dotnet tests ... done in 1 minutes and 33 seconds
(8/15) Running minio-go tests ... FAILED in 5 minutes and 9 seconds
{
  "args": {},
  "duration": 1329,
  "error": "The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.",
  "function": "CopyObjectPart(destination, source)",
  "message": "CompleteMultipartUpload call failed",
  "name": "minio-go: testUnencryptedToSSES3CopyObjectPart",
  "status": "FAIL"
}
(8/15) Running minio-java tests ... done in 6 minutes and 12 seconds
(9/15) Running minio-js tests ... done in 2 minutes and 36 seconds
(10/15) Running minio-py tests ... done in 10 minutes and 11 seconds
(11/15) Running s3cmd tests ... done in 1 minutes and 30 seconds
(12/15) Running s3select tests ... done in 50 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally
Error: No such image: minio/minio:10346-9dc4b9d