fix: incorrect O_DIRECT behavior for reads

[harshavardhana]

Description

fix: incorrect O_DIRECT behavior for reads

Motivation and Context

O_DIRECT behavior was broken and it was still
caching all the reads, this change properly fixes
this behavior.

How to test this PR?

Need to test on Linux kernel to observe the
reads and page-cache build up upon re-reads.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Optimization (provides speedup with no functional changes)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• Fixes a regression (If yes, please add commit-id or PR # here)
• Documentation updated
• Unit tests added/updated

[vadmeste]

Not sure why we required DMA for direct IO in the past, any idea ?

[harshavardhana]

Not sure why we required DMA for direct IO in the past, any idea ?

DMA means direct access - to refer to O_DIRECT

[vadmeste]

DMA means direct access - to refer to O_DIRECT

yes understood, but MinIO server won't start already if the disk doesn't support O_DIRECT, so I see this is more like a cleanup.

[minio-trusted]

Mint Automation

Test	Result
mint-large-bucket.sh	✔️
mint-fs.sh	✔️
mint-gateway-s3.sh	✔️
mint-erasure.sh	✔️
mint-dist-erasure.sh	✔️
mint-zoned.sh	✔️
mint-gateway-nas.sh	✔️
mint-compress-encrypt-dist-erasure.sh	✔️
Deleting image on docker hub
Deleting image locally