Hadoop-19801. Allow to skip recursive file deletion for non-empty directory

[EnricoMi]

Description of PR

This adds option fs.s3a.delete.non-empty-directory.enabled. When true, only one delete request is sent for the directory (key prefix). The S3 endpoint has to support this feature.

This is useful for S3 endpoints that support deletion of path prefixes (non-empty directories).

How was this patch tested?

Ran ./bin/hadoop fs -rm against an S3 endpoint. The logging showed the expected behaviour.

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

AI Tooling

No AI tooling used.