Honor path_style for non-AWS S3 buckets

[MichaelKatsoulis]

Proposed commit message

The aws-s3 input set HostnameImmutable = (getBucketARN() != "") when a custom
endpoint was configured. getBucketARN() also returns the bucket name for
non_aws_bucket_name, so any non-AWS S3-compatible bucket with a custom
endpoint ended up with HostnameImmutable=true. In the AWS SDK v2 an immutable
hostname forces path-style requests and ignores path_style parameter.

This broke providers that require virtual-hosted addressing (e.g. Huawei Cloud
OBS): every ListObjectsV2 failed with VirtualHostDomainRequired (403) and no
data was ingested, with no config-side workaround.

The fix is scope the immutable hostname to AWS bucket-ARN / access-point polling only:

e.HostnameImmutable = (c.getBucketARN() != "" && c.NonAWSBucketName == "")

For non-AWS buckets the hostname is now mutable, so path_style controls
addressing: virtual-hosted when path_style: false (default), path-style when
path_style: true.

Disruptive user impact

path_style now takes effect for non-AWS buckets where it was previously
ignored. Non-AWS users who relied on the old forced path-style
behavior and did not set path_style may need to set path_style: true.
But this fixes a real bug.

Related SDH

• https://github.com/elastic/sdh-beats/issues/7374

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
• /test : Run the Buildkite pipeline.

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @MichaelKatsoulis? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[infra-vault-gh-plugin-prod]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[botelastic]

This pull request doesn't have a Team:<team> label.

[MichaelKatsoulis]

@Kavindu-Dodan I anonymized the test

[Kavindu-Dodan]

LGTM

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[github-actions]

@Mergifyio backport 9.5 9.4 9.3 8.19

[mergify]

backport 9.5 9.4 9.3 8.19

✅ Backports have been created

Details

• #52072 [9.5](backport #52003) Honor path_style for non-AWS S3 buckets has been created for branch 9.5
• #52074 [9.4](backport #52003) Honor path_style for non-AWS S3 buckets has been created for branch 9.4
• #52075 [9.3](backport #52003) Honor path_style for non-AWS S3 buckets has been created for branch 9.3
• #52076 [8.19](backport #52003) Honor path_style for non-AWS S3 buckets has been created for branch 8.19