Option configure_s3_using_s3a doesn't work in certain deployments

[exalate-issue-sync]

In [https://h2oai.atlassian.net/browse/PUBDEV-8135|https://h2oai.atlassian.net/browse/PUBDEV-8135|smart-link] we added the option {{configure_s3_using_s3a}} to simplify switching between S3A to S3.

This option, however, only works in certain deployments. In deployments using a private S3 endpoint, it cannot be used because the public bucket “[www.h2o.ai|http://www.h2o.ai]” used to configure S3AFileSystem and extract the AmazonS3 client doesn’t work. In Hadoop deployments, there can be specific configurations per bucket.

The goal of this Jira is to refactor the code and make the option usable everywhere. Users should be able to provide their S3A configuration and then seamlessly switch between using S3 and S3A.

We also don’t have any documentation for the option yet (cc: [~accountid:5d1185d4f46aa30c271c7cc6] )

[h2o-ops]

JIRA Issue Details

Jira Issue: PUBDEV-8787
Assignee: Michal Kurka
Reporter: Michal Kurka
State: Resolved
Fix Version: 3.38.0.1
Attachments: N/A
Development PRs: Available

[h2o-ops]

Linked PRs from JIRA

#6262
https://github.com/h2oai/h2oai-serving/pull/1079