S3 file downloads not working with AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN

[exalate-issue-sync]

This is essentially a re-opening of [https://h2oai.atlassian.net/browse/PUBDEV-8169|https://h2oai.atlassian.net/browse/PUBDEV-8169|smart-link].

I upgraded to H2O version 3.37.0.5718 (one of the nightly builds with the fix from [https://h2oai.atlassian.net/browse/PUBDEV-8557|https://h2oai.atlassian.net/browse/PUBDEV-8557|smart-link] ), but the issue still exists. FWIW, I tried using both {{s3://}} and {{s3a://}} URLs, but both fail. However, the AWS cli is able to download these files from the running container on EKS just fine.

FWIW, we’re starting up a container-local H2O server from Python using {{h2o.init}} (which works) and then running {{h2o.import_file}} which fails. This command works just fine if we include the standard AWS_ACCESS_KEY and secret key environment variables, but fails if we try to use the service account token mount (even though the AWS CLI has no problems).

[exalate-issue-sync]

Michal Kurka commented: [~accountid:5c9943ec3a5542225fedb6b9] can you please try to reproduce the issue?

[exalate-issue-sync]

Marek Novotny commented: [~accountid:557058:04659f86-fbfe-4d01-90c9-146c34df6ee6] I’ve managed to reproduce the problem and also it seems I’ve found the cause of the problem. H2O fat jar doesn’t contain the [https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-sts|https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-sts|smart-link] library, which is required by [WebIdentityTokenCredentialsProvider|https://github.com/aws/aws-sdk-java/blame/1.12.127/aws-java-sdk-core/src/main/java/com/amazonaws/auth/WebIdentityTokenCredentialsProvider.java#L58].

[~accountid:5cbdf1602675f70e50cd543b] As workaround you can add the library to java classpath explicitly when calling h2o.init():
Python script:

{noformat}import h2o
h2o.init(extra_classpath=["/root/.m2/repository/com/amazonaws/aws-java-sdk-sts/1.12.127/aws-java-sdk-sts-1.12.127.jar",])
path = "s3://ai.h2o.sparkling/testing/prostate.csv"
h2o.import_file(path=path){noformat}

Dockerfile:

{noformat}FROM h2oai/h2o-open-source-k8s:3.36.0.2
RUN microdnf install python3
RUN microdnf install unzip
RUN microdnf install maven
RUN mvn dependency:get -Dartifact=com.amazonaws:aws-java-sdk-sts:1.12.127
RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" && unzip awscliv2.zip && ./aws/install
RUN pip3 install https://h2o-release.s3.amazonaws.com/h2o/master/5718/Python/h2o-3.37.0.5718-py2.py3-none-any.whl
COPY script.py /script.py
CMD env && python3 /script.py{noformat}

[exalate-issue-sync]

Michal Kurka commented: Thanks, [~accountid:5c9943ec3a5542225fedb6b9] for the fix. [~accountid:5cbdf1602675f70e50cd543b] it was merged an hour ago - it will be available in today’s nightly build if you want to test it.

[exalate-issue-sync]

Stephen Hopper commented: [~accountid:557058:04659f86-fbfe-4d01-90c9-146c34df6ee6] [~accountid:5c9943ec3a5542225fedb6b9] I finally got around to testing this with H2O 3.36.1.1. It’s working wonderfully. Thank you so much for your work on this!

[h2o-ops-ro]

JIRA Issue Details

Jira Issue: PUBDEV-8567
Assignee: Marek Novotny
Reporter: Stephen Hopper
State: Resolved
Fix Version: 3.36.0.3
Attachments: N/A
Development PRs: Available

[h2o-ops-ro]

Linked PRs from JIRA

#6068