HADOOP-18154. S3A Authentication to support WebIdentity #4070

jclarysse · 2022-03-14T15:33:48Z

Description of PR

The PR addresses a requirement to comply with AWS security concept IAM roles for service accounts (IRSA) while operating a service that isn't based on Apache Spark and that runs inside Amazon Elastic Kubernetes Service (EKS).

The code change consists in adding a new credentials provider class org.apache.hadoop.fs.s3a.OIDCTokenCredentialsProvider to the module hadoop-aws.

How was this patch tested?

No new unit-test or integration-test was created on-purpose. The patch was "only" tested based on Hadoop release 2.10.1, as part of our specific use-case based on Delta sharing service v0.4.0 along with the following Hadoop configuration (core-site.xml):

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>fs.s3a.aws.credentials.provider</name>
    <value>org.apache.hadoop.fs.s3a.OIDCTokenCredentialsProvider</value>
  </property>
  <property>
    <name>fs.s3a.jwt.path</name>
    <value>/var/run/secrets/eks.amazonaws.com/serviceaccount/token</value>
  </property>
  <property>
    <name>fs.s3a.assumed.role.arn</name>
    <value>my_iam_role_arn</value>
  </property>
  <property>
    <name>fs.s3a.assumed.role.session.name</name>
    <value>my_iam_session_name</value>
  </property>
  <property>
      <name>fs.s3a.server-side-encryption-algorithm</name>
      <value>SSE-KMS</value>
  </property>
  <property>
      <name>fs.s3a.server-side-encryption.key</name>
      <value>my_kms_key_id</value>
  </property>      
</configuration>

For code changes:

The title or this PR starts with the corresponding JIRA issue 'HADOOP-18154'
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
No new dependency was added to the code.

steveloughran

i've done the initial review.
thanks for this; it looks like a low risk new feature

i can't see any easy tests for it though.

what i will need is

docs
you to run all the integration tests using this as auth so we know what break.

things will break, but that should just be where the code has assumptions about authentication...the tests should recognise that these are only session credentials
and skip those tests

...op-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/OIDCTokenCredentialsProvider.java

steveloughran · 2022-03-15T10:02:32Z

...op-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/OIDCTokenCredentialsProvider.java

+    private String sessionName;
+    private IOException lookupIOE;
+
+    public OIDCTokenCredentialsProvider(Configuration conf) {


should credential providers be allowed to raise IOEs? we should be able to fix that

...op-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/OIDCTokenCredentialsProvider.java

steveloughran · 2022-03-15T10:04:03Z

...op-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/OIDCTokenCredentialsProvider.java

+                    .roleSessionName(sessionName)
+                    .build();
+            return credentialsProvider.getCredentials();
+        }


nit: same line as the }

steveloughran · 2022-03-15T10:10:07Z

...op-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/OIDCTokenCredentialsProvider.java

+        if (!StringUtils.isEmpty(jwtPath) && !StringUtils.isEmpty(roleARN)) {
+            final AWSCredentialsProvider credentialsProvider =
+                WebIdentityTokenCredentialsProvider.builder()
+                    .webIdentityTokenFile(jwtPath)


this will handle local files only, so won't work for jobs across a cluster unless the token is already there.

either cluster fs paths will be needed (download locally and then reference) or we require it on the host of the user launching a job and then include the token data in a delegation token which goes with it. that's a lot more powerful -but a lot more work. best to leave that for a followup patch

Not sure i get your point. Our service runs accross multiple Kubernetes pods (replicas) using a ServiceAccount, so that any of those pods is automatically attached a volume pointing to a token file created with the same ServiceAcocunt signature. Perhaps i am missing other use-cases within the Hadoop ecosystem?

i was just wondering how the secrets get around. for other credentials we can pick them up from the user launching, say, a distcp job, and they will get passed round. alternatively, they can go into a cluster FS like hdfs.

if it works with your k8s setup, then the docs should say "mount a shared volume in your containers". support for credential propagation can be added by someone else when they needed it

steveloughran · 2022-03-15T10:11:39Z

...op-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/OIDCTokenCredentialsProvider.java

@@ -0,0 +1,79 @@
+package org.apache.hadoop.fs.s3a;
+
+import org.apache.commons.lang3.StringUtils;


nit, we have some layout rules for imports. here's my full intellij settings for this if it helps
https://gist.github.com/steveloughran/817dd90e0f1775ce2b6f24684dfb078c

steveloughran · 2022-03-21T18:57:24Z

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java

@@ -142,6 +142,10 @@ private Constants() {
  public static final String ASSUMED_ROLE_CREDENTIALS_DEFAULT =
      SimpleAWSCredentialsProvider.NAME;

+  /**
+   * Absolute path to the web identity token file


nit, add a . at the end of the sentence. javadoc versions like that.

also in docs, say "path in local/mounted filesystem" so it is clear it is not a cluster fs like HDFS

steveloughran · 2022-03-21T18:58:12Z

...op-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/OIDCTokenCredentialsProvider.java

+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;


can you move to org.apache.hadoop.fs.s3a.auth

steveloughran · 2022-03-21T18:58:25Z

...op-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/OIDCTokenCredentialsProvider.java

+import com.amazonaws.auth.WebIdentityTokenCredentialsProvider;
+
+import org.apache.commons.lang3.StringUtils;
+


you can cut this line

steveloughran · 2022-03-21T18:58:41Z

...op-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/OIDCTokenCredentialsProvider.java

+              getClass().getSimpleName(),
+              jwtPath, roleARN, sessionName);
+  }
+}


nit, add a newline

steveloughran · 2022-03-21T18:59:27Z

...op-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/OIDCTokenCredentialsProvider.java

+                  .build();
+          return credentialsProvider.getCredentials();
+      }
+      else throw new CredentialInitializationException(


better to throw explicit errors when the options are unset.

dannycjones

Some small feedbacks. Forgive me, I'm not familiar with WebIdentity / JWT / OIDC.

dannycjones · 2022-03-22T10:21:57Z

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java

+  /**
+   * Absolute path to the web identity token file
+   */
+  public static final String JWT_PATH = "fs.s3a.jwt.path";


If its for OIDC / WebIdentity, can we change to something like fs.s3a.oidc.jwt.path / fs.s3a.webidentity.jwt.path?

Also, add @value to JavaDoc for IDE hints?

dannycjones · 2022-03-22T10:55:19Z

...op-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/OIDCTokenCredentialsProvider.java

+/**
+ * Support OpenID Connect (OIDC) token for authenticating with AWS.
+ *
+ * Please note that users may reference this class name from configuration
+ * property fs.s3a.aws.credentials.provider.  Therefore, changing the class name
+ * would be a backward-incompatible change.
+ *
+ * This credential provider must not fail in creation because that will
+ * break a chain of credential providers.
+ */


Does this credential provider actually support more than just Open ID Connect - anything that vends an identity under a JWT?

This provider is allowing users to configure Role ARN, JWT path, and session name for the SDK WebIdentityTokenCredentialsProvider. Should we move to similar naming?

steveloughran · 2022-03-22T17:05:49Z

Looking at the WebIdentityTokenCredentialsProvider I see that if it doesn't get the parameters then it will fall back to environment variables. We absolutely do not want to be picking up env vars as it will only create support issues where configurations only work on a certain machines. (actually, we can ignore the session name settings as they are harmless)

I'm going to propose we go with @dannycjones's suggestion and support the whole set of values and have the prefix fs.s3a.webidentity for all of them.

for the arn, we could have a property fs.s3a.webidentity.role.arn

but, what should we do if it wasn't set?

fail to initialize
have that null value force the env var lookup.

I don't see any way to a completely block the environment variable resolution, which is a pain.

I also see in the internal Library classes that sometimes roles are set up with an external ID, but it is not possible here. Is that an issue?

steveloughran · 2022-03-30T16:53:50Z

so as well as authing with a webidentidy token, we could use https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html to get role credentials for up to 12h. which could go into a delegation token.

steveloughran · 2022-04-04T10:05:12Z

@mehakmeet

vikaskoppineedi · 2022-07-20T14:28:41Z

we need this feature, can somebody please help?

insider89 · 2023-04-10T07:24:24Z

Hey @steveloughran. Could you please provide status update on this PR, so you plan to merge it, or it's already outdated?

steveloughran · 2023-04-12T13:04:56Z

@insider89 not been any changes on this PR since it was last reviewed. if the author takes up the pr, addresses those issues etc then we can work on getting it in to hadoop.

I would suggest asking the author what the status is, and then working with them and other interested parties to get it into state where the reviewers are happy. I am not set up to test this, so a key role of those who need this is to verify the patch works.

boblozano · 2023-08-14T15:47:31Z

@jclarysse - I am looking for this same capability - we're deploying trino / hive metastore (on top of Hadoop) in EKS, and of course service account -> iam role mappings do not work as is.

Is this PR merged anyplace where we can build and try out inside of hive metastore?
Thanks.

steveloughran · 2023-08-16T13:16:27Z

note that hadoop-trunk will, once #5872 is in, move to aws sdk 2 only, with the other credential providers. There will be support for v1 credential providers, but only if the v1 aws sdk is explicitly added to the classspath

a PR to add web identity to trunk based on the AWS SDK 2 code is welcome, with docs and tests; one using V1 classes not going to get in. sorry.
I'm not sure yet whether we will ever do another release with a 1.x AWS SDK; depends on timetables, motivation etc. Ideally we will be shipping a hadoop version on the v2 sdk later this year.

steveloughran · 2024-01-17T14:12:20Z

Trunk is on aws v2 sdk now; it's where features go in. once something is in there a patch for branch-3.3 may be considered; that'd have to be a v1 implementation of the same feature so I'd be reluctant to add it as it would be very different.

Ju Clarysse added 2 commits March 14, 2022 08:28

fixup

4aad0ae

fixup

d213b2c

steveloughran changed the title ~~Hadoop 18154 trunk~~ HADOOP-18154. S3A Authentication to support WebIdentity Mar 15, 2022

steveloughran requested changes Mar 15, 2022

View reviewed changes

Ju Clarysse added 5 commits March 15, 2022 12:09

fix indent

2c25b87

fix license

f720742

use external constant class

623ce66

help with logging

e624ea1

fix indent

9377361

apache deleted a comment from hadoop-yetus Mar 21, 2022

steveloughran requested changes Mar 21, 2022

View reviewed changes

dannycjones suggested changes Mar 22, 2022

View reviewed changes

apache deleted a comment from hadoop-yetus Mar 22, 2022

steveloughran self-assigned this Mar 23, 2022

jclarysse mentioned this pull request Apr 27, 2022

Hadoop 18154 branch 2.10.1 #4049

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-18154. S3A Authentication to support WebIdentity #4070

HADOOP-18154. S3A Authentication to support WebIdentity #4070

jclarysse commented Mar 14, 2022 •

edited

steveloughran left a comment

steveloughran Mar 15, 2022

steveloughran Mar 15, 2022

steveloughran Mar 15, 2022

jclarysse Mar 15, 2022

steveloughran Mar 21, 2022

steveloughran Mar 15, 2022

steveloughran Mar 21, 2022

steveloughran Mar 21, 2022

steveloughran Mar 21, 2022

steveloughran Mar 21, 2022

steveloughran Mar 21, 2022

dannycjones left a comment

dannycjones Mar 22, 2022 •

edited

dannycjones Mar 22, 2022

steveloughran commented Mar 22, 2022

steveloughran commented Mar 30, 2022

steveloughran commented Apr 4, 2022

vikaskoppineedi commented Jul 20, 2022

insider89 commented Apr 10, 2023

steveloughran commented Apr 12, 2023

boblozano commented Aug 14, 2023

steveloughran commented Aug 16, 2023

steveloughran commented Jan 17, 2024

		@@ -0,0 +1,79 @@
		package org.apache.hadoop.fs.s3a;

		import org.apache.commons.lang3.StringUtils;

		import com.amazonaws.auth.WebIdentityTokenCredentialsProvider;

		import org.apache.commons.lang3.StringUtils;

HADOOP-18154. S3A Authentication to support WebIdentity #4070

Are you sure you want to change the base?

HADOOP-18154. S3A Authentication to support WebIdentity #4070

Conversation

jclarysse commented Mar 14, 2022 • edited

Description of PR

How was this patch tested?

For code changes:

steveloughran left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dannycjones left a comment

Choose a reason for hiding this comment

dannycjones Mar 22, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steveloughran commented Mar 22, 2022

steveloughran commented Mar 30, 2022

steveloughran commented Apr 4, 2022

vikaskoppineedi commented Jul 20, 2022

insider89 commented Apr 10, 2023

steveloughran commented Apr 12, 2023

boblozano commented Aug 14, 2023

steveloughran commented Aug 16, 2023

steveloughran commented Jan 17, 2024

jclarysse commented Mar 14, 2022 •

edited

dannycjones Mar 22, 2022 •

edited