KNOX-2020 - AWS federation support added to hadoop-jwt cookie

[sharad-oss]

This contribution enhances the Hadoop JWT cookie to contain AWS credentials to interact with the AWS ecosystem. This is done via an optional configuration when using SAML for authentication. The account admin can set up AWS federation with SAML (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_enable-console-saml.html). The SAML Response in such a case will contain the Identity Provider (IDP) injected Role. Simple Token Service (STS) is invoked for AssumeRoleWithSaml using this SAML Response for credentials, and on success a cookie is added to the browser. The value for this cookie is the Base 64 encoded credentials. The TTL for this cookie matches the TTL for AWS credentials. From there on the usual Knox flow continues.

Knox will add the hadoop-jwt cookie that is understood by the Hadoop ecosystem. A JWT provides standard info about the subject, TTL and claims. We enhance the claims to contain the AWS credentials by using the info in the previously stashed aws cookie. This way the Hadoop ecosystem services can now interact with AWS services too. The lifetime for the cookies matches the life of AWS credentials.

What changes were proposed in this pull request?

The Pac4j filter has been modified to optionally do AWS federation. This change has no effect if the configuration is not enabled.
A new security module gateway-provider-security-aws handles all the logic to do SAML federation and adding an AWS cookie.
This module receives the SAML Response and federates into AWS, and assume the first role in the assertion. This could be enhanced in future to determine which Role to pick. AWS credentials are stashed in a cookie for further usage (after the redirect happens in the filter). At the end it returns back to the pac4j filter with a cookie added to the response.

Methods to issue JWT token have been enhanced to take custom claims as parameters.

In KnoxSSO we enhance the hadoop-jwt cookie with the contents from the AWS cookie.
This is done by using custom claims. The procured AWS credentials are added to the JWT claims. Please look at the testing section for how the cookie looks like after the change. Note the cookie is unaffected if the option to do AWS SAML federation is not turned on.

Knox token has also been modified to reflect the new token format.

How was this patch tested?

Ran all unit tests
Tested on an EC2 deployment (AWS EMR). The knoxtoken (/knoxtoken/api/v1/token) looks like

{
  "sub": "[sharad]",
  "aud": "tokenbased",
  "iss": "KNOXSSO",
  "aws-session": "eyJ1c2VybmFtZSI6InNoYXJrYWxhQGdtYWlsLmNvbSIsIkFXU0FjY2Vzc0tleUlkIjoiQVNJQVNTRkFaVFBBV0tGWTdVNFMiLCJBV1NTZWNyZXRLZXkiOiJYT01yS0FHdFBNakdaWi91SlJibzM2eFhzRFlwT0RzU1IvWlA1N2RQIiwic2Vzc2lvblRva2VuIjoiRlFvR1pYSXZZWGR6RUZFYURDNE8vS0NVZ2pNNWFYY0luaUtsQWh1dTNvK2NkTEoyNmwxMzdKNVI1dFNROUZKUHIzQ2Y4SGJhdHMzc1pWSzJZeUlGTFZGcjJUTXBCZjBxVjZTQ2ZYSzZwUzZ2L3hHeFZvZGl2L1VOVzZLclBEeUhkbEtPT3lKdVBncGdPRkdoakdSZUw3YTBqSmNpVUhZalVhRmo4UVd5R2lkM05xRVFzeTVQRlEwalM0SmFZUkU0K04yczlycWhxSWF4K1hreDBEb0gzWGl1Y2RDVUtBQVcwK0NNaHB3eGV1d2paZDIwZGJNOTFOTTVsR2dNbmxZdzBiNjdPdXlwUDI3T0NoeU9TYkdGVVJtbC9DY3YrbjNwMFA2RVY0Q3pJbnZlUjBRN28zL0tDQmRsdGF2Mi9zWXUzSDZ0N0pXVE9ZYUJvUHhralZzTENlZmN2OXRJc2Q3bGx4cmxsUEhmRUZBMmJIOXRaa0xOQlY2alE4NmRzL1NlS09ZTlJBTnNaTmJOckxZOTlJekR2NmdPQklaVWlTNXZ4aUY4ajZMVEs2VjBLSnV2NitzRiIsImV4cGlyYXRpb24iOjE1NjgzMzUyNzUwMDB9",
  "exp": 1568367676
}
The AWS credentials are in Base 64 encoded format, and the exp matches the AWS credentials exp.

[risdenk]

Thanks for the PR @sharad-oss. I went through quickly looking at a lot of the dependency handling stuff. I think there are a few places that can be improved to match the other modules in the Knox project.

I'll have to go through in more detail over the next few days and look at the cookie/aws parts in detail.

[risdenk]

@sharad-oss looks like the jdk11 Travis failure might be real as well:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project gateway-provider-security-aws: Fatal error compiling: java.lang.ExceptionInInitializerError: com.sun.tools.javac.code.TypeTags -> [Help 1]

[sharad-oss]

@risdenk I wasn't aware Knox was being built on JDK-11 too. My current tests were all done using JDK-8. I will invest some time to build with JDK-11.

[lmccay]

From my comment on JIRA KNOX-2020:

@sharad-oss - this looks interesting!
I'd actually like to see a one-pager type doc that describes the usecases, the design and security considerations. Please attach it to the JIRA itself.
One thing that I am concerned about is the inclusion of sensitive credentials in the JWT based cookie.
The cookie from KnoxSSO is intended for browsers and generally represents the authenticated user but doesn't include credentials. It is essentially in clear text since the JWT is merely base64 encoded. This is not sufficient protection for credentials that can be used outside of the scope of Knox itself.
In terms of usecases, I'd like to understand the full flow including how/where the credentials are actually used and what consumer will be provided in Knox for the credentials.

[lmccay]

I've left a comment on the PR itself and on the KNOX-2020 JIRA but will add it here as well as to request changes.
I'd like to see a one-pager design doc that articulates the usecases, security considerations and end-to-end flow including the provided consumer of the credentials within the gateway.
See the other comments for my security concerns regarding including the credentials in the token.

[sharad-oss]

@lmccay sure will work on the one pager doc soon, and update the JIRA.

[sharad-oss]

@risdenk Addressed your feedback. I am seeing some test failures but they don't seem related to my changes.

I am also working on a one pager as requested by @lmccay.

[risdenk]

@sharad-oss thanks for the cleanup. Looks a lot better. There are a few remaining items related to dependencies. Also looking forward to reading the design doc @lmccay when you have it ready.

[smolnar82]

This PR has not been touched for over a year ago; closing it.