adding oauth support to MinIO browser #8075

kanagarajkm · 2019-08-14T10:23:41Z

Description

Adding OpenID login support to MinIO Browser.

Motivation and Context

Users will be able to login MinIO Browser through the configured Identity Provider.

How to test this PR?

Setup an Identity Provider(wso2, keycloak, Auth0, etc)
Update config.json with openid configuration

  "openid": {
    "jwks": { "url": "<JWKS_URL>" },
    "configURL": "<CONFIC_URL>",
    "policyClaimPrefix: "<POLICY_CLAIM_PREFIX>"
  }

Sample URLs for Keycloak are
configURL - http://localhost:8080/auth/realms/demo/.well-known/openid-configuration
jwks url - http://localhost:8080/auth/realms/demo/protocol/openid-connect/certs

JWT token should include a custom claim for the policy, this is required to create a STS user in MinIO. The name of the custom claim could be either policy or <NAMESPACE_Prefix>policy.
If there is no namespace then policyClaimPrefix can be ingored. For example if the custom claim name is https://min.io/policy then, policyClaimPrefix should be set as https://min.io/

Open MinIO Browser and click Log in with OpenID
Type in Client ID obtained from Identity Provider and ENTER
The user will be redirected the IdP login page
Upon successful login on IdP the user will be automatically logged into MinIO Browser

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

Fixes a regression (If yes, please add commit-id or PR # here)
Documentation needed
Unit tests needed
Functional tests needed (If yes, add mint PR # here: )

pkg/iam/policy/constants.go

pkg/auth/credentials.go

cmd/config-versions.go

harshavardhana · 2019-09-11T00:34:30Z

can you get this ready for final merge @kanagarajkm with builds everything running? - I am yet to show this to @abperiasamy will do soon.

kanagarajkm · 2019-09-11T05:53:19Z

can you get this ready for final merge @kanagarajkm with builds everything running? - I am yet to show this to @abperiasamy will do soon.

@harshavardhana ok, i can do that.

codecov-io · 2019-09-13T06:02:41Z

Codecov Report

Merging #8075 into master will increase coverage by 0.16%.
The diff coverage is 12.5%.

@@            Coverage Diff             @@
##           master    #8075      +/-   ##
==========================================
+ Coverage   43.87%   44.04%   +0.16%     
==========================================
  Files         325      303      -22     
  Lines       52891    52093     -798     
==========================================
- Hits        23208    22944     -264     
+ Misses      27525    26991     -534     
  Partials     2158     2158

Impacted Files	Coverage Δ
cmd/web-router.go	`88% <ø> (ø)`	⬆️
cmd/auth-handler.go	`62.76% <0%> (+0.37%)`	⬆️
cmd/sts-handlers.go	`10.66% <0%> (+1.61%)`	⬆️
cmd/web-handler-context.go	`89.18% <0%> (-2.48%)`	⬇️
cmd/utils.go	`67.53% <0%> (-3.74%)`	⬇️
cmd/iam.go	`9.04% <0%> (-0.03%)`	⬇️
cmd/web-handlers.go	`50.46% <1.88%> (-2.31%)`	⬇️
cmd/jwt.go	`85.24% <80%> (-1.08%)`	⬇️
cmd/disk-usage.go	`0% <0%> (-78.95%)`	⬇️
cmd/posix-errors.go	`40.74% <0%> (-37.45%)`	⬇️
... and 88 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8700945...2149538. Read the comment docs.

kanagarajkm · 2019-09-13T06:08:38Z

@harshavardhana the problems are resolved and its ready for review.

harshavardhana

can you also update the sts documentation on how to configure this?

cmd/web-handlers.go

kanagarajkm · 2019-09-25T10:33:41Z

can you also update the sts documentation on how to configure this?

Done, can you please review?

harshavardhana · 2019-10-01T01:40:58Z

@kanagarajkm can you provide more instructions on using this with keycloack ?

kanagarajkm · 2019-10-01T06:51:14Z

@harshavardhana added docs for setting up keycloak, please have a look. Updated web-identity.go to add port as an optional input param since keycloak by default runs on port 8080.

minio-ops · 2019-10-01T07:13:06Z

Mint Automation

Test	Result
mint-compression-xl.sh	✔️
mint-xl.sh	✔️
mint-compression-dist-xl.sh	✔️
mint-compression-fs.sh	✔️
mint-worm.sh	✔️
mint-fs.sh	✔️
mint-gateway-nas.sh	✔️
mint-large-bucket.sh	❌ more...
mint-dist-xl.sh	❌ more...

8075-27d6961/mint-large-bucket.sh.log:

Running with
SERVER_ENDPOINT:      72.28.97.58:31735
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 2346bf3b1fc9:/mint/log /tmp/mint-logs'

(1/14) Running aws-sdk-go tests ... done in 2 seconds
(2/14) Running aws-sdk-java tests ... done in 1 seconds
(3/14) Running aws-sdk-php tests ... done in 59 seconds
(4/14) Running aws-sdk-ruby tests ... done in 16 seconds
(5/14) Running awscli tests ... done in 2 minutes and 15 seconds
(6/14) Running healthcheck tests ... done in 0 seconds
(7/14) Running mc tests ... done in 1 minutes and 33 seconds
(8/14) Running minio-dotnet tests ... done in 1 minutes and 26 seconds
(9/14) Running minio-go tests ... FAILED in 3 minutes and 58 seconds
{
  "args": {},
  "duration": 57716,
  "error": "Please reduce your request",
  "function": "testStorageClassMetadataCopyObject()",
  "message": "CopyObject failed on RRS",
  "name": "minio-go: testStorageClassMetadataCopyObject",
  "status": "FAIL"
}

Executed 8 out of 14 tests successfully.

8075-27d6961/mint-dist-xl.sh.log:

Running with
SERVER_ENDPOINT:      72.28.97.56:32366
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp eb28d9bae755:/mint/log /tmp/mint-logs'

(1/14) Running aws-sdk-go tests ... done in 0 seconds
(2/14) Running aws-sdk-java tests ... done in 1 seconds
(3/14) Running aws-sdk-php tests ... done in 44 seconds
(4/14) Running aws-sdk-ruby tests ... done in 2 seconds
(5/14) Running awscli tests ... done in 2 minutes and 11 seconds
(6/14) Running healthcheck tests ... done in 0 seconds
(7/14) Running mc tests ... done in 1 minutes and 15 seconds
(8/14) Running minio-dotnet tests ... done in 53 seconds
(9/14) Running minio-go tests ... done in 1 minutes and 19 seconds
(10/14) Running minio-java tests ... done in 1 minutes and 18 seconds
(11/14) Running minio-js tests ... FAILED in 23 seconds
{
  "name": "minio-js",
  "function": "fPutObject(bucketName, objectName, filePath, metaData)",
  "args": "bucketName:minio-js-test-dd44be49-5af1-4241-ab62-a8c3c6526050, objectName:datafile-65-MB, filePath:/tmp/datafile-65-MB",
  "duration": 4817,
  "status": "FAIL",
  "error": "Error: read ECONNRESET at exports._errnoException (util.js:1020:11) at TCP.onread (net.js:580:26)"
}

Executed 10 out of 14 tests successfully.

jonasscherer · 2019-10-01T07:50:18Z

Is it possible to just use existing oidc tokens from some auth headers for the authentication for the MinIO Browser? thx

kanagarajkm · 2019-10-01T08:59:28Z

Is it possible to just use existing oidc tokens from some auth headers for the authentication for the MinIO Browser? thx

@jonasscherer sorry i didn't understand your question, can you please elaborate.

MinIO browser receives the token from the Identity Provider through a redirect and then it uses this token to create an STS user then login as that user.

jonasscherer · 2019-10-01T10:04:38Z

@kanagarajkm we are using MinIO behind a auth proxy, which already handles the login, redirect etc.
So when the request reaches the MinIO Browser, the user is already logged in and the corresponding tokens are available through headers. This way Minio just has to verify the existing tokens against the authentication provider and create/login the user.

The question is, can I offer the MinIO browser an authentication header that, if present, logs the user in directly?

kanagarajkm · 2019-10-01T11:01:04Z

@jonasscherer what is the authentication provider used here? does it have support for OpenID Connect?

jonasscherer · 2019-10-01T11:11:21Z

@jonasscherer what is the authentication provider used here? does it have support for OpenID Connect?

Yes - it's actually Keycloak :)

kanagarajkm · 2019-10-01T11:14:57Z

@jonasscherer do you see any problems in configuring KeyCloak as openid provider for MinIO and let the MinIO browser take care of rest instead of using an auth proxy?

jonasscherer · 2019-10-01T12:30:36Z

@kanagarajkm Unfortunately I see some problems, because we want to offer a single sign-on for several services. In this case the user would have to log in to Minio again...
It's basically just like your workflow, except that the user is already logged in. How do you figure out if a request comes from a logged in user or not?

kanagarajkm · 2019-10-01T12:51:14Z

@jonasscherer let me clarify a few things here.

Let's take for example keycloak is running in https://auth.domain1.com, then you have a MinIO server running on https://minio.domain2.com then another xyz service running on https://xyz.domain3.com

So when a user tries to access https://minio.domain2.com, they are redirected to https://auth.domain1.com where they enter their credentials(this is not MinIO creds). Then they are redirected back to https://minio.domain2.com as logged in user, they don't login explicitly on MinIO, the user doesn't require MinIO creds. This same applies when they access the xyz service running on https://xyz.domain3.com.

jonasscherer · 2019-10-01T13:36:44Z

@kanagarajkm This is exactly what I meant :)
My question referred to how MinIO determines that a user is logged in (after redirected from Keycloak). Typically this is achieved by authentication cookies or headers. I would then have to provide the required information. I think it will probably work out of the box. Thanks for your effort, I'm looking forward to try the new feature.

kanagarajkm · 2019-10-01T17:23:20Z

MinIO server verifies the JWT token received from Keycloak using the public certs(JWKS url) of keycloak. Then creates an STS user with the provided policy. This user is now logged in automatically. So its not just about setting the auth header.

what exactly is your auth proxy? Is it keycloak gatekeeper?

harshavardhana · 2019-10-14T22:17:43Z

@kanagarajkm can you re-work this PR to work with #8392 such that, we can merge this into this branch as well.

harshavardhana · 2019-10-14T22:18:37Z

Send it to branch new-config-admin-v2 on minio

larsw · 2019-10-15T16:11:33Z

Hi,

First off, I must say that I very much welcome this PR!

Question 1: Isn't it better to infer the jwks URL from the .well-known/openid-configuration endpoint configured with configURL? E.g. the jwks configuration property is now redundant.

Question 2/advice: Please consider using a battle-hardened oidc package like https://www.npmjs.com/package/oidc-client - it even has a React wrapper: https://www.npmjs.com/package/react-oidc

Question 3: Consider adding a configurable drop-down menu for choosing the OpenID Connect Provider to use, so that the end-user doesn't need to enter the technical client name of it.

harshavardhana · 2019-10-15T19:14:09Z

Closing this, this functionality is merged into new-config-admin-v2 branch

harshavardhana requested changes Aug 14, 2019

View reviewed changes

pkg/iam/policy/constants.go Outdated Show resolved Hide resolved

pkg/auth/credentials.go Outdated Show resolved Hide resolved

cmd/config-versions.go Outdated Show resolved Hide resolved

kanagarajkm force-pushed the sts-browser branch 2 times, most recently from ae615d6 to 9f09fda Compare August 20, 2019 10:58

kanagarajkm force-pushed the sts-browser branch 2 times, most recently from d0b7c22 to bc295d2 Compare August 29, 2019 06:03

This was referenced Sep 11, 2019

Support namespaced JWT Claims #8107

Closed

Support OpenID configuration URL #7055

Closed

kanagarajkm force-pushed the sts-browser branch 3 times, most recently from 4c4896c to 2149538 Compare September 13, 2019 05:15

kanagarajkm changed the title ~~[WIP] adding oauth support to MinIO browser~~ [adding oauth support to MinIO browser Sep 13, 2019

kanagarajkm changed the title ~~[adding oauth support to MinIO browser~~ adding oauth support to MinIO browser Sep 13, 2019

kanagarajkm marked this pull request as ready for review September 13, 2019 06:07

kanagarajkm force-pushed the sts-browser branch from 2149538 to ab5b4cf Compare September 18, 2019 07:41

harshavardhana requested changes Sep 25, 2019

View reviewed changes

cmd/web-handlers.go Outdated Show resolved Hide resolved

kanagarajkm force-pushed the sts-browser branch 3 times, most recently from 78ba3bc to 51636ae Compare September 25, 2019 10:32

kanagarajkm added 2 commits October 1, 2019 10:48

adding oauth support to MinIO browser

70245f6

added keycloak quick start guide

27d6961

kanagarajkm force-pushed the sts-browser branch from 51636ae to 27d6961 Compare October 1, 2019 06:47

harshavardhana closed this Oct 15, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

adding oauth support to MinIO browser #8075

adding oauth support to MinIO browser #8075

kanagarajkm commented Aug 14, 2019 •

edited

harshavardhana commented Sep 11, 2019

kanagarajkm commented Sep 11, 2019

codecov-io commented Sep 13, 2019 •

edited

kanagarajkm commented Sep 13, 2019

harshavardhana left a comment

kanagarajkm commented Sep 25, 2019

harshavardhana commented Oct 1, 2019

kanagarajkm commented Oct 1, 2019

minio-ops commented Oct 1, 2019

jonasscherer commented Oct 1, 2019

kanagarajkm commented Oct 1, 2019

jonasscherer commented Oct 1, 2019

kanagarajkm commented Oct 1, 2019

jonasscherer commented Oct 1, 2019

kanagarajkm commented Oct 1, 2019

jonasscherer commented Oct 1, 2019

kanagarajkm commented Oct 1, 2019

jonasscherer commented Oct 1, 2019

kanagarajkm commented Oct 1, 2019

harshavardhana commented Oct 14, 2019

harshavardhana commented Oct 14, 2019 •

edited

larsw commented Oct 15, 2019 •

edited

harshavardhana commented Oct 15, 2019

adding oauth support to MinIO browser #8075

adding oauth support to MinIO browser #8075

Conversation

kanagarajkm commented Aug 14, 2019 • edited

Description

Motivation and Context

How to test this PR?

Types of changes

Checklist:

harshavardhana commented Sep 11, 2019

kanagarajkm commented Sep 11, 2019

codecov-io commented Sep 13, 2019 • edited

Codecov Report

kanagarajkm commented Sep 13, 2019

harshavardhana left a comment

Choose a reason for hiding this comment

kanagarajkm commented Sep 25, 2019

harshavardhana commented Oct 1, 2019

kanagarajkm commented Oct 1, 2019

minio-ops commented Oct 1, 2019

Mint Automation

8075-27d6961/mint-large-bucket.sh.log:

8075-27d6961/mint-dist-xl.sh.log:

jonasscherer commented Oct 1, 2019

kanagarajkm commented Oct 1, 2019

jonasscherer commented Oct 1, 2019

kanagarajkm commented Oct 1, 2019

jonasscherer commented Oct 1, 2019

kanagarajkm commented Oct 1, 2019

jonasscherer commented Oct 1, 2019

kanagarajkm commented Oct 1, 2019

jonasscherer commented Oct 1, 2019

kanagarajkm commented Oct 1, 2019

harshavardhana commented Oct 14, 2019

harshavardhana commented Oct 14, 2019 • edited

larsw commented Oct 15, 2019 • edited

harshavardhana commented Oct 15, 2019

kanagarajkm commented Aug 14, 2019 •

edited

codecov-io commented Sep 13, 2019 •

edited

harshavardhana commented Oct 14, 2019 •

edited

larsw commented Oct 15, 2019 •

edited