Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support IAP on GKE #60

Closed
7 tasks done
jlewi opened this issue Dec 21, 2017 · 8 comments
Closed
7 tasks done

Support IAP on GKE #60

jlewi opened this issue Dec 21, 2017 · 8 comments
Assignees
@ankushagarwal
Labels
platform/gcp priority/p1
Milestone
Kubecon Europe

Comments

@jlewi
Copy link
Contributor

jlewi commented Dec 21, 2017 •
edited by ankushagarwal

When Kubeflow is deployed on GKE it should be easy to configure it to setup IAP to secure remote access to Kubeflow services.

  • Ksonnet manifests for setting up ingress and other resources
  • Docs explaining how to enable IAP
  • Use Cloud Endpoints (or some other proxy) to validate JWT and reject traffic that bypassed IAP

Here are issues that need to be fixed to use Envoy as a proxy that handles JWT verification

  • istio/proxy#941 -Need support for ES256 algorithm which is what IAP uses
  • istio/proxy#939 - Need to support headers used by IAP
  • istio/proxy#929 - Envoy proxy needs to support rejecting all requests without valid JWT credentials
  • istio/proxy#930 - Need to support JWT validation without breaking GCP health checking
    • We can work around this just by having 2 envoy proxies in sequence
    • both can be running in the same pod
@jlewi
Copy link
Contributor Author

jlewi commented Jan 7, 2018

So I have a prototype of enabling IAP https://github.com/jlewi/kubeflow/tree/iap. The one issue I'm hitting is that the proxy doing IAP validation doesn't seem to reject traffic that didn't go through IAP.

In this implementation a sidecar in the JupyterHub pod validates the incoming request and reject it if it didn't go through IAP or the JWT isn't valid.

One question I have is how can we configure JupyterHub to use the identity in the JWT but not verify it? It looks like there's a JWTAuthenticator but I don't actually want to authenticate it since the request was already authenticated in the side car.

Is there an easy way to tell JupyterHub to get identity from the JWT but not authenticate it?

/cc @yuvipanda @mogthesprog @foxish

@mogthesprog
Copy link

I’m on mobile and out at the moment. But IIRC IAP sends some special headers X-Goog-User or something, so you could maybe use the Remote-User Authenticator for jupyter hub which takes the name of the user header as configuration? I can have another nose for the links when I get home.

That should do what you need on then jupyterhub end I believe.

@jlewi
Copy link
Contributor Author

jlewi commented Jan 7, 2018

@mogthesprog Thanks for the quick response let me try that out.

@jlewi
Copy link
Contributor Author

jlewi commented Jan 7, 2018

That worked thank you very much. I just had to configure the JupyterHub config as follows

c.JupyterHub.authenticator_class ='jhub_remote_user_authenticator.remote_user_auth.RemoteUserAuthenticator'
c.RemoteUserAuthenticator.header_name = 'x-goog-authenticated-user-email'

@mogthesprog
Copy link

Awesome! :) Glad it worked!

Yeah there are a couple of headers forwarded by IAP. I’ll have to check if there’s one that contains a signed JWT and maybe that can be used to by the jwtauthenticator.

@jlewi
Copy link
Contributor Author

jlewi commented Jan 7, 2018

Yup there is see here.

In our case, I think its safe to use x-goog-authenticated-user-{email,id} because we're relying on the sidecar to ensure traffic went through IAP and IAP will strip out any user supplied x-goog-* headers.

This was referenced Jan 14, 2018
Merged
Merged
@jlewi jlewi added this to the Kubecon Europe milestone Jan 20, 2018
@jlewi jlewi mentioned this issue Jan 29, 2018
Merged
jlewi added a commit that referenced this issue Jan 31, 2018
@jlewi
Support IAP on GKE using Envoy as a reverse proxy (#128)
763d18e 
To provide secure ingress to the services running on the cluster we will use https -> reverse proxy and the reverse proxy will enforce authentication by checking JWTs.

This PR adds a reverse proxy and JWT validation based on Envoy.

Envoy can't validate JWT's created by IAP because of a couple of blocking issues.
  * istio/proxy#941
  * istio/proxy#939
  * istio/proxy#929
  * istio/proxy#930

Nontheless, this PR allows us to turn on IAP and a reverse proxy. We can use this to continue developing our story for secure access to services in the cluster (e.g. TensorBoard and TF models)

While this PR focuses on GCP's IAP most of this code will work just as well for non GCP; see #11 for more details.

We make Kubespawner configurable so that when using IAP we use the user's Google identity as verified by IAP and do not make the user login to JupyterHub

This requires building a new JupyterHub image with the jhub_remote_user_authenticator added.
Related to #60 and #11 regarding secure ingress
jlewi added a commit that referenced this issue Feb 2, 2018
@jlewi
Use ambassador as a reverse proxy (#166)
da6d4b6 
Ambassdor provides a reverse proxy that can be dynamically configured; we can mappings just by creating K8s services and anotating them with the prefix we want to serve the path on.

Ambassador makes it really easy to configure our reverse proxy to direct traffic to an ephemeral service (e.g. a Tensorboard instance serving data for a particular job).

This PR adds Ambassador and adds some default routes; e.g. we add a route to serve the K8s dashboard at /k8s/ui/

In follow on PRs we can annotate other services (like the TFJobs UI) and TensorBoard deployments so that we can create mappings for those servers as well.

Related to:
#11 Secure Proxy
#60 IAP on GKE
#154 Contour vs. Ambassador.
@jlewi
Copy link
Contributor Author

jlewi commented Mar 8, 2018

The envoy issues have been addressed so we should be unblocked.

@jlewi
Copy link
Contributor Author

jlewi commented Mar 8, 2018

I believe this is the docker file I used
https://github.com/jlewi/proxy/blob/iap/docker/Dockerfile.kubeflow

But its probably unnecessary to build our own Docker image if they already built one with the changes.
If not I think this is there release script.
https://github.com/jlewi/proxy/blob/iap/script/release-docker

@jlewi jlewi mentioned this issue Mar 8, 2018
Closed
@ankushagarwal ankushagarwal self-assigned this Mar 9, 2018
@ankushagarwal ankushagarwal mentioned this issue Mar 9, 2018
Merged
k8s-ci-robot pushed a commit that referenced this issue Mar 9, 2018
@k8s-ci-robot
Use the correct jwt validation config (#401)
eb8afe5 
* Use the correct jwt validation config

It is defined in https://github.com/istio/proxy/blob/master/src/envoy/http/jwt_auth/config.proto in the JWT message

Related #394
Related #60

* Update envoy image
@ankushagarwal ankushagarwal mentioned this issue Mar 9, 2018
Merged
k8s-ci-robot pushed a commit that referenced this issue Mar 10, 2018
@k8s-ci-robot
JWT validation works with GCP IAP - remove outdated docs (#406)
0e392eb 
* JWT validation works with GCP IAP - remove outdated docs

Related to #60

* Updated to iap.md after JWT validation is supported
kimwnasptd pushed a commit to arrikto/kubeflow that referenced this issue Mar 5, 2019
@zabbasi @k8s-ci-robot
Update org.yaml (kubeflow#60)
f4efd65
yanniszark pushed a commit to arrikto/kubeflow that referenced this issue Feb 15, 2021
@YujiOshima @gaocegege
improve frontend (kubeflow#60)
dd6c87e 
* improve frontend

Signed-off-by: YujiOshima <yuji.oshima0x3fd@gmail.com>

* add sample config for TFBoard ingress host

Signed-off-by: YujiOshima <yuji.oshima0x3fd@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ankushagarwal ankushagarwal

Labels
platform/gcp priority/p1
Projects
None yet
Milestone
Kubecon Europe
Development

No branches or pull requests
4 participants
@jlewi @ankushagarwal @mogthesprog @lluunn