Secure proxy

[jlewi]

Opening this issue to track addition of a secure proxy.

Kubeflow will consist of many applications running insecure web apps. Already the stack contains Juptyer notebooks and TensorBoard.

It would be nice if Kubeflow included a secure proxy so that a user could access Kubeflow components at URLs like

https://my-kubeflow.my-domain.io/juptyerhub
https://my-kubeflow.my-domain.io/tf-jobs/my-job/tensorboard

Some issues to consider.

• Some of these apps are ephemeral. For example,
  * TfJob spins up a TensorBoard instance whose lifetime is tied to the life time of the job. So we'd like to dynamically add routes.
• We'd like to integrate nicely with different Clouds
  * For example, on GKE we'd like to use IAP

I think we can use Ambassador to do this

• Ambassador is a proxy based on Envoy
• Mappings can be dynamically added just by adding annotations to a K8s service

Here's a diagram illustrating what this would like

Using Ambassador everything sits behind a single ingress point. The ingress would be configured differently on different clouds but everything else should be the same.

On GKE we can secure the ingress using IAP

• Hopefully Ambassador supports (or will support) Envoy's JWT features so that we can use Ambassador to reject traffic that bypassed IAP

Some open questions

• How do people securely connect to non-GKE clusters?
  * Do people use VPN? Do we need an IAP equivalent?
• Can we configure Ambassador to do JWT validation?
• On non-GKE clusters I think we could use oauth-proxy to replace IAP
  * This could authenticate the request using a variety of providers (Github, Google, Facebook) and provide a signed JWT. At which point the solution is largely the same as in the GKE case (i.e. use Envoy to validate JWT and reject traffic without a JWT)
  * i think we'd want to integrate oauth-proxy into our envoy proxy
  * Ambassador already has support for external auth (but not oauth-proxy)
  * Not sure about contour
  • How much can we get done for Kubecon May?

Alternatives

• It looks like HeptIo has a similar project contour
  • I think contour is designed as an ingress controller

Not sure what the benefits of Contour vs. Ambassador might be. Anyone have an opinion?

• I've gotten Ambassador working so that's what I'm probably going to start with.

[jlewi]

I was told by someone at Redhat that ISTIO's egress functionality might provide a solution that doesn't involve a reverse proxy.

[jlewi]

Here are some relevant links
https://github.com/fabric8io/exposecontroller
https://blog.fabric8.io/accessing-your-applications-on-kubernetes-or-openshift-3e69be0ad3bc

[jlewi]

Also relevant

https://github.com/PalmStoneGames/kube-cert-manager

cert-manager in combination with external DNS were mentioned as one way to secure TLS services.

/cc @elsonrodriguez

[jlewi]

One question that has come up is why not use JupyterHub's built in support for TLS and OAuth authentication.

I think that would only be a partial solution because we have lots of web services (e.g TfJob UI, Argo UI, TensorBoard, etc...) that we need to secure in addition to JupyterHub.

The pattern in K8s seems to be to use a common sidecar which can serve as a secure proxy that sits in front of insecure web services. I think ISTIO's envoy recently added JWT validation. I was told that this was still in the early stages though and not production ready.

/cc @yuvipanda

[jimexist]

What about dex and go-oidc?

[jimexist]

Related: also at least some consideration should be taken into where to store the JWT token, cookie or local storage? This has some clear pros and cons and tradeoff points to take.

[pdmack]

Istio could provide east-west security to Ambassador's north-south. There are slightly different options available for the sidecar injection depending on the minimum kube requirement (1.8->manual injection; 1.9->automatic via mutating webhooks).

[jlewi]

I wonder if a simple initial solution (maybe in time for our 0.1 release) would be to use Ambassador's external authentication and use basic http auth.
https://www.getambassador.io/user-guide/auth-tutorial

[plombardi89]

@jlewi I work on Ambassador with @kflynn and I'm looking for your thoughts on this... It would be fairly easy to write an Ambassador HTTP Basic Authentication extauth module. The implementation reads the contents of password database and compares a given password with the hashed value in the database to determine whether a user has access to a protected resource.

Password Database Schema

• ${user_id} represents the authenticating users name, email or system ID
• ${bcrypt_password} is the bcrypt hashed password used for the authentication check.

---
users:
  ${user_id}:
    password: ${bcrypt_password}

Algorithm doesn't have to be bcrypt. Just throwing that out there as an idea.

Extra Tooling

• Maybe: A simple CLI tool to add / remove users from the password database and verify the database validity.

Theory of Operation

Deployment

1. Create the Password DB file.
2. Deploy the Password DB file as a Kubernetes v1.Secret.
3. Deploy the Extauth module and inject the Password DB as a volume.
4. Read the Password DB into memory.

Add / Remove User

1. Update the Password DB file.
2. Update the `v1.Secret' with the new Password DB contents.
3. The Extauth module should watch the mounted secret volume for file changes. When it changes then reload.

Runtime

If the user and password hash matches then Extauth returns a 200 OK otherwise 401 Unauthorized.

[jlewi]

@plombardi89 I would love this! I think this gives us a very simple story that works anywhere.

[plombardi89]

@jlewi We are scheduling some engineering time to work on this during our upcoming two week engineering cycle.

[plombardi89]

We've begun work on this in https://github.com/datawire/ambassador-auth-httpbasic

[plombardi89]

@jlewi We released 0.1.1 of https://github.com/datawire/ambassador-auth-httpbasic . Please give it a spin!

[jlewi]

Thank you so much.

[jlewi]

Would HTTP auth behind https be useful for any of the other K8s distributions (e.g. Azure, Openshift, or Canonical)?

Is anyone interested in taking this on?

/cc @wbuchwalter @willb @sfabel

[bhack]

Check emissary-ingress/emissary#28

[jlewi]

Thanks @bhack looks like someone created an OIDC plugin for Ambassador that uses keycloak
https://github.com/ajmyyra/ambassador-auth-oidc

[pdmack]

OpenShift makes use of sidecar proxies for securing endpoints that are otherwise insecure
https://github.com/openshift/oauth-proxy

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.