Support OAuthenticator for our hubs

[yuvipanda]

Description

Right now, we only support using Auth0 for our hub authentication. Individual hubs should be able to opt-in to using OAuthenticator for specific features, rather than being forced to just use Auth0.

Benefit

While our Auth0 setup allows us to create arbitrary new hubs without needing to provision external auth credentials (like GitHub OAuth secrets, Google OAuth secrets, etc), it doesn't allow us to use provider specific features present in the JupyterHub OAuthenticator - such as allowing users with a GitHub org or team to login (#598).

Many communities may want to use these features, particularly the more advanced ones. Most importantly, this would allow us to mirror Pangeo's authentication setup.

Implementation

• Note that if we set up OAuthentication for our hubs, this will be a manual step
• We should explore whether we can do the same thing but using OAuth instead of a manual OAuthentication

Tasks to complete

• Make a release of Z2JH (since changes to OAuthenticator are on master right now)
• Bump our z2jh image in basehub chart and /images/hub: Bump jupyterhub chart and hub image to version 1.1.3 #635
• Implementation proposal for how we'd support both Auth0 + OAuthenticator
• Support specifying secret config on a per-hub / per-cluster basis, to store auth credentials
• Determine policy for when we should use OAuthenticator directly vs Auth0

Updates

• 2021-09-14 - For this sprint, @sgibson91 is going to put together a design proposal for how we could balance using both Auth0 and the native authenticator in our hub deployments.
• 2021-09-21 - Give a shot at implementing native OAuthenticator on our hubs + using that to authenticate with GitHub Teams. Will open another issue for long-term options using Auth0.
• 2021-09-29 - Merging Support native JupyterHub OAuthenticator in 2i2c-managed hubs #706 caused some JSON validation errors so we rolled back the merge and will try to debug
  • Deploy manually, might be being bitten by obscured error output in CI
  • Use online JSON validation checker to make absolutely sure

[sgibson91]

Changelog for v1.1.3 of z2jh merged and tag release made jupyterhub/zero-to-jupyterhub-k8s#2361

[sgibson91]

Opened #635 to bump the version of z2jh we're using

[choldgraf]

Next steps

• Come up with a design proposal for how we'd implement this in our hub deployments.

[sgibson91]

I have a HackMD with the design proposal here: https://hackmd.io/@sgibson91/S1CWEnemY Feedback appreciated!

[choldgraf]

Hey @sgibson91 - I will take a closer look this week! But I had two really quick top-level questions:

1. Can we rule out the possibility of using Auth0 + GitHub Teams authentication? If this were possible it'd simplify things a bit (at least in the short term), but I'm assuming we've already checked and it isn't possible.
2. For the secret credentials, couldn't we just store those in the same hub config file, so long as they were encrypted? If I recall, sops still produces valid YAML even when the file is encrypted, so as long as we remembered to decrypt as part of deploys, maybe it'd be possible?

[yuvipanda]

I left a couple comments in the hackmd, @sgibson91. But overall +1 from me.

To get this to work with auth0, I think we'll need to add functionality that so that somehow when auth0 is authenticating with GitHub, it fetches information about 'groups' from GitHub, populates it into auth0, and then uses that for login authorization. I think auth0 has RBAC functionality (https://auth0.com/docs/authorization/rbac/) that can be used to do this and more. The question is how to get info from GitHub in auth0, and I haven't played around with it. I feel like it should be possible, since it is such an obvious use case.

However, the path forward that @sgibson91 has outlined here with sops and OAuthenticator is simpler to implement right now, and more of a known quantity to us. I'm happy to defer to her on wether the auth0 option should be investigated further or not.

[sgibson91]

Thanks for the comments @yuvipanda! I would have to do some further digging into the Auth0 use case. It's definitely a valid use case but, thinking in terms of our Right to Replicate, is authenticating with GitHub Teams via Auth0 something that a community would want to do? I understand we're doing it as it saves us the toil of manually creating client IDs and secrets for every auth connection. Are there benefits for a single hub community for taking this approach? I might recommend opening a longer term issue for investing GitHub Teams with Auth0 and come back to it if we find this is an overwhelmingly popular option and we can't keep up with the toil - if people are +1?

2. For the secret credentials, couldn't we just store those in the same hub config file, so long as they were encrypted? If I recall, sops still produces valid YAML even when the file is encrypted, so as long as we remembered to decrypt as part of deploys, maybe it'd be possible?

There are a couple of reasons that make me want to push-back against this:

1. We configure auth on a per-hub basis, not a per-cluster basis. So putting all config in one file and encrypting it would mean encrypting config for all hubs, even if they don't have the secret fields associated with the JupyterHub OAuthenticator. This is overkill.
2. While sops maintains the YAML structure, it still encrypts all values. And I think "remembering to decrypt" would get in the way of some quick operating tasks, such as updating image tags or domain names, that could presently be done in the browser if required.
3. I also think 2 begins to get in the way of our Right to Replicate. How can a community replicate their infrastructure, specifically without our support, if we've encrypted it? At least in the current proposal, the minimal amount of config is obscured and that can easily be deduced from the z2jh/OAuth docs.

[yuvipanda]

I might recommend opening a longer term issue for investing GitHub Teams with Auth0 and come back to it if we find this is an overwhelmingly popular option and we can't keep up with the toil - if people are +1?

Sounds good to me! Do we already have one?

[sgibson91]

Do we already have one?

None that I'm aware of. Will open one soon.

[choldgraf]

• 2021-09-21 - Give a shot at implementing native OAuthenticator on our hubs + using that to authenticate with GitHub Teams. Will open another issue for long-term options using Auth0.

[sgibson91]

• Will open another issue for long-term options using Auth0.

Issue opened here #698

[sgibson91]

I have a first pass at this here: #706
This is currently deployed and working on Pangeo staging, but I'd like some early feedback to check that what I'm doing isn't too hacky.

[sgibson91]

Reopening as merging #706 caused some validation errors that I'm going to look into with fresh eyes. I reverted the change in #721. CI isn't green across the board, but the errors are mostly timeout related I think due to hitting various APIs with a lot of traffic in a short amount of time. I re-ran once and got a different set of hubs failing with timeout errors so I think they're all up to date with post-721 merge, just being flakey

[sgibson91]

Sigh, broken again. See #728 which is actually not a complete bugfix.

[sgibson91]

Closing as this was fixed by #728 (the resulting CI failures where related to the prometheus version #616)