OAuth2 authorization

[nsmith-]

Dask provides a TLS client-scheduler secure connection mechanism that can serve double duty as a client authentication mechanism. However, x509 certificate+key pairs for both server and client can be cumbersome to set up, especially if many different users need to authenticate. A popular authentication flow these days is OAuth2 allowing to factorize authentication, authorization, and access to resources. It would be nice if a dask Security object implementing OAuth2 for client authorization to the scheduler were available. Then any subsequent communication would put the auth token in the headers. Note this does not help with connection security, which probably would best be served by 1-way TLS as used widely in https.

An alternative to putting OAuth support in distributed natively, after the introduction of websocket protocols thanks to @marcosmoyano in #4396, is to put a reverse proxy in front of the scheduler to authorize access, as there are reverse proxies that can handle authorization, e.g. nginx.

@oshadura @bbockelm @jacobtomlinson

[jrbourbeau]

Thanks @nsmith-! cc'ing @jcrist @ian-r-rose too

[jacobtomlinson]

I think Oauth2 support would be nice, but my preference would be to have an Oauth2 proxy sitting in front of Dask rather than building it directly into Dask.

I'm curious about the comment you made around multiple users connecting to one Dask scheduler.

Today Dask doesn't support this use case very well. Graphs are executed on a first-come-first-serve basis and if many users are trying to submit independent work to the scheduler they will lively have a bad time. Is this something you are doing today?

[marcosmoyano]

I think Oauth2 support would be nice, but my preference would be to have an Oauth2 proxy sitting in front of Dask rather than building it directly into Dask.

+1
I've been thinking about writing a plugable auth system sitting in front of Dask. And have the Security class accept some sort of auth policy object as well. On the other hand, this feels like something that should be handled outside of Dask. I keep going back and fwd with this idea 🤔. @jacobtomlinson I'm curious to hear your thoughts on this.

[mrocklin]

Also cc'ing @jcrist

[jcrist]

I think Oauth2 support would be nice, but my preference would be to have an Oauth2 proxy sitting in front of Dask rather than building it directly into Dask.

How would this work? Distributed's comms aren't http based, so any proxy and auth mechanism here would need to be custom. There's nothing preventing us from using OAuth2 to validate a token sent to the scheduler when a new client connection is created, but I'm also not sure I see the use case. It'd be a fair amount of custom code.

[nsmith-]

Today Dask doesn't support this use case very well. Graphs are executed on a first-come-first-serve basis and if many users are trying to submit independent work to the scheduler they will lively have a bad time. Is this something you are doing today?

I was a bit unclear: we spin up a separate cluster per user, but part of that operation is generating an x509 key system per user. If we want to allow users to connect to the cluster from their personal computer as well as from JHub, then they would have to download a client keypair and the CA. It would simplify operations a bit if the user just had to generate an access token (easy copy-paste) instead of downloading a pem file. Further, in principle, a library running on a personal machine could generate a new token through an interactive prompt.

[jcrist]

If we want to allow users to connect to the cluster from their personal computer as well as from JHub

FWIW, this is a use case that dask-gateway handles fairly well (it's one of the reasons I wrote it in the first place). Certs are automatically minted and distributed, and users can connect from either within the cluster (usually on jhub) or externally from their own laptop.

[oshadura]

@nsmith- about dask-gateway it looks something that could definitely fir us, I was going to try dask-gateway as one of todo's for coffea-casa (I think I mentioned it even in presentation at Dask summit :) ).

[bbockelm]

To be clear, OAuth2 is simply a protocol for distributing the tokens. I think what you're really proposing here @nsmith- is having a token-based authentication mechanism for authentication within the cluster? The distribution of the token to the client (via OAuth2) almost certainly should be done outside of Dask.

Two thoughts:

1. There's not a great way to add a token to the TLS transport itself. One possibility is to have the token added to the Dask layer and only send/accept bearer token-based authentication when the underlying transport is TLS.

• You can send the raw token over the TLS connection -- but then this potentially looks like a new protocol itself, needs framing, etc. I suspect having token auth in the Dask layer is better
• The downside of doing this in the Dask layer is that, to first order, there's no concept of identity or authorization inside Dask today. This would be a relatively large pickup.

2. One reason why tokens are often preferred versus mutual X509 is the distribution of the X.509 credential is a royal PITA (if nothing else, you can't copy/paste it easily!). Instead of changing the authentication layer, one can consider different mechanisms for easing the distribution of the X.509 credentials.

• Example: using ECDSA signatures, a certificate + CA certificate + private key can be shrunk down to about ~1.5K of base64-encoded text. If you encode it to one line as a JWT, this is something you can copy/paste with less hassle.
• Idea: perhaps OAuth2 itself can be used to distribute the certificate stack?

[jcrist]

Simple token based authorization may be in scope, and is something we could handle within the current distributed protocol (see #1686 for discussion of this). When configured, the client would send a token as part of the initial handshake (an existing part of the distributed protocol). A simple authorization implementation might then compare the token to a valid token (or tokens) configured on the server (this could be expanded if needed to support e.g. pinging some other server to get a yea/nay on whether the token is acceptable). I don't see what this gets you over using mTLS (except for easy copyability), but it's doable.

I'm less enthused about adding an authentication mechanism to distributed itself. Distributed has no concept of users (as @jacobtomlinson mentioned above), and adding one would be a larger effort.

---

That all said, dask-gateway handles this fully right now for you (including authentication), with no need to distribute certificates. The user authenticates with the dask-gateway api (as a third party) using a configurable authentication mechanism. When connecting/creating a cluster, the api server acts as a middleman, minting and distributing the mTLS certs to the user/cluster, so the user never needs to manage these themselves.

[bbockelm]

Simple token based authorization may be in scope, and is something we could handle within the current distributed protocol (see #1686 for discussion of this). When configured, the client would send a token as part of the initial handshake (an existing part of the distributed protocol).

Yes, agreed, that ticket is a reasonable way to go about it.

I don't see what this gets you over using mTLS (except for easy copyability), but it's doable.

I do think easier distribution is indeed something worth highlighting. Another thing that might be possible is having different authorized roles (e.g., client-only token, worker node-only token) without having to break into the concept of authentication and mapping client certificates to specific roles.

That all said, dask-gateway handles this fully right now for you (including authentication), with no need to distribute certificates.

Right. I think @ncsmith- might have other reasons to not use dask-gateway (if nothing else, the particular batch system used has support in dask-jobqueue but not dask-gateway!) but the architecture of having a separate agent distribute trust is reasonable.