[discuss] client auth protocol for push/pull

[cpuguy83]

In #34319, we are looking to add support for mirrors for private registries, which has brought up the topic of how to deal with auth.

Personally speaking, I don't want to see:

1. Auth credentials stored on the daemon side
2. Universal auth credentials for all clients
3. An even greater mess of the existing auth scenarios (Seriously, read through the code and tell me how long it takes you to figure out all that is happening).

[cpuguy83]

ping @stevvooe @dmcgowan @saschagrunert

[cpuguy83]

Just a thought on this topic...

Perahaps docker pull <registry>/<image> would trigger a minor back and forth to negotiate the remote to actually pull from.... so the daemon could respond with "oh I have these mirrors configured" and the client could choose the remote(s) to try by sending the auth creds for the remote to use.

Additionally, the client could send the remote and/or the list of remotes it wants to use with the creds for them along with the initial request.... however daemon configured mirrors gets a little weird here.

It may be helpful to have an endpoint on the daemon that can assist the client in resolving a remote?

[saschagrunert]

@cpuguy83 Thank you for taking care in opening the discussion here. If I take your input and put it into a picture I could imagine a solution like this:

So what are the main reasons in not doing this and what else is missing?

[dekkagaijin]

Having an auth negotiation between client and server would also help with the issue that docker build unnecessarily sends the entire universe of creds to the daemon

[thaJeztah]

@caervs PTAL

[tonistiigi]

Having an auth negotiation between client and server would also help with the issue that docker build unnecessarily sends the entire universe of creds to the daemon

This is not true anymore with BuildKit. BuildKit uses session endpoint to call back to the client for specific credentials per host, basically like the diagram above shows but without sending requests again.

This can be extended to pull/push endpoint, mirror credentials, and short-lived tokens instead. Extra security benefit would be that the password never leaves the client machine (master...simonferquel:auth-streaming actually attempted to add it long time ago).

[lajoie]

Has this work stalled (and thus stalled #34319)?

[cpuguy83]

This is really looking for contributors.

[tonistiigi]

@cpuguy83 fyi, latest development moby/buildkit#1660

[dmcgowan]

I agree the right direction is getting tokens directly on the client. Once this is in place, it can allow token servers to generate longer lived, tightly scoped tokens which could also be used by orchestrators.

[corhere]

For the push/pull case, I really like the solution outlined in #38591 (comment): the client optimistically requests a pull or push and is presented with the auth challenge to complete the operation, mirroring the authentication workflow between daemon and registry. The auth challenge could even be a direct copy of the registry's response, I think. It includes all the necessary information: the canonical name of the registry to authenticate for, the token server to authenticate against, and the auth challenge itself. The registry only needs to be reachable from the daemon and the token server from the client, which would enable the use-case of authenticating with the token server using client certificates, as discussed in #45409 (comment).

sequenceDiagram
    Client->>+Daemon: pull my.reg/image:tag
    Daemon->>+my.reg: pull my.reg/image:tag
    my.reg-->>-Daemon: 401 Unauthorized<br/>Www-authenticate: Bearer realm="my.auth/token",service="my.reg"
    Daemon-->-Client: 401 Unauthorized<br/>Www-authenticate: Bearer realm="my.auth/token",service="my.reg"
    Client->>+my.auth: POST /token
    my.auth-->>-Client: {"token": "..."}
    Client->>+Daemon: pull my.reg/image:tag<br/>Authorization: Bearer ...
    Daemon->>+my.reg: pull my.reg/image:tag<br/>Authorization: Bearer ...
    my.reg-->>-Daemon: 200 OK
    Daemon-->>-Client: 200 OK

[dmcgowan]

containerd also added support for a similar flow to this in containerd 1.7. Daemon side pulling with a client callback for auth. It also supports sending www-authenticate header back to client and receiving back the authorization header to send back to registry. https://github.com/containerd/containerd/blob/main/api/types/transfer/registry.proto#L44