Registry authentication with .docker/config

[rbrthogan]

As far as I am aware it is currently not possible to authenticate with docker registry that use OAuth authentication, with only basic authentication being supported. Is this correct?

Is it possible to read authorisation tokens from .docker/config file ?

[vsoch]

OAuth is a bit of a different flow, so I'm not sure how it would fit in nicely. Can you point me to more info on the docker config? That definitely seems reasonable to try!

[rbrthogan]

.docker/config.json simply stores your credentials ( https://docs.docker.com/engine/reference/commandline/login/) Some registries us OAuth so they require you send an authorisation header with the request. https://docs.docker.com/registry/spec/auth/token/ As far as I'm aware docker pull handles all that as long as your registry is configured correctly and it just looks in .docker/config.json for credentials. For a user point of view docker pull works just the same. I'm not sure exactly how you are handling this right now with the environment variables, so it might be more complicated than I'm aware. FYI: I'm using the stups framework https://github.com/zalando-stups, so pierone is the registry I'm talking about and has functionality for refreshing my tokens. I don't think that this issue is specific to Pierone however, should apply to any registry with OAuth flow, but just letting you know for some context. Sorry I can't be more helpful, hopefully someone more knowledgeable will chime in.

…

On May 25, 2017 6:15 PM, "Vanessa Sochat" ***@***.***> wrote: OAuth is a bit of a different flow, so I'm not sure how it would fit in nicely. Can you point me to more info on the docker config? That definitely seems reasonable to try! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#693 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIyaz__GQTEDWRTJ_RfbZXNbIx3H7pFkks5r9bcTgaJpZM4Nmf47> .

[vsoch]

hey @rbrthogan ! We start with the username and password (likely from the environment as SINGULARITY_DOCKER_USERNAME and SINGULARITY_DOCKER_PASSWORD and then put them together with a : and encode with base64 to turn into the initial token, eg:

def basic_auth_header(username, password):
    '''basic_auth_header will return a base64 encoded header object to
    generate a token
    :param username: the username
    :param password: the password
    '''
    s = "%s:%s" % (username, password)
    if sys.version_info[0] >= 3:
        s = bytes(s, 'utf-8')
        credentials = base64.b64encode(s).decode('utf-8')
    else:
        credentials = base64.b64encode(s)
    auth = {"Authorization": "Basic %s" % credentials}
    return auth

This auth object, which is part of the header, is then handed to our client that makes all the requests for docker manifests (eg, metadata, layers, etc.) Given that no authentication is needed, it just works! If authentication is needed then the server will respond with a 401, and then we look at the Www-Authenticate header to get the specific scope, registry, and repo that we need to ask permissions for. Then we update that in the header, and issue the request again, eg:

self.token_url = "%s?service=%s&expires_in=9000&scope=%s" %(realm,service,scope)

and if all goes well, this responds with a much (longer) token that will do the authentication, and we update the client headers with it:

token = json.loads(response)["token"]
token = {"Authorization": "Bearer %s" %(token) }

It looks like the ~/.docker/config.json file is specifically when you log into the docker daemon with "docker login" - funny enough I've been using Docker for years (meaning having the daemon locally) and never actually logged in. I'm not sure I see the dire need to add a "sniff" for this config, because we have taken an approach that doesn't have any dependency or integration with their daemon (just using the back end APIs in the same way) @gmkurtzer do you have thoughts on this? @rbrthogan what would be the pros/benefits of supporting this specific config over just using the environment variables?

[rbrthogan]

Ah ok, cool. In that case, maybe we can just extend the current approach to allow the user to pass the fully formed token explicitly, i.e., add a '--token' field to env variable that overrides the token creation you are doing and just gets passed directly. Does that make sense? I can fork and test it out when I'm back in work in the morning and make a PR if it works.

…

On May 25, 2017 8:06 PM, "Vanessa Sochat" ***@***.***> wrote: hey @rbrthogan <https://github.com/rbrthogan> ! We start with the username and password (likely from the environment as SINGULARITY_DOCKER_USERNAME and SINGULARITY_DOCKER_PASSWORD and then put them together with a : and encode with base64 to turn into the initial token, eg: def basic_auth_header(username, password): '''basic_auth_header will return a base64 encoded header object to generate a token :param username: the username :param password: the password ''' s = "%s:%s" % (username, password) if sys.version_info[0] >= 3: s = bytes(s, 'utf-8') credentials = base64.b64encode(s).decode('utf-8') else: credentials = base64.b64encode(s) auth = {"Authorization": "Basic %s" % credentials} return auth This auth object, which is part of the header, is then handed to our client that makes all the requests for docker manifests (eg, metadata, layers, etc.) Given that no authentication is needed, it just works! If authentication is needed then the server will respond with a 401, and then we look at the Www-Authenticate header to get the specific scope, registry, and repo that we need to ask permissions for. Then we update that in the header, and issue the request again, eg: self.token_url = "%s?service=%s&expires_in=9000&scope=%s" %(realm,service,scope) and if all goes well, this responds with a much (longer) token that will do the authentication, and we update the client headers with it: token = json.loads(response)["token"] token = {"Authorization": "Bearer %s" %(token) } It looks like the ~/.docker/config.json file is specifically when you log into the docker daemon with "docker login" - funny enough I've been using Docker for years (meaning having the daemon locally) and never actually logged in. I'm not sure I see the dire need to add a "sniff" for this config, because we have taken an approach that doesn't have any dependency or integration with their daemon (just using the back end APIs in the same way) @gmkurtzer <https://github.com/gmkurtzer> do you have thoughts on this? @rbrthogan <https://github.com/rbrthogan> what would be the pros/benefits of supporting this specific config over just using the environment variables? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#693 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIyaz-aIssLv3ewqBLCLGGS9OvJL_WiSks5r9dFDgaJpZM4Nmf47> .

[vsoch]

hey @rbrthogan - I just tested this out (comparing the value in the .docker/config.json with the one we are generating) and they are totally the same! What I can do is have this be a fallback for a user credential. Something like:

• upon init, if username and password are provided via environment, use that
• if no username and password, check for docker/config.json, and use that instead

This is a reasonable addition, and I think worth adding if it benefits a ton of users (that prefer using docker login instead of exporting things to the environment). Let me give it some tests and I'll put in a PR for us to look over.

[vsoch]

I thought more about this - I don't like the idea of automatically using any docker config found in the user's home - I think since Singularity is not just a container to carry around Docker, having this essentially hard coded in the software's functionality isn't a direction I like. The ability to specify a --token would also coincide with hard coding something docker in. Is it really a hassle to export the environment variables? That gives support for Docker, it's something you do intentionally, but it doesn't add extra variables/options to the core user interface/interaction of Singularity. Let me know your thoughts - and definitely if you have idea for implementation give it a go and we can discuss!

[vsoch]

okay, I just figured out a way to do it as a "last call resort" and will link the PR soon.

[rbrthogan]

thanks for the quick work @vsoch !

I'm not sure I understand your concern about hard coding the .docker/config.json stuff though. For sure, singularity is more than just a container on top of Docker, but the changes we are proposing are specifically related the docker api client which is already in the project.

I have some comments about the solution which I will repost on the PR to continue discussion there:

It seems to me that a much more elegant solution would be to just rely on the existence of user credentials in .docker/config.json. Right now, for Basic authentication, you have a 2 stage process, with the first simply reproducing data that would be contained in .docker/config.json if it exists followed by using this data to get access, and separately have a fall back to read this file anyway. I think that if .docker/config.json exists it should be first thing used, otherwise perhaps you could look for env variables, construct the first token, etc. This way, the api will execute entirely the same code, independent of registry Auth, as long as the correct credentials are present.

I personally don't love the env variable solution because it requires me to stored my password as plain text which is irksome. Perhaps you don't want to rely on the existence of .docker/config.json because it is generated by the docker daemon. I do think, however, that most users who often use private docker registries (either with Basic Authentication or OAuth) likely use the docker daemon to handle credentials for them. In any case, the env variable solution can always be there as fall back if .docker/config.json doesn't exist.

I hope this isn't just my personal bias speaking, I would be interested to hear community feedback.

[vsoch]

I disagree about use of the docker config - I've been using Docker since 2015 and this is the first time I used docker login to generate one. I advocate for the environment variable approach because you have to be specific about setting it, and importantly, it doesn't require the Docker daemon. A user on a shared resource would have to arbitrarily create one or copy from a host, and that's a lot more work than just exporting the environment variable.

[Quanteek]

Hey,

I am having quite same issues here and I do also advocate for a user environnement approach or a --token approach.

Our use case is that:

• we have a registry hosted with gitlab
• we have a web app with a user profile and some programs configurations
• we want to host some mechanists models for agronomy or neuroscience on the registry
• trigger these models on an HPC system for simulation or quantitative analysis

With the current approach it will ask me to ask and send plain user and password which I don't really like and having a token in environment variable would be more reliable for me.

We don't need to have a .docker/config.json on the HPC since we tested it (docker) and do not want users inside the docker group for docker login for example because of security reasons or because dockeris too messy with our /tmp configuration.

If I can be of any help (dev. or test) please tell me, it would be a pleasure :)

[vsoch]

Are you trying to get images from a docker registry hosted via gitlab? it looks like there is some extra headers and authentication potentially required: http://docs.gitlab.com/ce/user/project/container_registry.html if you could give me more details, and possibly some and points with different levels of permission to test development, I can see if I can resolve the issue with our current infrastructure or add some customization for Gitlab registries.

[Quanteek]

@vsoch yes we are trying to get images from a docker registry hosted via gitlab.centralesupelec.fr since gitlab has built-in registry capabilities.

I can create you and account and give you credentials on the platform via PM if you are interested to see the behavior of singularity with this platform ?

PS : sorry for the long delay :/

[vsoch]

no worries! Yes I think support for Gitlab would be great - send me the credentials and I should have some time later in the week to take a look.

[Quanteek]

Done !

You should have received a mail from gitlab.centralesupelec.fr :)

Enjoy :p and thanks for your work.