authentication to other registries fails (cannot push or pull private images)

[jessfraz]

This applies to (GCR) #106 (ACR) #79 and (ECR) #127

Tracking all in one issue.

• GCR pulling private image from gcr fails with 403 forbidden #106
• ACR Authentication to Azure ACR fails using admin user #79
• ECR ECR login fails #127 thanks to @pgavlin Do not fail on basic auth in login. #145

[northtyphoon]

Update for Azure ACR, we have deployed a fix. With the latest img build, you can pull/push to ACR.

[pgavlin]

As of #145, logging in to and pulling from/pushing to ECR should work.

[jessfraz]

Thank you @pgavlin !!!!

[oxypwn]

registry.gitlab.com also fails it seems.

nm.. it works with gitlabs registry using basic auth both push and pull.

[topjor]

I think this issue is related related to all repositories that require token based authentication
I just tried to pull and push to my own private repository that requires token based (jwt Bearer) auth but i am presented with a error 401

I have intercepted traffic between me and the repository (using burp suite and proxy env var) and noticed that after successfully logging in and getting a token back, it does not use this token on subsequent requests to the registry resulting in a 401 again (it retries a few times but never is it setting the Authorization header with the token.

The versions I am using:
img: v0.5.2 (0b967de)
docker: 18.06.1-ce (to verify the correct working of the registry and authentication)
registry image: registry:2

[topjor]

In addition to my previous comment, I just tried to pull again without trying to login first and it successfully pulled my image.
it tried to pull, got 401 because of missing token, requested anonymously a token, and used the received token to to pull the image.
Looking at the traffic in burp suite again displayed a correct authorization header when not logged in.
(tried to login and pull again but still the same as written before)

[jessfraz]

can you retest with master or the latest release i updated the vendored auth code in 3e187c6

[topjor]

Unfortunately it still does not seem to work

I have the idea that the program does something different when credentials are being used then without.
When using img om a registry that requires a token, but has no credentials, it will request a token without supplying credentials (anonymously), receives a token and correctly uses it in subsequent requests to the resources,
but when credentials are supplied it for some reason does not want to use the token that it requests.

I Have created a gist with my captured data
https://gist.github.com/topjor/7ae7e8428ff0b863dbb13bafc3c7edfe
You can see without credentials it uses basic authentication
with credentials it uses oauth style authentication

[topjor]

I have found the issue after I had a bright moment this morning.
As seen my auth server only sends the token in the token field, but for compatibility with oauth the access_token field should also set with the token.
and because I observed after logging in that it uses oauth params and not basic, it expects/uses the access_token field, which not all auth servers set (including my own).
After updating my auth server to also set the access_token field in the auth return and testing a pull, it works and it uses the token correctly now.

reference: https://docs.docker.com/registry/spec/auth/token/#requesting-a-token (token response fields)

[grayhemp]

img: v0.5.4

I have a similar issue with pushing to Docker hub:

# img pull registry-1.docker.io/grayhemp/base:master
Pulling registry-1.docker.io/grayhemp/base:master...
Pulled: sha256:ab0ec9f43df2828191ee19b62daf56e11982be3edb1485908deebb622f85d031
Size: 2.106MiB

# img login -u grayhemp --password-stdin registry-1.docker.io < /run/pwd 
Login succeeded.

# img push registry-1.docker.io/grayhemp/base:master
Pushing registry-1.docker.io/grayhemp/base:master...
server message: insufficient_scope: authorization failed

Please let me know if I can do something to help to debug it, or (with a hope) if there might possibly be a workaround.

[grayhemp]

Here is the gist of the same commands but with -d
https://gist.github.com/grayhemp/7870615842a2eefecb1945aabb39d66c

[PatrickMcAfee-Eagle]

I'm seeing the same issue as @grayhemp with Artifactory, his gist shows effectively my exact issues.

I'm on 0.5.6

[lefebsy]

Hello,
Bug with Jfrog/Artifactory:6.5.9 and img:0.5.6.

• "Login succeeded"
• "pull alpine:latest" : ok
• "Push -d alpine:latest " show errors 401 Unauthorized and 404 (only for some checksums)
• Push final message is something like : "failed commit on ref "config-sha256:5f9d3....": unexpected status: 404 Not Found"

[topjor]

@lefebsy, you probably should also send a issue to your registry provider as this is a issue that the back-end of img uses a oauth scheme for authenticating when pulling/pushing and the authentication server that your provider uses does not support oauth login, reference: https://docs.docker.com/registry/spec/auth/oauth/

as for the back-end of img that is responsible of pulling/pushing, it does not support falling back to basic auth when oauth fails.

[lefebsy]

@topjor Thank you very much for the explanation. I will try to open a ticket at Artifactory with these details.

@lefebsy, you probably should also send a issue to your registry provider as this is a issue that the back-end of img uses a oauth scheme for authenticating....

[matthiasak]

for those wanting to use gcr, you may be able to use the gcloud auth token feature of the CLI

gcloud auth print-access-token | img login -u oauth2accesstoken --password-stdin https://gcr.io

[lefebsy]

Hello all,
Little update for people who are trying to use img with artifactory. With skopeo I've been successfully able to push rootless my image made with img to artifactory. In summary : img build -> img save tarball -> untar in a directory -> skopeo copy dest-creds usr:pwd oci:dir docker://artifactory-url/image:version....
If someone is able to check the skopeo copy function and merge it with img push function... it should be very usefull :-)

---

@topjor Thank you very much for the explanation. I will try to open a ticket at Artifactory with these details.

@lefebsy, you probably should also send a issue to your registry provider as this is a issue that the back-end of img uses a oauth scheme for authenticating....

Hello, for information, issued JFrog Artifactory support for this bug, but not very sure it will help, jfrog seems not to be very concerned about this problem with img...
https://www.jfrog.com/jira/browse/RTFACT-18825
https://www.jfrog.com/jira/browse/RTFACT-18583

[HoffmannThomas]

Little update for people who are trying to use img with artifactory. With skopeo I've been successfully able to push rootless my image made with img to artifactory. In summary : img build -> img save tarball -> untar in a directory -> skopeo copy dest-creds usr:pwd oci:dir docker://artifactory-url/image:version....

Hi,

You can also directly refer to the tarball. Thanks to you this is currently in my Jenkinsfile (inside a withCredentials and running on Kubernetes):

def DOCKER_HOST="..."
def CONTAINER_NAME="..."
def CONTAINER_TAG="${env.BUILD_ID}"
def IMAGE_FILE_NAME="${CONTAINER_NAME}_${CONTAINER_TAG}.tar"

IMAGE_FILE_NAME = IMAGE_FILE_NAME.replaceAll("/","_")

printf '${PASSWORD}' | img login --username '${USERNAME}' --password-stdin '${DOCKER_HOST}'
img build -t ${DOCKER_HOST}/${CONTAINER_NAME}:${CONTAINER_TAG} .
img save -o ${IMAGE_FILE_NAME} ${DOCKER_HOST}/${CONTAINER_NAME}:${CONTAINER_TAG}
skopeo copy docker-archive:${IMAGE_FILE_NAME} docker://${DOCKER_HOST}/${CONTAINER_NAME}:${CONTAINER_TAG}
skopeo copy docker://${DOCKER_HOST}/${CONTAINER_NAME}:${CONTAINER_TAG} docker://${DOCKER_HOST}/${CONTAINER_NAME}:latest

I'm using a modified img-docker just adding skopeo and company CAs before the root user is locked:

RUN apk update && apk add ca-certificates curl wget openssl skopeo --no-cache
RUN wget -O /usr/local/share/ca-certificates/companyroot.crt http://companyCA.corp/RootCA.crt
RUN wget -O /usr/local/share/ca-certificates/companysub.crt http://companyCA.corp/SubCA.crt
RUN update-ca-certificates
RUN curl https://company.corp

[tlawrie]

Hi everyone. Did img push end up getting a fix to be able to push to an artifactory docker registry? If it should have parity with docker push (which works) we should not need extra cli tools such as skopeo.

[alexellis]

Can the checkboxes on this issue be updated yet? It seems like 1-2 of the [ ] empty checkboxes should have a [x] tick in them now.

I noticed that Artifactory from jFrog fails with some operations. Here is a related issue - containerd/containerd#3533

[champloo11]

Pending the approval and merge of #276, img will use use any matching creds helpers for a given domain (tested against ECR and GCR). Meaning that machines with

• docker-credential-gcloud (installed with google sdk)
• docker-credential-ecr

Installed should be able to authenticating automatically using img login as would occur with a docker login.

[champloo11]

@jessfraz GCP and ACR authentication helpers provided credentials are now supported as of our last merge. Mind resolving this?