unexpected status: 401 Unauthorized #327

blabla1337 · 2020-07-21T18:14:53Z

Hi All,

So I saw this was posted before but without a proper solution or breakdown why this happens. #139 (comment)
I'm also having this issue from time to time and its really unclear what it triggers.

First I do the docker login and that goes well, after that I start the buildx for the 2 images but I still get the 401 error in the end as you can see. I'm using it on Mac Docker Desktop 2.3.2.0 Edge. if you need additional information please let me know.

docker login
Authenticating with existing credentials...
Login Succeeded

docker buildx build -f Docker/alpine-cloud/api/Dockerfile  --platform linux/amd64,linux/arm/v7 -t blabla1337/skf-api:dev --push .
[+] Building 1483.8s (38/38) FINISHED                                                                                                                                                        
 => [internal] load build definition from Dockerfile                                                                                                                                    0.1s
 => => transferring dockerfile: 32B                                                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                                       0.1s
 => => transferring context: 2B                                                                                                                                                         0.0s
 => [linux/amd64 internal] load metadata for docker.io/library/python:3.7                                                                                                               9.2s
 => [linux/arm/v7 internal] load metadata for docker.io/library/python:3.7                                                                                                             11.5s
 => [internal] load build context                                                                                                                                                      89.0s
 => => transferring context: 967.06MB                                                                                                                                                  81.4s
 => [linux/arm/v7 stage-1  1/16] FROM docker.io/library/python:3.7@sha256:322ff8fa9fb1cc6d0cc17e2e2b966ca17cda5d036930496cac1a03810525cbb1                                              0.0s
 => => resolve docker.io/library/python:3.7@sha256:322ff8fa9fb1cc6d0cc17e2e2b966ca17cda5d036930496cac1a03810525cbb1                                                                     0.0s
 => [linux/amd64 stage-1  1/16] FROM docker.io/library/python:3.7@sha256:322ff8fa9fb1cc6d0cc17e2e2b966ca17cda5d036930496cac1a03810525cbb1                                               0.0s
 => CACHED [linux/amd64 stage-1  2/16] RUN apt update &&    apt install -y vim     libblas-dev     libblas-dev     liblapack-dev     libatlas-base-dev     gfortran     cython          0.0s
 => CACHED [linux/amd64 stage-1  3/16] RUN groupadd --gid 1000 user_api && useradd --uid 1000 --gid user_api -m user_api &&  mkdir -p /home/user_api                                    0.0s
 => [linux/amd64 stage-1  4/16] ADD ./ /home/user_api/                                                                                                                                 27.8s
 => CACHED [linux/arm/v7 stage-1  2/16] RUN apt update &&    apt install -y vim     libblas-dev     libblas-dev     liblapack-dev     libatlas-base-dev     gfortran     cython         0.0s
 => CACHED [linux/arm/v7 stage-1  3/16] RUN groupadd --gid 1000 user_api && useradd --uid 1000 --gid user_api -m user_api &&  mkdir -p /home/user_api                                   0.0s
 => CACHED [linux/arm/v7 stage-1  4/16] ADD ./ /home/user_api/                                                                                                                          0.0s
 => CACHED [linux/arm/v7 stage-1  5/16] COPY ./Docker/alpine-cloud/api/entrypoint.sh  /home/user_api/entrypoint.sh                                                                      0.0s
 => CACHED [linux/arm/v7 stage-1  6/16] RUN touch /home/user_api/skf/db/db.sqlite                                                                                                       0.0s
 => CACHED [linux/arm/v7 stage-1  7/16] RUN chmod a+rw /home/user_api/skf/db /home/user_api/skf/db/*                                                                                    0.0s
 => CACHED [linux/arm/v7 stage-1  8/16] WORKDIR /home/user_api/                                                                                                                         0.0s
 => CACHED [linux/arm/v7 stage-1  9/16] RUN mkdir .kube                                                                                                                                 0.0s
 => CACHED [linux/arm/v7 stage-1 10/16] RUN mkdir .config                                                                                                                               0.0s
 => CACHED [linux/arm/v7 stage-1 11/16] RUN mkdir .config/pip                                                                                                                           0.0s
 => CACHED [linux/arm/v7 stage-1 12/16] RUN echo "[global]" >> .config/pip/pip.conf                                                                                                     0.0s
 => CACHED [linux/arm/v7 stage-1 13/16] RUN echo "extra-index-url=https://www.piwheels.org/simple" >> .config/pip/pip.conf                                                              0.0s
 => CACHED [linux/arm/v7 stage-1 14/16] RUN pip3 install --upgrade pip &&    pip3 install --user nltk &&    pip3 install --user cython &&    pip3 install --user numpy                  0.0s
 => CACHED [linux/arm/v7 stage-1 15/16] RUN if [ `ps auwxf | grep qemu-arm | wc -l` = 4 ]; then   pip3 install --user scipy==1.5.0rc2 ; else   pip3 install --user scipy ; fi           0.0s
 => [linux/arm/v7 stage-1 16/16] RUN pip3 install --user  -r requirements.txt                                                                                                         325.9s
 => [linux/amd64 stage-1  5/16] COPY ./Docker/alpine-cloud/api/entrypoint.sh  /home/user_api/entrypoint.sh                                                                              1.2s
 => [linux/amd64 stage-1  6/16] RUN touch /home/user_api/skf/db/db.sqlite                                                                                                               0.2s
 => [linux/amd64 stage-1  7/16] RUN chmod a+rw /home/user_api/skf/db /home/user_api/skf/db/*                                                                                            0.2s
 => [linux/amd64 stage-1  8/16] WORKDIR /home/user_api/                                                                                                                                 0.0s
 => [linux/amd64 stage-1  9/16] RUN mkdir .kube                                                                                                                                         0.2s
 => [linux/amd64 stage-1 10/16] RUN mkdir .config                                                                                                                                       0.2s
 => [linux/amd64 stage-1 11/16] RUN mkdir .config/pip                                                                                                                                   0.2s
 => [linux/amd64 stage-1 12/16] RUN echo "[global]" >> .config/pip/pip.conf                                                                                                             0.2s
 => [linux/amd64 stage-1 13/16] RUN echo "extra-index-url=https://www.piwheels.org/simple" >> .config/pip/pip.conf                                                                      0.2s
 => [linux/amd64 stage-1 14/16] RUN pip3 install --upgrade pip &&    pip3 install --user nltk &&    pip3 install --user cython &&    pip3 install --user numpy                         22.0s
 => [linux/amd64 stage-1 15/16] RUN if [ `ps auwxf | grep qemu-arm | wc -l` = 4 ]; then   pip3 install --user scipy==1.5.0rc2 ; else   pip3 install --user scipy ; fi                  16.0s
 => [linux/amd64 stage-1 16/16] RUN pip3 install --user  -r requirements.txt                                                                                                           65.1s
 => ERROR exporting to image                                                                                                                                                         1056.5s
 => => exporting layers                                                                                                                                                               433.2s
 => => exporting manifest sha256:651a8c2a92c21b35939f84379ac3ce6a59a102ad12c1ffd77b87ae5943c10440                                                                                       0.0s
 => => exporting config sha256:75bc89b31a6e8e5bcb67ef00ed3afbec60af799da0412d17c13852eb956d4e5f                                                                                         0.0s
 => => exporting manifest sha256:92aad5879eb60c8e4a776ecca27906068d0ae6264b69afefad152e8b566756ad                                                                                       0.0s
 => => exporting config sha256:24273c4da9572c159210060f87c5889f2f1508feeab36708a27d55fae989945e                                                                                         0.0s
 => => exporting manifest list sha256:cbbd8a5daf155203ad74613fafddeb7dd1187b7d35da33682c150247c6871c27                                                                                  0.0s
 => => pushing layers                                                                                                                                                                 621.9s
 => => pushing manifest for docker.io/blabla1337/skf-api:dev                                                                                                                            1.2s
------
 > exporting to image:
------
failed to solve: rpc error: code = Unknown desc = failed commit on ref "manifest-sha256:651a8c2a92c21b35939f84379ac3ce6a59a102ad12c1ffd77b87ae5943c10440": unexpected status: 401 Unauthorized

The text was updated successfully, but these errors were encountered:

blabla1337 · 2020-07-21T18:41:12Z

So I tried it a couple more times again and now its working:

docker buildx build -f Docker/alpine-cloud/api/Dockerfile  --platform linux/amd64,linux/arm/v7 -t blabla1337/skf-api:dev --push .
[+] Building 1148.9s (38/38) FINISHED                                                                                                                                                        
 => [internal] load build definition from Dockerfile                                                                                                                                    0.0s
 => => transferring dockerfile: 32B                                                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                                       0.1s
 => => transferring context: 2B                                                                                                                                                         0.0s
 => [linux/amd64 internal] load metadata for docker.io/library/python:3.7                                                                                                               3.9s
 => [linux/arm/v7 internal] load metadata for docker.io/library/python:3.7                                                                                                              6.2s
 => [internal] load build context                                                                                                                                                     132.2s
 => => transferring context: 967.06MB                                                                                                                                                 122.9s
 => [linux/amd64 stage-1  1/16] FROM docker.io/library/python:3.7@sha256:322ff8fa9fb1cc6d0cc17e2e2b966ca17cda5d036930496cac1a03810525cbb1                                               0.0s
 => => resolve docker.io/library/python:3.7@sha256:322ff8fa9fb1cc6d0cc17e2e2b966ca17cda5d036930496cac1a03810525cbb1                                                                     0.0s
 => [linux/arm/v7 stage-1  1/16] FROM docker.io/library/python:3.7@sha256:322ff8fa9fb1cc6d0cc17e2e2b966ca17cda5d036930496cac1a03810525cbb1                                              0.0s
 => => resolve docker.io/library/python:3.7@sha256:322ff8fa9fb1cc6d0cc17e2e2b966ca17cda5d036930496cac1a03810525cbb1                                                                     0.0s
 => CACHED [linux/amd64 stage-1  2/16] RUN apt update &&    apt install -y vim     libblas-dev     libblas-dev     liblapack-dev     libatlas-base-dev     gfortran     cython          0.0s
 => CACHED [linux/amd64 stage-1  3/16] RUN groupadd --gid 1000 user_api && useradd --uid 1000 --gid user_api -m user_api &&  mkdir -p /home/user_api                                    0.0s
 => CACHED [linux/amd64 stage-1  4/16] ADD ./ /home/user_api/                                                                                                                           0.0s
 => CACHED [linux/amd64 stage-1  5/16] COPY ./Docker/alpine-cloud/api/entrypoint.sh  /home/user_api/entrypoint.sh                                                                       0.0s
 => CACHED [linux/amd64 stage-1  6/16] RUN touch /home/user_api/skf/db/db.sqlite                                                                                                        0.0s
 => CACHED [linux/amd64 stage-1  7/16] RUN chmod a+rw /home/user_api/skf/db /home/user_api/skf/db/*                                                                                     0.0s
 => CACHED [linux/amd64 stage-1  8/16] WORKDIR /home/user_api/                                                                                                                          0.0s
 => CACHED [linux/amd64 stage-1  9/16] RUN mkdir .kube                                                                                                                                  0.0s
 => CACHED [linux/amd64 stage-1 10/16] RUN mkdir .config                                                                                                                                0.0s
 => CACHED [linux/amd64 stage-1 11/16] RUN mkdir .config/pip                                                                                                                            0.0s
 => CACHED [linux/amd64 stage-1 12/16] RUN echo "[global]" >> .config/pip/pip.conf                                                                                                      0.0s
 => CACHED [linux/arm/v7 stage-1  2/16] RUN apt update &&    apt install -y vim     libblas-dev     libblas-dev     liblapack-dev     libatlas-base-dev     gfortran     cython         0.0s
 => CACHED [linux/arm/v7 stage-1  3/16] RUN groupadd --gid 1000 user_api && useradd --uid 1000 --gid user_api -m user_api &&  mkdir -p /home/user_api                                   0.0s
 => CACHED [linux/arm/v7 stage-1  4/16] ADD ./ /home/user_api/                                                                                                                          0.0s
 => CACHED [linux/arm/v7 stage-1  5/16] COPY ./Docker/alpine-cloud/api/entrypoint.sh  /home/user_api/entrypoint.sh                                                                      0.0s
 => CACHED [linux/arm/v7 stage-1  6/16] RUN touch /home/user_api/skf/db/db.sqlite                                                                                                       0.0s
 => CACHED [linux/arm/v7 stage-1  7/16] RUN chmod a+rw /home/user_api/skf/db /home/user_api/skf/db/*                                                                                    0.0s
 => CACHED [linux/arm/v7 stage-1  8/16] WORKDIR /home/user_api/                                                                                                                         0.0s
 => CACHED [linux/arm/v7 stage-1  9/16] RUN mkdir .kube                                                                                                                                 0.0s
 => CACHED [linux/arm/v7 stage-1 10/16] RUN mkdir .config                                                                                                                               0.0s
 => CACHED [linux/arm/v7 stage-1 11/16] RUN mkdir .config/pip                                                                                                                           0.0s
 => CACHED [linux/arm/v7 stage-1 12/16] RUN echo "[global]" >> .config/pip/pip.conf                                                                                                     0.0s
 => CACHED [linux/arm/v7 stage-1 13/16] RUN echo "extra-index-url=https://www.piwheels.org/simple" >> .config/pip/pip.conf                                                              0.0s
 => CACHED [linux/arm/v7 stage-1 14/16] RUN pip3 install --upgrade pip &&    pip3 install --user nltk &&    pip3 install --user cython &&    pip3 install --user numpy                  0.0s
 => CACHED [linux/arm/v7 stage-1 15/16] RUN if [ `ps auwxf | grep qemu-arm | wc -l` = 4 ]; then   pip3 install --user scipy==1.5.0rc2 ; else   pip3 install --user scipy ; fi           0.0s
 => [linux/arm/v7 stage-1 16/16] RUN pip3 install --user  -r requirements.txt                                                                                                         371.1s
 => [linux/amd64 stage-1 13/16] RUN echo "extra-index-url=https://www.piwheels.org/simple" >> .config/pip/pip.conf                                                                      3.7s
 => [linux/amd64 stage-1 14/16] RUN pip3 install --upgrade pip &&    pip3 install --user nltk &&    pip3 install --user cython &&    pip3 install --user numpy                         23.3s
 => [linux/amd64 stage-1 15/16] RUN if [ `ps auwxf | grep qemu-arm | wc -l` = 4 ]; then   pip3 install --user scipy==1.5.0rc2 ; else   pip3 install --user scipy ; fi                  12.3s
 => [linux/amd64 stage-1 16/16] RUN pip3 install --user  -r requirements.txt                                                                                                           65.4s
 => exporting to image                                                                                                                                                                638.5s
 => => exporting layers                                                                                                                                                               302.5s
 => => exporting manifest sha256:dd1ff1af15e0d11d75c276aa648a56a7debc16bb6242489cb95a9172f4524e7a                                                                                       0.0s
 => => exporting config sha256:50adb73b1fe9b5e04bf4dd764188f12fabe1a6f693d793a2b06d906edd7dc016                                                                                         0.0s
 => => exporting manifest sha256:0395b10a6c7045398acb39c4e94e9dc5c3870bf96f001772d0e72b403a523fdb                                                                                       0.0s
 => => exporting config sha256:15bad4ce8ffeda2ded7e88b7009209fad96589adbde0111a01c35f324601c3f8                                                                                         0.0s
 => => exporting manifest list sha256:8061b5764cfeeeebf0fc5acb8f0a3bad1475a018a06ca8c0412fbffe56f77ca1                                                                                  0.0s
 => => pushing layers                                                                                                                                                                 332.2s
 => => pushing manifest for docker.io/blabla1337/skf-api:dev                                                                                                                            3.6s

No idea what the issue can be...

jc21 · 2020-07-22T00:07:04Z

I'm experiencing this in my CI builds, after docker login then a docker buildx and it fails with unauthorized only 1 hour and 1 minute later. How long is this authentication token meant to last? I was sure it would be longer than this.

Here is a full build history if anyone is interested:
https://ci.nginxproxymanager.com/blue/organizations/jenkins/nginx-proxy-manager/detail/master/36/pipeline

EDIT: I should note that retriggering this build works as it uses the cached docker images and pushes faster.

emwjacobson · 2020-07-29T22:15:13Z

I ran into this problem today creating large images. What I did was use the --cache-to flag when building it, then when I go to push it to dockerhub I log in and use --cache-from.
An example of the workflow would be something like this:

# Build images and cache them to a local cache/ folder, this part takes the longest
docker buildx build --cache-to=type=local,dest=cache,mode=max --platform linux/amd64,linux/arm64 --tag myimagename:latest .
# Login
docker login -u username -p password
# Build again, but use cache from local source
docker buildx build --push --cache-from=type=local,src=cache --platform linux/amd64,linux/arm64 --tag myimagename:latest .

As long as it uploads the image somewhat quickly, it hopefully shouldn't time you out.

Note that this does take a lot more disk space. I made a VM and gave it 64gb of disk space, and with a 7gb docker image crosscompiled to x86 and arm it ran out of disk space while caching all of the layers.

Hope that helps a bit.

EDIT: You could also play around with caching to/from a registry. I mentioned local caching because my internet would be a bottleneck, but it might reduce disk usage if that is a concern, but I havent personally tested it.

tonistiigi · 2020-08-25T21:59:38Z

Please test with moby/buildkit#1636 docker buildx create --driver-opt image=moby/buildkit:master

andrewufrank · 2020-09-03T15:34:30Z

congratulation! now I can push. each push is a success, when before it mostly failed (after running for 10 minutes).
thank you very much!
andrew

See docker/buildx#327 Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>

tyranron · 2020-11-24T15:43:31Z

I'm hitting this at the moment with Quay.io. GitHub Container Registry works OK.

I'm able to reproduce this locally via CLI. The interesting thing that a regular docker push works OK.

Another interesting thing that the first push to Quay.io (when images not yet existed) worked OK.

shepmaster · 2020-12-04T03:11:05Z

@tyranron same deal here: pushing to Quay.io from a GitHub Actions workflow on the subsequent run.

tonistiigi · 2020-12-04T03:59:57Z

I think you guys are hitting containerd/containerd#4622 (comment)

rblaine95 · 2021-01-03T00:39:12Z

Another interesting thing that the first push to Quay.io (when images not yet existed) worked OK.

Exact same issue here with me.
Initial build (fresh repo, no existing tags) succeeds, subsequent builds fail
I've tried with the with.driver-opts: image=moby/buildkit:master in the setup stage to no avail.
Maybe because moby/buildkit#1921 hasn't been approved/merged?

ricardbejarano · 2021-01-19T20:47:02Z

Same issue here!

because we hit docker/buildx#327 (#patch)

Suggested fix when hitting the 401 authorized error that we are when trying to push new images: docker/buildx#327 Signed-off-by: David Bond <davidsbond93@gmail.com>

spiffxp · 2021-05-28T05:10:48Z

I think the fix for containerd/containerd#4622 (comment) found its way into containerd v1.5.0 via containerd/containerd#4854

It looks like moby/buildkit pulled in containerd v1.5.0 via moby/buildkit#2102 23 days ago

I'm wondering when we might expect a release of buildx that include this fix. Do we need to wait for a release of buildkit, or would y'all be open to pulling in something sooner?

crazy-max · 2021-06-28T15:48:05Z

Should be fixed since BuildKit 0.8.2.

See docker/buildx#327

Suggested fix when hitting the 401 authorized error that we are when trying to push new images: docker/buildx#327 Signed-off-by: David Bond <davidsbond93@gmail.com>

This was referenced Jul 21, 2020

unexpected status: 401 Unauthorized #139

Closed

Long running push to the hub end-up failing with 401 after successfully pushing the layers #324

Closed

smira added a commit to smira/talos that referenced this issue Oct 2, 2020

chore: attempt to fix image pushing for GitHub

dfa4587

See docker/buildx#327 Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>

smira mentioned this issue Oct 2, 2020

chore: attempt to fix image pushing for GitHub siderolabs/talos#2596

Merged

smira added a commit to smira/talos that referenced this issue Oct 2, 2020

chore: attempt to fix image pushing for GitHub

8d10d9e

See docker/buildx#327 Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>

talos-bot pushed a commit to siderolabs/talos that referenced this issue Oct 2, 2020

chore: attempt to fix image pushing for GitHub

c3d7277

See docker/buildx#327 Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>

thaJeztah mentioned this issue Dec 23, 2020

go.mod: github.com/containerd/containerd v1.4.1-0.20201221152511-dc207b654da0 moby/buildkit#1921

Closed

rblaine95 mentioned this issue Jan 2, 2021

Buildx pushing to quay - 401 unauthorized rblaine95/dockerhub-mirror#1

Closed

jekkel added a commit to kiwigrid/k8s-sidecar that referenced this issue Feb 16, 2021

More circleCI fixes, no-auto version bump, contribution guideline

dbb5409

because we hit docker/buildx#327 (#patch)

spiffxp mentioned this issue Feb 23, 2021

E2E test images: httpd images failed to push to staging kubernetes/test-infra#20884

Closed

j-fuentes mentioned this issue Apr 13, 2021

Github workflows fail to tag images after push jetstack/jetstack-secure#224

Open

davidsbond mentioned this issue Apr 27, 2021

Use moby/buildkit:master when setting up buildx jetstack/jetstack-secure#227

Merged

mlavacca mentioned this issue May 17, 2021

Revert buildx kubermatic/kubermatic#7044

Merged

crazy-max closed this as completed Jun 28, 2021

wonderflow mentioned this issue Aug 18, 2021

fix(CI): use buildkit as docker build engine to avoid 401 unauthorized kubevela/kubevela#2120

Merged

mgoltzsche added a commit to mgoltzsche/podman-static that referenced this issue Sep 25, 2021

fix docker auth using newer buildkit version.

2cd5e18

See docker/buildx#327

mgoltzsche mentioned this issue Sep 26, 2021

Push to GitHub fails - unexpected status: 401 Unauthorized #780

Closed

lehrig mentioned this issue Sep 13, 2022

Pushing large image fails with Gateway errors #1315

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

unexpected status: 401 Unauthorized #327

unexpected status: 401 Unauthorized #327

blabla1337 commented Jul 21, 2020

blabla1337 commented Jul 21, 2020

jc21 commented Jul 22, 2020 •

edited

emwjacobson commented Jul 29, 2020 •

edited

tonistiigi commented Aug 25, 2020

andrewufrank commented Sep 3, 2020

tyranron commented Nov 24, 2020

shepmaster commented Dec 4, 2020

tonistiigi commented Dec 4, 2020

rblaine95 commented Jan 3, 2021 •

edited

ricardbejarano commented Jan 19, 2021

spiffxp commented May 28, 2021

crazy-max commented Jun 28, 2021 •

edited

unexpected status: 401 Unauthorized #327

unexpected status: 401 Unauthorized #327

Comments

blabla1337 commented Jul 21, 2020

blabla1337 commented Jul 21, 2020

jc21 commented Jul 22, 2020 • edited

emwjacobson commented Jul 29, 2020 • edited

tonistiigi commented Aug 25, 2020

andrewufrank commented Sep 3, 2020

tyranron commented Nov 24, 2020

shepmaster commented Dec 4, 2020

tonistiigi commented Dec 4, 2020

rblaine95 commented Jan 3, 2021 • edited

ricardbejarano commented Jan 19, 2021

spiffxp commented May 28, 2021

crazy-max commented Jun 28, 2021 • edited

jc21 commented Jul 22, 2020 •

edited

emwjacobson commented Jul 29, 2020 •

edited

rblaine95 commented Jan 3, 2021 •

edited

crazy-max commented Jun 28, 2021 •

edited