buildx failed with: ERROR: failed to solve: process ... did not complete successfully: exit code: 100

[RunningLeon]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

• The documentation does not mention anything about my problem
• There are no open or closed issues that are related to my problem

Description

Failed to build docker image with

NVIDIA/nvidia-docker#4 4.944 Fetched 27.9 MB in 4s (7434 kB/s)
NVIDIA/nvidia-docker#4 4.944 Reading package lists...
NVIDIA/nvidia-docker#4 6.737 E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/Packages.gz  File has unexpected size (1127072 != 1126689). Mirror sync in progress? [IP: 152.195.19.142 443]
NVIDIA/nvidia-docker#4 6.737    Hashes of expected file:
NVIDIA/nvidia-docker#4 6.737     - Filesize:1126689 [weak]
NVIDIA/nvidia-docker#4 6.737     - SHA256:7cee2584ca6d97b2f07018ba4f9c3c473fb4e299ed170968a1f8c99c090cc59f
NVIDIA/nvidia-docker#4 6.737     - SHA1:4ee24fac5518a3fcc3702590a0dab32c95484c54 [weak]
NVIDIA/nvidia-docker#4 6.737     - MD5Sum:593faff511765d11055c9919bf2e3bf8 [weak]
NVIDIA/nvidia-docker#4 6.737    Release file created at: Thu, 17 Aug 2023 19:03:05 +0000
NVIDIA/nvidia-docker#4 6.737 E: Some index files failed to download. They have been ignored, or old ones used instead.
NVIDIA/nvidia-docker#4 ERROR: process "/bin/sh -c apt-get update && apt-get install -y --no-install-recommends     rapidjson-dev libgoogle-glog-dev gdb      && rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 100
------
 > [2/5] RUN apt-get update && apt-get install -y --no-install-recommends     rapidjson-dev libgoogle-glog-dev gdb      && rm -rf /var/lib/apt/lists/*:
3.275 Get:20 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [28.6 kB]

6.737 E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/Packages.gz  File has unexpected size (1127072 != 1126689). Mirror sync in progress? [IP: 152.195.19.142 443]
6.737    Hashes of expected file:
6.737     - Filesize:1126689 [weak]
6.737     - SHA256:7cee2584ca6d97b2f07018ba4f9c3c473fb4e299ed170968a1f8c99c090cc59f
6.737     - SHA1:4ee24fac5518a3fcc[370](https://github.com/RunningLeon/lmdeploy/actions/runs/5898044107/job/15998515422#step:11:374)2590a0dab32c95484c54 [weak]
6.737     - MD5Sum:593faff511765d11055c9919bf2e3bf8 [weak]
6.737    Release file created at: Thu, 17 Aug 2023 19:03:05 +0000
6.737 E: Some index files failed to download. They have been ignored, or old ones used instead.
------
./docker/Dockerfile:3
--------------------
   2 |     
   3 | >>> RUN apt-get update && apt-get install -y --no-install-recommends \
   4 | >>>     rapidjson-dev libgoogle-glog-dev gdb  \
   5 | >>>     && rm -rf /var/lib/apt/lists/*
   6 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c apt-get update && apt-get install -y --no-install-recommends     rapidjson-dev libgoogle-glog-dev gdb      && rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 100
Error: buildx failed with: ERROR: failed to solve: process "/bin/sh -c apt-get update && apt-get install -y --no-install-recommends     rapidjson-dev libgoogle-glog-dev gdb      && rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 100

Expected behaviour

Build docker image

Actual behaviour

Failed with building docker image

Repository URL

No response

Workflow run URL

https://github.com/RunningLeon/lmdeploy/actions/runs/5898044107/job/15998515422

YAML workflow

https://github.com/RunningLeon/lmdeploy/actions/runs/5898044107/workflow

Workflow logs

logs_38.zip

BuildKit logs

No response

Additional info

No response

[crazy-max]

Failed to build docker image with

Does not look to be a bug with buildx but apt repo https://developer.download.nvidia.com

[RunningLeon]

hi, thanks for your reply. But we can build successfully build the image on the local Ubuntu machine

[crazy-max]

But we can build successfully build the image on the local Ubuntu machine

Maybe because packages are cached and therefore you can't repro. Try with apt-get clean && apt-get update to make sure.

File has unexpected size (1127072 != 1126689). Mirror sync in progress? [IP: 152.195.19.142 443]

This sounds like a temporary repository server problem that should go away on its own.

[RunningLeon]

hi, changed and still not working

NVIDIA/nvidia-docker#4 5.279 E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/Packages.gz  File has unexpected size (1127072 != 1126689). Mirror sync in progress? [IP: 152.195.19.142 443]
NVIDIA/nvidia-docker#4 5.279    Hashes of expected file:
NVIDIA/nvidia-docker#4 5.279     - Filesize:1126689 [weak]
NVIDIA/nvidia-docker#4 5.279     - SHA256:7cee2584ca6d97b2f07018ba4f9c3c473fb4e299ed170968a1f8c99c090cc59f
NVIDIA/nvidia-docker#4 5.279     - SHA1:4ee24fac5518a3fcc3702590a0dab32c95484c54 [weak]
NVIDIA/nvidia-docker#4 5.279     - MD5Sum:593faff511765d11055c9919bf2e3bf8 [weak]
NVIDIA/nvidia-docker#4 5.279    Release file created at: Thu, 17 Aug 2023 19:03:05 +0000
NVIDIA/nvidia-docker#4 5.279 E: Some index files failed to download. They have been ignored, or old ones used instead.
NVIDIA/nvidia-docker#4 ERROR: process "/bin/sh -c apt-get clean && apt-get update && apt-get install -y --no-install-recommends     rapidjson-dev libgoogle-glog-dev gdb      && rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 100
------
 > [2/5] RUN apt-get clean && apt-get update && apt-get install -y --no-install-recommends     rapidjson-dev libgoogle-glog-dev gdb      && rm -rf /var/lib/apt/lists/*:
2.961 Get:20 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1092 kB]

5.279 E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/Packages.gz  File has unexpected size (1127072 != 1126689). Mirror sync in progress? [IP: 152.195.19.142 443]
5.279    Hashes of expected file:
5.279     - Filesize:1126689 [weak]
5.279     - SHA256:7cee2584ca6d97b2f07018ba4f9c3c473fb4e299ed170968a1f8c99c090cc59f
5.279     - SHA1:4ee24fac5518a3fcc[370](https://github.com/RunningLeon/lmdeploy/actions/runs/5901187377/job/16006876421#step:11:374)2590a0dab32c95484c54 [weak]
5.279     - MD5Sum:593faff511765d11055c9919bf2e3bf8 [weak]
5.279    Release file created at: Thu, 17 Aug 2023 19:03:05 +0000
5.279 E: Some index files failed to download. They have been ignored, or old ones used instead.
------
./docker/Dockerfile:3
--------------------
   2 |     
   3 | >>> RUN apt-get clean && apt-get update && apt-get install -y --no-install-recommends \
   4 | >>>     rapidjson-dev libgoogle-glog-dev gdb  \
   5 | >>>     && rm -rf /var/lib/apt/lists/*
   6 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c apt-get clean && apt-get update && apt-get install -y --no-install-recommends     rapidjson-dev libgoogle-glog-dev gdb      && rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 100
Error: buildx failed with: ERROR: failed to solve: process "/bin/sh -c apt-get clean && apt-get update && apt-get install -y --no-install-recommends     rapidjson-dev libgoogle-glog-dev gdb      && rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 100

[guzman109]

But we can build successfully build the image on the local Ubuntu machine

Maybe because packages are cached and therefore you can't repro. Try with apt-get clean && apt-get update to make sure.

File has unexpected size (1127072 != 1126689). Mirror sync in progress? [IP: 152.195.19.142 443]

This sounds like a temporary repository server problem that should go away on its own.

They meant to do this on your local computer to reproduce.

[crazy-max]

You probably can't repro locally because this is not using the same CDN as the one in GitHub infrastructure. After investigation, I found this page https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/ and looking at the "File has unexpected size" section, it says to report this issue to NVIDIA:

{{Packages.gz File has unexpected size (631054 != 481481). Mirror sync in progress? [IP: XXX.XXX.XXX.XXX 443]
Hashes of expected file:

* Filesize:481481 [weak]
* SHA256:8556d67c6d380c957f05057f448d994584a135d7ed75e5ae6bb25c3fc1070b0b
* SHA1:c5ea9556407a3b5daec4aac530cd038e9b490441 [weak]
* MD5Sum:a5513131dbd2d4e50f185422ebb43ac9 [weak]
* Release file created at: Mon, 25 Apr 2022 23:27:19 +0000
* E: Some index files failed to download. They have been ignored, or old ones used instead.}}

Solution: Report CDN issue to NVIDIA.

Similar to:

• Cuda repo update failing since yesterday NVIDIA/nvidia-container-toolkit#100
• File has unexpected size NVIDIA/nvidia-docker#1328
• Unable to build ubuntu20.04 docker NVIDIA/nvidia-docker#1521
• Public GPG key error NVIDIA/nvidia-container-toolkit#258

Suggest to open an issue on https://github.com/NVIDIA/nvidia-docker

[RunningLeon]

@crazy-max hi, many thanks for your efforts. Found same issue NVIDIA/cuda-repo-management#10 and solved by adding rm /etc/apt/sources.list.d/cuda*.list in Dockefile