Error running build.sh (Ubuntu 22.04) (Unable to locate package zstd)

[tensiondriven]

1. Git clone the repo
2. sudo ./build.sh
3. Wait
4. Error:

(willow-inference-server) (willow) j@textgen-working:~/willow-inference-server$ sudo ./build.sh 
[+] Building 1.3s (7/12)                                                                                                 
 => [internal] load build definition from Dockerfile                                                                0.0s
 => => transferring dockerfile: 873B                                                                                0.0s
 => [internal] load .dockerignore                                                                                   0.0s
 => => transferring context: 125B                                                                                   0.0s
 => [internal] load metadata for nvcr.io/nvidia/tensorrt:22.12-py3                                                  1.0s
 => [stage-0 1/8] FROM nvcr.io/nvidia/tensorrt:22.12-py3@sha256:d07df947208a069ce111b86d5b7044aa802bb26f67eebaa62b  0.0s
 => [internal] load build context                                                                                   0.0s
 => => transferring context: 140.87kB                                                                               0.0s
 => CACHED [stage-0 2/8] WORKDIR /app                                                                               0.0s
 => ERROR [stage-0 3/8] RUN apt-get update && apt-get install -y zstd  && rm -rf /var/lib/apt/lists/*               0.2s
------
 > [stage-0 3/8] RUN apt-get update && apt-get install -y zstd  && rm -rf /var/lib/apt/lists/*:
#0 0.168 Err:1 http://archive.ubuntu.com/ubuntu focal InRelease
#0 0.168   Could not connect to archive.ubuntu.com:80 (91.189.91.39). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (91.189.91.38). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (185.125.190.39). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (185.125.190.36). - connect (111: Connection refused)
#0 0.168 Err:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease
#0 0.168   Unable to connect to archive.ubuntu.com:http:
#0 0.168 Err:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease
#0 0.168   Unable to connect to archive.ubuntu.com:http:
#0 0.179 Err:4 http://security.ubuntu.com/ubuntu focal-security InRelease
#0 0.179   Could not connect to security.ubuntu.com:80 (185.125.190.36). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (91.189.91.39). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (185.125.190.39). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (91.189.91.38). - connect (111: Connection refused)
#0 0.185 Reading package lists...
#0 0.191 W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/InRelease  Could not connect to archive.ubuntu.com:80 (91.189.91.39). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (91.189.91.38). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (185.125.190.39). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (185.125.190.36). - connect (111: Connection refused)
#0 0.191 W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-updates/InRelease  Unable to connect to archive.ubuntu.com:http:
#0 0.191 W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-backports/InRelease  Unable to connect to archive.ubuntu.com:http:
#0 0.191 W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/focal-security/InRelease  Could not connect to security.ubuntu.com:80 (185.125.190.36). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (91.189.91.39). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (185.125.190.39). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (91.189.91.38). - connect (111: Connection refused)
#0 0.191 W: Some index files failed to download. They have been ignored, or old ones used instead.
#0 0.195 Reading package lists...
#0 0.200 Building dependency tree...
#0 0.201 Reading state information...
#0 0.201 E: Unable to locate package zstd
------
Dockerfile:9
--------------------
   7 |     
   8 |     # Install zstd for model compression and distribution
   9 | >>> RUN apt-get update && apt-get install -y zstd  && rm -rf /var/lib/apt/lists/*
  10 |     
  11 |     # Install our torch ver matching cuda
--------------------
ERROR: failed to solve: process "/bin/sh -c apt-get update && apt-get install -y zstd  && rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 100

The IP address mentioned in the error does answer a ping:

(willow-inference-server) (willow) j@textgen-working:~/willow-inference-server$ ping 91.189.91.38
PING 91.189.91.38 (91.189.91.38) 56(84) bytes of data.
64 bytes from 91.189.91.38: icmp_seq=1 ttl=51 time=39.0 ms
64 bytes from 91.189.91.38: icmp_seq=2 ttl=51 time=37.4 ms
64 bytes from 91.189.91.38: icmp_seq=3 ttl=51 time=36.5 ms
^C
--- 91.189.91.38 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 36.534/37.639/38.991/1.017 ms

This is a good example of why it would be nice to have some non-docker instructions available.

Another issue with build.sh is that it doesn't appear to use a local Dockerfile, so I can't try to fix this, even if I wanted to:

#!/bin/bash
set -e
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
cd "$SCRIPT_DIR"

if [ "$1" ]; then
    TAG="$1"
else
    TAG="latest"
fi

DOCKER_BUILDKIT=1 docker build -t willow-inference-server:"$TAG" .

As is, I'm a bit dead in the water here. I don't know how docker image names map to github repositories, but perhaps I can figure it out.

[tensiondriven]

D'oh! I am a big dummy. The Dockerfile is in fact right there. I will try editing it.

[tensiondriven]

According to https://packages.ubuntu.com/jammy/zstd the zstd package depends on liblz4-1. I changed the offending line in the Dockerfile to:

apt-get install -y liblz4-1  && rm -rf /var/lib/apt/lists/* 

and that seems to have resolved the problem

[kristiankielhofner]

apt-get (of course) installs necessary dependencies by default so I have no idea what the problem was there. More than likely you had some transient network connectivity issue because when you can't connect to Ubuntu mirrors it's definitely not a WIS issue - we haven't even really done anything yet.

You're also probably better served (at this point) using/testing the wisng branch for WIS. Our first (imminent) stable release will be based on that.

[tensiondriven]

@kristiankielhofner Thank you!

When I tried apt-get install -y zstd locally outside of the docker context, it worked fine - I suspect that the base docker image, nvcr.io/nvidia/tensorrt:22.12-py3, has something funky in the /etc/sources, where it is able to find liblz4-1 but not zstd.

[tensiondriven]

I suspect others may run into this in the future; for now, I'm happy to close.

[kristiankielhofner]

Except that it does work for all of the people who have installed WIS before you. I do it several times a day :).

Additionally, installing liblz4-1 eliminated your error but it's actually the wrong lib. LZ4 is a different compression algorithm, the dependant lib for zstd is libzstd1. Installing liblz4-1 does nothing other than occupy storage space.

When you try to use zstd inside of the container (as we do in occasional places) it will fail because the binary and dependant lib isn't actually installed.