Triton build failed due to apt-get update

[SalahEddineKouiri]

Description

I have encountered a bug while building triton server with docker. Using the ./build.py -v --enable-all command and encountered the following error message. Any idea what might be the solution to this?

 => [internal] load build definition from Dockerfile                                                                  0.0s
 => => transferring dockerfile: 7.12kB                                                                                0.0s
 => [internal] load .dockerignore                                                                                     0.0s
 => => transferring context: 103B                                                                                     0.0s
 => resolve image config for docker.io/docker/dockerfile:1.3                                                          1.1s
 => docker-image://docker.io/docker/dockerfile:1.3@sha256:42399d4635eddd7a9b8a24be879d2f9a930d0ed040a61324cfdf59ef13  1.8s
 => => resolve docker.io/docker/dockerfile:1.3@sha256:42399d4635eddd7a9b8a24be879d2f9a930d0ed040a61324cfdf59ef1357b3  0.0s
 => => sha256:42399d4635eddd7a9b8a24be879d2f9a930d0ed040a61324cfdf59ef1357b3b2 2.00kB / 2.00kB                        0.0s
 => => sha256:93f32bd6dd9004897fed4703191f48924975081860667932a4df35ba567d7426 528B / 528B                            0.0s
 => => sha256:e532695ddd93ca7c85a816c67afdb352e91052fab7ac19a675088f80915779a7 1.21kB / 1.21kB                        0.0s
 => => sha256:24a639a53085eb680e1d11618ac62f3977a3926fedf5b8471ace519b8c778030 9.67MB / 9.67MB                        1.1s
 => => extracting sha256:24a639a53085eb680e1d11618ac62f3977a3926fedf5b8471ace519b8c778030                             0.6s
 => [internal] load metadata for nvcr.io/nvidia/tritonserver:23.08-py3                                                0.8s
 => [internal] load metadata for docker.io/condaforge/mambaforge:latest                                               0.8s
 => [final 1/4] FROM nvcr.io/nvidia/tritonserver:23.08-py3@sha256:b88ed20fe7daa16c4d45cbc931b3e3fc18be8b206a4aac6f  985.6s
 => => resolve nvcr.io/nvidia/tritonserver:23.08-py3@sha256:b88ed20fe7daa16c4d45cbc931b3e3fc18be8b206a4aac6f04e40022  0.0s
 => => sha256:b88ed20fe7daa16c4d45cbc931b3e3fc18be8b206a4aac6f04e40022151eaa93 685B / 685B                            0.0s
 => => sha256:0ec682bf99715a9f88631226f3749e2271b8b9f254528ef61f65ed829984821c 128.73MB / 128.73MB                   26.7s
 => => sha256:9d19ee268e0d7bcf6716e6658ee1b0384a71d6f2f9aa1ae2085610cf7c7b316f 30.43MB / 30.43MB                     14.0s
 => => sha256:9ac855545fa90ed2bf3b388fdff9ef06ac9427b0c0fca07c9e59161983d8827e 150.07MB / 150.07MB                   50.4s
 => => sha256:57215e06cad9be6f65702831d1cc0b994f5539268522fbda022472a943a50e28 4.85kB / 4.85kB                        0.0s
 => => sha256:3e96065a3dcc1c5abac69a3439d23eb1a876230bcab9c777ddd24e9e50d9ff87 26.77kB / 26.77kB                      0.0s
 => => sha256:0a77dcbd0e648ddc4f8e5230ade8fdb781d99e24fa4f13ca96a360c7f7e6751f 15.11kB / 15.11kB                     14.2s
 => => extracting sha256:9d19ee268e0d7bcf6716e6658ee1b0384a71d6f2f9aa1ae2085610cf7c7b316f                             3.2s
 => => sha256:0a1756432df4a4350712d8ae5c003f1526bd2180800b3ae6301cfc9ccf370254 3.02GB / 3.02GB                      594.3s
 => => sha256:e9225f7ab6606813ec9acba98a064826ebfd6713a9645a58cd068538af1ecddb 11.53kB / 11.53kB                     26.8s
 => => extracting sha256:0ec682bf99715a9f88631226f3749e2271b8b9f254528ef61f65ed829984821c                            11.3s
 => => sha256:5b5b131e0f20db4cb8e568b623a95f8fc16ed1c6b322a9366df70b59a881f24f 182B / 182B                           26.9s
 => => sha256:238f69a43816e481f0295995fcf5fe74d59facf0f9f99734c8d0a2fb140630e0 5.72kB / 5.72kB                       26.9s
 => => sha256:4d04a8db404f16c2704fa10739cb6745a0187713a21a6ef0deb34b48629b54c1 139.29kB / 139.29kB                   27.1s
 => => sha256:7852b73ea931e3a8d3287ee7ef3cf4bad068e44f046583bfc2b81336fb299284 31.36MB / 31.36MB                     35.0s
 => => sha256:3e3e04011ebdba380ab129f0ee390626cb2a600623815ca756340c18bedb9517 87.01MB / 87.01MB                     54.2s
 => => sha256:f608e2fbff86e98627b7e462057e7d2416522096d73fe4664b82fe6ce8a4047d 503B / 503B                           50.5s
 => => extracting sha256:9ac855545fa90ed2bf3b388fdff9ef06ac9427b0c0fca07c9e59161983d8827e                            12.6s
 => => sha256:2ad84487f9d4d31cd1e0a92697a5447dd241935253d036b272ef16d31620c1e7 471.38MB / 471.38MB                  173.3s
 => => sha256:f9d96fc3a3ac8b088a739d04d83e86b906bdad948ac2e39ae2186ac3ad5fe67e 1.85kB / 1.85kB                       54.3s
 => => sha256:5d489c8c533d919cf38e76594190ffc179f376690f289d2a267ac1d514aa301b 44.54MB / 44.54MB                     65.8s
 => => extracting sha256:0a77dcbd0e648ddc4f8e5230ade8fdb781d99e24fa4f13ca96a360c7f7e6751f                             0.0s
 => => sha256:8bcf99987574ab9cb29ff73a11d3259146d031bc2d5c43b9d563267e610ff933 385.48MB / 385.48MB                  156.8s
 => => sha256:9d653565ab29dbd97678d4c529f8cbc54f42e3442d9cbc96e9215d2c5a2ca3f3 10.85kB / 10.85kB                    156.9s
 => => sha256:dc929aa7086d76b17633cab0284542eb205aae9ff387cebe07f30f6b331c636e 59.52MB / 59.52MB                    170.6s
 => => sha256:1f02cedb11f1e2c20950d6f7512989d113dc161f6dc2d5a0d26249465d70eaef 118B / 118B                          170.7s
 => => sha256:614754a205b6663ee531d2a62e864b886d481f957e9ce90122fba0c16fc03118 757B / 757B                          170.8s
 => => sha256:068a4d084b92def53a56512be4b47b6767f0ce521df81ee95de55f1809ae5fa8 2.27GB / 2.27GB                      570.9s
 => => sha256:f36e6df7cc9483f1e7b3ce65bd698e0a53a522f6729491071ef49309c8c91816 2.80MB / 2.80MB                      174.2s
 => => sha256:a29267856b0a60d11cfe6009f64607f87705a241859a62b12ab32150a66a12e7 2.84kB / 2.84kB                      174.2s
 => => extracting sha256:0a1756432df4a4350712d8ae5c003f1526bd2180800b3ae6301cfc9ccf370254                           174.6s
 => => extracting sha256:e9225f7ab6606813ec9acba98a064826ebfd6713a9645a58cd068538af1ecddb                             0.0s
 => => extracting sha256:5b5b131e0f20db4cb8e568b623a95f8fc16ed1c6b322a9366df70b59a881f24f                             0.0s
 => => extracting sha256:238f69a43816e481f0295995fcf5fe74d59facf0f9f99734c8d0a2fb140630e0                             0.0s
 => => extracting sha256:4d04a8db404f16c2704fa10739cb6745a0187713a21a6ef0deb34b48629b54c1                             0.0s
 => => extracting sha256:7852b73ea931e3a8d3287ee7ef3cf4bad068e44f046583bfc2b81336fb299284                             3.4s
 => => extracting sha256:3e3e04011ebdba380ab129f0ee390626cb2a600623815ca756340c18bedb9517                             6.9s
 => => extracting sha256:f608e2fbff86e98627b7e462057e7d2416522096d73fe4664b82fe6ce8a4047d                             0.0s
 => => extracting sha256:2ad84487f9d4d31cd1e0a92697a5447dd241935253d036b272ef16d31620c1e7                            23.1s
 => => extracting sha256:f9d96fc3a3ac8b088a739d04d83e86b906bdad948ac2e39ae2186ac3ad5fe67e                             0.0s
 => => extracting sha256:5d489c8c533d919cf38e76594190ffc179f376690f289d2a267ac1d514aa301b                             4.3s
 => => extracting sha256:8bcf99987574ab9cb29ff73a11d3259146d031bc2d5c43b9d563267e610ff933                            22.5s
 => => extracting sha256:9d653565ab29dbd97678d4c529f8cbc54f42e3442d9cbc96e9215d2c5a2ca3f3                             0.0s
 => => extracting sha256:dc929aa7086d76b17633cab0284542eb205aae9ff387cebe07f30f6b331c636e                             5.5s
 => => extracting sha256:1f02cedb11f1e2c20950d6f7512989d113dc161f6dc2d5a0d26249465d70eaef                             0.0s
 => => extracting sha256:614754a205b6663ee531d2a62e864b886d481f957e9ce90122fba0c16fc03118                             0.0s
 => => extracting sha256:068a4d084b92def53a56512be4b47b6767f0ce521df81ee95de55f1809ae5fa8                           148.9s
 => => extracting sha256:f36e6df7cc9483f1e7b3ce65bd698e0a53a522f6729491071ef49309c8c91816                             0.1s
 => => extracting sha256:a29267856b0a60d11cfe6009f64607f87705a241859a62b12ab32150a66a12e7                             0.0s
 => [internal] load build context                                                                                     0.1s
 => => transferring context: 179.42kB                                                                                 0.0s
 => [conda-base 1/6] FROM docker.io/condaforge/mambaforge@sha256:615e43f72b49a6d46d3f93f7fadf7b7e22f6de934b8e1c46b  221.3s
 => => resolve docker.io/condaforge/mambaforge@sha256:615e43f72b49a6d46d3f93f7fadf7b7e22f6de934b8e1c46b01becb2816e8f  0.0s
 => => sha256:1461e0a1fa14431128dc95d921655fd6cd0b9147b4ec757c6d99e02776e82b47 676B / 676B                            0.0s
 => => sha256:253ccc1ed3c48f907a1396c616b34eae0cc71aacbca8795018529390d2268821 3.88kB / 3.88kB                        0.0s
 => => sha256:615e43f72b49a6d46d3f93f7fadf7b7e22f6de934b8e1c46b01becb2816e8fc7 2.36kB / 2.36kB                        0.0s
 => => sha256:96d54c3075c9eeaed5561fd620828fd6bb5d80ecae7cb25f9ba5f7d88ea6e15c 27.51MB / 27.51MB                    179.5s
 => => extracting sha256:96d54c3075c9eeaed5561fd620828fd6bb5d80ecae7cb25f9ba5f7d88ea6e15c                             3.0s
 => => sha256:5f1c8a17ad9ee2015fbec1b8b65495c0bfda1dddb3ae6bc8cf8cac88734f8f3e 120.41MB / 120.41MB                  208.0s
 => => extracting sha256:5f1c8a17ad9ee2015fbec1b8b65495c0bfda1dddb3ae6bc8cf8cac88734f8f3e                            13.0s
 => [conda-base 2/6] COPY ./ops/gpuci_conda_retry /usr/bin/gpuci_conda_retry                                          1.5s
 => [conda-base 3/6] COPY ./ops/gpuci_mamba_retry /usr/bin/gpuci_mamba_retry                                          0.0s
 => [conda-base 4/6] RUN chmod +x /usr/bin/gpuci_conda_retry /usr/bin/gpuci_mamba_retry                               0.4s
 => [conda-base 5/6] RUN mkdir /conda                                                                                 0.5s
 => [conda-base 6/6] RUN gpuci_mamba_retry install -c conda-forge conda-pack=0.7                                     37.3s
 => [conda-dev 1/4] COPY ./conda/environments/rapids_triton_dev.yml /conda/environment.yml                            0.0s
 => [conda-dev 2/4] RUN gpuci_mamba_retry env update -f /conda/environment.yml  && rm /conda/environment.yml         19.6s
 => [conda-dev 3/4] RUN conda-pack -n rapids_triton_dev -o /tmp/env.tar  && mkdir /conda/dev/  && cd /conda/dev/  &&  8.8s
 => [conda-dev 4/4] RUN /conda/dev/bin/conda-unpack                                                                   1.6s
 => [base 2/3] RUN  if command -v curl;  then [ $(uname -m) = 'x86_64' ]  && curl -o /tmp/cuda-keyring.deb       htt  5.5s
 => [final 2/4] RUN mkdir /models                                                                                     5.2s
 => [final 3/4] RUN if [ -d /opt/tritonserver/backends/fil ];     then       rm -rf /opt/tritonserver/backends/fil/*  0.6s
 => ERROR [base 3/3] RUN apt-get update     && apt-get install --no-install-recommends -y       build-essential       6.9s
------
 > [base 3/3] RUN apt-get update     && apt-get install --no-install-recommends -y       build-essential       ca-certificates       git     && apt-get clean     && rm -rf /var/lib/apt/lists/*:
0.859 Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
0.865 Get:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease [1581 B]
0.881 Get:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  InRelease [1581 B]
0.983 Err:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease
0.983   At least one invalid signature was encountered.
1.077 Err:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  InRelease
1.077   At least one invalid signature was encountered.
1.205 Err:1 http://security.ubuntu.com/ubuntu jammy-security InRelease
1.205   At least one invalid signature was encountered.
6.214 Get:4 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
6.470 Get:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
6.532 Get:6 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB]
6.568 Err:4 http://archive.ubuntu.com/ubuntu jammy InRelease
6.568   At least one invalid signature was encountered.
6.700 Err:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
6.700   At least one invalid signature was encountered.
6.816 Err:6 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
6.816   At least one invalid signature was encountered.
6.840 Reading package lists...
6.867 W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease: At least one invalid signature was encountered.
6.867 E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease' is not signed.
6.867 W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  InRelease: At least one invalid signature was encountered.
6.867 E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  InRelease' is not signed.
6.867 W: GPG error: http://security.ubuntu.com/ubuntu jammy-security InRelease: At least one invalid signature was encountered.
6.867 E: The repository 'http://security.ubuntu.com/ubuntu jammy-security InRelease' is not signed.
6.867 W: GPG error: http://archive.ubuntu.com/ubuntu jammy InRelease: At least one invalid signature was encountered.
6.867 E: The repository 'http://archive.ubuntu.com/ubuntu jammy InRelease' is not signed.
6.867 W: GPG error: http://archive.ubuntu.com/ubuntu jammy-updates InRelease: At least one invalid signature was encountered.
6.867 E: The repository 'http://archive.ubuntu.com/ubuntu jammy-updates InRelease' is not signed.
6.867 W: GPG error: http://archive.ubuntu.com/ubuntu jammy-backports InRelease: At least one invalid signature was encountered.
6.867 E: The repository 'http://archive.ubuntu.com/ubuntu jammy-backports InRelease' is not signed.
------
Dockerfile:83
--------------------
  82 |
  83 | >>> RUN apt-get update \
  84 | >>>     && apt-get install --no-install-recommends -y \
  85 | >>>       build-essential \
  86 | >>>       ca-certificates \
  87 | >>>       git \
  88 | >>>     && apt-get clean \
  89 | >>>     && rm -rf /var/lib/apt/lists/*
  90 |
--------------------
ERROR: failed to solve: process "/bin/sh -c apt-get update     && apt-get install --no-install-recommends -y       build-essential       ca-certificates       git     && apt-get clean     && rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 100
make[2]: *** [CMakeFiles/fil_docker.dir/build.make:74: fil/libtriton_fil.so] Error 1
make[2]: Leaving directory '/tmp/tritonbuild/fil/build'
make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/fil_docker.dir/all] Error 2
make[1]: Leaving directory '/tmp/tritonbuild/fil/build'
make: *** [Makefile:124: all] Error 2
error: build failed

[dyastremsky]

Are you building off the main branch? We test the main branch during builds, so the public key should work. If you are building off an older branch, it is possible that some of the versions have changed and there are key errors, so you would need to modify build.py to account for those (e.g. this error and possible solutions are discussed in the nvidia-docker GitHub repo here).

[SalahEddineKouiri]

Yes, I'm using the main branch. Are there any other branches I should be considering?

[dyastremsky]

Nope, that is correct. Maybe pull to ensure the branch is up to date with main on the Triton server GitHub?

[SalahEddineKouiri]

Still getting the same issue. Not sure what I'm doing wrong here.

[dyastremsky]

Thanks for letting me know. I have created a ticket for us to investigate.

Ticket reference: DLIS-5728.

[latchukarthick98]

Is there an update on this issue?

[dyastremsky]

Not yet. This is in our queue.

One thing that might be worthwhile is replacing --enable-all with only the components you need, in case one of the components is specifically causing this and specifically building the others resolves this issue. There are other routes to using Triton (e.g. using the containers, customizing Triton by building specific components with the CMake instructions in their repositories' READMEs, etc.), in case that is helpful.