Debian Buster arm32: "Manifest not found" using curl / TLS

[pdcastro]

Image build fails at build.tpl#L56-L57 when curl can't verify host certificate:

+ mkdir -p /usr/src/dbus-python
+ curl -SL http://dbus.freedesktop.org/releases/dbus-python/dbus-python-1.2.4.tar.gz -o dbus-python.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   368  100   368    0     0    286      0  0:00:01  0:00:01 --:--:--   908
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

It only occurs on Debian Buster arm32 images when downloading with curl (other archs are fine) so looks like we can't use curl here until it's fixed upstream by the package maintainers. We can change to wget for now.

Note: credit to @nghiant2710 for the investigation! 👍

[balena-ci]

[pdcastro] This issue has attached support thread https://jel.ly.fish/#/support-thread~af4e153f-79f1-4b28-8179-960a9f528d68

[imrehg]

This is discussed at https://www.flowdock.com/app/rulemotion/i-baseimages/threads/ItT-a3BTywFhOXsGMrUrlJDTvXu and relevant upstream issue is debuerreotype/docker-debian-artifacts#68

[balena-ci]

[imrehg] This issue has attached support thread https://jel.ly.fish/#/support-thread~b0b90313-33c6-4f42-a623-6bd8c93c1094

[balena-ci]

[chrisys] This issue has attached support thread https://jel.ly.fish/#/support-thread~06cc4d80-5f5c-43a3-8858-415cc89a793c

[zrzka]

Might be related.

It works with curl when I RUN c_rehash.

[main]     Step 1/25 : FROM balenalib/raspberrypi3-debian as base
[main]      ---> 409a5ebb1eea
[main]     Step 2/25 : ENV INITSYSTEM on
[main]     Using cache
[main]      ---> ae5d10096947
[main]     Step 3/25 : ENV DEBIAN_FRONTEND noninteractive
[main]     Using cache
[main]      ---> 5e5417b9aed9
[main]     Step 4/25 : FROM base as rust
[main]      ---> 5e5417b9aed9
[main]     Step 5/25 : RUN apt-get -q update && apt-get install -yq --no-install-recommends build-essential curl file
[main]     Using cache
[main]      ---> 0969ee84ae23
[main]     Step 6/25 : RUN c_rehash
[main]      ---> Running in 6d52d10122e1
[main]     Doing /usr/lib/ssl/certs
[main]     WARNING: Skipping duplicate certificate ca-certificates.crt
[main]     
[main]     WARNING: Skipping duplicate certificate ca-certificates.crt
[main]     
[main]      ---> fe153490b377
[main]     Removing intermediate container 6d52d10122e1
[main]     Step 7/25 : ENV PATH /root/.cargo/bin:$PATH
[main]      ---> Running in 0db1a5d0359d
[main]      ---> 84a614e49d21
[main]     Removing intermediate container 0db1a5d0359d
[main]     Step 8/25 : RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
[main]      ---> Running in d0dc0ce74525
[main]     info: downloading installer
[main]     
[main]     info: syncing channel updates for 'stable-armv7-unknown-linux-gnueabihf'
[main]     
[main]     info: latest update on 2019-07-04, rust version 1.36.0 (a53f9df32 2019-07-03)
[main]     info: downloading component 'rustc'

[balena-ci]

[pdcastro] This issue has attached support thread https://jel.ly.fish/#/support-thread~ca6f7d4f-9043-4494-9a55-87b1910f15ac

[seemethere]

Also having issues with this, was trying to test out the latest nightlies for docker:

❯ docker run --rm -it balenalib/rpi-raspbian:buster sh -c 'curl -fsSL get.docker.com | CHANNEL=nightly sh'
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

[pdcastro]

A customer has reported that the wifi-connect project is affected by this issue, and that it was "fixed" by pinning the base image in the Dockerfile.template to "stretch":

FROM balenalib/%%RESIN_MACHINE_NAME%%-debian:stretch

[brownjohnf]

Was this issue actually resolved in the base images, or is the pinning workaround still required? This issue is still referenced in https://github.com/balena-io-examples/balena-rust-hello-world/blob/master/Dockerfile.template#L19.

[nghiant2710]

hey @brownjohnf, this was fixed a while ago. Maybe the Dockerfile in the rust example is not updated (last change was on Oct 3, 2019)

[rfay]

I seem to still have this problem with arm/v7, looks to me like maybe it's not fixed.

[pdcastro]

@rfay, do you have an example of how to reproduce the issue? I have just tested with the following two Dockerfiles, using pinned / versioned base images. The old base image of May 2019 reproduces the issue, while a recent base image of August 2020 does not have the issue. Note that the raspberrypi3 base image is for the armv7hf architecture, which should match arm/v7.

# Dockerfile
FROM balenalib/raspberrypi3-debian:buster-20190529
RUN curl -SL "http://dbus.freedesktop.org/releases/dbus-python/dbus-python-$PYTHON_DBUS_VERSION.tar.gz" -o dbus-python.tar.gz

$ docker build .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM balenalib/raspberrypi3-debian:buster-20190529
 ---> 6528b0c421fb
Step 2/2 : RUN curl -SL "http://dbus.freedesktop.org/releases/dbus-python/dbus-python-$PYTHON_DBUS_VERSION.tar.gz" -o dbus-python.tar.gz
 ---> Running in 2495ddcd03fa
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   363  100   363    0     0    812      0 --:--:-- --:--:-- --:--:--   836
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
The command '/bin/sh -c curl -SL "http://dbus.freedesktop.org/releases/dbus-python/dbus-python-$PYTHON_DBUS_VERSION.tar.gz" -o dbus-python.tar.gz' returned a non-zero code: 60

# Dockerfile
FROM balenalib/raspberrypi3-debian:buster-20200813
RUN curl -SL "http://dbus.freedesktop.org/releases/dbus-python/dbus-python-$PYTHON_DBUS_VERSION.tar.gz" -o dbus-python.tar.gz

$ docker build .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM balenalib/raspberrypi3-debian:buster-20200813
 ---> e0f8b96d7af7
Step 2/2 : RUN curl -SL "http://dbus.freedesktop.org/releases/dbus-python/dbus-python-$PYTHON_DBUS_VERSION.tar.gz" -o dbus-python.tar.gz
 ---> Using cache
 ---> b7bc68074dc9
Successfully built b7bc68074dc9

With a recent version of Docker or a very recent / latest version of the balena CLI, you could also use the --pull command-line option to re-pull the base image, in case an older version is cached locally.

[rfay]

Sorry, My problem was with the debian:buster-slim image and arm/v7, you just helped me find my way to solving it. I appreciate that very much. Sorry to stir things up. I didn't totally understand the context here, but the answer (RUN c_rehash ) sure was the right one.