Fix glibc not found problem

[kathoef]

Currently, building the image-registry-checker image succeeds, but the "service" fails to start with these error messages,

$ grep "FROM" Dockerfile
FROM golang:${GO_VERSION} as crane-builder
FROM rust:${RUST_VERSION} as rust-builder
FROM gcr.io/distroless/cc
$ docker build .
Sending build context to Docker daemon  67.58kB
Step 1/15 : ARG GO_VERSION=1.20
Step 2/15 : ARG RUST_VERSION=1
[...]
Successfully built ca7a4e54b74c
$ docker run -it --rm -p 8080:8080 ca7a4e54b74c
./image-registry-checker: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by ./image-registry-checker)
./image-registry-checker: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./image-registry-checker)
./image-registry-checker: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./image-registry-checker)

Approaching the problem was a bit "hard", especially since I do not know anything about Rust and because all the specified unit tests were succeeding (see "development environment" added to the source code directory). Steps to follow,

$ docker build -t rustdev:latest .
Step 1/7 : ARG GO_VERSION=1.20.0
Step 2/7 : ARG RUST_VERSION=1.67.0
[...]
Successfully built ffb5f2c30478
Successfully tagged rustdev:latest
$ docker run -it --rm -v $PWD:/app -w /app rustdev:latest bash
root@6cf340409721:/app# cargo test
    Updating crates.io index
  Downloaded atty v0.2.14
[...]
   Compiling image-registry-checker v0.1.0 (/app)
    Finished test [unoptimized + debuginfo] target(s) in 1m 39s
     Running unittests src/main.rs (target/debug/deps/image_registry_checker-abca1758d69d4dfb)

running 3 tests
test image_exist::test::check_image_slug_returns_error_on_failed_spawn ... ok
test image_exist::test::check_image_slug_returns_true_on_success ... ok
test image_exist::test::check_image_slug_returns_false_on_invalid_slug ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.39s

To approach the root cause, I then started to write a few curl-based "integration tests" and played around with starting from other base container images and also updating Rust package versions. Turns out, the problem is rooted in using the gcr.io/distroless/cc image, which is a pointer to gcr.io/distroless/cc-debian11.

By using the newer gcr.io/distroless/cc-debian12 (see here for "valid" image tags) the image_registry_checker service at least starts successfully again. Steps to demonstrate,

$ docker pull gcr.io/distroless/cc
$ docker pull gcr.io/distroless/cc-debian11
$ docker pull gcr.io/distroless/cc-debian12
$ docker images
REPOSITORY                                                          TAG                                      IMAGE ID       CREATED             SIZE
gcr.io/distroless/cc-debian12                                       latest            8007e4e3052a   N/A              23.4MB
gcr.io/distroless/cc                                                latest            6883c9e0f352   N/A              22.9MB
gcr.io/distroless/cc-debian11                                       latest            6883c9e0f352   N/A              22.9MB
$ grep "FROM" Dockerfile
FROM golang:${GO_VERSION} as crane-builder
FROM rust:${RUST_VERSION} as rust-builder
FROM gcr.io/distroless/cc-debian12
$ docker build .
Sending build context to Docker daemon  60.42kB
Step 1/15 : ARG GO_VERSION=1.20
Step 2/15 : ARG RUST_VERSION=1
[...]
Successfully built 0f984d9b81bf
$ docker run -it --rm -p 8080:8080 0f984d9b81bf
 2024-04-18T14:35:08.370Z INFO  image_registry_checker > Cannot read environment from .env: path not found
 2024-04-18T14:35:08.371Z INFO  warp::server           > Server::run; addr=0.0.0.0:8080
 2024-04-18T14:35:08.371Z INFO  warp::server           > listening on http://0.0.0.0:8080

Please note, the "integration test environment" added via tests.sh is a bit of a "fun with Bash" dev project... I decided to add it, because it might help to ensure that the service does indeed what it should do (especially for people without any Rust knowledge), i.e. by testing it in a more language-agnostic way. 😉

This is also motivated by the following observation: Interestingly, the image_registry_checker service build from gcr.io/distroless/cc-debian12 passes all specified "integration tests" in tests.sh. See here,

$ grep "FROM" Dockerfile
FROM golang:${GO_VERSION} as crane-builder
FROM rust:${RUST_VERSION} as rust-builder
FROM gcr.io/distroless/cc-debian12
#FROM debian:12
$ bash tests.sh
[...]

Ensuring that service is running...
test_if_service_is_running ... [ running = running ] ... PASSED

Starting integration tests...
test_if_health_endpoint_is_served ... [ 200 = 200 ] ... PASSED
test_if_swagger_endpoint_is_served ... [ 200 = 200 ] ... PASSED
test_if_apidoc_endpoint_is_served ... [ 200 = 200 ] ... PASSED
test_http_status_return_codes ... [ 200 = 200 ] ... PASSED
test_http_status_return_codes ... [ 404 = 404 ] ... PASSED
test_openolat_return_code_interoperability ... [ 200 -ge 200 -a 200 -lt 300 ] ... PASSED
test_openolat_return_code_interoperability ... [ 404 = 404 ] ... PASSED

Fetching service logs...
 2024-04-18T15:05:34.197Z INFO  image_registry_checker > Cannot read environment from .env: path not found
 2024-04-18T15:05:34.197Z INFO  warp::server           > Server::run; addr=0.0.0.0:8080
 2024-04-18T15:05:34.197Z INFO  warp::server           > listening on http://0.0.0.0:8080
 2024-04-18T15:05:44.223Z INFO  image_registry_checker > GET /health (19.14µs) 200 OK
 2024-04-18T15:05:44.228Z INFO  image_registry_checker > GET /swagger-ui/index.html (24.52µs) 200 OK
 2024-04-18T15:05:44.232Z INFO  image_registry_checker > GET /api-doc.json (35.50µs) 200 OK
 2024-04-18T15:05:45.109Z INFO  image_registry_checker > GET /exists (872.18ms) 200 OK
 2024-04-18T15:05:46.467Z INFO  image_registry_checker > GET /exists (1.35s) 404 Not Found
 2024-04-18T15:05:47.309Z INFO  image_registry_checker > GET /exists (835.70ms) 200 OK
 2024-04-18T15:05:48.584Z INFO  image_registry_checker > GET /exists (1.27s) 404 Not Found

[...]

But it fails to do so if build from e.g. debian:12, where the image_registry_checker service for some reason always returns "404 Not Found", even if a container image is known to exist in a registry. See here,

$ grep "FROM" Dockerfile
FROM golang:${GO_VERSION} as crane-builder
FROM rust:${RUST_VERSION} as rust-builder
#FROM gcr.io/distroless/cc-debian12
FROM debian:12
$ bash tests.sh
[...]

Ensuring that service is running...
test_if_service_is_running ... [ running = running ] ... PASSED

Starting integration tests...
test_if_health_endpoint_is_served ... [ 200 = 200 ] ... PASSED
test_if_swagger_endpoint_is_served ... [ 200 = 200 ] ... PASSED
test_if_apidoc_endpoint_is_served ... [ 200 = 200 ] ... PASSED
test_http_status_return_codes ... [ 404 = 200 ] ... FAILED
test_http_status_return_codes ... [ 404 = 404 ] ... PASSED
test_openolat_return_code_interoperability ... [ 404 -ge 200 -a 404 -lt 300 ] ... FAILED
test_openolat_return_code_interoperability ... [ 404 = 404 ] ... PASSED

Fetching service logs...
 2024-04-18T15:03:23.779Z INFO  image_registry_checker > Cannot read environment from .env: path not found
 2024-04-18T15:03:23.779Z INFO  warp::server           > Server::run; addr=0.0.0.0:8080
 2024-04-18T15:03:23.779Z INFO  warp::server           > listening on http://0.0.0.0:8080
 2024-04-18T15:03:33.807Z INFO  image_registry_checker > GET /health (22.42µs) 200 OK
 2024-04-18T15:03:33.813Z INFO  image_registry_checker > GET /swagger-ui/index.html (32.63µs) 200 OK
 2024-04-18T15:03:33.818Z INFO  image_registry_checker > GET /api-doc.json (32.80µs) 200 OK
 2024-04-18T15:03:34.055Z INFO  image_registry_checker > GET /exists (231.68ms) 404 Not Found
 2024-04-18T15:03:34.288Z INFO  image_registry_checker > GET /exists (227.51ms) 404 Not Found
 2024-04-18T15:03:34.518Z INFO  image_registry_checker > GET /exists (223.20ms) 404 Not Found
 2024-04-18T15:03:34.747Z INFO  image_registry_checker > GET /exists (223.00ms) 404 Not Found

[...]

Any clues about that, @martinclaus?

[martinclaus]

Hi @kathoef! Thank you for all your effort. Much appreciated!

The server returns a 404 if the subprocess with the crane command can be spawned successfully but returns a non-zero exit code.
Additional logging of the output of the command would be helpful to debug the problem when using debian:12.

[martinclaus]

Changes from #6 suggest that debian:12 has certificate issues:

...
2024-04-19T20:52:13.868Z ERROR image_registry_checker::image_exist > "/crane" failed with status code 1: Error: fetching manifest docker.io/alpine: Get "https://index.docker.io/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority

 2024-04-19T20:52:13.868Z INFO  image_registry_checker              > GET /exists (242.68ms) 404 Not Found
 2024-04-19T20:52:14.137Z ERROR image_registry_checker::image_exist > "/crane" failed with status code 1: Error: fetching manifest docker.io/non-existent: Get "https://index.docker.io/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority

 2024-04-19T20:52:14.137Z INFO  image_registry_checker              > GET /exists (232.77ms) 404 Not Found
 2024-04-19T20:52:14.405Z ERROR image_registry_checker::image_exist > "/crane" failed with status code 1: Error: fetching manifest docker.io/alpine: Get "https://index.docker.io/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority

 2024-04-19T20:52:14.405Z INFO  image_registry_checker              > GET /exists (245.38ms) 404 Not Found
 2024-04-19T20:52:14.652Z ERROR image_registry_checker::image_exist > "/crane" failed with status code 1: Error: fetching manifest docker.io/non-existent: Get "https://index.docker.io/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority

 2024-04-19T20:52:14.652Z INFO  image_registry_checker              > GET /exists (225.12ms) 404 Not Found
...