[Disconnected env] Could we replace tag for hardcoded busybox image by digest to make it work in disconnected env?

[tkrishtop]

Bug Report

In disconnected environment image pull by tag doesn't work. The mirroring could be done using digest but not tag because we have no access to remote registry to resolve the tag.

The problem is that scorecard code contains image hardcoded by tag. As a result, scorecard pod doesn't start because it tries to pull an image hardcoded by tag here.

$ oc describe pod scorecard-test-l5k6

Events:
  Type     Reason          Age                 From               Message
  ----     ------          ----                ----               -------
  Normal   Scheduled       114s                default-scheduler  Successfully assigned default/scorecard-test-l5k6 to worker-3
  Warning  Failed          52s                 kubelet            Failed to pull image "docker.io/busybox:1.33.0": rpc error: code = Unknown desc = error pinging docker registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp 3.209.182.229:443: i/o timeout
  Warning  Failed          52s                 kubelet            Error: ErrImagePull
  Normal   BackOff         51s                 kubelet            Back-off pulling image "docker.io/busybox:1.33.0"
  Warning  Failed          51s                 kubelet            Error: ImagePullBackOff
  Normal   Pulling         40s (x2 over 112s)  kubelet            Pulling image "docker.io/busybox:1.33.0"

The solution would be to replace a reference by tag docker.io/busybox:1.33.0 by a digest reference with sha.

What did you do?

I run scorecard in disconnected environment

operator-sdk scorecard --output json --selector=test=basic-check-spec-test --kubeconfig path/to/kubeconfig --namespace scorecard-testing --service-account default --config /tmp/ansible.1p913asescorecard_tmp_dir/artifacts/scorecard-test-config.yaml --verbose --wait-time 300s "/tmp/ansible.1p913asescorecard_tmp_dir/scorecard_operator_dir"

level=debug msg="Debug logging is set"
Error: error running tests context deadline exceeded

Environment

$ operator-sdk version

quay.io/operator-framework/scorecard-test:v1.13

Possible Solution

The solution would be to replace a reference by tag docker.io/busybox:1.33.0 by a digest reference with sha.

[jmccormick2001]

#4886 that issue/fix would also fix this one. @jmrodri @theishshah

[tkrishtop]

Hello, I managed to have operator-sdk scorecard working in my disconnected env using the following diff

$ git diff
diff --git a/internal/scorecard/storage.go b/internal/scorecard/storage.go
index 3ec2e537..a57d6ddd 100644
--- a/internal/scorecard/storage.go
+++ b/internal/scorecard/storage.go
@@ -149,7 +149,7 @@ func addStorageToPod(podDef *v1.Pod, mountPath string) {
        // add the storage sidecar container
        storageContainer := v1.Container{
                Name:            StorageSidecarContainer,
-               Image:           "busybox",
+               Image:           "busybox@sha256:c71cb4f7e8ececaffb34037c2637dc86820e4185100e18b4d02d613a9bd772af",
                ImagePullPolicy: v1.PullIfNotPresent,
                Args: []string{
                        "/bin/sh",
diff --git a/internal/scorecard/testpod.go b/internal/scorecard/testpod.go
index 5862a83c..82b9388f 100644
--- a/internal/scorecard/testpod.go
+++ b/internal/scorecard/testpod.go
@@ -33,7 +33,7 @@ const (
 
        // The image used to untar bundles prior to running tests within a runner Pod.
        // This image tag should always be pinned to a specific version.
-       scorecardUntarImage = "registry.access.redhat.com/ubi8/ubi:8.4"
+       scorecardUntarImage = "registry.access.redhat.com/ubi8@sha256:910f6bc0b5ae9b555eb91b88d28d568099b060088616eba2867b07ab6ea457c7"
 )

Here is more detailed explanation.

@jmccormick2001 @estroz @jmrodri @theishshah what do you think? Could we have a release or dedicated branch with the following patch?

[estroz]

At this point scorecard (and probably all commands hard-coded image names) should permit configuring all images using explicit flags, since there are a variety of cases where the hard-coded names are not pullable. This should be easy to do.

[jmccormick2001]

yeah, I was thinking the same, we might have a flag like --untar-image="registry.access.redhat.com/ubi8/ubi:8.4", if set then that would override a default.

[tkrishtop]

Hi @estroz @jmccormick2001 ok, let's work on the fix you prefer. To sum up, there will be two flags exposed: one for busybox and one for ubi8:

--busybox busybox@sha256:c71cb4f7e8ececaffb34037c2637dc86820e4185100e18b4d02d613a9bd772af
--ubi8 registry.access.redhat.com/ubi8@sha256:910f6bc0b5ae9b555eb91b88d28d568099b060088616eba2867b07ab6ea457c7

Is it fine for you?

[tkrishtop]

/assign

[jmccormick2001]

I like the "--untar-image" flag name since it tells the purpose of that image within scorecard, whether it is busybox or ubi8 is just because they both support a tar command, so the actual image could vary for this purpose.

[tkrishtop]

hi @jmccormick2001 my tests show that both busybox and ubi8 images are pulled and used when operator-sdk scorecard --selector=test=basic-check-spec-test is running. That means that we need two different tags. Concerning tag names, I'm fine to use untar-image for both:

--untar-image-busybox busybox@sha256:c71cb4f7e8ececaffb34037c2637dc86820e4185100e18b4d02d613a9bd772af
--untar-image registry.access.redhat.com/ubi8@sha256:910f6bc0b5ae9b555eb91b88d28d568099b060088616eba2867b07ab6ea457c7

[jmccormick2001]

yep, I suspect the ubi8 image is being pulled because that is the base of the scorecard test container. The untar is being performed by the busybox container in the Pods initcontainer.

https://github.com/operator-framework/operator-sdk/blob/master/images/scorecard-test/Dockerfile#L20

for sure you would be able to override the initContainer's image using:

--untar-image

that is because the test Pods are created when you run tests so that value would be easy to override at runtime.

[tkrishtop]

@jmccormick2001 my issue is not to rewrite any of the images used in Dockerfiles, these are already inside of scorecard-test image, and I do not care at all about them.
My issue concerns two images which are explicitly pulled during scorecard job execution:

busybox

operator-sdk/internal/scorecard/storage.go

Line 152 in 82e8ebb

Image: "busybox",

ubi8

operator-sdk/internal/scorecard/testpod.go

Line 36 in 50caa74

scorecardUntarImage = "registry.access.redhat.com/ubi8/ubi:8.4"

[jmccormick2001]

ah, sorry, I forgot about the storage container sidecar using the ubi image, ok so you would maybe end up with 2 flags as you suggest, something like:
--untar-image
--storage-image

[tkrishtop]

Thank you for checking @jmccormick2001, I'm not familiar with the architecture and certainly used wrong terminology :) Thank you for the clarification, I'll start to work on the implementation then