Cirrus: Install docker from package cache

[cevich]

What type of PR is this?

/kind flake

What this PR does / why we need it:

Installing packages at runtime (from an external source) is problematic
for many reasons. Specifically in the case of buildah/docker
conformance testing, it means the current "latest" pacakges are
always installed. This is a problem as new release branches are
created, because it presents an opportunity for test-environment changes
to happen after buildah/test code is stabilized.

Fix this by using new/special VM images which cache the required docker
packages. At runtime then, the required packages may be installed from
this cache instead of reaching out to the repository. Since images used
by tests on release branches never change, this will also serve to
stabilize the package versions for that specific environment.

How to verify it

The automated 'conformance' testing will pass in this PR

Which issue(s) this PR fixes:

None

Special notes for your reviewer:

Ref: containers/automation_images#75

Does this PR introduce a user-facing change?

None

[cevich]

Uh oh, I think I broke it. @edsantiago every platform (a guess - I only looked at two or three) is failing this bud with --runtime and --runtime-flag test in the same way. I don't think (?) it has anything to do with my script changes (here), but may have something to do with brand-new/updated VM images. Do you have/know/see any clues that may indicate what I broke?

[edsantiago]

Sorry, I took a quick look at this last week but decided it was nothing I could have an opinion on.

Diving into it now, this is almost certainly a problem with the new VM update. I can reproduce this on my laptop, with buildah@master. My laptop updated runc 1.0.0-377.rc93 -> -378.rc95 on 2021-05-29.

My uneducated guess is that the test developer chose a string ("nsexec started") that looked reliable, but is actually not; and runc changed so that it no longer spits out that string; and someone will have to fix the buildah test.

[cevich]

@edsantiago thanks for taking a look, that helps. I haven't looked at the test recently...anything you can think of so the test doesn't need to rely on (perhaps frequently changing) debug-level messages? If not, I'll try fixing the test with the message you suggest and see what happens.

[edsantiago]

I would bring in a runc expert, because I have no idea what guarantees runc makes or does not make about its debug messages; or what problems we (buildah gating tests) will run into if the new test runs with an old runc (e.g. on RHEL). My naïve first reaction is to simply search for the presence of level=debug, which (to me) guarantees that the runtime-flag was passed along to something (whether that is runc or not, the test cannot determine, but I don't know how important it is to care about that. If it's important, the test could check for "level=debug.*msg=.*/runc".)

[cevich]

@giuseppe what's your opinion (see test failure due to new runc). Can I update the test to just check for level=debug or is something more extensive needed in the system-tests?

[cevich]

Added proposed fix (in case it's acceptable to @giuseppe) and updated VM images yet again, to help confirm podman 3.2.1 addition to Ubuntu images is functional (cc @lsm5)

[edsantiago]

LGTM

[edsantiago]

Looks like seccomp again:

# time="2021-06-16T21:35:29Z" level=error msg="container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall bdflush: permission denied" func=main.fatal file="/usr/src/packages/BUILD/src/github.com/opencontainers/runc/utils.go:57"
# time="2021-06-16T21:35:29Z" level=debug msg="container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall bdflush: permission denied" func=main.fatal file="/usr/src/packages/BUILD/src/github.com/opencontainers/runc/utils.go:59"

Log

[cevich]

sigh, yep and rebasing didn't help (was worth a try).

[+0731s] # time="2021-06-17T13:53:08Z" level=error msg="container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall bdflush: permission denied" func=main.fatal file="/usr/src/packages/BUILD/src/github.com/opencontainers/runc/utils.go:57"
[+0731s] # time="2021-06-17T13:53:08Z" level=debug msg="container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall bdflush: permission denied" func=main.fatal file="/usr/src/packages/BUILD/src/github.com/opencontainers/runc/utils.go:59"
[+0731s] # : exit status 1

[cevich]

Opened test PR in podman to see if these several-days newer images reproduce similar failures containers/podman#10709

[cevich]

Yay, a result! This problem is reproducing in podman testing on Ubuntu 20.10 (CGv1 + runc): https://cirrus-ci.com/task/6152440987779072?logs=main#L272

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cevich

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [cevich]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[cevich]

Okay assuming tests pass, this should be ready to go. Though the real fruits will only appear over time in new release branches, it might only have a minor improvement to stability on 'main'. Once this merges, I will investigate possibility of doing something similar for already released branches (possible in theory, practice is another matter).

[rhatdan]

/lgtm