CI: use local registry #5584

edsantiago · 2024-06-11T13:44:59Z

CI VMs now include a local registry prepopulated with all
container images needed by tests.

Signed-off-by: Ed Santiago santiago@redhat.com

None

packit-as-a-service · 2024-06-11T13:49:48Z

Ephemeral COPR build failed. @containers/packit-build please check.

edsantiago

I was not expecting this to be so easy!

@containers/buildah-maintainers this is not yet ready for merge -- I need to figure out a way to conditionalize on-my-desktop runs vs in-CI-using-cache runs -- but it's very close. Feedback welcome.

edsantiago · 2024-06-12T20:31:11Z

tests/bud.bats

@@ -4577,9 +4577,11 @@ EOM
 }

 @test "bud copy chown with newuser" {
-  _prefetch quay.io/fedora/fedora
+  _prefetch $SAFEIMAGE


Eliminate unused image

edsantiago · 2024-06-12T20:31:46Z

tests/bud.bats

  skip_if_no_runtime
  skip_if_in_container
-  _prefetch alpine debian
+  _prefetch alpine ubuntu


eliminate unused image (for all purposes here, the ubuntu image works the same as debian)

edsantiago · 2024-06-12T20:33:16Z

tests/digest.bats

@@ -29,35 +29,6 @@ fromreftest() {
  rm -fr ${pushdir}
 }

-@test "from-by-digest-s1" {


Welp. Docker registry image no longer supports v1 schema. Code has been excised. There is no way to push a v2s1 image to this registry. Is it OK to just remove all this v1 testing?

edsantiago · 2024-06-12T20:33:39Z

tests/from.bats

+  # FIXME FIXME FIXME: I don't see the point of these. Any reason not to delete?
+#  run_buildah from --quiet --pull=true $WITH_POLICY_JSON docker.io/centos:7
+#  run_buildah rm $output
+#  run_buildah rmi docker.io/centos:7


These all look like dups of the code above.

edsantiago · 2024-06-12T20:34:23Z

tests/mkcw.bats

@@ -58,7 +58,7 @@ function mkcw_check_image() {
    skip "cryptsetup not found"
  fi
  _prefetch busybox
-  _prefetch bash
+  _prefetch $SAFEIMAGE


If there is/was anything special about bash, it was not obvious from comments, so it gets nuked

The bash image has three layers, compared to, say busybox, which has one. The test attempts to confirm that content from the base image and content added at conversion time both show up in the disk image that mkcw is generating.

Thank you! $SAFEIMAGE has two layers, would that be sufficient? If not, could you suggest a three-layer image from this list? https://github.com/containers/automation_images/pull/357/files#diff-7b013edc1f5a127319d75ae1f8ae443e0bfa4d891960c27ee284bbb107ddd290R56

One of those layers is empty, but it should be fine. The error we were attempting to avoid was causing the disk image to lose content from layers that weren't the last one (and as far as this test is concerned, for images with one layer, the only layer is also the last layer), and the presence of the second layer, empty as it is, means the image still meets the test expectations.

Thanks. Repushed with a warning comment

...to minimize hiccups. RUN-2091 in Jira. Network registries are too unreliable; they cause too many flakes in CI. Here we set up a registry running on each VM, prepopulated with all container images used in podman and buildah tests. Related PRs: containers/podman#22726 containers/buildah#5584 Once those merge, podman and buildah CI tests will fetch images from this local registry. Signed-off-by: Ed Santiago <santiago@redhat.com>

edsantiago · 2024-07-08T21:15:32Z

Two flakes, both in what looks like network pulls (dnf, image pull from k8s.io). The k8s.io one is probably a good candidate for future caching.

edsantiago · 2024-07-11T19:25:48Z

Podman PR has merged. I'm considering this ready. @containers/buildah-maintainers PTAL.

rhatdan · 2024-07-11T20:03:44Z

/approve
LGTM
@nalind @giuseppe @flouthoc @mheon PTAL

flouthoc

LGTM

As of containers/automation_images#357 our CI VMs include a local registry preloaded with all(*) images used in tests. * where "all" means "most". This commit defines a new registries-cached.conf, used in tests, that redirects docker and quay to the new local registry. The hope is that this will reduce CI flakes. Since tests change over time, and new tests may require new images, this commit also adds a mechanism for pulling in remote images at test run time. Obviously this negates the purpose of the cache, since it introduces a flake pain point. The idea is: DO NOT DO THIS UNLESS ABSOLUTELY NECESSARY, and then, if we have to do this, hurry up and spin new CI VMs that include the new image(s). Signed-off-by: Ed Santiago <santiago@redhat.com>

Make tests pass when using a local cache registry (as triggered by $CI_USE_REGISTRY_CACHE being non-empty). Mostly just change images: consolidate to a smaller set, like, replace the few instances of debian with ubuntu, use $SAFEIMAGE where we can. The most significant change was with some s1 (schema 1?) images. Those exist on quay, but cannot be mirrored locally: the local registry forbids schema 1 images. I choose to skip these tests when running against a local registry. Second most significant change is running containerized tests with --net=host. We need this to access the registry running on a localhost port. (One alternative would be to bind the registry on all interfaces, cringe, then perform sed magic on the registries-cache.conf file changing 127.0.0.1 to host.containers.internal, more cringe). Also, some cleanup. Signed-off-by: Ed Santiago <santiago@redhat.com>

edsantiago · 2024-07-18T14:08:18Z

ping, this code has been working well in podman. I'd like to get this merged so I can get working on the required treadmill changes (for vendoring into podman).

giuseppe

LGTM

openshift-ci · 2024-07-18T14:31:45Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: edsantiago, flouthoc, giuseppe, rhatdan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [flouthoc,giuseppe,rhatdan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

rhatdan · 2024-07-19T01:10:39Z

/lgtm

edsantiago marked this pull request as draft June 11, 2024 13:45

openshift-ci bot added the do-not-merge/work-in-progress label Jun 11, 2024

edsantiago force-pushed the ci-vms-with-local-registry branch from 78d2d8d to 206e178 Compare June 12, 2024 14:39

edsantiago commented Jun 12, 2024

View reviewed changes

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 24, 2024

edsantiago force-pushed the ci-vms-with-local-registry branch from 206e178 to 6880b88 Compare June 24, 2024 11:12

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 24, 2024

edsantiago force-pushed the ci-vms-with-local-registry branch 3 times, most recently from c641c36 to 379e37b Compare July 1, 2024 21:56

edsantiago force-pushed the ci-vms-with-local-registry branch from 379e37b to 9021f5e Compare July 2, 2024 18:38

edsantiago force-pushed the ci-vms-with-local-registry branch 2 times, most recently from bd2573a to 177edd7 Compare July 8, 2024 14:24

edsantiago force-pushed the ci-vms-with-local-registry branch 2 times, most recently from 3551994 to 379d816 Compare July 11, 2024 10:57

edsantiago mentioned this pull request Jul 11, 2024

Create a local registry containers/automation_images#357

Open

edsantiago marked this pull request as ready for review July 11, 2024 19:25

openshift-ci bot removed the do-not-merge/work-in-progress label Jul 11, 2024

openshift-ci bot added the approved label Jul 11, 2024

edsantiago force-pushed the ci-vms-with-local-registry branch from 379d816 to e8ab724 Compare July 11, 2024 21:33

flouthoc approved these changes Jul 11, 2024

View reviewed changes

edsantiago force-pushed the ci-vms-with-local-registry branch from e8ab724 to f28b5d1 Compare July 15, 2024 19:40

edsantiago force-pushed the ci-vms-with-local-registry branch 2 times, most recently from b6eeba6 to d9260e2 Compare July 17, 2024 18:17

edsantiago force-pushed the ci-vms-with-local-registry branch from d9260e2 to 3ad0fd1 Compare July 17, 2024 18:27

giuseppe approved these changes Jul 18, 2024

View reviewed changes

openshift-ci bot assigned rhatdan Jul 19, 2024

openshift-ci bot added the lgtm label Jul 19, 2024

openshift-merge-bot bot merged commit 16bd9a3 into containers:main Jul 19, 2024
36 of 37 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: use local registry #5584

CI: use local registry #5584

edsantiago commented Jun 11, 2024

packit-as-a-service bot commented Jun 11, 2024

edsantiago left a comment

edsantiago Jun 12, 2024

edsantiago Jun 12, 2024

edsantiago Jun 12, 2024

edsantiago Jun 12, 2024

edsantiago Jun 12, 2024

nalind Jul 11, 2024

edsantiago Jul 11, 2024

nalind Jul 11, 2024

edsantiago Jul 11, 2024

edsantiago commented Jul 8, 2024

edsantiago commented Jul 11, 2024

rhatdan commented Jul 11, 2024

flouthoc left a comment

edsantiago commented Jul 18, 2024

giuseppe left a comment

openshift-ci bot commented Jul 18, 2024

rhatdan commented Jul 19, 2024

CI: use local registry #5584

CI: use local registry #5584

Conversation

edsantiago commented Jun 11, 2024

packit-as-a-service bot commented Jun 11, 2024

edsantiago left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edsantiago commented Jul 8, 2024

edsantiago commented Jul 11, 2024

rhatdan commented Jul 11, 2024

flouthoc left a comment

Choose a reason for hiding this comment

edsantiago commented Jul 18, 2024

giuseppe left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Jul 18, 2024

rhatdan commented Jul 19, 2024