Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tests: stop using ubi8 (this is the "0x3c" flake) #4318

Closed
edsantiago opened this issue Oct 5, 2022 · 4 comments
Closed

tests: stop using ubi8 (this is the "0x3c" flake) #4318

edsantiago opened this issue Oct 5, 2022 · 4 comments
Labels
locked - please file new issue/PR

Comments

@edsantiago
Copy link
Collaborator

Use of ubi8 images is causing one of our big flakes, mostly in podman (because it runs many more CI jobs than buildah) but also in buildah CI:

$ buildah build -f Containerfile --pull=false -q --arch=amd64 -t image-amd --signature-policy /var/tmp/go/src/github.com/containers/buildah/tests/./policy.json /var/tmp/buildah_tests.sufdm8/my-dir
error creating build container: copying system image from manifest list:
     reading signatures:
     parsing signature https://access.redhat.com/webassets/docker/content/sigstore/ubi8-minimal@sha256=SHA/signature-7:
     unrecognized signature format, starting with binary 0x3c

I think this is also the cause of the horrid multiple-platform-no-run flake, or at least one of the causes.

We can't rely on flaky registries. Can all tests that use ubi8-* be fixed to use a different image?

[Integration] 295 bud --pull=false --arch test

[Containerized] 295 bud --pull=false --arch test

...and, in podman-land:

[bud] 277 bud --pull=false --arch test

  • fedora-36 : bud podman fedora-36 root host
  • fedora-36 : bud remote fedora-36 root host [remote]

[bud] 268 bud --pull=false --arch test

[bud] 261 bud --pull=false --arch test
@edsantiago
Copy link
Collaborator Author

Every. Single. Integration. Job. In. My. PR.

@mtrmac
Copy link
Collaborator

mtrmac commented Oct 10, 2022

For the record: This is the signature server serving 200 OK with HTML content; it is supposed to serve 200 OK with a signature, or 404 — or some non-200 error at least.

@edsantiago edsantiago mentioned this issue Oct 25, 2022
Merged
@edsantiago
Copy link
Collaborator Author

All CI (buildah and podman), all gating tests, are now broken because ubi8 changed something.

✗ bud --pull=false --arch test
   #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
   #|     FAIL: buildah inspect --format {{ index .Docker.Config.Labels "architecture" }} image-arm
   #| expected: =~ 'arm64'
   #|   actual:    'aarch64'
   #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

When we use images we don't control, on registries we don't control, bad things happen.

@edsantiago edsantiago mentioned this issue Oct 26, 2022
Merged
edsantiago added a commit to edsantiago/buildah that referenced this issue Oct 26, 2022
@edsantiago
Define and use a safe, reliable test image
7aa34b8 
The ubi8 image changed some architecture string yesterday and
broke all our CI on all branches, including podman because
it too runs bud.bats tests.

Solution: use a safe, reliable, trustworthy test image (the
one built and used for podman). Because that image does not
have the same labels, confirm pull using .Architecture
instead.

We should stop using ubi8 and registry.redhat (containers#4318). They
are unreliable. This PR doesn't fix that; it's just an
emergency patch for one specific CI break. We can use
this as a basis for future removals of ubi8.

Signed-off-by: Ed Santiago <santiago@redhat.com>
edsantiago added a commit to edsantiago/buildah that referenced this issue Oct 26, 2022
@edsantiago
Define and use a safe, reliable test image
4a10a8c 
The ubi8 image changed some architecture string yesterday and
broke all our CI on all branches, including podman because
it too runs bud.bats tests.

Solution: use a safe, reliable, trustworthy test image (the
one built and used for podman). Because that image does not
have the same labels, confirm pull using .Architecture
instead.

We should stop using ubi8 and registry.redhat (containers#4318). They
are unreliable. This PR doesn't fix that; it's just an
emergency patch for one specific CI break. We can use
this as a basis for future removals of ubi8.

Signed-off-by: Ed Santiago <santiago@redhat.com>
@edsantiago edsantiago mentioned this issue Oct 26, 2022
Merged
edsantiago added a commit to edsantiago/buildah that referenced this issue Oct 26, 2022
@edsantiago
Define and use a safe, reliable test image
a351c6a 
The ubi8 image changed some architecture string yesterday and
broke all our CI on all branches, including podman because
it too runs bud.bats tests.

Solution: use a safe, reliable, trustworthy test image (the
one built and used for podman). Because that image does not
have the same labels, confirm pull using .Architecture
instead.

We should stop using ubi8 and registry.redhat (containers#4318). They
are unreliable. This PR doesn't fix that; it's just an
emergency patch for one specific CI break. We can use
this as a basis for future removals of ubi8.

Signed-off-by: Ed Santiago <santiago@redhat.com>
@edsantiago edsantiago mentioned this issue Oct 26, 2022
Closed
edsantiago added a commit to edsantiago/buildah that referenced this issue Oct 27, 2022
@edsantiago
Define and use a safe, reliable test image
33e25e8 
The ubi8 image changed some architecture string yesterday and
broke all our CI on all branches, including podman because
it too runs bud.bats tests.

Solution: use a safe, reliable, trustworthy test image (the
one built and used for podman). Because that image does not
have the same labels, confirm pull using .Architecture
instead.

We should stop using ubi8 and registry.redhat (containers#4318). They
are unreliable. This PR doesn't fix that; it's just an
emergency patch for one specific CI break. We can use
this as a basis for future removals of ubi8.

Signed-off-by: Ed Santiago <santiago@redhat.com>
This was referenced Oct 27, 2022
Merged
Merged
@edsantiago
Copy link
Collaborator Author

Fixed in #4384 and #4397.

flouthoc pushed a commit to flouthoc/buildah that referenced this issue Nov 23, 2022
@edsantiago @flouthoc
Define and use a safe, reliable test image
93402c1 
The ubi8 image changed some architecture string yesterday and
broke all our CI on all branches, including podman because
it too runs bud.bats tests.

Solution: use a safe, reliable, trustworthy test image (the
one built and used for podman). Because that image does not
have the same labels, confirm pull using .Architecture
instead.

We should stop using ubi8 and registry.redhat (containers#4318). They
are unreliable. This PR doesn't fix that; it's just an
emergency patch for one specific CI break. We can use
this as a basis for future removals of ubi8.

Signed-off-by: Ed Santiago <santiago@redhat.com>
Signed-off-by: Aditya R <arajan@redhat.com>
flouthoc pushed a commit to flouthoc/buildah that referenced this issue Nov 23, 2022
@edsantiago @flouthoc
Define and use a safe, reliable test image
9fa3c16 
The ubi8 image changed some architecture string yesterday and
broke all our CI on all branches, including podman because
it too runs bud.bats tests.

Solution: use a safe, reliable, trustworthy test image (the
one built and used for podman). Because that image does not
have the same labels, confirm pull using .Architecture
instead.

We should stop using ubi8 and registry.redhat (containers#4318). They
are unreliable. This PR doesn't fix that; it's just an
emergency patch for one specific CI break. We can use
this as a basis for future removals of ubi8.

Signed-off-by: Ed Santiago <santiago@redhat.com>
Signed-off-by: Aditya R <arajan@redhat.com>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 30, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
locked - please file new issue/PR
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@edsantiago @mtrmac