Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New f39 (official, not beta) image #312

Merged
merged 1 commit into from
Nov 16, 2023
Merged

Conversation

edsantiago
Copy link
Member

Signed-off-by: Ed Santiago santiago@redhat.com

None

@cevich
Copy link
Member

cevich commented Nov 14, 2023

Re: Building wheel for aiohttp (pyproject.toml): finished with status 'error'

Ugh, damn python dependency breakage. The tooling images are nearly all critical to CI. This might be a problem with missing/incompatible C libraries on the system, based on:

      gcc -fno-strict-overflow -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -fcf-protection -fexceptions -fcf-protection -fexceptions -fcf-protection -fexceptions -fPIC -I/usr/share/automation/lib/ccia.venv/include -I/usr/include/python3.12 -c aiohttp/_websocket.c -o build/temp.linux-x86_64-cpython-312/aiohttp/_websocket.o
      aiohttp/_websocket.c: In function ‘__pyx_pf_7aiohttp_10_websocket__websocket_mask_cython’:
      aiohttp/_websocket.c:1475:3: warning: ‘Py_OptimizeFlag’ is deprecated [-Wdeprecated-declarations]
       1475 |   if (unlikely(!Py_OptimizeFlag)) {
            |   ^~
      In file included from /usr/include/python3.12/Python.h:48,
...cut...

I normally turn to google first, to see if anybody else reported this/similar (including on other OSes). Otherwise somebody probably needs to sit down with registry.fedoraproject.org/fedora-minimal:latest and have a long heart-to-heart. Meaning, try to unscramble the gcc failure if possible.

@edsantiago
Copy link
Member Author

Pushed same fix as #311. I was just being (stupidly) optimistic by hoping that the bug would be fixed by now.

@edsantiago edsantiago force-pushed the f39_released branch 2 times, most recently from 29ee77e to 3d35ebe Compare November 14, 2023 18:27
@edsantiago
Copy link
Member Author

apt-get -qq upgrade grub-common failed. The -qq hides any possible understanding of why. Re-pushed, and I assume this one will fail too but at least we might see why.

@cevich
Copy link
Member

cevich commented Nov 14, 2023

BTW, Thanks for taking this on @edsantiago

I was going to ask this afternoon, then noticed you're already on top of it 👍

@edsantiago edsantiago force-pushed the f39_released branch 5 times, most recently from 8d5121e to f3360b1 Compare November 15, 2023 15:53
Copy link

Cirrus CI build successful. Found built image names and IDs:

Stage Image Name IMAGE_SUFFIX
base debian do-not-use
base fedora do-not-use
base fedora-aws do-not-use
base fedora-aws-arm64 do-not-use
base image-builder do-not-use
base prior-fedora do-not-use
cache build-push c20231115t223006z-f39f38d13
cache debian c20231115t223006z-f39f38d13
cache fedora c20231115t223006z-f39f38d13
cache fedora-aws c20231115t223006z-f39f38d13
cache fedora-netavark c20231115t223006z-f39f38d13
cache fedora-netavark-aws-arm64 c20231115t223006z-f39f38d13
cache fedora-podman-aws-arm64 c20231115t223006z-f39f38d13
cache fedora-podman-py c20231115t223006z-f39f38d13
cache prior-fedora c20231115t223006z-f39f38d13
cache rawhide c20231115t223006z-f39f38d13
cache win-server-wsl c20231115t223006z-f39f38d13

@edsantiago edsantiago force-pushed the f39_released branch 2 times, most recently from abbdf4a to 595727b Compare November 16, 2023 00:47
Copy link

Cirrus CI build successful. Found built image names and IDs:

Stage Image Name IMAGE_SUFFIX
base debian do-not-use
base fedora do-not-use
base fedora-aws do-not-use
base fedora-aws-arm64 do-not-use
base image-builder do-not-use
base prior-fedora do-not-use
cache build-push c20231116t004657z-f39f38d13
cache debian c20231116t004657z-f39f38d13
cache fedora c20231116t004657z-f39f38d13
cache fedora-aws c20231116t004657z-f39f38d13
cache fedora-netavark c20231116t004657z-f39f38d13
cache fedora-netavark-aws-arm64 c20231116t004657z-f39f38d13
cache fedora-podman-aws-arm64 c20231116t004657z-f39f38d13
cache fedora-podman-py c20231116t004657z-f39f38d13
cache prior-fedora c20231116t004657z-f39f38d13
cache rawhide c20231116t004657z-f39f38d13
cache win-server-wsl c20231116t004657z-f39f38d13

@edsantiago
Copy link
Member Author

@cevich I think this is ready for review. Podman test PR is containers/podman#20610

@edsantiago
Copy link
Member Author

Nope, still broken, bad crun on prior-fedora. I've written a script to extract and dump package versions, to make it easier to recognize these problems in the future.

buildah

Distro Version
debian 1.32.2+ds1-1
fedora 1.32.2-1
fedora-aws 1.32.2-1
prior-fedora 1.32.2-1
rawhide 1.32.2-1

conmon

Distro Version
debian 2.1.6+ds1-1
fedora 2:2.1.8-2
fedora-aws 2:2.1.8-2
prior-fedora 2:2.1.7-2
rawhide 2:2.1.7-3

container-selinux

Distro Version
debian -
fedora 2:2.224.0-1
fedora-aws 2:2.224.0-1
prior-fedora 2:2.224.0-1
rawhide 2:2.224.0-1

containers-common

Distro Version
debian -
fedora 4:1-95
fedora-aws 4:1-95
prior-fedora 4:1-89
rawhide 4:1-100

criu

Distro Version
debian 3.17.1-3
fedora 3.18-3
fedora-aws 3.18-3
prior-fedora 3.18-1
rawhide 3.18-3

crun

Distro Version
debian 1.11.1-1
fedora 1.11.2-1
fedora-aws 1.11.1-1
prior-fedora 1.11-1
rawhide 1.11.2-1

golang

Distro Version
debian -
fedora 1.21.3-1
fedora-aws 1.21.3-1
prior-fedora 1.20.10-2
rawhide 1.21.3-1

netavark

Distro Version
debian 1.4.0-4
fedora 1.8.0-2
fedora-aws 1.8.0-2
prior-fedora -
rawhide 1.8.0-2

passt

Distro Version
debian 0.0~git20231107.74e6f48-1
fedora -
fedora-aws 0^20231004.gf851084-1
prior-fedora 0^20231004.gf851084-1
rawhide 0^20231110.g5ec3634-1

podman

Distro Version
debian 4.7.2+ds1-2
fedora 5:4.7.2-1
fedora-aws 5:4.7.2-1
prior-fedora 5:4.7.2-1
rawhide 5:4.7.2-1

runc

Distro Version
debian 1.1.10+ds1-1
fedora 2:1.1.8-1
fedora-aws 2:1.1.8-1
prior-fedora 2:1.1.8-1
rawhide 2:1.1.9-1

skopeo

Distro Version
debian 1.13.3+ds1-2
fedora 1:1.13.3-1
fedora-aws 1:1.13.3-1
prior-fedora 1:1.13.3-1
rawhide 1:1.13.3-1

slirp4netns

Distro Version
debian 1.2.1-1
fedora 1.2.2-1
fedora-aws 1.2.2-1
prior-fedora 1.2.2-1
rawhide 1.2.2-1

systemd

Distro Version
debian -
fedora -
fedora-aws -
prior-fedora -
rawhide 254.5-2

Copy link

Cirrus CI build successful. Found built image names and IDs:

Stage Image Name IMAGE_SUFFIX
base debian do-not-use
base fedora do-not-use
base fedora-aws do-not-use
base fedora-aws-arm64 do-not-use
base image-builder do-not-use
base prior-fedora do-not-use
cache build-push c20231116t131200z-f39f38d13
cache debian c20231116t131200z-f39f38d13
cache fedora c20231116t131200z-f39f38d13
cache fedora-aws c20231116t131200z-f39f38d13
cache fedora-netavark c20231116t131200z-f39f38d13
cache fedora-netavark-aws-arm64 c20231116t131200z-f39f38d13
cache fedora-podman-aws-arm64 c20231116t131200z-f39f38d13
cache fedora-podman-py c20231116t131200z-f39f38d13
cache prior-fedora c20231116t131200z-f39f38d13
cache rawhide c20231116t131200z-f39f38d13
cache win-server-wsl c20231116t131200z-f39f38d13

@edsantiago
Copy link
Member Author

Tighter format. crun now looks good on all Fedoras, but not Debian.

Distro buildah conmon container-selinux containers-common
debian 1.32.2+ds1-1 2.1.6+ds1-1 - -
fedora 1.32.2-1 2:2.1.8-2 2:2.224.0-1 4:1-95
fedora-aws 1.32.2-1 2:2.1.8-2 2:2.224.0-1 4:1-95
prior-fedora 1.32.2-1 2:2.1.7-2 2:2.224.0-1 4:1-89
rawhide 1.32.2-1 2:2.1.7-3 2:2.224.0-1 4:1-100
Distro criu crun golang netavark
debian 3.17.1-3 1.11.1-1 - 1.4.0-4
fedora 3.18-3 1.11.2-1 1.21.3-1 1.8.0-2
fedora-aws 3.18-3 1.11.2-1 1.21.3-1 1.8.0-2
prior-fedora 3.18-1 1.11.2-1 1.20.10-2 -
rawhide 3.18-3 1.11.2-1 1.21.3-1 1.8.0-2
Distro passt podman runc skopeo
debian 0.0~git20231107.74e6f48-1 4.7.2+ds1-2 1.1.10+ds1-1 1.13.3+ds1-2
fedora - 5:4.7.2-1 2:1.1.8-1 1:1.13.3-1
fedora-aws 0^20231004.gf851084-1 5:4.7.2-1 2:1.1.8-1 1:1.13.3-1
prior-fedora 0^20231004.gf851084-1 5:4.7.2-1 2:1.1.8-1 1:1.13.3-1
rawhide 0^20231110.g5ec3634-1 5:4.7.2-1 2:1.1.9-1 1:1.13.3-1
Distro slirp4netns systemd
debian 1.2.1-1 -
fedora 1.2.2-1 -
fedora-aws 1.2.2-1 -
prior-fedora 1.2.2-1 -
rawhide 1.2.2-1 254.5-2

IMPORT_IMG_SFX Show resolved Hide resolved
# has python-3.12, which causes something to barf:
# aiohttp/_websocket.c:3744:45: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
# Possible cause: https://github.com/cython/cython/issues/5238
ARG BASE_TAG=38
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oof, another one. Maybe add the time-bomb to ci/validate.sh based on a search of all files in the repo?

Stepping-back, my main concern here is the adage "There's nothing more permanent than temporary". Maybe this is an annoyance that can be considered later and/or in a separate PR.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cannot add timebomb here... and am reluctant to add a complicated special-case to validation. It would need to find FIXMEs, then look for a YMD, then parse, then just yuk. Some day this will blow up again, and whoever fixes it can revert to latest (if that's really desirable) or f39 or do whatever is necessary then. This really can't be futureproofed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, thanks for your considerations and explaining your thinking.

@cevich
Copy link
Member

cevich commented Nov 16, 2023

I've written a script

Wow, that would be really cool if ran automatically and the output followed the table of images/IDs. Should we collaborate in a separate PR to figure a way to make that happen?

Copy link
Member

@cevich cevich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - assuming it fixes the crun problem, I'm willing to live with this as-is. I'm sensitive to what a PITA it is to get a new Distro version built and passing tests.

Unless you feel my other comments should not be addressed and/or in separate PRs?

If not, please feel free to merge as appropriate. Oh, and assuming podman-CI is happy, would you mind tagging this version?
(That's the magic that makes Renovate deploy it across all other repos.)

@edsantiago
Copy link
Member Author

I've written a script

Wow, that would be really cool if ran automatically and the output followed the table of images/IDs. Should we collaborate in a separate PR to figure a way to make that happen?

Some day, but not now. It's too kludgey and heuristic. A better fix will be to rpm -qa|sort or dpkg -s, save and preserve as artifact, then process that.Some rainy day.

Since this is a complicated new push, I won't tag any images yet.

First step: create new base images:

  1minutetip$ make IMPORT_IMG_SFX
  1minutetip$ make image_builder_debug ....

Second step:

  home$ make IMG_SFX

Commit and push. Subsequent emergency management steps:

  1) Change "-qq" to "-q" in debian apt-get, so we have some
     hope of figuring out what is failing.

  2) debian update of grub no longer works. Try a new way.
     (We can no longer update grub-common, due to dependency
     error. Old grub fails with a "version_find_latest" error.
     So, new solution is to provide version_find_latest).

     2a) New timebomb() function will ensure that temporary
         workarounds like this one do not accumulate.

  3) force-update crun on f38 so we get 1.11.2.
     3a) use new timebomb(), see 2a above.

  4) ccia is failing due to cython issue in newer Fedora.
     Force using f38, which works. Cannot timebomb().

  5) fedora-aws build kept timing out. Discover and add
     AWS_SOMETHING envariables to .cirrus.yml

Signed-off-by: Ed Santiago <santiago@redhat.com>
Copy link

Cirrus CI build successful. Found built image names and IDs:

Stage Image Name IMAGE_SUFFIX
base debian do-not-use
base fedora do-not-use
base fedora-aws do-not-use
base fedora-aws-arm64 do-not-use
base image-builder do-not-use
base prior-fedora do-not-use
cache build-push c20231116t174419z-f39f38d13
cache debian c20231116t174419z-f39f38d13
cache fedora c20231116t174419z-f39f38d13
cache fedora-aws c20231116t174419z-f39f38d13
cache fedora-netavark c20231116t174419z-f39f38d13
cache fedora-netavark-aws-arm64 c20231116t174419z-f39f38d13
cache fedora-podman-aws-arm64 c20231116t174419z-f39f38d13
cache fedora-podman-py c20231116t174419z-f39f38d13
cache prior-fedora c20231116t174419z-f39f38d13
cache rawhide c20231116t174419z-f39f38d13
cache win-server-wsl c20231116t174419z-f39f38d13

EOF
exit 1
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! I haven't seen a stderr redirect done like that before, interesting.

Now you've got me thinking I should add this into the common automation library. There's probably lots of places we could use this.

Thanks!

@cevich cevich merged commit 416e87b into containers:main Nov 16, 2023
39 checks passed
@cevich
Copy link
Member

cevich commented Nov 16, 2023

I pushed a tag for this too BTW.

Thanks again Ed.

@edsantiago edsantiago deleted the f39_released branch November 16, 2023 21:01
@edsantiago
Copy link
Member Author

Distro base buildah conmon container-selinux
debian 13 1.32.2+ds1-1 2.1.6+ds1-1 -
prior-fedora 38-1.6 1.32.2-1 2:2.1.7-2 2:2.224.0-1
fedora 39-1.5 1.32.2-1 2:2.1.8-2 2:2.224.0-1
fedora-aws - 1.32.2-1 2:2.1.8-2 2:2.224.0-1
rawhide 40-0 1.32.2-1 2:2.1.8-2 2:2.224.0-1
Distro containers-common criu crun golang
debian - 3.17.1-3 1.11.1-1 2:1.21~2
prior-fedora 4:1-89 3.18-1 1.11.2-1 1.20.10-2
fedora 4:1-95 3.18-3 1.11.2-1 1.21.3-1
fedora-aws 4:1-95 3.18-3 1.11.2-1 1.21.3-1
rawhide 4:1-100 3.18-3 1.11.2-1 1.21.3-1
Distro netavark passt podman runc
debian 1.4.0-4 0.0~git20231107.74e6f48-1 4.7.2+ds1-2 1.1.10+ds1-1
prior-fedora - 0^20231004.gf851084-1 5:4.7.2-1 2:1.1.8-1
fedora 1.8.0-2 - 5:4.7.2-1 2:1.1.8-1
fedora-aws 1.8.0-2 0^20231004.gf851084-1 5:4.7.2-1 2:1.1.8-1
rawhide 1.8.0-2 0^20231110.g5ec3634-1 5:4.7.2-1 2:1.1.9-1
Distro skopeo slirp4netns systemd-container
debian 1.13.3+ds1-2 1.2.1-1 255~rc2-1
prior-fedora 1:1.13.3-1 1.2.2-1 253.12-1
fedora 1:1.13.3-1 1.2.2-1 254.5-2
fedora-aws 1:1.13.3-1 1.2.2-1 254.5-2
rawhide 1:1.13.3-1 1.2.2-1 254.5-2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants