Packit builds fail for Fedora 41 / rawhide

[saschagrunert]

The packit CI fails with the following error:

-g -O2 -g -ldl -O2 -g -ldl -Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes
/usr/bin/ld: /tmp/go-link-1701599834/go.o: warning: relocation against `github.com/tetratelabs/wazero/internal/engine/wazevo/backend/isa/amd64.entrypoint' in read-only section `.text'
/usr/bin/ld: /tmp/go-link-1701599834/go.o: in function `local.github.com/cri-o/cri-o/vendor/github.com/tetratelabs/wazero/internal/engine/wazevo.(*callEngine).callWithStack':
go.go:(.text+0x136c683): undefined reference to `github.com/tetratelabs/wazero/internal/engine/wazevo/backend/isa/amd64.entrypoint'
/usr/bin/ld: /tmp/go-link-1701599834/go.o: in function `local.github.com/cri-o/cri-o/vendor/github.com/tetratelabs/wazero/internal/engine/wazevo.(*callEngine).callWithStack':
/builddir/build/BUILD/cri-o-1.32.0-build/cri-o-1.32.0/_build/src/github.com/cri-o/cri-o/vendor/github.com/tetratelabs/wazero/internal/engine/wazevo/call_engine.go:302:(.text+0x136c7f8): undefined reference to `github.com/tetratelabs/wazero/internal/engine/wazevo/backend/isa/amd64.afterGoFunctionCallEntrypoint'
/usr/bin/ld: /builddir/build/BUILD/cri-o-1.32.0-build/cri-o-1.32.0/_build/src/github.com/cri-o/cri-o/vendor/github.com/tetratelabs/wazero/internal/engine/wazevo/call_engine.go:314:(.text+0x136c8c6): undefined reference to `github.com/tetratelabs/wazero/internal/engine/wazevo/backend/isa/amd64.afterGoFunctionCallEntrypoint'
/usr/bin/ld: /builddir/build/BUILD/cri-o-1.32.0-build/cri-o-1.32.0/_build/src/github.com/cri-o/cri-o/vendor/github.com/tetratelabs/wazero/internal/engine/wazevo/call_engine.go:373:(.text+0x136c9c1): undefined reference to `github.com/tetratelabs/wazero/internal/engine/wazevo/backend/isa/amd64.afterGoFunctionCallEntrypoint'
/usr/bin/ld: /builddir/build/BUILD/cri-o-1.32.0-build/cri-o-1.32.0/_build/src/github.com/cri-o/cri-o/vendor/github.com/tetratelabs/wazero/internal/engine/wazevo/call_engine.go:335:(.text+0x136cab1): undefined reference to `github.com/tetratelabs/wazero/internal/engine/wazevo/backend/isa/amd64.afterGoFunctionCallEntrypoint'
/usr/bin/ld: /builddir/build/BUILD/cri-o-1.32.0-build/cri-o-1.32.0/_build/src/github.com/cri-o/cri-o/vendor/github.com/tetratelabs/wazero/internal/engine/wazevo/call_engine.go:427:(.text+0x136cb0a): undefined reference to `github.com/tetratelabs/wazero/internal/engine/wazevo/backend/isa/amd64.afterGoFunctionCallEntrypoint'
/usr/bin/ld: /tmp/go-link-1701599834/go.o:/builddir/build/BUILD/cri-o-1.32.0-build/cri-o-1.32.0/_build/src/github.com/cri-o/cri-o/vendor/github.com/tetratelabs/wazero/internal/engine/wazevo/call_engine.go:407: more undefined references to `github.com/tetratelabs/wazero/internal/engine/wazevo/backend/isa/amd64.afterGoFunctionCallEntrypoint' follow
/usr/bin/ld: warning: creating DT_TEXTREL in a PIE
collect2: error: ld returned 1 exit status

The issue occured after the merge of the NRI update in #8855.

I can't reproduce it with plain Fedora 41/rawhide packages, so I'm assuming that it's coming from redhat-hardened-ld.

The symbol is specified using go:linkname in:
https://github.com/tetratelabs/wazero/blob/610c20/internal/engine/wazevo/entrypoint_amd64.go#L14

[saschagrunert]

@cri-o/cri-o-maintainers @lsm5 is there a way to build the packit packages locally? Do we know where the specfile is hosted?

[sohankunkerkar]

@LorbusChris do you have any pointers here?

[buckaroogeek]

The package repository for the cri-o rpm is at https://src.fedoraproject.org/rpms/cri-o. I will note that this package has been replaced by "versioned" repositories for CRI-O (i.e. rawhide and Fedora 41 now have separate packages for cri-o1.29, cri-o1.30, cri-o1.31, and now cri-o1.32. (see for example https://src.fedoraproject.org/fork/buckaroogeek/rpms/cri-o1.31. This parallel similar packages for Kubernetes and CRI-Tool so that a Fedora user can use Fedora as the host OS for any supported version of Kubernetes and CRI-O using the standard Fedora repositories.

I just started to work on the cri-o1.32 package and am seeing the warning and errors above. No solution yet.

With respect to packit - I need to submit a PR to CRI-O to remove packit from CRI-O and use a packit pull-from-upstream job which has the .packit.yaml file residing in the Fedora repositories.

[buckaroogeek]

I found this in /usr/share/doc/redhat-rpm-config/buildflags.md on my Fedora workstation.

In some cases (such as when a DSO is loaded as a plugin and is
expected to bind to symbols in the main executable), undefined symbols
are expected.  In this case, you can add

    %undefine _strict_symbol_defs_build

to the RPM spec file to disable these strict checks.  Alternatively,
you can pass `-z undefs` to ld (written as `-Wl,-z,undefs` on the gcc
command line).  The latter needs binutils 2.29.1-12.fc28 or later.

I added %global __golang_extldflags -Wl,-z,undefs to the cri-o1.32 spec file (not the spec file used by Packit at this time) and was able to get a successful build.

I am able to generate rpms and will be pushing the new cri-o1.32 rpm into rawhide and, soon, Fedora 41. Please note, that I have not yet installed the rpm yet or tested it in a Kubernetes 1.32 cluster. Perhaps later today or tomorrow.

I will also submit a PR for the cri-o rpm with an update. My recommendation to @haircommander has been to retire the cri-o package in rawhide as this has been done with the kubernetes rpm. The versioned rpms are suitable replacements which uncouple the tight interdependency between Fedora version and Kubernetes/CI-O/CRI-Tools versions.

[buckaroogeek]

I set up a single node cluster on a Fedora rawhide VM using a locally built cri-o1.32-1.32.0-2.fc42.x86_64.rpm and kubernetes1.32 rpms from rawhide. Followed https://docs.fedoraproject.org/en-US/quick-docs/using-kubernetes-kubeadm/ with slight modifications. No issues.

[buckaroogeek]

PR submitted to fix unversioned CRI-O rpm in Fedora rawhide. Versioned rpms submitted.

[lsm5]

@cri-o/cri-o-maintainers @lsm5 is there a way to build the packit packages locally? Do we know where the specfile is hosted?

Try packit build locally .

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[saschagrunert]

This is resolved now.