Bug 1894144: baremetal: pin libvirt to 4.5.0 #4339

stbenjam · 2020-11-03T16:11:47Z

The latest version of RHEL ships with libvirt 6.x, which are now in
yum repositories used to build the CI containers. However, when running
an installer built that's linked against it on an older RHEL host, the
installer fails:

$ ocp/ostest/openshift-baremetal-install --dir ocp/ostest --log-level=debug create manifests
ocp/ostest/openshift-baremetal-install: /lib64/libvirt.so.0: version
```LIBVIRT_5.2.0' not found (required by
ocp/ostest/openshift-baremetal-install)
[...snip...]

According to the RHEL 8 ABI Guarantees,
"an application is only guaranteed to run correctly if it executes in an
environment that is as new as the environment it was built in or newer."
Therefore, we must build openshift baremetal installer against the
oldest version of the libvirt libraries we expect it to run against,
which is 4.5.x.

Long term, the libvirt team is looking at not linking to libvirt at
compile time, but rather use dlopen("libvirt.so") at runtime, and then
dlsym("funcname") to resolve functions at time they are first invoked.

berrange · 2020-11-03T16:42:46Z

Theoretically, RHEL is not supposed to do this,
as the X.Y of core software should remain to ensure a stable ABI, but I
guess as libvirt is now distributed as part of AppStream there are looser
requirements.

RHEL ABI compatibility only applies in the forwards direction. ie if you build something in RHEL-8.2, you can run it in 8.3, 8.4, ...9.x, etc. There is no expectation of backwards ABI compatibility, so you can not expect to run on 8.1, 8.0, 7.x, if you built on 8.2.

IOW, to take advantage of the RHEL ABI compatibility you need to build against the oldest version that you intend to be able to run on. Libvirt API is append-only, so you will get full forwards compatibility.

The main limitation with this is that you cannot opt-in to use of newer APIs than is available on your oldest platform.

stbenjam · 2020-11-03T16:47:55Z

Theoretically, RHEL is not supposed to do this,
as the X.Y of core software should remain to ensure a stable ABI, but I
guess as libvirt is now distributed as part of AppStream there are looser
requirements.

RHEL ABI compatibility only applies in the forwards direction. ie if you build something in RHEL-8.2, you can run it in 8.3, 8.4, ...9.x, etc. There is no expectation of backwards ABI compatibility, so you can not expect to run on 8.1, 8.0, 7.x, if you built on 8.2.

IOW, to take advantage of the RHEL ABI compatibility you need to build against the oldest version that you intend to be able to run on. Libvirt API is append-only, so you will get full forwards compatibility.

The main limitation with this is that you cannot opt-in to use of newer APIs than is available on your oldest platform.

Thanks for clarifying - I just had a similar discussion on Slack with @cfergeau and came to the same conclusion after reading https://access.redhat.com/articles/rhel8-abi-compatibility.

What's not clear is how OpenShift CI handles this, we're sort of stuck at whatever repos we're given in the particular container, and it seems CI pushes them forward automatically. As now all of a sudden they have the latest libvirt versions.

The nightly brew OCP versions do not have this problem, it seems to be built consistently against RHEL 8.0 or something close, as it has libvirt 4.5.0.

stbenjam · 2020-11-03T16:49:23Z

My attempt at pinning to 4.5.0 seems to have worked, but now I have some frankenstein with mostly RHEL 8.2-ish packages, and then an older libvirt.

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_installer/4339/pull-ci-openshift-installer-master-images/1323664566072119296/artifacts/build-logs/baremetal-installer.log

stbenjam · 2020-11-03T16:54:34Z

@cfergeau suggested maybe we try building with libvirt-go 4.5 but https://github.com/dmacvicar/terraform-provider-libvirt/ is pinned to at least 5.0, so it may be an accident that any of this actually works today.

berrange · 2020-11-03T16:56:41Z

FWIW, I'm increasingly of the opinion that we need to change the way the libvirt-go bindings work. Instead of linking to libvirt at compile time, I think it'll be more useful to Go app developers if we instead use dlopen("libvirt.so") at runtime, and then dlsym("funcname") to resolve functions at time they are first invoked. This would avoid the need to have libvirt installed on the host at all when building libvirt-go, and users would only need it installed at runtime if they actually intend to use its functionality.

stbenjam · 2020-11-03T17:12:31Z

FWIW, I'm increasingly of the opinion that we need to change the way the libvirt-go bindings work. Instead of linking to libvirt at compile time, I think it'll be more useful to Go app developers if we instead use dlopen("libvirt.so") at runtime, and then dlsym("funcname") to resolve functions at time they are first invoked. This would avoid the need to have libvirt installed on the host at all when building libvirt-go, and users would only need it installed at runtime if they actually intend to use its functionality.

@berrange This would solve a big problem for us. Today, we ship an entirely separate installer for baremetal (which uses a libvirt bootstrapping VM). What would be involved in such a change?

Also: do you think the way this PR is done makes sense to work around the problem we're currently seeing?

berrange · 2020-11-03T17:19:17Z

@berrange This would solve a big problem for us. Today, we ship an entirely separate installer for baremetal (which uses a libvirt bootstrapping VM). What would be involved in such a change?

I've been doing experimentation on this part time. Mostly I've been trying to make it possible to auto-generate alot of the required binding code, because it is way too much work to do the conversion manually. It has been low priority because no one expressed a clear and urgent need for it. Since I know your use case now though, I can justify making it a higher priority on my todo list.

Also: do you think the way this PR is done makes sense to work around the problem we're currently seeing?

I don't know much about your CI/deployment needs, but superficially it looks like a reasonable minimum effort approach to try to solve it.

The latest version of RHEL ships with libvirt 6.x, which are now in yum repositories used to build the CI containers. However, when running an installer built that's linked against it on an older RHEL host, the installer fails: ``` $ ocp/ostest/openshift-baremetal-install --dir ocp/ostest --log-level=debug create manifests ocp/ostest/openshift-baremetal-install: /lib64/libvirt.so.0: version ```LIBVIRT_5.2.0' not found (required by ocp/ostest/openshift-baremetal-install) [...snip...] ``` According to the [RHEL 8 ABI Guarantees](https://access.redhat.com/articles/rhel8-abi-compatibility), "an application is only guaranteed to run correctly if it executes in an environment that is as new as the environment it was built in or newer." Therefore, we must build openshift baremetal installer against the oldest version of the libvirt libraries we expect it to run against, which is 4.5.x. Long term, the libvirt team is looking at not linking to libvirt at compile time, but rather use `dlopen("libvirt.so")` at runtime, and then `dlsym("funcname")` to resolve functions at time they are first invoked.

stbenjam · 2020-11-03T17:46:20Z

/retitle Bug 1894144: baremetal: pin libvirt to 4.5.0

openshift-ci-robot · 2020-11-03T17:46:44Z

@stbenjam: This pull request references Bugzilla bug 1894144, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.7.0) matches configured target release for branch (4.7.0)
bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1894144: baremetal: pin libvirt to 4.5.0

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hardys · 2020-11-03T17:58:42Z

Seems like a reasonable workaround to unblock CI while we figure out a better long-term plan.

/lgtm

staebler · 2020-11-03T19:47:14Z

/approve

openshift-ci-robot · 2020-11-03T19:47:32Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: staebler

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~images/baremetal/OWNERS~~ [staebler]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

stbenjam · 2020-11-03T19:48:18Z

/cherry-pick release-4.6

We’ll likely need this for 4.6, too.

openshift-cherrypick-robot · 2020-11-03T19:48:19Z

@stbenjam: once the present PR merges, I will cherry-pick it on top of release-4.6 in a new PR and assign it to you.

In response to this:

/cherry-pick release-4.6

We’ll likely need this for 4.6, too.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2020-11-03T20:08:27Z

/retest