Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

version check fails on RHEL / CentOS 7.x #14

Closed
sspans opened this issue Dec 20, 2016 · 24 comments
Closed

version check fails on RHEL / CentOS 7.x #14

sspans opened this issue Dec 20, 2016 · 24 comments

Comments

@sspans
Copy link

@sspans sspans commented Dec 20, 2016

Installing the openvswitch datapath modules via DKMS fails on redhat/centos 7 because the original modules are tagged with a checksum in the srcversion field:

# modinfo /lib/modules/3.10.0-514.el7.x86_64/kernel/net/openvswitch/openvswitch.ko
filename:       /lib/modules/3.10.0-514.el7.x86_64/kernel/net/openvswitch/openvswitch.ko
license:        GPL
description:    Open vSwitch switching datapath
rhelversion:    7.3
srcversion:     B31AE95554C9D9A0067F935
depends:        nf_conntrack,nf_nat,libcrc32c,nf_nat_ipv6,nf_nat_ipv4,nf_defrag_ipv6
intree:         Y
vermagic:       3.10.0-514.el7.x86_64 SMP mod_unload modversions
signer:         CentOS Linux kernel signing key
sig_key:        D4:88:63:A7:C1:6F:CC:27:41:23:E6:29:8F:74:F0:57:AF:19:FC:54
sig_hashalgo:   sha256

Resulting failure:

#dkms autoinstall openvswitch/2.6.2

openvswitch:
Running module version sanity check.
Error! Module version 2.6.2 for openvswitch.ko
is not newer than what is already found in kernel 3.10.0-514.el7.x86_64 (B31AE95554C9D9A0067F935).
You may override by specifying --force.

Quick hack which resolves the issue for me:

--- /usr/sbin/dkms.orig	2016-12-20 11:31:59.368461247 +0100
+++ /usr/sbin/dkms	2016-12-20 11:32:03.059461265 +0100
@@ -723,7 +723,7 @@
             res[2]=${vals[2]}
             ;;
         srcversion:)
-        res[1]=${vals[1]}
+        #res[1]=${vals[1]}
         ;;
     esac
     done < <(modinfo $1)
@GoPerry
Copy link
Collaborator

@GoPerry GoPerry commented Dec 21, 2016

`#dkms autoinstall openvswitch/2.6.2

openvswitch:
Running module version sanity check.
Error! Module version 2.6.2 for openvswitch.ko
is not newer than what is already found in kernel 3.10.0-514.el7.x86_64 (B31AE95554C9D9A0067F935).
You may override by specifying --force.`

I want to know which kernel version you are using to build the openvswitch.ko module ?

@sspans
Copy link
Author

@sspans sspans commented Dec 21, 2016

The default kernel from RHEL 7.3 version: 3.10.0-514.el7.x86_64

However this issue seems to hit every RHEL7 / CentOS 7 kernel, they all have similar modinfo output.

@scaronni
Copy link
Collaborator

@scaronni scaronni commented Dec 21, 2016

Same issue as here: #5

@sspans
Copy link
Author

@sspans sspans commented Dec 21, 2016

It would make sense to make dkms handle 'broken' modules, given that this affects the mainline kernels from RHEL/CentOS. POLA and all.

@GoPerry
Copy link
Collaborator

@GoPerry GoPerry commented Jan 3, 2017

@sspans @scaronni
Could you help to provide the broken modules source code to test ?

@GoPerry
Copy link
Collaborator

@GoPerry GoPerry commented Jan 3, 2017

I have the RHEL7.3 build environment to reproduce the installing error,if the issue can be reproduced and make sure it is a dkms bug ,we will find out the workable solution to fix it .

@superm1
Copy link
Contributor

@superm1 superm1 commented Jan 3, 2017

@yuzaipiaofei I believe this should be the source https://github.com/openvswitch/ovs/tree/master/datapath

Here's the DKMS package spec file:
https://github.com/openvswitch/ovs/blob/75e2077e0c43224bcca92746b28b01a4936fc101/rhel/openvswitch-dkms.spec.in

@sspans You might be able to provide something a little more direct about the exact version you are failing with, but it sounds like this should be a generic failure.

@sspans
Copy link
Author

@sspans sspans commented Jan 4, 2017

The failure is triggered by the upstream openvswitch kernel module - it's not dependend on the dkms source tarball. A sample of affected kernels would be:

kernel-3.10.0-514.2.2.el7.x86_64
kernel-3.10.0-327.el7.x86_64
kernel-3.10.0-327.36.3.el7.x86_64

Modinfo which triggers the issue looks like this:

> modinfo /var/lib/dkms/openvswitch/2.6.2/3.10.0-514.2.2.el7.x86_64/x86_64/module/openvswitch.ko
filename:       /var/lib/dkms/openvswitch/2.6.2/3.10.0-514.2.2.el7.x86_64/x86_64/module/openvswitch.ko
version:        2.6.2
license:        GPL
description:    Open vSwitch switching datapath
rhelversion:    7.3
srcversion:     3A8FCA763C880FAD0F8AD6A
depends:        nf_conntrack,nf_nat,nf_defrag_ipv6,udp_tunnel,libcrc32c,nf_nat_ipv6,nf_nat_ipv4
vermagic:       3.10.0-514.2.2.el7.x86_64 SMP mod_unload modversions
@sspans
Copy link
Author

@sspans sspans commented Jan 4, 2017

@superm1 please see issue #5 for a longer description. There are upstream modules in distribution kernels which trigger DKMS failures because the modules lack a version field.

Fixing the upstream modules is a noble goal, but would not help for the kernels already out there.
The fix to DKMS mentionedi in issue #5 is trivial and would solve the issue.

@jared-dominguez
Copy link

@jared-dominguez jared-dominguez commented Jan 4, 2017

@sspans I already stated why the patch in #5 will not be accepted and gave acceptable alternatives, including other changes to DKMS (even though openvswitch really should fix their module). Also, for what it's worth, distributions update kernels more frequently than DKMS, so I don't follow your argument about kernels already in the field, especially since the bug is in openvswitch anyway.

@sspans
Copy link
Author

@sspans sspans commented Jan 5, 2017

I'm pretty sure this will not be changed on the rhel/centos side.

What would be an acceptable change to DKMS to deal with this?

@GoPerry
Copy link
Collaborator

@GoPerry GoPerry commented Jan 14, 2017

Before change i also can get below install error
[root@localhost src]# dkms install openvswitch -v 1.4

openvswitch:
Running module version sanity check.
Error! Module version B31AE95554C9D9A0067F935 for openvswitch.ko
is not newer than what is already found in kernel 3.10.0-514.el7.x86_64 (B31AE95554C9D9A0067F935).
You may override by specifying --force.
Adding any weak-modules

depmod...

Backing up initramfs-3.10.0-514.el7.x86_64.img to /boot/initramfs-3.10.0-514.el7.x86_64.img.old-dkms
Making new initramfs-3.10.0-514.el7.x86_64.img
(If next boot fails, revert to initramfs-3.10.0-514.el7.x86_64.img.old-dkms image)
dracut..........

DKMS: install completed.

@GoPerry
Copy link
Collaborator

@GoPerry GoPerry commented Jan 14, 2017

apply patch code :

diff --git a/dkms b/dkms
index 65286de..a66af89 100644
--- a/dkms
+++ b/dkms
@@ -803,7 +803,7 @@ check_version_sanity()
return 1
fi

  • if [[ $kernels_info && $dkms_info && ! ( $(VER $dkms_info) > $(VER $kernels_info) ) && ! $force ]]; then
  • if [[ $kernels_info && $dkms_info && ! ( $(VER $dkms_info) -ge $(VER $kernels_info) ) && ! $force ]]; then
    error $"Module version $dkms_info for ${4}$module_suffix"
    $"is not newer than what is already found in kernel $1 ($kernels_info)."
    $"You may override by specifying --force."
    @@ -3681,7 +3681,7 @@ while (($# > 0)); do
    exec >/dev/null 2>&1
    ;;
    --version|-V)
@GoPerry
Copy link
Collaborator

@GoPerry GoPerry commented Jan 14, 2017

Then install the module again :
[root@localhost src]# dkms install openvswitch -v 1.4

openvswitch:
Running module version sanity check.

  • Original module
    • Found /lib/modules/3.10.0-514.el7.x86_64/updates/openvswitch.ko
    • Storing in /var/lib/dkms/openvswitch/original_module/3.10.0-514.el7.x86_64/x86_64/
    • Archiving for uninstallation purposes
  • Installation
    • Installing to /lib/modules/3.10.0-514.el7.x86_64/extra/
      Adding any weak-modules

depmod...

Backing up initramfs-3.10.0-514.el7.x86_64.img to /boot/initramfs-3.10.0-514.el7.x86_64.img.old-dkms
Making new initramfs-3.10.0-514.el7.x86_64.img
(If next boot fails, revert to initramfs-3.10.0-514.el7.x86_64.img.old-dkms image)
dracut..........

DKMS: install completed.

@GoPerry
Copy link
Collaborator

@GoPerry GoPerry commented Jan 14, 2017

In my opinion,if the kernel module is built by user or developer, maybe the module source codes are same or changed.

  1. If the module codes are same as the module codes built and installed in kernel . i think dkms should allow user to install it again .
  2. if the codes are changed and built with module version crc ,or somehow the code built without essential kernel module version . Dkms should also allow the user to install their modules without previous error warning .

This is my suggestion to fix the issue.

Mario & Jared . How do you think this solution @superm1 @jared-dominguez

@jared-dominguez
Copy link

@jared-dominguez jared-dominguez commented Jan 16, 2017

The semantics for "--force" shouldn't be changed since it already is in use. A new parameter is preferable. Perhaps "--force-unversioned"? It should output a warning message too.

The new behavior being described here should be restricted to using this --force-unversioned parameter.

@sspans
Copy link
Author

@sspans sspans commented Jan 17, 2017

If this also would be available as a configuration option then our issue would be resolved.
Most invocations of dkms are via the automated kernel upgrades, so a way to configure this behaviour for the openvswitch module would be needed.

@jared-dominguez
Copy link

@jared-dominguez jared-dominguez commented Jan 17, 2017

@sspans Yes, sorry, that was implied, but again only if a warning message is output. And again, openvswitch really should fix their bug as that's the real issue.

@GoPerry
Copy link
Collaborator

@GoPerry GoPerry commented May 23, 2017

After i review the issue again, i found that this issue root cause is the openvswitch dkms package source code version is same as the RHEL7.3 GA kernel module source codes. DKMS will compare the module info and found the new built openvswitch module info is same as the installed one .

Current DKMS sanity check policy will refuse the duplicated installation .
If you still want to install your module , you can use "--force" to install it.
So i do not think we need add more codes for this issue .
Sorry about that .

@GoPerry GoPerry closed this May 23, 2017
@sspans
Copy link
Author

@sspans sspans commented May 23, 2017

Please reopen this issue, you seem to have misunderstood the issue.

There are two versions of the openvswitch kernel module, the in-tree version (as shipped by redhat - with no versioning) and the out of tree version (from https://github.com/openvswitch/ovs). The latter has functionality which isn't included in the in-tree version.

With dkms as-is it isn't possible to install the out of tree - versioned ovs module, because the in-tree module doesn't have a version number.

Please considder merging one of the suggested patches.

@GoPerry GoPerry reopened this May 23, 2017
@celesteking
Copy link

@celesteking celesteking commented Jun 2, 2017

Can we get this fixed finally? Comparing on md5 hash is stupid.

tonyhutter added a commit to tonyhutter/dkms that referenced this issue Sep 17, 2018
DKMS wasn't checking the module version when installing newer
kernel modules.  This fixes the issue.

Fixes: dell#14
@tonyhutter
Copy link
Contributor

@tonyhutter tonyhutter commented Sep 17, 2018

I've posted a fix (including a reproducer script) for this issue in #60. Please take a look at it when you get a chance.

@scaronni scaronni closed this in ffc0fee Oct 8, 2018
@scaronni
Copy link
Collaborator

@scaronni scaronni commented Oct 8, 2018

Sounds good! Many thanks

@GabeSJones
Copy link

@GabeSJones GabeSJones commented Oct 10, 2018

I don't think the posted fix addresses the original reporter's problem. I am facing a similar predicament. We have made modifications to an in-tree driver in recent 4.1x kernels. We'd like for these changes to be available out-of-tree to users of distributions with much older kernels, so we anticipate distributing them via a DKMS RPM or DEB package.

The code has never used MODULE_VERSION. We want to change that upstream, but none of the existing releases of this driver are versioned, so any current distribution that has this driver installed has an unversioned module. Attempting to install a DKMS package with a versioned module results in the error that the original reporter posted, because check_version_sanity() ends up comparing the version of the new module with the srcversion hash of the old module.

If, e..g, the existing module has no version, and the new module is version 4.20, you end up with something like:

kernels_info = (D461CC2083F4A806A36EBD1)
dkms_info = (4.20 89CAAC69BAB21FECAA04D17)

The following ends up comparing 4.20 with D461CC2083F4A806A36EBD1, which seems invalid:

if [[ $kernels_info && $dkms_info && ! ( $(VER $dkms_info) > $(VER $kernels_info) ) && ! $force ]]; then

Further, in the case where neither module is versioned, the code above compares the two srcversion hashes against each other with something other than equality, which doesn't make any sense. As a result, if I leave the new module unversioned as well, the install may or may not work depending upon whether the srcversion hash on the new module happens to compare favorably to the one on the existing module. The code above only works consistently if both the existing module and the module being installed both have version numbers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants