New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

version check fails on RHEL / CentOS 7.x #14

Closed
sspans opened this Issue Dec 20, 2016 · 24 comments

Comments

8 participants
@sspans
Copy link

sspans commented Dec 20, 2016

Installing the openvswitch datapath modules via DKMS fails on redhat/centos 7 because the original modules are tagged with a checksum in the srcversion field:

# modinfo /lib/modules/3.10.0-514.el7.x86_64/kernel/net/openvswitch/openvswitch.ko
filename:       /lib/modules/3.10.0-514.el7.x86_64/kernel/net/openvswitch/openvswitch.ko
license:        GPL
description:    Open vSwitch switching datapath
rhelversion:    7.3
srcversion:     B31AE95554C9D9A0067F935
depends:        nf_conntrack,nf_nat,libcrc32c,nf_nat_ipv6,nf_nat_ipv4,nf_defrag_ipv6
intree:         Y
vermagic:       3.10.0-514.el7.x86_64 SMP mod_unload modversions
signer:         CentOS Linux kernel signing key
sig_key:        D4:88:63:A7:C1:6F:CC:27:41:23:E6:29:8F:74:F0:57:AF:19:FC:54
sig_hashalgo:   sha256

Resulting failure:

#dkms autoinstall openvswitch/2.6.2

openvswitch:
Running module version sanity check.
Error! Module version 2.6.2 for openvswitch.ko
is not newer than what is already found in kernel 3.10.0-514.el7.x86_64 (B31AE95554C9D9A0067F935).
You may override by specifying --force.

Quick hack which resolves the issue for me:

--- /usr/sbin/dkms.orig	2016-12-20 11:31:59.368461247 +0100
+++ /usr/sbin/dkms	2016-12-20 11:32:03.059461265 +0100
@@ -723,7 +723,7 @@
             res[2]=${vals[2]}
             ;;
         srcversion:)
-        res[1]=${vals[1]}
+        #res[1]=${vals[1]}
         ;;
     esac
     done < <(modinfo $1)
@yuzaipiaofei

This comment has been minimized.

Copy link
Collaborator

yuzaipiaofei commented Dec 21, 2016

`#dkms autoinstall openvswitch/2.6.2

openvswitch:
Running module version sanity check.
Error! Module version 2.6.2 for openvswitch.ko
is not newer than what is already found in kernel 3.10.0-514.el7.x86_64 (B31AE95554C9D9A0067F935).
You may override by specifying --force.`

I want to know which kernel version you are using to build the openvswitch.ko module ?

@sspans

This comment has been minimized.

Copy link
Author

sspans commented Dec 21, 2016

The default kernel from RHEL 7.3 version: 3.10.0-514.el7.x86_64

However this issue seems to hit every RHEL7 / CentOS 7 kernel, they all have similar modinfo output.

@scaronni

This comment has been minimized.

Copy link
Collaborator

scaronni commented Dec 21, 2016

Same issue as here: #5

@sspans

This comment has been minimized.

Copy link
Author

sspans commented Dec 21, 2016

It would make sense to make dkms handle 'broken' modules, given that this affects the mainline kernels from RHEL/CentOS. POLA and all.

@yuzaipiaofei

This comment has been minimized.

Copy link
Collaborator

yuzaipiaofei commented Jan 3, 2017

@sspans @scaronni
Could you help to provide the broken modules source code to test ?

@yuzaipiaofei

This comment has been minimized.

Copy link
Collaborator

yuzaipiaofei commented Jan 3, 2017

I have the RHEL7.3 build environment to reproduce the installing error,if the issue can be reproduced and make sure it is a dkms bug ,we will find out the workable solution to fix it .

@superm1

This comment has been minimized.

Copy link
Member

superm1 commented Jan 3, 2017

@yuzaipiaofei I believe this should be the source https://github.com/openvswitch/ovs/tree/master/datapath

Here's the DKMS package spec file:
https://github.com/openvswitch/ovs/blob/75e2077e0c43224bcca92746b28b01a4936fc101/rhel/openvswitch-dkms.spec.in

@sspans You might be able to provide something a little more direct about the exact version you are failing with, but it sounds like this should be a generic failure.

@sspans

This comment has been minimized.

Copy link
Author

sspans commented Jan 4, 2017

The failure is triggered by the upstream openvswitch kernel module - it's not dependend on the dkms source tarball. A sample of affected kernels would be:

kernel-3.10.0-514.2.2.el7.x86_64
kernel-3.10.0-327.el7.x86_64
kernel-3.10.0-327.36.3.el7.x86_64

Modinfo which triggers the issue looks like this:

> modinfo /var/lib/dkms/openvswitch/2.6.2/3.10.0-514.2.2.el7.x86_64/x86_64/module/openvswitch.ko
filename:       /var/lib/dkms/openvswitch/2.6.2/3.10.0-514.2.2.el7.x86_64/x86_64/module/openvswitch.ko
version:        2.6.2
license:        GPL
description:    Open vSwitch switching datapath
rhelversion:    7.3
srcversion:     3A8FCA763C880FAD0F8AD6A
depends:        nf_conntrack,nf_nat,nf_defrag_ipv6,udp_tunnel,libcrc32c,nf_nat_ipv6,nf_nat_ipv4
vermagic:       3.10.0-514.2.2.el7.x86_64 SMP mod_unload modversions
@sspans

This comment has been minimized.

Copy link
Author

sspans commented Jan 4, 2017

@superm1 please see issue #5 for a longer description. There are upstream modules in distribution kernels which trigger DKMS failures because the modules lack a version field.

Fixing the upstream modules is a noble goal, but would not help for the kernels already out there.
The fix to DKMS mentionedi in issue #5 is trivial and would solve the issue.

@jared-dominguez

This comment has been minimized.

Copy link
Member

jared-dominguez commented Jan 4, 2017

@sspans I already stated why the patch in #5 will not be accepted and gave acceptable alternatives, including other changes to DKMS (even though openvswitch really should fix their module). Also, for what it's worth, distributions update kernels more frequently than DKMS, so I don't follow your argument about kernels already in the field, especially since the bug is in openvswitch anyway.

@sspans

This comment has been minimized.

Copy link
Author

sspans commented Jan 5, 2017

I'm pretty sure this will not be changed on the rhel/centos side.

What would be an acceptable change to DKMS to deal with this?

@yuzaipiaofei

This comment has been minimized.

Copy link
Collaborator

yuzaipiaofei commented Jan 14, 2017

Before change i also can get below install error
[root@localhost src]# dkms install openvswitch -v 1.4

openvswitch:
Running module version sanity check.
Error! Module version B31AE95554C9D9A0067F935 for openvswitch.ko
is not newer than what is already found in kernel 3.10.0-514.el7.x86_64 (B31AE95554C9D9A0067F935).
You may override by specifying --force.
Adding any weak-modules

depmod...

Backing up initramfs-3.10.0-514.el7.x86_64.img to /boot/initramfs-3.10.0-514.el7.x86_64.img.old-dkms
Making new initramfs-3.10.0-514.el7.x86_64.img
(If next boot fails, revert to initramfs-3.10.0-514.el7.x86_64.img.old-dkms image)
dracut..........

DKMS: install completed.

@yuzaipiaofei

This comment has been minimized.

Copy link
Collaborator

yuzaipiaofei commented Jan 14, 2017

apply patch code :

diff --git a/dkms b/dkms
index 65286de..a66af89 100644
--- a/dkms
+++ b/dkms
@@ -803,7 +803,7 @@ check_version_sanity()
return 1
fi

  • if [[ $kernels_info && $dkms_info && ! ( $(VER $dkms_info) > $(VER $kernels_info) ) && ! $force ]]; then
  • if [[ $kernels_info && $dkms_info && ! ( $(VER $dkms_info) -ge $(VER $kernels_info) ) && ! $force ]]; then
    error $"Module version $dkms_info for ${4}$module_suffix"
    $"is not newer than what is already found in kernel $1 ($kernels_info)."
    $"You may override by specifying --force."
    @@ -3681,7 +3681,7 @@ while (($# > 0)); do
    exec >/dev/null 2>&1
    ;;
    --version|-V)
@yuzaipiaofei

This comment has been minimized.

Copy link
Collaborator

yuzaipiaofei commented Jan 14, 2017

Then install the module again :
[root@localhost src]# dkms install openvswitch -v 1.4

openvswitch:
Running module version sanity check.

  • Original module
    • Found /lib/modules/3.10.0-514.el7.x86_64/updates/openvswitch.ko
    • Storing in /var/lib/dkms/openvswitch/original_module/3.10.0-514.el7.x86_64/x86_64/
    • Archiving for uninstallation purposes
  • Installation
    • Installing to /lib/modules/3.10.0-514.el7.x86_64/extra/
      Adding any weak-modules

depmod...

Backing up initramfs-3.10.0-514.el7.x86_64.img to /boot/initramfs-3.10.0-514.el7.x86_64.img.old-dkms
Making new initramfs-3.10.0-514.el7.x86_64.img
(If next boot fails, revert to initramfs-3.10.0-514.el7.x86_64.img.old-dkms image)
dracut..........

DKMS: install completed.

@yuzaipiaofei

This comment has been minimized.

Copy link
Collaborator

yuzaipiaofei commented Jan 14, 2017

In my opinion,if the kernel module is built by user or developer, maybe the module source codes are same or changed.

  1. If the module codes are same as the module codes built and installed in kernel . i think dkms should allow user to install it again .
  2. if the codes are changed and built with module version crc ,or somehow the code built without essential kernel module version . Dkms should also allow the user to install their modules without previous error warning .

This is my suggestion to fix the issue.

Mario & Jared . How do you think this solution @superm1 @jared-dominguez

@jared-dominguez

This comment has been minimized.

Copy link
Member

jared-dominguez commented Jan 16, 2017

The semantics for "--force" shouldn't be changed since it already is in use. A new parameter is preferable. Perhaps "--force-unversioned"? It should output a warning message too.

The new behavior being described here should be restricted to using this --force-unversioned parameter.

@sspans

This comment has been minimized.

Copy link
Author

sspans commented Jan 17, 2017

If this also would be available as a configuration option then our issue would be resolved.
Most invocations of dkms are via the automated kernel upgrades, so a way to configure this behaviour for the openvswitch module would be needed.

@jared-dominguez

This comment has been minimized.

Copy link
Member

jared-dominguez commented Jan 17, 2017

@sspans Yes, sorry, that was implied, but again only if a warning message is output. And again, openvswitch really should fix their bug as that's the real issue.

@yuzaipiaofei

This comment has been minimized.

Copy link
Collaborator

yuzaipiaofei commented May 23, 2017

After i review the issue again, i found that this issue root cause is the openvswitch dkms package source code version is same as the RHEL7.3 GA kernel module source codes. DKMS will compare the module info and found the new built openvswitch module info is same as the installed one .

Current DKMS sanity check policy will refuse the duplicated installation .
If you still want to install your module , you can use "--force" to install it.
So i do not think we need add more codes for this issue .
Sorry about that .

@sspans

This comment has been minimized.

Copy link
Author

sspans commented May 23, 2017

Please reopen this issue, you seem to have misunderstood the issue.

There are two versions of the openvswitch kernel module, the in-tree version (as shipped by redhat - with no versioning) and the out of tree version (from https://github.com/openvswitch/ovs). The latter has functionality which isn't included in the in-tree version.

With dkms as-is it isn't possible to install the out of tree - versioned ovs module, because the in-tree module doesn't have a version number.

Please considder merging one of the suggested patches.

@yuzaipiaofei yuzaipiaofei reopened this May 23, 2017

@celesteking

This comment has been minimized.

Copy link

celesteking commented Jun 2, 2017

Can we get this fixed finally? Comparing on md5 hash is stupid.

tonyhutter added a commit to tonyhutter/dkms that referenced this issue Sep 17, 2018

Fix version check when installing modules
DKMS wasn't checking the module version when installing newer
kernel modules.  This fixes the issue.

Fixes: dell#14
@tonyhutter

This comment has been minimized.

Copy link
Contributor

tonyhutter commented Sep 17, 2018

I've posted a fix (including a reproducer script) for this issue in #60. Please take a look at it when you get a chance.

@scaronni scaronni closed this in ffc0fee Oct 8, 2018

@scaronni

This comment has been minimized.

Copy link
Collaborator

scaronni commented Oct 8, 2018

Sounds good! Many thanks

@GabeSJones

This comment has been minimized.

Copy link

GabeSJones commented Oct 10, 2018

I don't think the posted fix addresses the original reporter's problem. I am facing a similar predicament. We have made modifications to an in-tree driver in recent 4.1x kernels. We'd like for these changes to be available out-of-tree to users of distributions with much older kernels, so we anticipate distributing them via a DKMS RPM or DEB package.

The code has never used MODULE_VERSION. We want to change that upstream, but none of the existing releases of this driver are versioned, so any current distribution that has this driver installed has an unversioned module. Attempting to install a DKMS package with a versioned module results in the error that the original reporter posted, because check_version_sanity() ends up comparing the version of the new module with the srcversion hash of the old module.

If, e..g, the existing module has no version, and the new module is version 4.20, you end up with something like:

kernels_info = (D461CC2083F4A806A36EBD1)
dkms_info = (4.20 89CAAC69BAB21FECAA04D17)

The following ends up comparing 4.20 with D461CC2083F4A806A36EBD1, which seems invalid:

if [[ $kernels_info && $dkms_info && ! ( $(VER $dkms_info) > $(VER $kernels_info) ) && ! $force ]]; then

Further, in the case where neither module is versioned, the code above compares the two srcversion hashes against each other with something other than equality, which doesn't make any sense. As a result, if I leave the new module unversioned as well, the install may or may not work depending upon whether the srcversion hash on the new module happens to compare favorably to the one on the existing module. The code above only works consistently if both the existing module and the module being installed both have version numbers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment