3rd party kernel installation failed. "error: Multiple subdirectories found in: usr/lib/modules: ..."

[harryg02]

Host system details

Provide the output of rpm-ostree status.
$ rpm-ostree status
State: idle
Deployments:
● fedora:fedora/35/x86_64/silverblue
Version: 35.20220318.0 (2022-03-18T17:53:05Z)
BaseCommit: a492d0056fc144ff71888859f58c822d105df6be0f31942ede6d05bcd49e5d39
GPGSignature: Valid signature by 787EA6AE1147EEE56C40B30CDB4639719867C58F
LayeredPackages: 'dnf-command(copr)' dnf

fedora:fedora/35/x86_64/silverblue
Version: 35.1.2 (2021-10-26T02:25:55Z)
Commit: 34b3a6ad08a934f5db9031929a1f64a788033d151e9074596c6a53af2edf68ac
GPGSignature: Valid signature by 787EA6AE1147EEE56C40B30CDB4639719867C58F

Expected vs actual behavior
Actual:

# rpm-ostree override remove kernel kernel-core kernel-modules kernel-modules-extra --install kernel-longterm
Checking out tree a492d00... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4 updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2021-09-21T18:07:30Z solvables: 4
rpm-md repo 'updates' (cached); generated: 2022-03-18T19:56:48Z solvables: 28575
rpm-md repo 'fedora' (cached); generated: 2021-10-26T05:31:27Z solvables: 65732
rpm-md repo 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' (cached); generated: 2022-03-18T02:44:06Z solvables: 44
rpm-md repo 'updates-archive' (cached); generated: 2022-03-18T20:40:10Z solvables: 34248
Resolving dependencies... done
Will download: 4 packages (112.1 MB)
⠁ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠙ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠦ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠋ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠁ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠂ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠄ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠖ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠓ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠁ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠒ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠤ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠠ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠐ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠉ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠉ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠒ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4'... done
Downloading from 'updates'... done
Importing packages... done
Applying 4 overrides and 19 overlays
Processing packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
error: Multiple subdirectories found in: usr/lib/modules: 5.4.185-300.fc35.x86_64 5.16.15-201.fc35.x86_64+debug

Expected:

rpm-ostree override remove kernel kernel-core kernel-modules kernel-modules-extra --install kernel-longterm
Checking out tree a492d00... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4 updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2021-09-21T18:07:30Z solvables: 4
rpm-md repo 'updates' (cached); generated: 2022-03-18T19:56:48Z solvables: 28575
rpm-md repo 'fedora' (cached); generated: 2021-10-26T05:31:27Z solvables: 65732
rpm-md repo 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' (cached); generated: 2022-03-18T02:44:06Z solvables: 44
rpm-md repo 'updates-archive' (cached); generated: 2022-03-18T20:40:10Z solvables: 34248
Resolving dependencies... done
Will download: 4 packages (112.1 MB)
⠁ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠙ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠦ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠋ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠁ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠂ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠄ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠖ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠓ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠁ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠒ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠤ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠠ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠐ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠉ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠉ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' ⠒ Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4' Downloading from 'copr:copr.fedorainfracloud.org:kwizart:kernel-longterm-5.4'... done
Downloading from 'updates'... done
Importing packages... done
Applying 4 overrides and 19 overlays
Processing packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
...
Added:
kernel-longterm-5.4.185-300.fc35.x86_64
kernel-longterm-core-5.4.185-300.fc35.x86_64
kernel-longterm-modules-5.4.185-300.fc35.x86_64
kernel-longterm-modules-extra-5.4.185-300.fc35.x86_64
Changes queued for next boot. Run "systemctl reboot" to start a reboot

Steps to reproduce it

1. Install DNF and copr plugin.
   $ rpm-ostree install dnf -A
   $ rpm-ostree install 'dnf-command(copr)' -A

2. Add Kernel-longterm copr using dnf.
   $ sudo dnf copr enable kwizart/kernel-longterm-5.4

3. install kernel-longterm.
   $ rpm-ostree override remove kernel kernel-core kernel-modules kernel-modules-extra --install kernel-longterm

This command worked a month ago for replacing the kernel, but now it doesn't work anymore. Any help would be appreciated.

[harryg02]

Update: The command 'rpm-ostree override remove kernel kernel-core kernel-modules kernel-modules-extra --install kernel-longterm' worked fine with kwizart/kernel-longterm-5.15 but not kwizart/kernel-longterm-5.4. I also tried both installing 5.14 and 5.4 kernel from koji according to this fedora silverblue official doc. 5.14 kernel was successfully installed but 5.4 kernel failed to install with the error: Multiple subdirectories found in: usr/lib/modules: ... It seems like there's problem with installing older kernel on fedora silverblue.

[jcdickinson]

The version of ostree that ships in the ISO has this bug, you'll get further if you update: fedora-silverblue/issue-tracker#214.

[BeeGrech]

I am encountering this issue within rhivos (centos stream 9) ostree deployment. Does anyone know if its only been fixed downstream?

[cgwalters]

I am encountering this issue within rhivos (centos stream 9) ostree deployment. Does anyone know if its only been fixed downstream?

Are you trying to ship multiple kernels?

[BeeGrech]

This is from a testing perspective for kernel, we will need to install an instrumented kernel build. Eventually downstream we would build with the kernel, but at this stage of testing we need to use a upstream image.

[cgwalters]

Sure, that's fine and supported. You just currently need to be sure you're removing all traces of the previous kernel.

This relates to #2542 as well as #3364

[secretmango]

I have a different problem installing the hardened Kernel:

rpm-ostree override remove kernel kernel-core kernel-modules kernel-modules-extra --install kernel-hardened
rpm-md repo 'copr:copr.fedorainfracloud.org:samsepi0l:HardHatOS' (cached); generated: 2023-03-18T15:06:31Z solvables: 8

# all going well

Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
error: Multiple kernels (vmlinuz) found in: usr/lib/modules: 6.1.20-hardened1-1-hardened 6.1.18-200.fc37.x86_64+debug