Panic on 12.1-RC2 when kernel module is loaded

[nomadlogic]

As per details bellow, on 12.1-RC2 when I have the upstream FreeBSD drm-kmod pkg installed I get a panic on boot.

Here's info from the panic:

<6>[drm] Connector DP-1: get mode from tunables:
<6>[drm]   - kern.vt.fb.modes.DP-1
<6>[drm]   - kern.vt.fb.default_mode
<6>[drm] Connector HDMI-A-1: get mode from tunables:
<6>[drm]   - kern.vt.fb.modes.HDMI-A-1
<6>[drm]   - kern.vt.fb.default_mode


Fatal trap 12: page fault while in kernel mode
cpuid = 5; apic id = 05
fault virtual address	= 0x1
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff836b790f
stack pointer	        = 0x28:0xfffffe0073af7810
frame pointer	        = 0x28:0xfffffe0073af7880
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 0 (softirq_5)
trap number		= 12
panic: page fault
cpuid = 5
time = 1571832290
KDB: stack backtrace:
#0 0xffffffff80c1d297 at kdb_backtrace+0x67
#1 0xffffffff80bd05cd at vpanic+0x19d
#2 0xffffffff80bd0423 at panic+0x43
#3 0xffffffff810a7d2c at trap_fatal+0x39c
#4 0xffffffff810a7d79 at trap_pfault+0x49
#5 0xffffffff810a736f at trap+0x29f
#6 0xffffffff8108175c at calltrap+0x8
#7 0xffffffff837b3480 at tasklet_handler+0x100
#8 0xffffffff80c1bb54 at gtaskqueue_run_locked+0x144
#9 0xffffffff80c1b7b8 at gtaskqueue_thread_loop+0x98
#10 0xffffffff80b90c23 at fork_exit+0x83
#11 0xffffffff8108279e at fork_trampoline+0xe
Uptime: 4s
Dumping 799 out of 16097 MB:..3%..11%..21%..31%..41%..51%..61%..71%..81%..91%

__curthread () at /usr/home/pete/git/freebsd/sys/amd64/include/pcpu.h:234
234		__asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (OFFSETOF_CURTHREAD));

not super helpful unfortunately but I have a core available if we want to poke at it.

here is my system info

FreeBSD duke.nomadlogic.org 12.1-RC2 FreeBSD 12.1-RC2 6ec59e86775(releng/12.1) GENERIC  amd64

$ pkg info drm-fbsd12.0-kmod
drm-fbsd12.0-kmod-4.16.g20190814
Name           : drm-fbsd12.0-kmod
Version        : 4.16.g20190814
Installed on   : Wed Oct 23 12:02:09 2019 PDT
Origin         : graphics/drm-fbsd12.0-kmod
Architecture   : FreeBSD:12:amd64
Prefix         : /usr/local
Categories     : graphics kld
Licenses       : BSD2CLAUSE, MIT, GPLv2
Maintainer     : x11@FreeBSD.org
WWW            : https://github.com/FreeBSDDesktop/kms-drm
Comment        : DRM modules for the linuxkpi-based KMS components
Options        :
	DEBUG          : off
Annotations    :
	FreeBSD_version: 1200086
	repo_type      : binary
	repository     : FreeBSD
Flat size      : 9.89MiB
Description    :
amdgpu, i915, and radeon DRM modules for the linuxkpi-based KMS components.
Currently corresponding to Linux 4.16 DRM.
This version is for FreeBSD 12.0.
amdgpu and radeonkms are known to fail with EFI boot.

WWW: https://github.com/FreeBSDDesktop/kms-drm

[valpackett]

The 12.0-kmod package is built on 12.0 and so is not compatible with 12.1. You should build drm-v4.16-fbsd12.0 or drm-v5.0-fbsd12.1 (branches of this repo) from source. Just clone, checkout the branch, make (-jN), sudo|doas|etc make install.

// @johalun why is this module even loading on 12.1? Aren't version checks supposed to prevent that?

[nomadlogic]

Yes that was the purpose of this exercise. We are preparing for the 12.1-RELEASE and wanted to determine that the existing binaries won't work, and then to come up with a solution for users after the release is live.

[zeising]

This will be big problem once 12.1 is out and people start upgrading. We must make it so that binary packages work across minor FreeBSD version updates, even kmod packages. It was the same issue when 11.3 was released, iirc.

@myfreeweb I know that the fix is fairly simple, for you, an experienced FreeBSD user and developer. But as @nomadlogic points out, this isn't about that, this is about making things for 12.1 work out of the box. Having people doing freebsd-update to 12.1 just to have the system immediately panicing, is quite bad, and will drive people away. Pointing people to the ports tree and tell them to build the ports from there might work. Telling them to clone a random git repository and use that won't work. At all.

[valpackett]

Oh sure, the original post just didn't look like "an exercise" at all, it totally looks like just "I have this problem right now" :)

Again, first of all it's odd that there's a panic instead of the loader refusing to load the module. The failure should be softer.

Anyway I guess we're bound to reinvent Linux's DKMS thing eventually..

Other than that, an interesting solution for shipping the correct module would be to have an additional pkg repo (enabled out of the box) just for this module (or all kmod packages eventually). The main repo doesn't distinguish between minor releases but this one would.

// Yeah, it doesn't have to be from git right now, rebuilding from ports should work too.

[zeising]

Oh sure, the original post just didn't look like "an exercise" at all, it totally looks like just "I have this problem right now" :)

I guess the missing context is that this was a followup from a discussion at our meeting, which is quite hard for you to know. :)

Again, first of all it's odd that there's a panic instead of the loader refusing to load the module. The failure should be softer.

Agreed. I believe the check is only if the module is too new for the kernel, not the opposite. I think the rationale is that it's easier to be backwards compatible (making the kernel work with older modules) than the other way around. This might need to change though.

Anyway I guess we're bound to reinvent Linux's DKMS thing eventually..

Yeah, I think that's probably the best solution, to be honest.

Other than that, an interesting solution for shipping the correct module would be to have an additional pkg repo (enabled out of the box) just for this module (or all kmod packages eventually). The main repo doesn't distinguish between minor releases but this one would.

This is an interesting solution as well, I wonder if it's possible.

[nomadlogic]

Oh sure, the original post just didn't look like "an exercise" at all, it totally looks like just "I have this problem right now" :)

that's a good point, I should have put more context into this ticket when i opened it. I just wanted to make sure we had documented what happens when trying to load the binary, but to people who were part of that discussion it def looks like a generic bug report that needs a fix now.

[0mp]

I'm also experiencing the panic after the upgrade to 12.1-RC2. The machine just reboots immediately after loading the module. I can provide more information if desired.

[kappat]

Not a solution but how about having drm-v5.0-fbsd12.1 comitted to the ports tree with the NO_PACKAGE option then unlock it when 12.0 gets EOL'd? I think it would be a bit more approachable to type make install clean than pulling from here.

[zeising]

drm-fbsd12.0-kmod works on 12.1, you just have to compile it from ports as well.

[therontarigo]

Not a solution but how about having drm-v5.0-fbsd12.1 comitted to the ports tree with the NO_PACKAGE option then unlock it when 12.0 gets EOL'd? I think it would be a bit more approachable to type make install clean than pulling from here.

What would be useful (instead of an identical port) is a slave port that downloads and builds against 12.1 source tree. Then even the 12.0 package builder may place a 12.1-compatible package into the repository. It doesn't ideally fix the problem since user must take care to install the correct package despite both appearing available. Building the download-its-own-12.1 slave on a real 12.1 installation should also be disabled with a warning. Since this isn't normally done, maybe the potential for confusion makes it more trouble than it's worth?

Rebuilding a kernel module during package install, on a system which otherwise uses only downloaded binaries for everything else (what Linux DKMS does) does not seem like a good idea.

Ultimately a decision will be needed between either engineering the kernel to ensure compatibility with older modules or building all kmod ports for each release, yes?

[therontarigo]

Rebuilding a kernel module during package install, on a system which otherwise uses only downloaded binaries for everything else (what Linux DKMS does) does not seem like a good idea.

Thinking through this further... There can be a "fat" kmod package containing the module for all releases supported (this should compress well) and the appropriate binary may be selected automatically during installation. At the very list this is better from the user's perspective than having it build from source, and this shouldn't require any changes to pkg or ports infrastructure.

[nomadlogic]

This will not resolve the primary issue that official pkgs are built against a major versions least common denominator. The crux of the problem is not that we can't distribute multiple versions of the kmod (we do that already via the drm-kmod meta port), but that the package builders for 12.x for example don't build against the 12.1-RELEASE source, but rather against 12.0-RELEASE.

[therontarigo]

I addressed that in my first comment: Have the port itself download the 12.1-RELEASE source. And I wasn't discussing multiple versions of the kmod, I was discussing the same version of the kmod built against multiple differing kernel sources.

[nomadlogic]

Ok I must have been confused when you said "Rebuilding a kernel module during package install, on a system which otherwise uses only downloaded binaries for everything else (what Linux DKMS does) does not seem like a good idea."

I think the general consensus so far has been that we will need to ship a sub-set of the source needed for the drm-kmod to build correctly ala DKMS. This seems to be the only reasonable way for us to ensure people on newer releases are able to install the official upstream pkgs. It also has quite a bit of precedent in both the Linux and Unix world before that when working with drivers.

[rozhuk-im]

A bit offtopic: I got crashes on 12.1 with drm-v5.0-fbsd12.1 on video playback until update bios to latest.
Older 4.16 does not work with newer bios - system hang after amdgpu load.

[cederom]

i have upgraded from 12.0 to 12.1 and yes my machine reboots when loading drm-kmod :-(

this is not how things were ever made in FreeBSD :-(

why did you remove the dem-kmod-stable? that would provide some compatibility among all 12.

now you have to create a dedicated port for every release. the 12.1 is out and the port is not there.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241787

:-(

[nomadlogic]

There is no need for a new port specifically for 12.1. The issue is, if you would read the entire issue, that prebuilt packages are currently built against the oldest support release for a major version. So for FreeBSD-12 packages are built against 12.0-RELEASE.

Unfortunately for the DRM drivers (as well as some other ports) which requires building kmods necessary changes were introduced to the KBI for 12.1 that are not backwards compatible with 12.0. This is what is causing the panic on kernel load you reference.

This issue can be resolved by rebuilding the drm-kmod package on your local 12.1 machine. I use "make package" personally to achieve this.

[cederom]

the issue is that i did a release upgrade and my machine is useless out of the box.

even worse it crashes the os with mounted root in endless loop what eventually results in data corruption!

the issue is that it should work out of the box without "reading the entire issue". this is not Linux. i avoid Linux for that.

i came here after reporting a bug in bugtracker and reading mailing list with dozens people with the same issue.

FreeBSD was always "it works stable or its not part of stable" (not to mention a release).

We should not accept, ever, this rush, just because others do. We avoid Windows and Linux for that. We never should accept a situation like this in FreeBSD!

btw if the port is for 12 stable then please rename it to 12 from 12.0 because 12.0 suggests it will not work on 12.1 (as it would not work on 11.2).

My two cents to your discussion maybe can come handy:

• if you want to have multiple ports versions i would suggest using suffix concatenation and most variable part at the end i.e. drm-kmod-fbsd12-5.0:
• good naming: drm-kmod, drm-kmod-devel, drm-kmod-current, drm-kmod-fbsd11, drm-kmod-fbsd12, drm-kmod-fbsd12-4.16, drm-kmod-fbsd12-5.0.
• bad naming: drm-kmod, drm-devel-kmod, drm-current-kmod, drm-fbsd11.2-kmod, drm-fbsd12.0-kmod.
• do you consider a burden of maintenance of so many ports in so many variants?
• why not having one port that would work among different releases? lots of versioning would introduce Linux like confusion.
• for some time I am using PKG with binary packages, I would prefer to have one DRM port, and no post-install compilation (that would lead to dramatic dependency tree for a simple thing.. not really acceptable i.e. on embedded system).
• some additional and external repositories for kernel components are really not good idea.

UPDATE: portsnap fetch update; cd graphics/drm-fbsd12.0-kmod; make deinstall; make clean; make install did fix the issue on my machine, thank you @nomadlogic and everyone for useful hints! Still updating client machines remotely seems risky.

[rozhuk-im]

I prefer to have ports:
drm-v5.0-fbsd12 - better works with 2200g
drm-v4.16-fbsd12 - better for notebook 2500u

To always have synced kernel and modules from ports I add to make.conf:

PORTS_MODULES+= graphics/gpu-firmware-kmod
PORTS_MODULES+= graphics/drm-fbsd12.0-kmod

and do not use binary updates.
To build modules in parallel use patch: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290
and always set -j, for example:

make -j`getconf NPROCESSORS_ONLN` -C /usr/src/ -s -DALWAYS_CHECK_MAKE buildworld
make -j`getconf NPROCESSORS_ONLN` -C /usr/src/ -s -DALWAYS_CHECK_MAKE buildkernel

[therontarigo]

Seems like kms-drm and the ports are all "works as intended" in this situation.

The real bugs are FreeBSD bugs in general:

• Kernel allows loading of modules which are incompatible because they were compiled for a different minor release. port bug, see comments below
• FreeBSD pkg tool and package build infrastructure is incapable of building kernel module ports for every minor release.

[cederom]

Good point @therontarigo I would slightly update it to:

• Kernel allows loading modules that are incompatible. Incompatibility not necessarily means a different minor release. DRM 12.0 builds and works fine both on 12.0 and 12.1. Some more general blacklist mechanism could be more versatile (i.e. DRM-12.0 is blacklisted on 12.1). In case of ports there is a variable that can tell if the current release is old/new enough.
• PKG should provide capability to build a package for a specific release using base and sources of that specific release. That would solve problem of 12.1 module being built on 12.0 that causes panic on 12.1. That would provide one package variant for 12.0 that was built on 12.0, or 12.1 built on 12.1.

I just wonder is it the right way to go. What other ports/packages/modules has this kind of issues? Maybe the issue is really with the current DRM implementation or design if it enforces such a big changes around not the otherwise? Maybe DRM design can be reconsidered rather than whole set of kernel changes for DRM? :-)

For me it looks like a problem that appeared in Linux 2.6.9 -> 2.6.10 when Kernel API started to change between each release. such a maintenance disaster. Please do not follow this path on FreeBSD.

[cederom]

Maybe this is the good place and time to ask this question: why DRM drivers provided with generic kernel distribution are different from DRM provided with Ports?

I can understand that Ports version provides development snapshots that can provide features preview integrated with system build automation.

But why then those features do not land in kernel master? Wouldn't that solve all mentioned build/distribution/maintenance problems? Users would get functional video drivers out of the box!

[emaste]

Yes, this is a system failure in the way we produce and release packages. I am sorry that this was not addressed prior to the release of 12.1.

Kernel allows loading of modules which are incompatible because they were compiled for a different minor release.

The kernel has infrastructure to only allow compatible modules to load; the issue here is that the module built on 12.0 claims that it is compatible with any 12.x. This is an issue with the port itself.

FreeBSD pkg tool and package build infrastructure is incapable of building kernel module ports for every minor release.

Yes, one way to solve this would be to have the concept of overlay repos or a hierarchy of repos, so that the kernel module package could come from a 12.0 or 12.1 repo as appropriate, while the rest of the packages would come from a shared repo as today.

[therontarigo]

why DRM drivers provided with generic kernel distribution are different from DRM provided with Ports?

Please search "drm2" in freebsd-current@ archives.

[rozhuk-im]

Not only DRM kernel modules in ports, there are vbox, netgraph, mac and other.
These things should not distribute as binary packages to avoid api/abi incapability and kernel panics as result.

[cederom]

I would just like you to reconsider those frequent Kernel API changes. It already resulted in some problems. This is a kind of solution that generates more new problems. Look at Windows or Linux what mess it is. No one really even cares anymore. I really love FreeBSD for its elegance, minimal ZEN like optimal simplicity and integrity. I am sure there are better ways to handle this situation rather than pushing towards avalanche of even more complexity.

[rozhuk-im]

I make drm-fbsd12-devel-kmod port for drm-v5.0-fbsd12.1 branch, if some one need this - can post to fbsd bug tracker.

[cederom]

pkg lock if your friend when you update the packages on 12.1 and you again get updated binary built on 12.0, that causes infinite crash and boot loop on a mounted working silesystems.

THIS MODULE IS A MESS AND SHOULD NOT BE PART OF A RELEASE NOR THIS MESSY LINUX APPROACH :-(

Sorry, but I have wasted several days on crashed production systems with this.

Looks like this is a bug tracker thread to follow: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241101

[costis-t]

Pretty sad situation. Wasted time too, for basically no reason.

[arrowd]

Just bumped into this problem too on my $WORK after upgrading a dozen of machines to 12.1. This is not a FreeBSD way, indeed.

[therontarigo]

This issue technically pertains only to the panic on startup, not to the various troubles around ports, pkgs, and supported releases. How close is anyone to fixing this panic bug?

If a kernel module cannot bail out while loading, shouldn't it instead disable itself such that no minor-release-specific code paths are entered, with a dmesg warning for why the mismatched module will do nothing?

[therontarigo]

Hi, I welcome feedback / rejection / enhancements for an approach to making sure packages are built for each minor release. https://reviews.freebsd.org/D23881

[m-ildefons]

Dear maintainers,

I think I ran into this issue too, when I upgraded my Laptop from 12.1-RELEASE to 12.2-RELEASE. I can't precisely tell because before an error message could appear, the text output got corrupted and I could no longer read my screen.
After much digging, rebooting the old kernel, disabling the i915kms driver, booting the new kernel, removing drm-kmod and drm-fbsd12.0-kmod packages via pkg and replacing them from ports did solve the issue and I could eventually complete my upgrade. But I am still quite frustrated with that experience, mostly because I had to spend way too much time reading vaguely related forum threads to find a solution in the first place. Then I came across other users on IRC who knew of this problem and reported that this had happened in previous releases as well.

The ideal solution for me would be to simply have the correct package available during upgrade, so this whole disable-uninstall-portsbuild dance can be avoided. If that is not possible right now, I'd still wish to hear about such issues in the release notes - especially since this was known beforehand.

Relevant System Specs:
Intel i5-4200U (Intel® HD Graphics 4400)
FreeBSD 12.1-RELEASE/FreeBSD 12.2-RELEASE

[ctipper]

I’ve searched the Internet for an explanation of why drm-current-kmod requires FreeBSD 13-CURRENT, the issue remains obscure, there’s no discussion on any mailing list I can find. I’ve had to remove this port because this is too annoying and my Ryzen 3000 is not supported, and never will be until I upgrade to FreeBSD 13. I am likely not going to do an in-place upgrade when the time comes, and but this at the moment is the main compelling reason to do so.

[cederom]

@ctipper just build it and install manually (do not use pkg binary) from the ports and it should work on 12 :-)

# cd /usr/ports/graphics/drm-fbsd12.0-kmod/
# make clean reinstall

If you work on AMD GPU you need amdgpu driver but this one crashes my RX580 station :-(
This may come handy: https://wiki.freebsd.org/Graphics/

13 will bring new features, it may work better, it will come up soon :-) You may even try the beta already: https://download.freebsd.org/ftp/releases/amd64/amd64/ISO-IMAGES/13.0/
https://www.freebsd.org/releases/13.0R/schedule/

[zeising]

@ctipper @cederom This discussion is off topic for this PR.
As has been suggested in this PR, if you are using STABLE, or upgrading from 12.1 to 12.2, you might need to build drm-fbsd12.0-kmod locally. This has been solved now if you are on FreeBSD 12.2 since package cluster is using FreeBSD 12.2 to build packages.
FreeBSD 13.0 (which is due to be released in March) carry a lot of improvements in the drm-kmod area, as well as an upgrade to Linux 5.4, which supports more recent Intel and AMD GPUs. It is currently in beta or RC stage, and I suggest you try it out. It is not possible to backport these changes to FreeBSD 12.
The reason drm-current-kmod needs FreeBSD current is because it uses lkpi functions that are only in current.
Closing this bug report.