Panic when running sysctl sys.device.drmn0.pcie_replay_count with amdgpu

[NorwegianRockCat]

There seems to be a panic when running the sysctl device.drmn0.pcie_replay_count with the amdgpu loaded (a Navi10 card).

The dump does not look very helpful:

panic: page fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 23; apic id = 17
fault virtual address   = 0x0
fault code              = supervisor read instruction, page not present
instruction pointer     = 0x20:0x0
stack pointer           = 0x28:0xfffffe00f834e7f8
frame pointer           = 0x28:0xfffffe00f834e810
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 3330 (sysctl)
trap number             = 12
panic: page fault
cpuid = 23
time = 1595050508
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00f834e4b0
vpanic() at vpanic+0x182/frame 0xfffffe00f834e500
panic() at panic+0x43/frame 0xfffffe00f834e560
trap_fatal() at trap_fatal+0x387/frame 0xfffffe00f834e5c0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00f834e610
trap() at trap+0x271/frame 0xfffffe00f834e720
calltrap() at calltrap+0x8/frame 0xfffffe00f834e720
--- trap 0xc, rip = 0, rsp = 0xfffffe00f834e7f8, rbp = 0xfffffe00f834e810 ---
??() at 0/frame 0xfffffe00f834e810
sysctl_handle_attr() at sysctl_handle_attr+0x70/frame 0xfffffe00f834e860
sysctl_root_handler_locked() at sysctl_root_handler_locked+0x91/frame 0xfffffe00f834e8b0
sysctl_root() at sysctl_root+0x249/frame 0xfffffe00f834e930
userland_sysctl() at userland_sysctl+0x173/frame 0xfffffe00f834e9e0
sys___sysctl() at sys___sysctl+0x5f/frame 0xfffffe00f834ea90
amd64_syscall() at amd64_syscall+0x119/frame 0xfffffe00f834ebb0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00f834ebb0
--- syscall (202, FreeBSD ELF64, sys___sysctl), rip = 0x800414caa, rsp = 0x7fffffffc3b8, rbp = 0x7fffffffc3f0 ---
KDB: enter: panic

I put the core.txt up on pastebin if more information is wanted.

[evadot]

I cannot reproduce on my amd machine.
Does that happens everytime for you ?
Do you have other kernel modules that could cause this ?

[NorwegianRockCat]

It seems to happen every time I tried this. This core dump was done right after I loaded the system. I typed sysctl -a to see that it worked.

then

kldload amdgpu

then sysctl -a and the system crashed.

Otherwise, cpuctl and amdtemp are the only other modules I loaded explicitly. I think the core dump might show what other required modules were loaded.

I am away from the system until Friday, but I can try and investigate a little more when I am back.

[NorwegianRockCat]

Did a new rebuild on the system and it still happens when amdgpu is loaded. I'll see if I can dig more.

Perusing the source code I did find this interesting comment in amdgpu_pm.c:418:

   /* sysctl -a can panic if this data is uninitialized */
    memset(&data, 0, sizeof(struct pp_states_info));

So, I wonder if there is some other Navi 10-specific structures that might have a similar issue? Any suggestions where one could start looking? Are all snprintf() calls likely being fed to sysctl?

[NorwegianRockCat]

I had a little bit of time and read the sysctl manpage. It seems that oid that causes the problem is sys.device.drmn0.pcie_replay_count. I've updated the bug report to reflect that information.

[DarkKirb]

i poked around with kgdb a bit and it appears that https://github.com/freebsd/drm-kmod/blob/master/drivers/gpu/drm/amd/amdgpu/nv.c#L564-L582 nv_asic_funcs is not fully defined, namely it is missing the fields:

• int (*get_pcie_lanes)(struct amdgpu_device *adev);
• void (*set_pcie_lanes)(struct amdgpu_device *adev, int lanes);
• uint64_t (*get_pcie_replay_count)(struct amdgpu_device *adev);

This bug affects Navi GPUs (the rx 5xxx series). The other instantiations of the amdgpu_asic_funcs structure contain the get_pcie_replay count field, meaning it is not reproducable there.

[NorwegianRockCat]

Great sleuthing @DarkKirb! Thanks for tracking this down! I've been meaning to track this more, but my current machine with the Navi card is currently packed away waiting for a move.

[NorwegianRockCat]

Hi @evadot, this problem is still present in 13-STABLE (and -BETA I imagine).

You mentioned that you would cherry-pick torvalds/linux@2af8153 from 5.5 to 5.4-lts (ref #37) . The cherry-pick indeed solves the problem.

It would be nice to have this working for 13-RELEASE as sysctl -a |grep … is a pattern that shows up from time to time in some of my workflows.

Can you do the cherry-pick, please?

[evadot]

Mhm, I somehow was sure that I did cherry-picked the patch but it seems not ...
Anyway, it's done now.
I have a few other stuff to include before I cut a new release but if you could confirm that building form the 5.4-lts branch fixes the issue for you.
Thanks.

[NorwegianRockCat]

I already have that patch applied locally to the 5.4-lts branch, and I can confirm that it does fix the problem.

I thought you had applied the patch too, but when I looked at the code, I couldn't see it. So hence my nudge here, glad it was finally dealt with before the 13.0-RELEASE.

I'll let you do the honors of closing the issue :-).