systemd-network segfault

[tavianator]

systemd version the issue has been seen with

255.4

Used distribution

Arch Linux

Linux kernel version used

6.8.4-arch1-1

CPU architectures issue was seen on

x86_64

Component

systemd-networkd

Expected behaviour you didn't see

No segfaults

Unexpected behaviour you saw

I was trying out different qdisc settings earlier today. Just now I noticed a systemd-network segfault in my logs. coredumpctl gdb shows it's an infinite recursion between tclass_drop() and qdisc_drop():

# coredumpctl gdb
Core was generated by `/usr/lib/systemd/systemd-networkd'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  qdisc_drop.isra.0 (qdisc=0x562c5f558e60) at ../systemd-stable/src/network/tc/qdisc.c:289
289             TClass *tclass;                                                                                                                                                              
(gdb) frame 1
#1  0x0000562c5e33b2fb in tclass_drop.isra.0 (tclass=0x562c5f56eb80) at ../systemd-stable/src/network/tc/tclass.c:268
268                     qdisc_drop(qdisc);                                                                                                                                                   
(gdb) frame 2
#2  0x0000562c5e33b19e in qdisc_drop.isra.0 (qdisc=0x562c5f558e60) at ../systemd-stable/src/network/tc/qdisc.c:301
301                     tclass_drop(tclass);
(gdb) bt
#0  qdisc_drop.isra.0 (qdisc=0x562c5f558e60) at ../systemd-stable/src/network/tc/qdisc.c:289
#1  0x0000562c5e33b2fb in tclass_drop.isra.0 (tclass=0x562c5f56eb80) at ../systemd-stable/src/network/tc/tclass.c:268
#2  0x0000562c5e33b19e in qdisc_drop.isra.0 (qdisc=0x562c5f558e60) at ../systemd-stable/src/network/tc/qdisc.c:301
#3  0x0000562c5e33b2fb in tclass_drop.isra.0 (tclass=0x562c5f56eb80) at ../systemd-stable/src/network/tc/tclass.c:268
#4  0x0000562c5e33b19e in qdisc_drop.isra.0 (qdisc=0x562c5f558e60) at ../systemd-stable/src/network/tc/qdisc.c:301
#5  0x0000562c5e33b2fb in tclass_drop.isra.0 (tclass=0x562c5f56eb80) at ../systemd-stable/src/network/tc/tclass.c:268
#6  0x0000562c5e33b19e in qdisc_drop.isra.0 (qdisc=0x562c5f558e60) at ../systemd-stable/src/network/tc/qdisc.c:301
#7  0x0000562c5e33b2fb in tclass_drop.isra.0 (tclass=0x562c5f56eb80) at ../systemd-stable/src/network/tc/tclass.c:268
#8  0x0000562c5e33b19e in qdisc_drop.isra.0 (qdisc=0x562c5f558e60) at ../systemd-stable/src/network/tc/qdisc.c:301
#9  0x0000562c5e33b2fb in tclass_drop.isra.0 (tclass=0x562c5f56eb80) at ../systemd-stable/src/network/tc/tclass.c:268
#10 0x0000562c5e33b19e in qdisc_drop.isra.0 (qdisc=0x562c5f558e60) at ../systemd-stable/src/network/tc/qdisc.c:301
...
#87353 0x0000562c5e33b2fb in tclass_drop.isra.0 (tclass=0x562c5f56eb80) at ../systemd-stable/src/network/tc/tclass.c:268
#87354 0x0000562c5e33b19e in qdisc_drop.isra.0 (qdisc=0x562c5f556a60) at ../systemd-stable/src/network/tc/qdisc.c:301
#87355 0x0000562c5e29502f in manager_rtnl_process_qdisc (rtnl=<optimized out>, message=<optimized out>, m=0x562c5f542f70) at ../systemd-stable/src/network/tc/qdisc.c:549
#87356 0x00007f7026296643 in process_match (m=0x562c5f59ef90, nl=0x562c5f544f70) at ../systemd-stable/src/libsystemd/sd-netlink/sd-netlink.c:314
#87357 process_running (ret=0x0, nl=0x562c5f544f70) at ../systemd-stable/src/libsystemd/sd-netlink/sd-netlink.c:344
#87358 sd_netlink_process (nl=0x562c5f544f70, ret=ret@entry=0x0) at ../systemd-stable/src/libsystemd/sd-netlink/sd-netlink.c:374
#87359 0x00007f7026296ba8 in io_callback (s=<optimized out>, fd=<optimized out>, revents=<optimized out>, userdata=<optimized out>)
    at ../systemd-stable/src/libsystemd/sd-netlink/sd-netlink.c:655
#87360 0x00007f70262cfa11 in source_dispatch (s=s@entry=0x562c5f5450c0) at ../systemd-stable/src/libsystemd/sd-event/sd-event.c:4187
#87361 0x00007f70262cfd4e in sd_event_dispatch (e=<optimized out>, e@entry=0x562c5f5432b0) at ../systemd-stable/src/libsystemd/sd-event/sd-event.c:4808
#87362 0x00007f70262d1849 in sd_event_run (e=<optimized out>, e@entry=0x562c5f5432b0, timeout=timeout@entry=18446744073709551615)
    at ../systemd-stable/src/libsystemd/sd-event/sd-event.c:4869
#87363 0x00007f70262d1a20 in sd_event_loop (e=<optimized out>) at ../systemd-stable/src/libsystemd/sd-event/sd-event.c:4891
#87364 0x0000562c5e25eaf4 in run (argv=<optimized out>, argc=<optimized out>) at ../systemd-stable/src/network/networkd.c:112
#87365 main (argc=<optimized out>, argv=<optimized out>) at ../systemd-stable/src/network/networkd.c:119

Steps to reproduce the problem

I haven't been able to reproduce it since, but the commands I was running at the time were like this:

# tc qdisc replace dev eth0 root cake
# tc qdisc replace dev eth0 root cake ethernet bandwidth 1gbit
# tc qdisc replace dev eth0 root fq
# tc qdisc replace dev eth0 root fq pacing

Additional program output to the terminal or log subsystem illustrating the issue

Apr 12 08:27:36 tachyon kernel: systemd-network[2201]: segfault at 7fff46098ff8 ip 0000562c5e33b12e sp 00007fff46098ff0 error 6 in systemd-networkd[562c5e259000+e4000] likely on CPU 16 (co>
Apr 12 08:27:36 tachyon kernel: Code: 66 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 41 55 41 54 55 53 48 83 ec 38 64 48 8b 04 25 28 00 00 00 48 89 44 24 28 31 c0 <48> c7 44 24 08 00 00 0>
Apr 12 08:27:36 tachyon systemd[1]: Created slice Slice /system/drkonqi-coredump-processor.
Apr 12 08:27:36 tachyon systemd[1]: Created slice Slice /system/systemd-coredump.
Apr 12 08:27:36 tachyon systemd[1]: Started Process Core Dump (PID 162006/UID 0).
Apr 12 08:27:36 tachyon systemd[1]: Started Pass systemd-coredump journal entries to relevant user for potential DrKonqi handling.
Apr 12 08:27:36 tachyon tailscaled[2680]: Accept: TCP{100.116.220.124:32934 > 100.107.249.85:22} 88 tcp non-syn
Apr 12 08:27:37 tachyon systemd-coredump[162008]: [🡕] Process 2201 (systemd-network) of user 981 dumped core.
                                                  
                                                  Stack trace of thread 2201:
                                                  #0  0x0000562c5e33b12e n/a (systemd-networkd + 0xf112e)
                                                  #1  0x0000562c5e33b2fb n/a (systemd-networkd + 0xf12fb)
                                                  #2  0x0000562c5e33b19e n/a (systemd-networkd + 0xf119e)
                                                  #3  0x0000562c5e33b2fb n/a (systemd-networkd + 0xf12fb)
                                                  #4  0x0000562c5e33b19e n/a (systemd-networkd + 0xf119e)
...

[evverx]

It's probably a duplicate of #32247

[tavianator]

It's probably a duplicate of #32247

Oh it must be! I didn't find that in a search because I looked for "systemd-network segfault", not systemd-networkd. Oops.

[yuwata]

Thank you for the detailed report. But yeah, #32247 is faster. Let's link this from #32247, and close this.