Kernel panic on FreeBSD-CURRENT with loopback

[larseggert]

Trying to use netmap over loopback crashes the kernel under FreeBSD-CURRENT. Here is the console log:

414.499376 [ 389] nm_os_generic_find_num_desc called, in tx 1024 rx 1024
414.574290 [ 397] nm_os_generic_find_num_queues called, in txq 0 rxq 0
414.649158 [1252] generic_netmap_attach     Created generic NA 0xfffff80004643400 (prev 0)
414.915872 [ 448] generic_netmap_register   Generic adapter 0xfffff80004643400 goes on
415.005343 [ 495] generic_netmap_register   RX ring 0 of generic adapter 0xfffff80004643400 goes on
415.110371 [ 502] generic_netmap_register   TX ring 0 of generic adapter 0xfffff80004643400 goes on


Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address	= 0x0
fault code		= supervisor read instruction, page not present
instruction pointer	= 0x20:0x0
stack pointer	        = 0x28:0xfffffe0476ef11e8
frame pointer	        = 0x28:0xfffffe0476ef1220
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 2025 (test_warp)
[ thread pid 2025 tid 100180 ]
Stopped at      0
db> bt
Tracing pid 2025 tid 100180 td 0xfffff8001b4b1a00
??() at 0
drbr_enqueue_drv() at drbr_enqueue_drv+0x54c/frame 0xfffffe0476ef1220
nm_os_generic_xmit_frame() at nm_os_generic_xmit_frame+0x48/frame 0xfffffe0476ef1230
generic_netmap_attach() at generic_netmap_attach+0xe79/frame 0xfffffe0476ef1310
netmap_ioctl() at netmap_ioctl+0x752/frame 0xfffffe0476ef16f0
nm_os_selrecord() at nm_os_selrecord+0x1de/frame 0xfffffe0476ef1720
dev2udev() at dev2udev+0x2b40/frame 0xfffffe0476ef1770
VOP_IOCTL_APV() at VOP_IOCTL_APV+0x89/frame 0xfffffe0476ef17a0
sys_posix_fadvise() at sys_posix_fadvise+0x4f1/frame 0xfffffe0476ef18b0
dev2udev() at dev2udev+0x312f/frame 0xfffffe0476ef18d0
kern_ioctl() at kern_ioctl+0x208/frame 0xfffffe0476ef1940
sys_ioctl() at sys_ioctl+0x16e/frame 0xfffffe0476ef1a20
amd64_syscall() at amd64_syscall+0x31b/frame 0xfffffe0476ef1bb0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0476ef1bb0
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x800dc6ada, rsp = 0x7fffffffe198, rbp = 0x7fffffffe390 ---
db>

[vmaffione]

It is probably unrelated to the bug, but I see from the log you are not using the latest code. Please pull the latest HEAD.

[larseggert]

Done, but installworld fails at the moment for me. I'll try again tomorrow.

[larseggert]

Here is the console output from today's -CURRENT:

779.992274 [1251] generic_netmap_attach     Emulated adapter for lo0 created (prev was 0)
780.256451 [ 443] generic_netmap_register   Emulated adapter for lo0 activated
780.337612 [ 488] generic_netmap_register   Emulated adapter: ring 'lo0 RX0' activated
780.429114 [ 495] generic_netmap_register   Emulated adapter: ring 'lo0 TX0' activated


Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address	= 0x0
fault code		= supervisor read instruction, page not present
instruction pointer	= 0x20:0x0
stack pointer	        = 0x28:0xfffffe047700e238
frame pointer	        = 0x28:0xfffffe047700e270
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 2454 (test_warp)
[ thread pid 2454 tid 100238 ]
Stopped at      0
db> bt
Tracing pid 2454 tid 100238 td 0xfffff8001ba67000
??() at 0
drbr_enqueue_drv() at drbr_enqueue_drv+0x54c/frame 0xfffffe047700e270
nm_os_generic_xmit_frame() at nm_os_generic_xmit_frame+0x48/frame 0xfffffe047700e280
generic_netmap_attach() at generic_netmap_attach+0x4a9/frame 0xfffffe047700e360
netmap_ioctl() at netmap_ioctl+0x7c3/frame 0xfffffe047700e750
nm_os_selrecord() at nm_os_selrecord+0x1de/frame 0xfffffe047700e780
dev2udev() at dev2udev+0x2b40/frame 0xfffffe047700e7d0
VOP_IOCTL_APV() at VOP_IOCTL_APV+0x89/frame 0xfffffe047700e800
sys_posix_fadvise() at sys_posix_fadvise+0x4f1/frame 0xfffffe047700e910
dev2udev() at dev2udev+0x312f/frame 0xfffffe047700e930
kern_ioctl() at kern_ioctl+0x208/frame 0xfffffe047700e9a0
sys_ioctl() at sys_ioctl+0x16e/frame 0xfffffe047700ea80
amd64_syscall() at amd64_syscall+0x27e/frame 0xfffffe047700ebb0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe047700ebb0
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x800deb6da, rsp = 0x7fffffffe0f8, rbp = 0x7fffffffe2e0 ---

[vmaffione]

Netmap code in HEAD is slightly behind the github master, so you should update it.
I get a slightly different crash, but it seems a bug in the FreeBSD code (not in netmap). The ifp->if_start is sometimes called without checking if it's not NULL.

diff --git a/sys/net/if.c b/sys/net/if.c
index fa55662b423..e27772bf9ec 100644
--- a/sys/net/if.c
+++ b/sys/net/if.c
@@ -3626,8 +3626,9 @@ if_printf(struct ifnet *ifp, const char * fmt, ...)
 void
 if_start(struct ifnet *ifp)
 {
-
-       (*(ifp)->if_start)(ifp);
+       if (ifp->if_start) {
+               (*(ifp)->if_start)(ifp);
+       }
 }
 
 /*
@@ -3670,7 +3671,7 @@ if_handoff(struct ifqueue *ifq, struct mbuf *m, struct ifnet *ifp, int adjust)
        }
        _IF_ENQUEUE(ifq, m);
        IF_UNLOCK(ifq);
-       if (ifp != NULL && !active)
+       if (ifp != NULL && !active && ifp->if_start)
                (*(ifp)->if_start)(ifp);
        return (1);
 }

[larseggert]

Are there any instructions on how to compile FreeBSD with a newer netmap from this repo?

Also, are you committing the above fix to -CURRENT?

[vmaffione]

Just copy the content of sys/dev/netmap and sys/net/netmap from the git repo to the corresponding directories in the freebsd source tree (usually /usr/src/sys/dev/netmap and /usr/src/sys/net/netmap), and recompile FreeBSD following the instructions from the manual.

I'm not a FreeBSD committer, but I've informed one of the committer about this patch. I will also forward this to freebsd-net.

[hselasky]

Why not set a dummy if_start method instead. Saves code in the fast-path.

[vmaffione]

Yes, I agree that would be the best choice.

[bu7cher]

Loopback is not an ethernet interface, is netmap supposed to work with non-ethernet interfaces?

[vmaffione]

Netmap works on any software or hardware interface, by means of the "emulated netmap adapter".

[bu7cher]

Some interfaces may behave as ethernet device, but other doesn't. Loopback interface has not if_transmit or if_start method, frames that are passed trough loopback interface are not ethernet frames, they are not started with ether_header. So, the question is still the same, is netmap supposed to work with such mbufs?
Also, there are gif, ipfwlog, pflog, enc, and etc. pseudo interfaces, that have some specific purpose and that are not ethernet-compatibel, do you think they all should be supported by netmap?
IMHO, generic_netmap_attach() should check interface type and return EOPNOTSUPP error for such interfaces.

[vmaffione]

Yes, I agree. My point was avoiding the crash, and not about supporting loopback interface in netmap.
We can do the change, but is there a generic way to recognize that an ifp does not correspond to and ethernet interface?

[bu7cher]

For FreeBSD you can check that ifp->if_type == IFT_ETHER.

[larseggert]

FWIW, netmap works fine over loopback on Linux. And being able to test apps written against the netmap API over loopback is invaluable during development. I have no opinion whether netmap should support any of the other pseudo interfaces, but please don't prevent it from working over loopback.

[vmaffione]

yet another option would be to implement a built-in netmap loopback device (e.g. "netmap:lo"), which can be implemented to be as fast as netmap pipes.

[loos-br]

About the crash and in this specific case I think the right fix would be add a dummy if_transmit() to avoid adding the packet to interface queue and the call to if_start().

[loos-br]

A fix was committed to FreeBSD to avoid the use of the loopback and thus, avoid the crash.

[larseggert]

I'm not a fan of this resolution, but so be it. For consistency, you probably also want to prevent it on Linux.

[vmaffione]

@larseggert , why shouldn't you using a pair of pipes to achieve the same behaviour as a loopback device? If you are using two processes (e.g. one transmitting and one receiving), and so two nm_open()s, you can just open an end of a pipe in the first process and the other end in the other process. The pipe pair would basically behave as a loopback device.
Or maybe you are using a single process (and single nm_open())?

[larseggert]

I use both - one process during unit testing, and two during other (manual) testing. In the latter case, how do I open pipes?

[vmaffione]

For instance

  pleft = nm_open("pi{1", NULL, 0 , NULL);
  pright = nm_open("pi}1", NULL, 0 , NULL);

and then whatever you write into pleft can be read from pright and the other way around.