Traffic graph does not show the bandwidth of IPS enabled interfaces

[JasMan78]

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

[x] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md

[x] I have searched the existing issues and I'm convinced that mine is new.

Describe the bug
The "Traffic Graph" does not show the current bandwidth of physical interfaces, on which IPS is enabled and no VLANs are configured. The table below the graph shows the correct bandwitdh.
As soon as I disable IPS (not IDS) or remove the affected interface from the IDS/IPS settings, the bandwidth is shown.

Physical interfaces, which have additional VLANs configured, are both shown correctly.

This happens since I've switched to a new hardware appliance with Intel network interfaces.
On my old hardware appliance with Realtek interfaces and the same IDS/IPS and VLAN configuration, the graph shows the bandwidth for all interfaces correctly.

I've found another user in the forum which is having the same issue (3): https://forum.opnsense.org/index.php?topic=18496.msg84177#msg84177

To Reproduce
Steps to reproduce the behavior:

1. Go to "Services: Intrusion Detection: Administration"
2. Enable IDS and IPS.
3. Add a physical interface which has no VLANs configured
4. Switch to the "Traffic graph"
5. As soon as all IDS/IPS rules are loaded, the bandwidth in the graph for the interface drops down to zero
6. The table at the bottom shows the correct current bandwidth

Expected behavior
No idea :)

Screenshots

Relevant log files
Please let me know if you need a log file to identify the issue.

Environment
OPNsense 20.7.1-amd64
Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz (4 cores)/NRG IPU662
Networ Interfaces: Intel i211AT

[AdSchellevis]

Best try the test kernel when it comes out (https://forum.opnsense.org/index.php?topic=17363.msg84952#msg84952), although I'm not sure if this issue already has an upstream fix. Intel 211 doesn't seem to obey the bpf zerocopy toggle (https://docs.opnsense.org/troubleshooting/network.html). We're not using i211 ourselves, i210 works like a charm.

[JasMan78]

I tried the test kernel but unfortunately the behaviour was the same. The graph still not shown the bandwidth as long as IPS is active.

root@JaWall:~ # uname -a
FreeBSD JasWall.mgmt.home.arpa 12.1-RELEASE-p8-HBSD FreeBSD 12.1-RELEASE-p8-HBSD opnsense/core#3 87f253a(master)-dirty: Sat Aug 15 09:29:08 PDT 2020 root@bsd12_openssl:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64

[JasMan78]

Same with OPNsense 20.7.3-amd64.

[AdSchellevis]

tried the new netmap test kernel ? (https://forum.opnsense.org/index.php?topic=19175.msg88695#msg88695)

[JasMan78]

@AdSchellevis Ooops, sorry. I didn't get an info about your answer/request.

I've installed the 20.7.4 update today, still no WAN traffic shown in the traffic graph.

[AdSchellevis]

@JasMan78 hmm, this is odd, I really thought I tested the graph with an i210 and zerocopy_enable (which already worked), but it seems that the new netmap kernel broke bpf in this case as well now. I probably didn't wait long enough for suricata to startup, maybe @muratbalaban43 has an idea, I think they where also looking into this subject.

[muratbalaban43]

@AdSchellevis , you're right. We are on this. Looks like netmap comes into the scene earlier than bpf.

[AdSchellevis]

@muratbalaban43 it's odd, using an earlier kernel I can't seem to get a functional bpf at all anymore (on my intel i210), so either my initial tests where flawed or I'm overlooking something completely different.

[muratbalaban43]

@AdSchellevis thanks for the update. Hopefully, I'll have some news next week.

[ghost235]

Same with e1000 AND vmxnet3 on VMWare 6.5.0 with 20.7.4
EDIT: on a ohter box with vmxnet3 and 20.1.9_1 on problem

[gitterdoneplease]

Seeing this exact behavior with em drivers too. Intel 82574L

[swhite2]

Please see #95 for a fix to this issue. Keep in mind that if a driver is NOT integrated with iflib, the driver is responsible for supplying its statistics to the ifnet structure. Based on this, they may or may not cause the traffic graph to flatline.

All drivers mentioned in this issue are integrated with iflib, as such they should now report traffic in IPS mode.

Tested and confirmed on e1000.

EDIT:
Ideally this change should be upstreamed, though a patch should suffice for now in my opinion.

[fichtner]

There is still some sort of issue here, reopen briefly to reassess https://forum.opnsense.org/index.php?topic=21168.0

[swhite2]

There is still some sort of issue here, reopen briefly to reassess https://forum.opnsense.org/index.php?topic=21168.0

The fix related to the traffic graph in IPS-mode was related to Netmap-specific code in iflib. Users reporting new missing traffic counters (like the link mentioned above) in any interface mode should specify the NIC/driver in question, as many of them are not integrated with iflib. As such, the missing counters are often caused by developers not implementing the FreeBSD-specific counter mechanism. Whether this is true in all cases is unclear, as it is quite difficult to reproduce with the limited set of hardware available to me. Users should at least check if netstat -idb [interface] or ifinfo [interface] reports any in/out-bytes/packets. Should no bytes/packets be reported, the issue will be driver-related.

[Famaku]

Using Mellenox ConnectX-3 NICs (driver: mlx4en) this issue still exists. Oddly enough, this only affects setups in which VLANs are defined on virtual functions (i.e. VFs, see https://docs.google.com/spreadsheets/d/1RVj8K3XOzWi-Bkjq6hUxWudu7Cxd8FFTqjLiBMzZWEM/edit#gid=0). When VLANs are assigned in OPNSense (which has it's drawbacks compared to VFs), it works as intended. It would be great if anyone could have a look at this.

[fichtner]

mlx4en doesn't support native netmap which suggest the generic netmap driver does not support packet/byte accounting as well. At this point the accounting patch for iflib/netmap combination was added in FreeBSD not long ago, but I'm not sure if it should be our effort to address this other issue in the OS. Maybe @muratbalaban43 and team can help out on this one.

Cheers,
Franco

[Famaku]

At this point the accounting patch for iflib/netmap combination was added in FreeBSD not long ago, but I'm not sure if it should be our effort to address this other issue in the OS.

So you're saying there's already a patch in the pipeline within FreeBSD? Just to clarify.
I'm at a loss how netmap gets broken by not adding VLANs in OPNSense, when VLANs are already configured on the actual NIC (i.e. on the Virtual Function level).

[fichtner]

The patch for this closed ticket’s problem is in FreeBSD now, yes.

It’s not so complicated really. if_vlan is a clean implementation with an if_transmit hook netmap can grab, but if_vlan will still account for packets. For bare Ethernet drivers that doesn’t work because netmap assumes it’s always accounted outside of it’s scope.

[muratbalaban43]

@fichtner @Famaku , we were looking to see if there could be another way to pass packets to netmap independent of the interface driver. Ideally, it can be an evolved version of the netmap emulated/generic driver. Not so much progress right now; but we'll revisit this in the coming year.