Only receive direction is captured by packet capture

[ghost]

Version: 17.1.6
URL: https://192.168.1.1/diag_packet_capture.php

When I use the packet capture it captured only the receive direction, but I need both directions.
Example:
20:03:24.451730 IP 192.168.1.4.16956 > 192.168.1.1.53: UDP, length 37
20:03:24.896679 IP 192.168.1.4.51763 > 192.168.1.1.53: UDP, length 44
20:03:27.903316 IP 192.168.1.4.26805 > 192.168.1.1.53: UDP, length 31

It's a bug or a feature? ;-)

[AdSchellevis]

@CyberC123 what filter's have you setup?

[ghost]

Interface LAN
Promiscuous unchecked
Address Family any
Protocol any
Host Address
Port 53
Packet Length
Count 100
Level of Detail full
Reverse DNS Lookup unchecked

[AdSchellevis]

maybe your traffic is returning via another path?
same kind of test on my end

[ghost]

The client is only on LAN interfaces connected. Which path can go otherwise?:(
Only on WAN interface both directions are recorded.

[AdSchellevis]

You'll probably have to investigate some more on your end, it's unlikely an issue with OPNsense (checked two boxes, both working fine).

[ghost]

okay, thanks for your quickly help.

[ghost]

When the IDS is running on the LAN interface, captured only the receive direction.
If I disable the IDS on LAN, the packet capture operates normal.

[AdSchellevis]

looks like #1913 is the same issue, likely a bug in netmap preventing packets from passing through in some scenario's.

[fichtner]

is this IPS mode on or off?

[ghost]

The IPS mode is on

[AdSchellevis]

Finally got a bit of time to look into this one, the issue lies in the netmap pcap support introduced in FreeBSD 11, just build a kernel with PCAP_SUPPORT_NETMAP=0 which works like it should.

We might consider turning netmap pcap support off in our standard kernel, although this has a performance penalty. I haven't measured anything yet, but I'm not sure if netmap+pcap and suricata can work in combination. @fichtner what do you think.

[fichtner]

Better to turn it off. Pcap speed is what it is. It shouldn’t be super fast to begin with. Also don’t want to taint this with netmap code paths for the obvious reason of not pulling in regressions like this one. How ironic. ;)

[AdSchellevis]

@fichtner I was thinking the same thing, need to do some testing, but will commit the config change when I'm ready.

[mimugmail]

I'd love to test this too, have many customer where I only see one direction when IPS is enabled, so not sure if IPS will stay untouched with this kernel change.

[AdSchellevis]

@mimugmail suricata with netmap should stay as it is, IDS mode might be slower (not sure yet)

[mimugmail]

But it's reproduceable that when Suricata is enabled, I can only see packets in one direction in tcpdump. As soon as I disable Suri I can see them ...

[fichtner]

IMO it’s a bug that when netmap was pulled into FreeBSD 11 this feature was automatically pulled in as well. 😔

[mimugmail]

Ok, just let me know when there's a kernel to test before public release. :)

[fichtner]

Monday. :)

[AdSchellevis]

I haven't tried 12 (yet), mainly because I really was under the impression that this did work at some point in time...
The X710/netmap likely is driver related, where the pcap issues seem to be generally broken, I wouldn't want to blame netmap specifically for that.

[AdSchellevis]

Tried the latest FreeBSD 12 kernel, but unfortunately netmap doesn't appear to be working at all there (using suricata).

[fichtner]

I suspect this will help opnsense/src@6fcf512a

Every native driver and the generic driver need to be fixed in order for this to work for everybody as expected. I'm testing re(4) tomorrow so this is more or less a POC, not a full fix.

[AdSchellevis]

that would be cool!

[fichtner]

Good and also a bit sobering news... The infamous re(4) driver (at least in its latest upstream version) does not have this defect. Simply could not reproduce. Traffic graphs show in both directions and packet capture does as well. Have IDS+IPS running on that interface.

So please, for the next reports from users, we need to know the NIC driver type. I can test and possibly fix em(4) and igb(4) when I have time.

[AdSchellevis]

@fichtner both em and igb suffer have the issue, I can easily test em from my dev vm.

[marjohn56]

Is this related?

https://forum.opnsense.org/index.php?topic=7666.msg45184#msg45184

[fichtner]

I'm not sure. Which NIC driver is this? The bug affects outgoing traffic visibility but shouldn't block packets from going out unless there is a second issue with the Netmap / Intel combination.

[mimugmail]

@marjohn56 was able to reproduce with IPS enabled, when only in IDS mode it works. Must be something related to netmap

[marjohn56]

Using Intel i211s on my Qotom. No idea what the forum user has. I'll run wireshark on it and see what gives.

[fichtner]

i211s is igb?

[marjohn56]

fink so.

[marjohn56]

OK, there are dhcp6 solicit packets going out. Need to rig up another ethernet port to see what's happing... another report soon..

[marjohn56]

Well dhcp6c packets go out according to wireshark, and they are responded to. Internal packet capture shows no dhcp6c at all, in either direction. dhcpd logs show dhcp6c running and sending solicits.

[mimugmail]

Thats exactly the issue, its not displayed via tcpdump, but in general it's working

[marjohn56]

Except that it isn't, well not dhcp6c anyway. This is even more bizarre than it first appears. If IPS is turned off then the dhcp6 server responds with a REQUEST as expected, if it's turned on, then it does not respond at all, yet the solicit packets look identical to me.

[marjohn56]

Here are the packet captures, just filtered on dhcp6

packets.zip

[mimugmail]

Stupid question, any rules hitting? :)

[marjohn56]

No rules at all. As I said, this is a packet capture on the WAN, with IPS off the server responds, with it on, it doesn't... go figure, I'm baffled... doesn't take much though. Oh I should add these are wireshark captures.... not internal.

[marjohn56]

Let me just give the setup an method used to capture these. packet capture is directly on the switch, so I am seeing the traffic from the whole LAN side of my network, hence the filter for dhcpv6 only. Process is unset IPS, kill dhcp6c and then start capture, we see the solicit and the request respons. Now turn on IPS, kill dhcp6c and restart the capture and restart dhcp6c, we see the solicits, we never see a request response.

[fichtner]

do you capture on the box or in front? maybe the server answers but suricata discards the response silently. that was always the suspicion in earlier reports regarding "IPv6 not working with IPS"

[marjohn56]

Watch my lips..." Oh I should add these are wireshark captures.... not internal." :)

[fichtner]

I am inclined to believe you :)

[marjohn56]

However, I see no reason why the server should respond when IPS is off and ignore it when IPS is on. Perhaps someone else can verify this to prove I'm not completely barking.

[fichtner]

Looks like this is it... https://reviews.freebsd.org/D17896

[AdSchellevis]

tested, it is.

[marjohn56]

Nice one Ad...