Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Only receive direction is captured by packet capture #1632

Closed
ghost opened this issue May 9, 2017 · 53 comments

Comments

Projects
None yet
4 participants
@ghost
Copy link

commented May 9, 2017

Version: 17.1.6
URL: https://192.168.1.1/diag_packet_capture.php

When I use the packet capture it captured only the receive direction, but I need both directions.
Example:
20:03:24.451730 IP 192.168.1.4.16956 > 192.168.1.1.53: UDP, length 37
20:03:24.896679 IP 192.168.1.4.51763 > 192.168.1.1.53: UDP, length 44
20:03:27.903316 IP 192.168.1.4.26805 > 192.168.1.1.53: UDP, length 31

It's a bug or a feature? ;-)

@AdSchellevis

This comment has been minimized.

Copy link
Member

commented May 9, 2017

@CyberC123 what filter's have you setup?

@ghost

This comment has been minimized.

Copy link
Author

commented May 9, 2017

Interface LAN
Promiscuous unchecked
Address Family any
Protocol any
Host Address
Port 53
Packet Length
Count 100
Level of Detail full
Reverse DNS Lookup unchecked

@AdSchellevis

This comment has been minimized.

Copy link
Member

commented May 9, 2017

maybe your traffic is returning via another path?
same kind of test on my end
image

@ghost

This comment has been minimized.

Copy link
Author

commented May 9, 2017

The client is only on LAN interfaces connected. Which path can go otherwise?:(
Only on WAN interface both directions are recorded.

@AdSchellevis

This comment has been minimized.

Copy link
Member

commented May 9, 2017

You'll probably have to investigate some more on your end, it's unlikely an issue with OPNsense (checked two boxes, both working fine).

@ghost

This comment has been minimized.

Copy link
Author

commented May 9, 2017

okay, thanks for your quickly help.

@ghost ghost closed this May 9, 2017

@ghost

This comment has been minimized.

Copy link
Author

commented Mar 18, 2018

When the IDS is running on the LAN interface, captured only the receive direction.
If I disable the IDS on LAN, the packet capture operates normal.

@ghost ghost reopened this Mar 18, 2018

@AdSchellevis

This comment has been minimized.

Copy link
Member

commented Mar 25, 2018

looks like #1913 is the same issue, likely a bug in netmap preventing packets from passing through in some scenario's.

@fichtner fichtner added the upstream label Mar 25, 2018

@fichtner

This comment has been minimized.

Copy link
Member

commented Mar 25, 2018

is this IPS mode on or off?

@fichtner fichtner added the support label Mar 25, 2018

@ghost

This comment has been minimized.

Copy link
Author

commented Mar 26, 2018

The IPS mode is on

@AdSchellevis

This comment has been minimized.

Copy link
Member

commented Aug 24, 2018

Finally got a bit of time to look into this one, the issue lies in the netmap pcap support introduced in FreeBSD 11, just build a kernel with PCAP_SUPPORT_NETMAP=0 which works like it should.

We might consider turning netmap pcap support off in our standard kernel, although this has a performance penalty. I haven't measured anything yet, but I'm not sure if netmap+pcap and suricata can work in combination. @fichtner what do you think.

@fichtner

This comment has been minimized.

Copy link
Member

commented Aug 24, 2018

Better to turn it off. Pcap speed is what it is. It shouldn’t be super fast to begin with. Also don’t want to taint this with netmap code paths for the obvious reason of not pulling in regressions like this one. How ironic. ;)

@fichtner fichtner removed the support label Aug 24, 2018

@fichtner fichtner self-assigned this Aug 24, 2018

@fichtner fichtner added this to the 19.1 milestone Aug 24, 2018

@AdSchellevis

This comment has been minimized.

Copy link
Member

commented Aug 25, 2018

@fichtner I was thinking the same thing, need to do some testing, but will commit the config change when I'm ready.

@mimugmail

This comment has been minimized.

Copy link
Member

commented Aug 28, 2018

I'd love to test this too, have many customer where I only see one direction when IPS is enabled, so not sure if IPS will stay untouched with this kernel change.

@AdSchellevis

This comment has been minimized.

Copy link
Member

commented Aug 28, 2018

@mimugmail suricata with netmap should stay as it is, IDS mode might be slower (not sure yet)

@mimugmail

This comment has been minimized.

Copy link
Member

commented Aug 28, 2018

But it's reproduceable that when Suricata is enabled, I can only see packets in one direction in tcpdump. As soon as I disable Suri I can see them ...

@fichtner

This comment has been minimized.

Copy link
Member

commented Aug 28, 2018

IMO it’s a bug that when netmap was pulled into FreeBSD 11 this feature was automatically pulled in as well. 😔

@mimugmail

This comment has been minimized.

Copy link
Member

commented Aug 28, 2018

Ok, just let me know when there's a kernel to test before public release. :)

@fichtner

This comment has been minimized.

Copy link
Member

commented Aug 28, 2018

Monday. :)

@AdSchellevis

This comment has been minimized.

Copy link
Member

commented Sep 2, 2018

Tried the latest FreeBSD 12 kernel, but unfortunately netmap doesn't appear to be working at all there (using suricata).

fichtner added a commit to opnsense/src that referenced this issue Sep 22, 2018

@fichtner

This comment has been minimized.

Copy link
Member

commented Sep 22, 2018

I suspect this will help opnsense/src@6fcf512

Every native driver and the generic driver need to be fixed in order for this to work for everybody as expected. I'm testing re(4) tomorrow so this is more or less a POC, not a full fix.

@AdSchellevis

This comment has been minimized.

Copy link
Member

commented Sep 23, 2018

that would be cool!

@fichtner

This comment has been minimized.

Copy link
Member

commented Sep 23, 2018

Good and also a bit sobering news... The infamous re(4) driver (at least in its latest upstream version) does not have this defect. Simply could not reproduce. Traffic graphs show in both directions and packet capture does as well. Have IDS+IPS running on that interface.

So please, for the next reports from users, we need to know the NIC driver type. I can test and possibly fix em(4) and igb(4) when I have time.

@AdSchellevis

This comment has been minimized.

Copy link
Member

commented Sep 23, 2018

@fichtner both em and igb suffer have the issue, I can easily test em from my dev vm.

@marjohn56

This comment has been minimized.

Copy link
Member

commented Oct 11, 2018

@fichtner

This comment has been minimized.

Copy link
Member

commented Oct 11, 2018

I'm not sure. Which NIC driver is this? The bug affects outgoing traffic visibility but shouldn't block packets from going out unless there is a second issue with the Netmap / Intel combination.

@mimugmail

This comment has been minimized.

Copy link
Member

commented Oct 11, 2018

@marjohn56 was able to reproduce with IPS enabled, when only in IDS mode it works. Must be something related to netmap

@marjohn56

This comment has been minimized.

Copy link
Member

commented Oct 11, 2018

Using Intel i211s on my Qotom. No idea what the forum user has. I'll run wireshark on it and see what gives.

@fichtner

This comment has been minimized.

Copy link
Member

commented Oct 11, 2018

i211s is igb?

@marjohn56

This comment has been minimized.

Copy link
Member

commented Oct 11, 2018

fink so.

@marjohn56

This comment has been minimized.

Copy link
Member

commented Oct 11, 2018

OK, there are dhcp6 solicit packets going out. Need to rig up another ethernet port to see what's happing... another report soon..

@marjohn56

This comment has been minimized.

Copy link
Member

commented Oct 11, 2018

Well dhcp6c packets go out according to wireshark, and they are responded to. Internal packet capture shows no dhcp6c at all, in either direction. dhcpd logs show dhcp6c running and sending solicits.

@mimugmail

This comment has been minimized.

Copy link
Member

commented Oct 11, 2018

Thats exactly the issue, its not displayed via tcpdump, but in general it's working

@marjohn56

This comment has been minimized.

Copy link
Member

commented Oct 11, 2018

Except that it isn't, well not dhcp6c anyway. This is even more bizarre than it first appears. If IPS is turned off then the dhcp6 server responds with a REQUEST as expected, if it's turned on, then it does not respond at all, yet the solicit packets look identical to me.

@marjohn56

This comment has been minimized.

Copy link
Member

commented Oct 11, 2018

Here are the packet captures, just filtered on dhcp6

packets.zip

@mimugmail

This comment has been minimized.

Copy link
Member

commented Oct 11, 2018

Stupid question, any rules hitting? :)

@marjohn56

This comment has been minimized.

Copy link
Member

commented Oct 11, 2018

No rules at all. As I said, this is a packet capture on the WAN, with IPS off the server responds, with it on, it doesn't... go figure, I'm baffled... doesn't take much though. Oh I should add these are wireshark captures.... not internal.

@marjohn56

This comment has been minimized.

Copy link
Member

commented Oct 11, 2018

Let me just give the setup an method used to capture these. packet capture is directly on the switch, so I am seeing the traffic from the whole LAN side of my network, hence the filter for dhcpv6 only. Process is unset IPS, kill dhcp6c and then start capture, we see the solicit and the request respons. Now turn on IPS, kill dhcp6c and restart the capture and restart dhcp6c, we see the solicits, we never see a request response.

@fichtner

This comment has been minimized.

Copy link
Member

commented Oct 11, 2018

do you capture on the box or in front? maybe the server answers but suricata discards the response silently. that was always the suspicion in earlier reports regarding "IPv6 not working with IPS"

@marjohn56

This comment has been minimized.

Copy link
Member

commented Oct 11, 2018

Watch my lips..." Oh I should add these are wireshark captures.... not internal." :)

@fichtner

This comment has been minimized.

Copy link
Member

commented Oct 11, 2018

I am inclined to believe you :)

@marjohn56

This comment has been minimized.

Copy link
Member

commented Oct 11, 2018

However, I see no reason why the server should respond when IPS is off and ignore it when IPS is on. Perhaps someone else can verify this to prove I'm not completely barking.

@fichtner

This comment has been minimized.

Copy link
Member

commented Dec 11, 2018

Looks like this is it... https://reviews.freebsd.org/D17896

@AdSchellevis

This comment has been minimized.

Copy link
Member

commented Dec 11, 2018

tested, it is.

@marjohn56

This comment has been minimized.

Copy link
Member

commented Dec 12, 2018

Nice one Ad...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.