Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Npcap strange behavior with WireShark #334

Closed
jeffrimko opened this issue Oct 17, 2019 · 23 comments
Closed

Npcap strange behavior with WireShark #334

jeffrimko opened this issue Oct 17, 2019 · 23 comments

Comments

@jeffrimko
Copy link

@jeffrimko jeffrimko commented Oct 17, 2019

Been seeing strange behavior with Wireshark and the Npcap Loopback Adapater. After starting a Wireshark capture on that interface, websites no longer load in a browser. For example, trying to browse to Wikipedia might result in a "Can’t reach this page" in Microsoft Edge or a ERR_CONNECTION_ABORTED/ERR_CONNECTION_RESET in Chrome. This issue will continue after the capture has stopped until I manually stop Npcap in a terminal using "net stop npcap".

It's possible that just having Npcap running is the issue but the issue seems to consistently happen after starting a Wireshark capture.

I added a screen capture of the issue here: https://youtu.be/eUuBBvwRU0g

Windows 10 version 1803
Wireshark version 3.0.5 (v3.0.5-0-g752a55954770)
Npcap version 0.9983
DiagReport-20191017-164533.txt

@dmiller-nmap
Copy link
Contributor

@dmiller-nmap dmiller-nmap commented Oct 23, 2019

Since you have Npcap 0.9983 installed, you no longer need the Npcap Loopback Adapter in order to do loopback capture. This dummy adapter was needed in previous releases, but had an unfortunate side effect of causing routing problems especially on systems with multiple network adapters or VPNs. The only reason it is left in is for legacy support for certain programs that need it, primarily Nmap 7.80 and earlier. Try reinstalling Npcap and un-checking the "Legacy loopback" installation option. If this does not solve the problem, we will see what remains to be done.

@jeffrimko
Copy link
Author

@jeffrimko jeffrimko commented Oct 24, 2019

Thanks for the response! I tried the following but am still seeing the issue:

  • Uninstalled Nmap 7.7.
  • Uninstalled Npcap 0.9983.
  • Restarted PC.
  • Confirmed that Wireshark captures on other interfaces did not show the reported weird behavior.
  • Reinstalled Npcap 0.9983 (left all 4 installation option boxes unchecked, including the legacy loopback support box).
  • Restarted PC.
  • Tried Wireshark capture on "Adapter for loopback traffic capture".

A few observations:

  • When Npcap is installed, Wireshark will not properly load all of the capture interfaces unless Npcap is running.
  • Stopping Npcap after the interfaces load and then attempting to start a capture on one results an error message: The capture session could not be initiated on interface '\Device\NPF_{9701151C-7BE6-470A-9E0D-67ECF4EFA37D}' (Error opening adapter: The system cannot find the device specified. (20)).
  • The issue happens even when capturing on a non-loopback interface. Note that Wireshark requires Npcap to be running to even start a capture on any interface. This might not be the case, might be due to having started a loopback capture first then switching adapters.

@dmiller-nmap
Copy link
Contributor

@dmiller-nmap dmiller-nmap commented Oct 24, 2019

Yes, Wireshark requires Npcap to be running in order to do capture because Npcap is what does the capturing, so your observations are as expected. I will look into this and see if I can reproduce your issue here.

@dmiller-nmap
Copy link
Contributor

@dmiller-nmap dmiller-nmap commented Oct 24, 2019

I have not been able to reproduce this issue, so I have a few more diagnostic questions or suggestions:

  1. Is your VPN connected when you experience this issue? If so, does the issue persist if the VPN is not disconnected?
  2. Have you experienced the issue on any other systems, and if so, what do those have in common with this one?
  3. You may have leftover Npcap Loopback adapters from previous installations of Npcap. Remove them according to #55 and try again.
  4. There may be a conflict or problem with Citrix's Deterministic Network Enhancer (DNE), so removing that or disabling it may fix your problem. Let us know if this is the case.

@jeffrimko
Copy link
Author

@jeffrimko jeffrimko commented Oct 24, 2019

Answering your questions in order:

  1. The issue shows up both connected and disconnected to the VPN.
  2. Think issue has been seen on another work PC but I can't confirm at the moment. Could be a weird interaction with some company policy software, not sure.
  3. Will do. Very certain there were older versions of Npcap installed on this PC previously.
  4. Okay, I'm not familiar with DNE but will look into it.

Big thanks again and I'll report back with any new info soon!

@jeffrimko
Copy link
Author

@jeffrimko jeffrimko commented Oct 28, 2019

Okay, some updates related to the previous list of questions:

  1. I didn't find any leftover Npcap Loopback adapters from previous installs.
  2. I'm still not sure what DNE is but suspect it's not a factor here.

The issue is definitely a direct result of the "C:\Windows\System32\Npcap\wpcap.dll" file. When I move this file to a temporary location, the issue goes away and the loopback adapter does not show up in Wireshark. When this file is in its normal location, the issue occurs immediately after starting Wireshark and it loads the list of interfaces (i.e. no need to start a capture).

I did notice some strange Wireshark traffic graphs with npcap enabled and disabled (by moving the wpcap.dll):
image

Maybe there is some strange interaction with Hyper-V (this is Windows 10 Pro, I forgot to mention that previously, sorry). I might try to temporarily disable Hyper-V later using these instructions found on StackOverflow. I cannot permanently disable Hyper-V since Docker is needed on this machine.

@guyharris
Copy link

@guyharris guyharris commented Nov 1, 2019

If Npcap is disabled in the second screenshot, presumably WinPcap is being used - some capture driver, packet.dll for that driver, and capture DLL for Wireshark to use is present.

@jeffrimko
Copy link
Author

@jeffrimko jeffrimko commented Nov 18, 2019

Quick update, I haven't investigated this much further recently. After some reading, was scared away from disabling Hyper-V since some people report issues afterwards. If anyone has advice on how to debug this, please give a holler.

@daulis
Copy link

@daulis daulis commented Jan 22, 2020

I believe that I'm having the same issue with users here as well.

Web browsing stops working after starting Wireshark/Npcap 0.9986

Setup:

  1. Clean system: Uninstall previous Wireshark + Npcap. Windows 10, version 1803.
  2. Install Wireshark-win64-3.2.1.exe (This includes Npcap 0.9986). Default options to use Npcap were used (all boxes, including Legacy loopback adapter were NOT checked)
  3. Confirmed via Device Manager that no previous Loopback adapters were installed.
  4. Web browsing works
  5. Start Wireshark. But, don't start capturing anything.
  6. Web browsing stops working
  7. "net stop npcap" - This allows web browsing to work again

Other notes:

  1. This is not reproducible on all PCs, just certain users.
  2. This issue is present with WinPcap API-compatible mode enabled OR disabled.
  3. Issue seems similar to: https://github.com/nmap/nmap/issues/1173. The workaround for Issue 1173 was to not install the loopback adapter. But, the new issue happens even without installing the Legacy loopback adapter. Is it possible that the new method of capturing loopback traffic (https://github.com/nmap/npcap/releases/tag/v0.9983), now triggers this defect, even though the loopback adapter isn't "installed"?

DiagReport-20200122-135533.txt

@dmiller-nmap
Copy link
Contributor

@dmiller-nmap dmiller-nmap commented Jan 23, 2020

When Wireshark starts up, it begins packet capture on all interfaces, using the statistics to create the "sparkline" traffic graphs next to each interface. This is likely the event that is triggering the network interruption, which as @daulis points out is likely unavoidable now that the Npcap Loopback Adapter is not required in order to do loopback capture. You may be able to avoid this by using the command-line to start Wireshark, specifying a particular interface with the -i option, which ought to skip the sparkline display. Of course this would only be a workaround; we would rather identify the root cause of the problem and fix it.

I have just a few more questions that I would like to have answered to help me narrow down the scope of the issue:

  1. Does closing Wireshark (and therefore turning off the sparkline graphs and closing their packet capture handles) restore connectivity? Or is stopping the npcap driver really the only way to restore it?
  2. If the above workaround (using Wireshark's -i command line option) successfully avoids interruption, then does interruption happen if you use -i to specify the loopback adapter AND -p to avoid "promiscuous mode"?

@daulis
Copy link

@daulis daulis commented Jan 23, 2020

I tried these things on a different PC than my previous report, so I can confirm that it's reproducible on at least 2x PCs here.
For all of my testing, I used Wireshark 3.2.1. I left Wireshark installed, and only changed Npcap versions for my testing below.

Some extra info that I did before answering your questions above:

  • I tried npcap-0.9982 (and un-checked 'Support loopback traffic'), and it didn't exhibit this behavior. When I uninstalled 0.9982, and tried 0.9983, the problem started. The problem seems to have been introduced in 0.9983, which makes sense based on the new feature in the release notes (https://github.com/nmap/npcap/releases/tag/v0.9983).
  • I then uninstalled 0.9983, and went back to 0.9982. When I did this, Wireshark showed "No Interfaces found.". I couldn't get them to appear again with 0.9982. I seem to remember having a similar issue at home, but I couldn't reliably get enough info to log an issue.

Answers to your questions. For these steps, I installed 0.9986, and unchecked "Install Npcap in WinPcap API-compatible Mode". I did this to match what Wireshark sets as defaults during install.

  1. "Does closing Wireshark (and therefore turning off the sparkline graphs and closing their packet capture handles) restore connectivity? Or is stopping the npcap driver really the only way to restore it?"
    Answer: No, closing Wireshark does not help. Only stopping the npcap service worked. When I started the npcap service after this, web browsing worked (until I started Wireshark again)

  2. "If the above workaround (using Wireshark's -i command line option) successfully avoids interruption, then does interruption happen if you use -i to specify the loopback adapter AND -p to avoid "promiscuous mode"?"
    Test Results:

  • "Wireshark.exe -i 1" - This still triggered the problem. The "-i" by itself seems to do nothing.
  • "Wireshark.exe -i 1 -k" - This still triggered the problem. The "-k" starts capturing on the selected interface right away
  • "Wireshark.exe -i 1 -k -p" - This still triggered the problem.
  • "dumpcap.exe -D" - This just lists interfaces. This triggered the problem.

It appears just the act of Listing Interfaces is enough to trigger the issue.

@guyharris
Copy link

@guyharris guyharris commented Jan 23, 2020

Note that giving -i with a number involves listing interfaces, so that tcpdump/TShark/Wireshark/dumpcap/etc. can translate ordinal numbers into interface names.

You'd have to give the Big Ugly String With a GUID as the -i argument in order to avoid listing interfaces.

@daulis
Copy link

@daulis daulis commented Jan 24, 2020

@guyharris When I tried using GUID, Wireshark still does the "Finding local interfaces" step. I tried using Wireshark.exe -i \Device\NPF_{EC3585DF-28AE-4F3B-BBFC-7F120F22046D} -k

@guyharris
Copy link

@guyharris guyharris commented Jan 24, 2020

Then you'll have to try it with dumpcap -i and the GUID string.

@daulis
Copy link

@daulis daulis commented Jan 24, 2020

dumpcap -i \Device\NPF_{EC3585DF-28AE-4F3B-BBFC-7F120F22046D} triggers the same bug

@guyharris
Copy link

@guyharris guyharris commented Jan 24, 2020

Unfortunately, the code that processes the -i option in all programs in the Wireshark suite calls pcap_findalldevs() to get additional information about the interface (libpcap lacks a "get information about this interface" call; I intend to fix that at some point), so there's no way to avoid pcap_findalldevs() with programs in the Wireshark suite.

This is a job for tcpdump, but there's currently no convenient binary build for recent tcpdump; you might try windump, which may require installation of npcap in WinPcap-compatibility mode.

@dmiller-nmap
Copy link
Contributor

@dmiller-nmap dmiller-nmap commented Jan 24, 2020

@guyharris Thanks for jumping in with suggestions and explanations. It seems simply opening the Loopback device is sufficient to trigger the bug, so unless @daulis is looking for a workaround that involves separate packet capture with windump and subsequent analysis with Wireshark, it seems the ball is in our court to figure out a fix. We are looking into a separate privately-reported crash related to repeated opening of the loopback adapter, so I'm hopeful we can improve things generally and take care of this bug in the process.

@guyharris
Copy link

@guyharris guyharris commented Jan 24, 2020

@guyharris Thanks for jumping in with suggestions and explanations. It seems simply opening the Loopback device is sufficient to trigger the bug,

Or, at least, that any call to pcap_findalldevs() is sufficient to trigger the bug, given that the tests have all been done with programs in the wireshark suite and that, currently, doing any capturing with programs in that suite involves at least one pcap_findalldevs() call.

Given that the pcap_findalldevs() in the current state of the git@github.com:nmap/libpcap.git repository will, on Windows with Npcap (and WinPcap), call PacketOpenAdapter() on each adapter it finds, in order to get interface flags, that means that any use of Wireshark-suite programs to do capture will involve a PacketOpenAdapter() call on every adapter it finds. If it finds a loopback adapter, it'll call PacketOpenAdapter() on it.

Presumably the bug not showing up if there is no loopback adapter is what shows that the bug is probably triggered by opening the loopback adapter.

(Unless 0.9982 has been tested with "Support loopback traffic" checked, it has not been established that the bug was introduced in 0.9983; if, with 0.9982 installed with loopback traffic support enabled, the problem shows up, what has been demonstrated is that 0.9983 removed the one reliable way of preventing the bug from being triggered, at the expense of having loopback capture support removed - which might be a reasonable tradeoff for some users.)

so unless @daulis is looking for a workaround that involves separate packet capture with windump and subsequent analysis with Wireshark,

WinDump will only call pcap_findalldevs() if either 1) the -D flag is specified (the whole point of that flag is to print a list of devices on which you can capture, so calling pcap_findalldevs() is a requirement), 2) an interface name that's just a number is specified (as it has to get the list of devices to turn the ordinal number in that list into an interface name), or 3) no -i flag is specified for a capture (as it has to get the list of devices in order to pick the first device in the list). So a capture with WinDump, using the -i flag with a GUID-based device name, should avoid opening the loopback device and not trigger the bug.

@daulis
Copy link

@daulis daulis commented Jan 28, 2020

@dmiller-nmap I can pull extra data from one of the PCs that is having this issue, and I can also try out test builds of npcap if that helps you.

@dmiller-nmap
Copy link
Contributor

@dmiller-nmap dmiller-nmap commented Feb 4, 2020

As a workaround, Npcap 0.9987 released today includes the above commit, which will allow you to use the Windows Registry to disable the loopback WFP callouts, which likely contain the problem code that we have not yet identified. Just set HKLM\SYSTEM\CurrentControlSet\Services\npcap\Parameters\LoopbackSupport to 0 and restart the driver. This is only intended to be a workaround, not a permanent solution, so the Npcap installer does not have a way to set this value, and it will overwrite it with the default value of 1 on reinstall or upgrade. We will continue investigating this issue and hope to find a real fix.

@daulis
Copy link

@daulis daulis commented Feb 4, 2020

Two observations when using 0.9987:

  1. At one point, when I tried to stop npcap, it failed. After that, it appeared stuck. I had to reboot my machine to recover. I wasn't able to reproduce this.
C:\WINDOWS\system32>net stop npcap
The Npcap Packet Driver (NPCAP) service is stopping........
The Npcap Packet Driver (NPCAP) service could not be stopped.

C:\WINDOWS\system32>net stop npcap
The service is starting or stopping.  Please try again later.
  1. Setting the registry key did work, so it does appear to be caused by the loopback adapter code. (I can't easily use that workaround for my use cases though)

@dmiller-nmap
Copy link
Contributor

@dmiller-nmap dmiller-nmap commented Mar 5, 2020

This issue should be resolved by Npcap 0.9988, released today. We would appreciate your feedback so we know whether to close this issue.

@jeffrimko
Copy link
Author

@jeffrimko jeffrimko commented Mar 6, 2020

Just updated to Npcap 0.9988 and the issue appears to be resolved! Thank you greatly!

@fyodor fyodor transferred this issue from nmap/nmap May 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants