-
Notifications
You must be signed in to change notification settings - Fork 43
Collect a Network Trace
The main tools we will be discussing are NETMON and NETSH, though WireShark is a good alternative and is also available on Linux systems. Your company may have a preferred capture tool already approved or in-place and it would be best to use that.
Depending on the scenario you want, different tools may offer some advantages over other tools.
| Scenario | Recommendation |
|---|---|
| Easily Reproducible Issue | If the issue is reliably reproducible, testing using WireShark or NETMON using the GUI is a relatively simple method for performing the capture. |
| Long-Running Capture | Use NMCAP or WireShark with chain files. You can also use a circular capture, but have to terminate the capture quickly to prevent relevant data from being overwritten. Chain files are preferred. You may have to monitor how much disk space they take up and delete older files if your system is low on storage. WireShark can limit the number of chain files, NMCAP does not. |
| Cannot Install On the Server | NETSH is installed on every version of Window since 2008 R2. You can capture a trace without having to install another tool. |
| Winsock Details | By using SCENARIO=NETCONNECTION, NETSH will also log WinSock and other low-level TCP events to the capture file. |
| Linux or MAC Systems | You can use WireShark on these systems or any other capture tool that is compatible with the PCAP format or other format that WireShark can open. |
| VPN Traffic | Use WireShark for best results. |
| Capture loopback/local network traffic | Installing NPCAP as part of the WireShark installation allows you to select the loopback adapter for capturing local network traffic, i.e. client and server are on the same machine. |
Note: Chain files are additional files generated once the capture has recorded a predetermined amount of data. Instead of having 6GB in a single file, which most tools won't open, you could have 12 x 500MB or 30 x 200MB capture files. SQL Network Analyzer can analyze all the chained trace files at once as if they were a single larger file. The report will tell you which file the issue is in, and the smaller files open and filter a lot more quickly than the larger files.
Note: To read NETSH files, you need to use NETMON. You can also convert from ETL to PCAPNG using this utility: https://github.com/microsoft/etl2pcapng
Note: Do not capture to the SQL Server data disk. Do not capture to a network share. If possible, avoid capturing to the C: drive as it could impact paging performance, etc.
Note: You must run the application or command prompt "As Administrator" in order to perform a packet capture. To analyze a captured file, you can run as an ordinary user.
| Tool | Download Location |
|---|---|
| NETMON |
http://www.microsoft.com/en-us/download/details.aspx?id=4865 Perform a complete install. |
| WireShark | www.wireshark.org |
| NETSH | Built into Windows 7 and Windows 2008 R2 and later. |
Note: NETSH must be run from the Admin command prompt.
Note: NETSH does not allow for chained captures. See batch commands below for a workaround.
Note: Unless explicitly specified, the trace file ends up in %LocalAppData%\temp\NetTraces\NetTrace.etl
Note: Using the SCENARIO=NETCONNECTION command-line option results in a CAB file with the ETL trace file in it, plus some other files containing the basic network configuration of the machine.
| Description | Command |
|---|---|
| Stop the capture |
NETSH TRACE STOP Some post processing must be performed and may take a minute or two, so do not close the window until the command prompt appears again. |
| Show if a trace is running | NETSH TRACE SHOW STATUS |
| Basic capture, unlimited growth | NETSH TRACE START CAPTURE=YES MAXSIZE=0 |
| Capture with socket events | NETSH TRACE START SCENARIO=NETCONNECTION CAPTURE=YES TRACEFILE=c:\temp\%computername%.etl |
| Circular capture and truncate packets |
NETSH TRACE START SCENARIO=NETCONNECTION CAPTURE=YES TRACEFILE=c:\temp\%computername%.etl FILEMODE=CIRCULAR MAXSIZE=2048 PACKETTRUNCATEBYTES=250 NETSH has a larger header than other capture formats, so do not truncate much below the amount suggested. |
| Filter the data during capture |
NETSH TRACE START CAPTURE=YES IPv4.Address=10.10.10.10 Protocol=!DNS MAXSIZE=0 Note the use of '=' instead of '==' in the filter. For more help on filtering options: NETSH TRACE SHOW CAPTUREFILTERHELP. |
NETSH does not allow for a chained capture, but LOGMAN does. LOGMAN cannot initiate a network capture, but can log the same provider as NETSH. By combining the commands into a batch file, you can get the best of both worlds:
StartChainedCap.cmd
These commands clear various caches before starting the trace. A TASKLIST helps correlate process ID's in the trace to application names.
Pasting the commands directly into the command window won't work because of the % symbols in the command are treated differently.
cd /d %~dp0
IPCONFIG /flushdns
NBTSTAT -R
Powershell -command "Get-WmiObject Win32_LogonSession | Where-Object {$_.AuthenticationPackage -ne 'NTLM'} | ForEach-Object {c:\windows\system32\klist.exe purge -li ([Convert]::ToString($_.LogonId, 16))}"
netsh trace start capture=yes maxsize=1 TRACEFILE=.\deleteme.etl
logman start ndiscap -p Microsoft-Windows-NDIS-PacketCapture -mode newfile -max 200 -o .\%computername%%%d.etl -ets
tasklist > .\TasklistAtStart.txt
Note: The CD command moves the "current directory" to the location of the cmd file.
The NETSH command starts the capture into a small circular file called deleteme.etl. LOGMAN piggy-backs this and logs to the computername file with a numeric suffix every 200MB (-max 200). Keep this size small to allow faster opening and filtering. Both files are stored in the current directory.
StopChainedCap.cmd
Delete the temporary etl file created by NETSH and give the CAB file a better name.
cd /d %~dp0
tasklist > .\TasklistAtEnd.txt
logman stop ndiscap -ets
netsh trace stop
del .\deleteme.etl
rename .\deleteme.cab network_settings.cab
Additional NETSH and LOGMAN command ideas
Note: NMCAP gets installed with NETMON.
Note: NMCAP must be run from an Admin command prompt.
| Description | Command |
|---|---|
| Stop the capture |
CTRL+C Do not close the window until the command-prompt reappears. Doing so will prevent the frame table from being written and the file will contain no identifiable frame data. If you capture a chained trace and the last file is bad, closing the window too soon is usually the cause. |
| Basic non-chained capture with Process names | NMCAP /network * /capture /file c:\data\%computername%.cap /captureprocesses |
| Basic chained capture |
NMCAP /network * /capture /file c:\data\%computername%.chn:100M Note CHN file extension in the command. Produces multiple files of 100MB each. |
| Chained capture with packet truncation | NMCAP /network * /capture /file c:\data\%computername%.chn:100M /maxframelength 180 |
| Auto-stop after 4 hours | NMCAP /network * /capture /file c:\data\%computername%.chn:100M /maxframelength 180 /stopwhen /timeafter 240 min |
| Auto-stop at a set time | NMCAP /network * /capture /file c:\data\%computername%.chn:100M /maxframelength 180 /stopwhen /time 3:00:00 PM 9/10/2020 |
| Auto-start at 3 AM and stop 2 hours later | NMCAP /network * /startwhen /time 3:00:00 AM 9/10/2016 /capture /file c:\data%computername%.chn:100M /stopwhen /timeafter 120Min |
| Filter packets during capture | NMCAP /network * /capture (!ARP AND !ICMP AND !NBTNS AND !BROWSER) /file NoNoise.cap |
| Filter words in Frame description | NMCAP /network * /capture contains(.Property.Description, "Continuation") /File TCPContinuations.cap |
| Start and stop on a filter condition | NMCAP /network * /startwhen /frame tcp.flags.syn==true and ipv4.address=10.10.10.10 /capture /file c:\data\%computername%.chn:100M /stopwhen /frame (tcp.flags.fin==true or tcp.flags.reset==true) and ipv4.address==10.10.10.10 |
Note: In Auto-start scenarios, NMCAP will begin capturing immediately, but will throw away packets until the capture start condition is met.
NMCAP can also be used to process existing files, such as joining several captures into one file or splitting a large file into several smaller files.
| Description | Command |
|---|---|
| Extract a conversation | NMCAP /inputcapture mytrace.cap /capture ipv4.address==10.10.10.10 AND tcp.port==1433 /file filteredoutput.cap |
| Extract a conversation in several files |
NMCAP /inputcapture mytrace.cap mytrace(1).cap mytrace(2).cap /capture ipv4.address==10.10.10.10 AND tcp.port==1433 /file filteredoutput.cap Wildcards are not supported for /inputcapture, files must be specified individually. |
| Split a large file into smaller ones | NMCAP /inputcapture mybigtrace.cap /capture /file mysmallfiles.chn:100M |
The PowerShell script below automatically builds the NMCAP command above that extracts a conversation across multiple input files. This is especially useful if there are a large number of input files.
#-------------------------------
#
# Extract-NetConversation.ps1
#
# A PowerShell script to extract a conversation using NMCAP that is spread across multiple chained files.
#
# Requirements: NETMON must be installed to use the NMCAP tool.
#
# Arguments:
#
# File Filter: *.cap - avoid using this if you re-run the script as it will also gather the filtered files.
# Use something more specific, such as sqlprod01(*).cap or even sql*.cap
#
# Capture Filter: Any valid NETMON filter. Surround with "quotes" if the filter string contains spaces.
# Use the client's IP address and port number for the most narrow filter.
#
# Output File: Any file name. You can incorporate the IP address and port number to distinguish from other filtered files.
#
#
# USAGE: .\Extract-NetConversation sql*.cap "IPV4.Address==10.10.10.10 AND tcp.port==1433" filtered.10.10.10.10.1433.cap
#
#-------------------------------
param ([string]$FileFilter = "*.cap", [string]$CaptureFilter = "", [string]$OutFileName = "filteredoutput.cap")
Set-ExecutionPolicy Unrestricted -Scope CurrentUser
# Get the list of files and concatenate the names into a string
$filestring = ""
$direntries = get-childitem -filter $FileFilter | sort-object -property LastWriteTime
foreach ($direntry in $direntries) { $filestring += "$($direntry.Name) " }
# Build and execute the command string - cannot just execute it directly because of the concatenation
$command = "NMCAP /inputcapture $filestring /capture $CaptureFilter /file $OutFileName"
$command | cmd # Executes the command via cmd.exe
The EDITCAP command does not split on file size, but rather on frame count, so the files may come out a bit uneven in size. A frame count of 200,000 should keep the split files under 1GB in size. If using Jumbo Frames, then you may want to lower this number.
Note: Omit the -F pcap argument if splitting PCAPNG files. pcap must be in lower case and switches are case-sensitive.
"C:\Program Files\Wireshark\editcap.exe" -c 200000 -F pcap BigCapture.pcap SplitCapture.pcap
Note: Omit the -F pcap argument if splitting PCAPNG files. pcap must be in lower case and switches are case-sensitive.
Note: The filter string must be quoted if it contains any spaces.
You should include some aspect the filter in the output file name to be able to distinguish between files.
"C:\Program Files\Wireshark\tshark.exe" -r sourcetracefile.pcap
-Y "ip.addr==10.10.10.10 and tcp.port==50288"
-w filteredoutput.10.10.10.10.50288.pcap -F pcap
Where to get TCPDUMP if it is not installed.
| Distribution | Command |
|---|---|
| Debian, Ubuntu, Linux Mint | sudo apt install tcpdump |
| CentOS, Fedora, AlmaLinux, Red Hat | sudo dnf install tcpdump |
| Arch Linux, Manjaro | sudo pacman -S tcpdump |
TCPDUMP command options
| Option | Description | Example |
|---|---|---|
| -i (Interface) | Select the interface you want to trace | -i eth0, -i any (for all interfaces) |
| -s #number | Set packet truncation | 0=no truncation, max truncation=65535 |
| -C #number | Sets the File size in MB | -C 200 |
| -W #number | Sets the number if files | -W 20 |
| -w $path | Sets the path and file name for trace | -w /tmp/trace.pcap |
Examples:
Consistent issue
tcpdump -i any -s 0 -w /tmp/MSTS/networktrace.pcap
Intermittent issue Issues that require a longer duration for the traces to be run without overwriting data: This command will create a chain trace of 50 files at 200MB.
tcpdump -i any -s 0 -C 200 -W 50 -w /tmp/MSTS/networktrace.pcap
For the full list of command option: https://www.tcpdump.org/manpages/tcpdump.1.html