Nmap on Windows with -sT shows 'filtered' instead of 'closed'

[dmiller-nmap]

Describe the bug
Nmap on Windows with -sT does not show closed ports, even though Wireshark shows RST packets in the loopback traffic capture.

To Reproduce
nmap -sT localhost -F

Expected behavior
Closed ports will show "closed" not "filtered"

Version info (please complete the following information):
OS: Windows 10

Nmap version 7.80 ( https://nmap.org )
Platform: i686-pc-windows-windows
Compiled with: nmap-liblua-5.3.5 openssl-1.0.2s nmap-libssh2-1.8.2 nmap-libz-1.2.11 nmap-libpcre-7.6 Npcap-0.9997 nmap-libdnet-1.12 ipv6
Compiled without:
Available nsock engines: iocp poll select

Additional context
Also affects Nmap 7.60. Have not checked older versions or other versions of Windows.

[dmiller-nmap]

Important updates:

• ALL connect scans are affected, not only localhost.
• I have checked and Nmap 6.40 and Nmap 5.35DC1 are also affected.

I believe this is related to how Windows handles non-blocking socket connect; something is going wrong between the connect() call returning WSAEWOULDBLOCK (which is mapped to EAGAIN in our code) and the select() code which is supposed to spot connect failures. I think maybe the problem is the getsockopt call which is supposed to get the error code. This might not be portable.

[dmiller-nmap]

Ok, this is really weird. When debugging, the outcome depends on where I place the breakpoints relative to the select() call.

• If I place the breakpoint just prior to the call to select(), then the call succeeds, returning 1 and setting the appropriate FD in the exception set. The port is marked "closed."
• If I place the breakpoint just after the call to select(), then the call does not succeed, returning 0 and failing to set the appropriate FD in any set. The port is marked "filtered" because the FD never comes up to be checked.

I verified and the FD set correctly contains the appropriate FD number in both cases. The timeout is a reasonable value (usually 0 seconds 999000 usecs, regardless of breakpoint placement).

[dmiller-nmap]

Adding a usleep(1500000) (1.5 seconds) immediately preceding the select() results in correct behavior, but usleep(1000000) (1 second) fails.

[dmiller-nmap]

The timeout is too short. Apparently WinSock is delaying the error notification for RST by up to 1.5 seconds. We'll have to add an artificial delay to compensate. We should turn this delay off if user chooses --defeat-rst-ratelimit.

[dmiller-nmap]

It's delaying because it's doing 3 retransmissions before giving up. This can be controlled with the TcpMaxConnectRetransmissions Registry setting. The TCP_MAXRT socket option sets a timeout on this behavior, but 0 means the default behavior, and the resolution is in seconds. Still, 1 second is more reasonable and may work, plus Win10 1607 introduced TCP_MAXRTMS so we could set it in milliseconds.

[dmiller-nmap]

Unfortunately, setting TCP_MAXRT means that we only ever get WSAETIMEDOUT error code back, even if a RST was received. This means that we can't distinguish an actual timeout from a connection reset, which puts us back to square 1.

[fyodor]

Update: As of 9/23/24, me and @dmiller-nmap can still reproduce this with Nmap 7.95. The issue is just that Windows takes so long to notify us about the RST. If we use --min-rtt-timeout it does work, but the scan is then super slow. We have ideas for potentially fixing this, though it is a bit of an edge case. Npcap is so good now that it's a lot rarer that people need to resort to -sT. And the ones who need more subtle details like whether a port is closed or filtered are probably more likely to scan from a system that can use raw sockets. So we're keeping this open, but for now it might just be considered another unfortunate limitation of using -sT on Windows.

[dmiller-nmap]

When I was looking into this last time, I asked on StackOverflow: Windows sockets: How to immediately detect TCP RST on nonblocking connect()?. No answer as yet, but a useful comment:

BSD-derived TCPs drop incoming SYNs if the backlog queue is full, which is indistinguishable at the client from host or network down, so connect() retries, which is plausible: if connect() gets an RST it means no listening socket, so it returns immediately. Windows on the other hand issues an RST if the backlog queue is full, so the client can't distinguish that condition from no listening socket, so it retries on both.

[dmiller-nmap]

Recent research has yielded all the necessary parts to fix this issue, so I'm dumping them here for later. There are 4 parameters to consider:

1. The overall connect timeout value, set with the TCP_MAXRT socket option. If the connection is incomplete when this time is reached, we get WSAETIMEDOUT, regardless of whether any RST was received.
2. The number of SYN retries, set with the SIO_TCP_INITIAL_RTO ioctl. If all of these retries receive RST before the connect timeout, we get WSACONNREFUSED. As yet undetermined what happens if only 1 receives RST, especially if the last retry times out without RST.
3. The intial timeout, which is used to set the timeout of the first SYN. Each timeout doubles after that up to 60 seconds.

So we need to set a timeout such that if a RST comes in response to the last SYN sent (the worst case), the connection will not timeout. The sum of the timeouts for n SYNs is therefore the sum from 0 to n - 1 of 2^n * RTO. The defaults appear to be 1s RTO and 5 attempts (4 retries), but a timeout of 21 seconds, hence our problem with distinguishing the responses.

Other observations:

• When a RST is received, Windows waits 0.5 seconds before sending the next SYN retry.
• Nmap doesn't enforce any particular timeout on connect attempts, leaving it up to the OS platform. Windows is the only place this has turned out to be a problem, but if our attempts to fix it result in improvements to scan times or quality, we may want to investigate adjusting timeouts on other platforms if available (TCP_SYNCNT on Linux, for example).

(Edited to correct: SIO_TCP_INITIAL_RTO actually interprets the TCP_INITIAL_RTO_PARAMETERS.Rtt as a timeout directly, rather than tripling it as I had initially thought.)

[dmiller-nmap]

For comparison, Linux kernel docs for tcp_syn_retries state:

Default value is 6, which corresponds to 63seconds till the last retransmission with the current initial RTO of 1second. With this the final timeout for an active TCP connection attempt will happen after 127seconds.

I was incorrect in saying Nmap doesn't enforce a timeout for connect scan. It closes the socket after the probe timeout, which is 1 second by default. In other words, Nmap is ensuring and expecting only 1 SYN is sent per connect() call, which makes our job of choosing parameters easier: for a given probe timeout, we can round up to the nearest whole second and then set SIO_TCP_INITIAL_RTO with { to_s * 1000, TCP_INITIAL_RTO_NO_SYN_RETRANSMISSIONS } and TCP_MAXRT set to to_s seconds. For Windows 10 1607 and later, we could use TCP_MAXRTMS and round to the nearest millisecond instead, but that would require determining at runtime whether TCP_MAXRTMS is available.