Host Discovery: Ignore TCP RST #1616

tsellers-r7 · 2019-05-30T23:40:41Z

This PR adds a new option, --host-discovery-ignore-tcp-reset, that tells Nmap to ignore TCP reset packets for the purpose host discovery. Firewalls can sometimes be configured to send resets instead of dropping packets when an ACL is triggered. This can cause hosts to be flagged as up when they are not. The intent of the patch is to provide an option to ignore these RST packets.

Note: This patch was originally written by Paul Miseiko at Rapid7 (my employer). I'm submitting this upstream with permission and by request. Any errors with the PR are likely mine.

I can't point to a device on the Internet that is doing this but we can simulate the behavior with a closed port on scanme.nmap.org and using TCP SYN discovery.

Without the option

Note the host is up due to a RST response: Host is up, received reset ttl 50 (0.074s latency).

$ sudo ./nmap -d -p 8080 -PS8080 -n --packet-trace  scanme.nmap.org

Starting Nmap 7.70SVN ( https://nmap.org ) at 2019-05-30 18:16 CDT
--------------- Timing report ---------------
  hostgroups: min 1, max 100000
  rtt-timeouts: init 1000, min 100, max 10000
  max-scan-delay: TCP 1000, UDP 1000, SCTP 1000
  parallelism: min 0, max 0
  max-retries: 10, host-timeout: 0
  min-rate: 0, max-rate: 0
---------------------------------------------
Initiating Ping Scan at 18:16
Scanning scanme.nmap.org (45.33.32.156) [1 port]
Packet capture filter (device en4): dst host 172.16.1.192 and (icmp or icmp6 or ((tcp or udp or sctp) and (src host 45.33.32.156)))
SENT (0.0361s) TCP 172.16.1.192:45225 > 45.33.32.156:8080 S ttl=59 id=48831 iplen=44  seq=1143953030 win=1024 <mss 1460>
RCVD (0.1101s) TCP 45.33.32.156:8080 > 172.16.1.192:45225 RA ttl=50 id=0 iplen=40  seq=0 win=0 
We got a TCP ping packet back from 45.33.32.156 port 8080 (trynum = 0)
Completed Ping Scan at 18:16, 0.07s elapsed (1 total hosts)
Overall sending rates: 13.35 packets / s, 587.33 bytes / s.
Initiating SYN Stealth Scan at 18:16
Scanning scanme.nmap.org (45.33.32.156) [1 port]
Packet capture filter (device en4): dst host 172.16.1.192 and (icmp or icmp6 or ((tcp or udp or sctp) and (src host 45.33.32.156)))
SENT (0.1114s) TCP 172.16.1.192:45481 > 45.33.32.156:8080 S ttl=59 id=15252 iplen=44  seq=1065690232 win=1024 <mss 1460>
RCVD (0.1857s) TCP 45.33.32.156:8080 > 172.16.1.192:45481 RA ttl=50 id=0 iplen=40  seq=0 win=0 
Completed SYN Stealth Scan at 18:16, 0.07s elapsed (1 total ports)
Overall sending rates: 13.31 packets / s, 585.80 bytes / s.
Nmap scan report for scanme.nmap.org (45.33.32.156)
Host is up, received reset ttl 50 (0.074s latency).
Other addresses for scanme.nmap.org (not scanned): 2600:3c01::f03c:91ff:fe18:bb2f
Scanned at 2019-05-30 18:16:26 CDT for 0s

PORT     STATE  SERVICE    REASON
8080/tcp closed http-proxy reset ttl 50
Final times for host: srtt: 74133 rttvar: 55632  to: 296661

Nmap done: 1 IP address (1 host up) scanned in 0.19 seconds
           Raw packets sent: 2 (88B) | Rcvd: 2 (80B)

With the option

Nmap thinks the host is down: Note: Host seems down. If it is really up,..

$ sudo ./nmap -d -p 8080 -PS8080 -n --packet-trace --host-discovery-ignore-tcp-reset scanme.nmap.org

Starting Nmap 7.70SVN ( https://nmap.org ) at 2019-05-30 18:15 CDT
--------------- Timing report ---------------
  hostgroups: min 1, max 100000
  rtt-timeouts: init 1000, min 100, max 10000
  max-scan-delay: TCP 1000, UDP 1000, SCTP 1000
  parallelism: min 0, max 0
  max-retries: 10, host-timeout: 0
  min-rate: 0, max-rate: 0
---------------------------------------------
Initiating Ping Scan at 18:15
Scanning scanme.nmap.org (45.33.32.156) [1 port]
Packet capture filter (device en4): dst host 172.16.1.192 and (icmp or icmp6 or ((tcp or udp or sctp) and (src host 45.33.32.156)))
SENT (0.0410s) TCP 172.16.1.192:54985 > 45.33.32.156:8080 S ttl=46 id=42332 iplen=44  seq=4020378568 win=1024 <mss 1460>
RCVD (0.1167s) TCP 45.33.32.156:8080 > 172.16.1.192:54985 RA ttl=50 id=0 iplen=40  seq=0 win=0 
SENT (1.0467s) TCP 172.16.1.192:54986 > 45.33.32.156:8080 S ttl=57 id=10575 iplen=44  seq=4020444105 win=1024 <mss 1460>
RCVD (1.1216s) TCP 45.33.32.156:8080 > 172.16.1.192:54986 RA ttl=47 id=0 iplen=40  seq=0 win=0 
Completed Ping Scan at 18:15, 2.02s elapsed (1 total hosts)
Overall sending rates: 0.99 packets / s, 43.44 bytes / s.
Nmap scan report for scanme.nmap.org (45.33.32.156) [host down, received no-response]
Other addresses for scanme.nmap.org (not scanned): 2600:3c01::f03c:91ff:fe18:bb2f
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 2.07 seconds
           Raw packets sent: 2 (88B) | Rcvd: 2 (80B)

With the option and using a port that responds with a SYN/ACK

In the example below I'm using SYN discovery against 80 (open) and 8080 (closed). We see a RST/ACK on 8080 which we ignore and a SYN/ACK on 80 which indicates that the host is up and that the new option didn't break the expected behavior. Note that I'm not actually scanning port 80 in order to reduce the potential impact on the results.

Host is up, received syn-ack ttl 47 (0.073s latency).

$ sudo ./nmap -d -p8080 -PS80,8080 -n --packet-trace --host-discovery-ignore-tcp-reset scanme.nmap.org

Starting Nmap 7.70SVN ( https://nmap.org ) at 2019-05-30 18:34 CDT
--------------- Timing report ---------------
  hostgroups: min 1, max 100000
  rtt-timeouts: init 1000, min 100, max 10000
  max-scan-delay: TCP 1000, UDP 1000, SCTP 1000
  parallelism: min 0, max 0
  max-retries: 10, host-timeout: 0
  min-rate: 0, max-rate: 0
---------------------------------------------
Initiating Ping Scan at 18:34
Scanning scanme.nmap.org (45.33.32.156) [2 ports]
Packet capture filter (device en4): dst host 172.16.1.192 and (icmp or icmp6 or ((tcp or udp or sctp) and (src host 45.33.32.156)))
SENT (0.0345s) TCP 172.16.1.192:62184 > 45.33.32.156:80 S ttl=57 id=64395 iplen=44  seq=2894489787 win=1024 <mss 1460>
SENT (0.0345s) TCP 172.16.1.192:62184 > 45.33.32.156:8080 S ttl=40 id=4801 iplen=44  seq=2894489787 win=1024 <mss 1460>
RCVD (0.1065s) TCP 45.33.32.156:8080 > 172.16.1.192:62184 RA ttl=47 id=0 iplen=40  seq=0 win=0 
RCVD (0.1072s) TCP 45.33.32.156:80 > 172.16.1.192:62184 SA ttl=47 id=0 iplen=44  seq=2668706749 win=29200 <mss 1460>
We got a TCP ping packet back from 45.33.32.156 port 80 (trynum = 0)
Completed Ping Scan at 18:34, 0.07s elapsed (1 total hosts)
Overall sending rates: 27.20 packets / s, 1196.86 bytes / s.
Initiating SYN Stealth Scan at 18:34
Scanning scanme.nmap.org (45.33.32.156) [1 port]
Packet capture filter (device en4): dst host 172.16.1.192 and (icmp or icmp6 or ((tcp or udp or sctp) and (src host 45.33.32.156)))
SENT (0.1086s) TCP 172.16.1.192:62440 > 45.33.32.156:8080 S ttl=46 id=51138 iplen=44  seq=2418066826 win=1024 <mss 1460>
RCVD (0.1801s) TCP 45.33.32.156:8080 > 172.16.1.192:62440 RA ttl=47 id=0 iplen=40  seq=0 win=0 
Completed SYN Stealth Scan at 18:34, 0.07s elapsed (1 total ports)
Overall sending rates: 13.78 packets / s, 606.39 bytes / s.
Nmap scan report for scanme.nmap.org (45.33.32.156)
Host is up, received syn-ack ttl 47 (0.073s latency).
Other addresses for scanme.nmap.org (not scanned): 2600:3c01::f03c:91ff:fe18:bb2f
Scanned at 2019-05-30 18:34:21 CDT for 0s

PORT     STATE  SERVICE    REASON
8080/tcp closed http-proxy reset ttl 47
Final times for host: srtt: 72606 rttvar: 54840  to: 291966

Nmap done: 1 IP address (1 host up) scanned in 0.18 seconds
           Raw packets sent: 3 (132B) | Rcvd: 3 (124B)

tsellers-r7 · 2019-05-30T23:43:15Z

@dmiller-nmap - Should this PR include updates to the documentation ( docs/nmap.usage.txt, etc ) or is this automatically generated?

tsellers-r7 · 2019-06-04T18:58:07Z

@dmiller-nmap @bonsaiviking - Any notes on the documentation side of this?

dmiller-nmap · 2019-12-30T16:35:08Z

Nmap-dev mailing list thread here: https://seclists.org/nmap-dev/2019/q2/31

dmiller-nmap

This does need documentation in docs/refguide.xml. Something like this would be great in the list right after --disable-arp-ping:

      <varlistentry>
        <term>
        <option>--discovery-ignore-rst</option> 
          <indexterm><primary><option>--discovery-ignore-rst</option></primary></indexterm>
        </term>
        <listitem>

          <para>In some cases, firewalls may spoof TCP reset (RST) replies in 
            response to probes to unoccupied or disallowed addresses. Since
            Nmap ordinarily considers RST replies to be proof that the target
            is up, this can lead to wasted time scanning targets that aren't
            there. Using the <option>--discovery-ignore-rst</option> will
            prevent Nmap from considering these replies during host discovery.
            You may need to select extra host discovery options to ensure you
            don't miss targets in this case.</para>
        </listitem>
      </varlistentry>

dmiller-nmap · 2020-01-09T20:42:58Z

nmap.cc

@@ -622,6 +623,7 @@ void parse_options(int argc, char **argv) {
    {"nsock-engine", required_argument, 0, 0},
    {"proxies", required_argument, 0, 0},
    {"proxy", required_argument, 0, 0},
+    {"host-discovery-ignore-tcp-reset", no_argument, 0, 0},


We should shorten this option name to --discovery-ignore-rst.

"host" is not needed, since "discovery" is unambiguous among Nmap scan phases. "reset" should be shortened to "rst" to match --defeat-rst-ratelimit, and since "rst" doesn't refer to any ICMP type/code or UDP flag, we can leave off "tcp."

dmiller-nmap · 2020-01-09T20:44:49Z

nmap.cc

@@ -271,6 +271,7 @@ static void printusage() {
         "  -PS/PA/PU/PY[portlist]: TCP SYN/ACK, UDP or SCTP discovery to given ports\n"
         "  -PE/PP/PM: ICMP echo, timestamp, and netmask request discovery probes\n"
         "  -PO[protocol list]: IP Protocol Ping\n"
+         "  --host-discovery-ignore-tcp-reset: Ignore reset during host discovery\n"


This is a specialized-enough option that it should not be added to the default usage statement. We generally want to limit this to the most widely-useful options, and it's complex enough that a 1-line description doesn't seem appropriate.

dmiller-nmap · 2020-01-09T21:05:49Z

scan_engine_raw.cc

-      if (!(tcp->th_flags & TH_RST)
-          && ((tcp->th_flags & (TH_SYN | TH_ACK)) != (TH_SYN | TH_ACK)))
-        continue;
+      if (o.host_discovery_ignore_tcp_reset


Should this also affect TCP Connect host discovery in scan_engine_connect.cc? We would avoid setting newhoststate = HOST_UP under case ECONNREFUSED: if USI->ping_scan && o.discovery_ignore_rst.

If we don't make this additional change, we should be very clear in the docs that the option only affects raw TCP host discovery probes, not unprivileged TCP host discovery. (There's not a way to use TCP Connect for host discovery if you're privileged; it just uses raw SYN for -PS, and -PT means ACK.)

dmiller-nmap · 2020-01-09T21:06:49Z

@tsellers-r7 I've done a review here. I can make the changes myself if you need me to, otherwise please address them and then commit. Thanks!

* Implement Ncat proxy creds via environment variable. Fixes nmap#2060, closes nmap#2073 * Fix --resume from IPv6 scans * Use correct default buffer position. Closes nmap#2084 * Clarify upper boundary for variable-length numerical fields * Make maximize_fdlimit return rlim_t on appropriate platforms. Closes nmap#2085. Fixes nmap#2079 * Credential object is creds.Account, not brute.Account. See nmap#2086 * Clarify location of the Error object * Use correct default buffer position. Closes nmap#2086 * Minor optimization of url.parse_query() * Output of matched fingerprints in http-default-accounts. Fixes nmap#2077 * Document that --open implies --defeat-rst-ratelimit since 7.40 * SNMP scripts are enabled on non-standard ports. See nmap#1473 * Increases SQL Server version resolution * Eliminate reflection false positives in http-shellshock. Closes nmap#2089 * Unify AFP pathname serialization * Correct AFP name extraction from responses. Closes nmap#2091 FPGetFileDirParms and FPEnumerateExt2 could crash due to unpacking from out-of-bounds positions. This latent issue got exposed by converting from bin.unpack to more stringent string.unpack * Clarified parsing of the volume list in AFP FPGetSrvrParms * Add cross references between the 2 whois scripts * Streamline Boolean expressions * Centralize AFP timestamp conversion to string * Fix a word-wrapping issue * Prevent SSH2 KEX confusion. Fixes nmap#2105 * Add ssh2.fetch_host_key() support for group 16 * Handle case of corrupted TCP options with length 0. Fixes nmap#2104 * Add iDRAC9 fingerprint to http-default-accounts. Closes nmap#2096 * fix license url: http -> https * Implementation of TLS SNI override in Ncat Closes nmap#2087, closes nmap#1928, fixes nmap#1927, fixes nmap#1974 * Fix off-by-one issue in last change. Fixes nmap#2107 * Be more strict with TCP options parsing, avoid reading off the end of TCP options. See nmap#2107 * Remove nmap-update This feature was never publicly released, and has not been distributed in our binary builds for a couple versions now. It needed to be removed in order to reduce the number of places Nmap looks for data files. See nmap#2051 * If fetchfile didn't find the XSL, use a relative path on all platforms. * Do not search NMAPDATADIR on Windows as it is not defined. See nmap#2051 * Remove an unused variable * Require trailing '/' to match a directory name with --script. See nmap#2051 * Stop using Shellshock in header name. Fixes nmap#1983 * Fix line wrapping * Speed improvement for script afp-ls. Closes nmap#2098 * New option --discovery-ignore-rst. Closes nmap#1616 * Nbase is needed for __attribute__ on Windows * include string_pool in Windows build * Use larger buffer size for socket errors (WSAETIMEDOUT was longer). * Allow multiple UDP payloads per port. Closes nmap#1859 (payloads to be committed later) * New UDP payloads. Closes nmap#1860 * Use ASCII chars for some payload data where it makes sense * Pass error along instead of printing (link error) * OpenSSL 1.1.X renamed libs: libeay32->libcrypto ssleay32->libssl * More OpenSSL DLL name changes * One last libeay32->libcrypto name change * Fix loopback detection on Windows with new Npcap * Add some popular favicon hashes * Update nmap-mac-prefixes * Update nmap-services from IANA * Handle too-short response in s7-info. See nmap#2117 * Remove a todo item that is done (--resolve-all) * Update dated 'class' network terms to CIDR. Closes nmap#2054 * Call superclass's init method from derived class * Use signed value for tcp header offset and option lengths to detect underflow * Correctly check for unsigned subtraction underflow. * Add some missing changelog entries * Tell LGTM to use the correct version of Python (2) * Process new Linux and OpenBSD fingerprints * Only get SSL options if we use them, currently for NO_SSLv2 * Process a few service fingerprint submissions * Add a requested feature * Try to make sure enough data is present before parsing. See nmap#2117 * Replace hyphens in the client SSH banner Hyphen is not allowed in the software version string (RFC 4253, section 4.2) * Update the SSH protocol flow. Closes nmap#1460 Allows the server to start the key exchange before the protocol version exchange (banner exchange) is completed * Silence static analysis warning LGTM points out that since comparison with sizeof(buf) coerces n to unsigned, all negative values become very large values, which are necessarily larger than sizeof(buf), so the test is redundant. We still want the test in our code to be explicit that we are checking for it, so reordering the comparisons should silence the warning. A good optimizing compiler should be able to combine the two conditions anyway. See github/codeql#4249 * Be explicit about truncating division (timeout is in whole milliseconds) * Improve docs on -Pn and host discovery "Host discovery" is the preferred term over "ping scan" because of confusion with ICMP Echo Request, a.k.a. "ping" as used by the "ping" utility. Warn when users use -Pn because it has negative impact on scan times since ultrascan timing parameters fall back to slow initial defaults. * Fix a config issue with LGTM (libverbs not linked in libpcap) * Update IPv6 classifier based on new submissions through 2020-09-14 * Fix a meaningless error message when parsing IPv6 extension headers. * Allow %F date format to mean YYYY-mm-dd like GNU date * Remove duplicate test conditionals already tested in enclosing block * Properly handle pcap reads in iocp engine. Fixes nmap#2126 Still has an odd code smell, but this fixes my test case with Nping. * Add missing prototype * Make IOCP the default Nsock engine on Windows. See nmap#2126 * Update macosx build to OpenSSL 1.1.1h, use jhbuild for all build steps * Default rule base for script mysql-audit. See nmap#2125 * Avoid masked use of date before 1/1/1970 UTC. Fixes nmap#2136, closes nmap#2137 * Fix a CHANGELOG typo * Reintegrate Nmap 7.90 release branch * Bump version and regen docs for 7.90SVN post-release * Only warn about protocol specs in port list with -p. Fixes nmap#2135 * Handle a weird IOCP error for UDP sockets. Fixes nmap#2140 Co-authored-by: nnposter <nnposter@e0a8ed71-7df4-0310-8962-fdc924857419> Co-authored-by: dmiller <dmiller@e0a8ed71-7df4-0310-8962-fdc924857419>

Host Discovery: Ignore RST

ab5184b

dmiller-nmap reviewed Jan 9, 2020

View reviewed changes

bendikro mentioned this pull request Jul 31, 2020

Add option to report the result of all the host discovery techniques #2090

Closed

nmap-bot closed this in 3d382bd Aug 31, 2020

res0nat0r mentioned this pull request Oct 7, 2020

stef res0nat0r/nmap#1

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Host Discovery: Ignore TCP RST #1616

Host Discovery: Ignore TCP RST #1616

tsellers-r7 commented May 30, 2019

tsellers-r7 commented May 30, 2019

tsellers-r7 commented Jun 4, 2019

dmiller-nmap commented Dec 30, 2019

dmiller-nmap left a comment

dmiller-nmap Jan 9, 2020

dmiller-nmap Jan 9, 2020

dmiller-nmap Jan 9, 2020

dmiller-nmap commented Jan 9, 2020

Host Discovery: Ignore TCP RST #1616

Host Discovery: Ignore TCP RST #1616

Conversation

tsellers-r7 commented May 30, 2019

Without the option

With the option

With the option and using a port that responds with a SYN/ACK

tsellers-r7 commented May 30, 2019

tsellers-r7 commented Jun 4, 2019

dmiller-nmap commented Dec 30, 2019

dmiller-nmap left a comment

Choose a reason for hiding this comment

dmiller-nmap Jan 9, 2020

Choose a reason for hiding this comment

dmiller-nmap Jan 9, 2020

Choose a reason for hiding this comment

dmiller-nmap Jan 9, 2020

Choose a reason for hiding this comment

dmiller-nmap commented Jan 9, 2020