New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Abort in stopTimeOutClock #1361

Closed
djcater opened this Issue Oct 18, 2018 · 6 comments

Comments

Projects
None yet
3 participants
@djcater

djcater commented Oct 18, 2018

$ sudo nmap -V 
Nmap version 7.70SVN ( https://nmap.org )
Platform: x86_64-unknown-linux-gnu
Compiled with: liblua-5.3.3 openssl-1.1.0g nmap-libssh2-1.8.0 libz-1.2.8 libpcre-8.39 libpcap-1.8.1 nmap-libdnet-1.12 ipv6
Compiled without:
Available nsock engines: epoll poll select
$ sudo nmap -4 -vvv -ddd -sn -PR 8.8.8.8 
Starting Nmap 7.70SVN ( https://nmap.org ) at 2018-10-18 22:18 BST
Fetchfile found /usr/local/bin/../share/nmap/nmap.xsl
The max # of sockets we are using is: 0
--------------- Timing report ---------------
  hostgroups: min 1, max 100000
  rtt-timeouts: init 1000, min 100, max 10000
  max-scan-delay: TCP 1000, UDP 1000, SCTP 1000
  parallelism: min 0, max 0
  max-retries: 10, host-timeout: 0
  min-rate: 0, max-rate: 0
---------------------------------------------
Fetchfile found /usr/local/bin/../share/nmap/nmap-payloads
Initiating Ping Scan at 22:18
Scanning 8.8.8.8 [0 ports]
**TIMING STATS** (0.0015s): IP, probes active/freshportsleft/retry_stack/outstanding/retranwait/onbench, cwnd/ssthresh/delay, timeout/srtt/rttvar/
   Groupstats (1/1 incomplete): 0/*/*/*/*/* 10.00/75/* 1000000/-1/-1
Current sending rates: 0.00 packets / s.
Overall sending rates: 0.00 packets / s.
Moving 8.8.8.8 to completed hosts list with 0 outstanding probes.
nmap: Target.cc:503: void Target::stopTimeOutClock(const timeval*): Assertion `htn.toclock_running == true' failed.
Aborted

Obviously doing an ARP scan of an address which isn't in the current subnet doesn't make a whole lot of sense, but it still shouldn't cause a crash.

I'm guessing this is due to d8ff55b which deferred starting the timeout clock until the first probe is sent, and in this case, no probes are sent, so when Nmap tries to stop the clock the assertion fails?

This is with Nmap built from the current master. It doesn't happen in Nmap 7.70.

@dmiller-nmap

This comment has been minimized.

dmiller-nmap commented Oct 19, 2018

Wow, this is a weird one. You're correct, d8ff55b is what turned this into an assertion failure, but there was a problem before that. Essentially, here's what happens:

  1. Nmap does address resolution and IP range and CIDR expansion to get IP addresses to fill up a hostgroup. It groups them according to outgoing interface using the system's routing information, deferring any that need to use a different interface until later.
  2. If the outgoing interface is Ethernet-like and ARP ping is allowed (either implicitly or because you used -PR) AND the target is directly connected as defined by the system's routing tables, ARP ping scan is performed ("Initiating ARP Ping Scan at XX:XX" will be seen).
  3. Otherwise, do a regular ping scan.

What's happening in this case is that the target address is not directly-connected according to the routing tables, so ARP ping is never done. Instead, it falls through to number 3, and a regular ping scan is done. But that scan is done according to the options on the command line, so it's really a ARP ping scan, but without proper setup. The target has no probes to send ([0 ports] in your output), so it's already done. Because the first probe is never sent, the timeout clock is never started, but because the target was supposed to be scanned, Nmap tries to turn off the timeout clock when it's "done" and hits the assertion failure.

Here some approaches we could take to fix this:

  1. Remove the -PR option. ARP ping is only ever used when it would be used by default anyway, since there's no way currently to force ARP ping for a not-directly-connected target.
  2. Make -PR take precedence and force an ARP ping even if the target is not directly connected. I think this is what you'd expect to happen, but it's the most prone to user error if someone adds -PR to an ordinary scan. Lots of breakage and weird layer-2 traffic.
  3. Remove targets from the list of targets to scan if they have no probes to send. This might avoid the assertion, but it's also ignoring what is almost certainly a bad thing. Maybe instead we should assert that the target has probes to send if it's being scanned?

I'm not sure which of these I will do, but I'm leaning towards 1 and 3 (with assert). It turns out there is a way to force an ARP host discovery for addresses on networks that are not configured on your system: use the -e and -S options to force a source address on whatever interface you want. Here's how it worked for me:

$ sudo ./nmap -sn -S1.1.1.2 -e eth0 1.1.1.1 -n -d
WARNING: If -S is being used to fake your source address, you may also have to use -e <interface> and -Pn .  If you are using it to specify your real source address, you can ignore this warning.
Starting Nmap 7.70SVN ( https://nmap.org ) at 2018-10-19 20:10 UTC
--------------- Timing report ---------------
  hostgroups: min 1, max 100000
  rtt-timeouts: init 1000, min 100, max 10000
  max-scan-delay: TCP 1000, UDP 1000, SCTP 1000
  parallelism: min 0, max 0
  max-retries: 10, host-timeout: 0
  min-rate: 0, max-rate: 0
---------------------------------------------
Initiating ARP Ping Scan at 20:10
Scanning 1.1.1.1 [1 port]
Packet capture filter (device eth0): arp and arp[18:4] = 0xF23C91AE and arp[22:2] = 0xFCB8
Completed ARP Ping Scan at 20:10, 0.45s elapsed (1 total hosts)
Overall sending rates: 4.42 packets / s, 185.82 bytes / s.
Nmap scan report for 1.1.1.1 [host down, received no-response]
Read from /home/miller/nmap/nmap: nmap-payloads.
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 0.51 seconds
           Raw packets sent: 2 (56B) | Rcvd: 0 (0B)
@djcater

This comment has been minimized.

djcater commented Oct 21, 2018

I actually think option 2 would be preferable, from a pen-tester perspective.

Sometimes a client might give you a static IP address and subnet mask for your test device, say 10.0.4.51, 255.255.255.192 (10.0.4.51/26), a default gateway address (say 10.0.4.1), and a list of targets to assess, say 10.0.4.0/22.

But what if the layer 2 subnet is actually bigger than they've said (say, /24, or the whole /22 that is to be assessed)?

It would be nice to be able to determine that with a set of ping scans. E.g.

sudo nmap -4 -vvv -d -sn -PR -oA arp 10.0.4.0/22
sudo nmap -4 -vvv -d -sn --disable-arp-ping -PE -oA icmp-echo 10.0.4.0/22
sudo nmap -4 -vvv -d -sn --disable-arp-ping -PP -oA icmp-timestamp 10.0.4.0/22

etc. etc. with the other ping options.

(I often do something like the above to get an initial feel for the network before doing more thorough host discovery with larger TCP and UDP scans).

In this case, an actual ARP scan would show that the subnet is bigger than we've been told, which might be useful to know for subsequent testing.

I just tested this scenario, and Nmap won't ARP scan the other hosts, whereas something like arping (from iputils) will (without having to spoof the source address).

@dmiller-nmap

This comment has been minimized.

dmiller-nmap commented Oct 22, 2018

The trouble with that approach is that Nmap then has to make the assumption of what source address and interface to use for those targets. For the ones in the /26 net that you configured in the OS, it can consult the routing tables to know what to use. For the others, it can't know. On a system that only has one non-loopback IP address, it seems like you could use that, but that would have to be handled as an edge case, with the default being to fail without knowing the source address to use. I think option 2 is out because of this.

Fortunately for your use case, providing -e eth0 -S 10.0.4.51 will force an ARP scan to be used for any destination address. It actually seems to me like maybe we ought to have options for setting a default gateway and netmask as well, for situations where Nmap can't get proper routing info from the OS.

nmap-bot pushed a commit that referenced this issue Oct 22, 2018

ultrascan: bail early if there are no probes in the current scan
This shouldn't ever happen, but it currently does for host discovery
scans using -PR where the target is not directly connected. See #1361
@fyodor

This comment has been minimized.

fyodor commented Oct 23, 2018

I like Dan's #1 approach--just make -PR do nothing since it is already the default behavior now whenever Nmap detects that it can use ARP scan. It's already not in the nmap -h output and we should probably remove it from the man page (or at least just put a sentence noting that it was removed since it is the default). But we should probably keep some of the text from -PR describing the process, and just move that to where we describe the default host discovery techniques.

@nmap-bot nmap-bot closed this in d03b10e Oct 24, 2018

@djcater

This comment has been minimized.

djcater commented Oct 24, 2018

@dmiller-nmap Thanks for the detailed responses.

I just cloned from Git to test your change, and there still seems to be one case where I can't get it to work as you described. Here's all the output which hopefully can help.

In these scenarios 192.168.1.1 is the gateway host in our subnet, 192.168.1.2 is our own IP address on eth0 in the subnet, 8.8.8.8 is an IP address outside of the subnet, and 10.99.99.99 is a spoofed source address not within our actual subnet nor within the subnet of the target host.

Version:

$ nmap -V
Nmap version 7.70SVN ( https://nmap.org )
Platform: x86_64-unknown-linux-gnu
Compiled with: liblua-5.3.3 openssl-1.1.0g nmap-libssh2-1.8.0 libz-1.2.8 libpcre-8.39 libpcap-1.8.1 nmap-libdnet-1.12 ipv6
Compiled without:
Available nsock engines: epoll poll select

Unspecified ping method of host within the subnet. Uses ARP (as expected), and host is deemed to be up (as expected).

$ sudo nmap -vvv -d -4 -n -sn -e eth0 192.168.1.1
Starting Nmap 7.70SVN ( https://nmap.org ) at 2018-10-24 23:19 BST
--------------- Timing report ---------------
  hostgroups: min 1, max 100000
  rtt-timeouts: init 1000, min 100, max 10000
  max-scan-delay: TCP 1000, UDP 1000, SCTP 1000
  parallelism: min 0, max 0
  max-retries: 10, host-timeout: 0
  min-rate: 0, max-rate: 0
---------------------------------------------
Initiating ARP Ping Scan at 23:19
Scanning 192.168.1.1 [1 port]
Packet capture filter (device eth0): arp and arp[18:4] = 0xXXXXXXXX and arp[22:2] = 0xXXXX
Completed ARP Ping Scan at 23:19, 0.05s elapsed (1 total hosts)
Overall sending rates: 19.81 packets / s, 832.16 bytes / s.
Nmap scan report for 192.168.1.1
Host is up, received arp-response (0.00026s latency).
MAC Address: XX:XX:XX:XX:XX:XX (XXX XXX)
Final times for host: srtt: 258 rttvar: 5000  to: 100000
Read from /usr/local/bin/../share/nmap: nmap-mac-prefixes nmap-payloads.
Nmap done: 1 IP address (1 host up) scanned in 0.12 seconds
           Raw packets sent: 1 (28B) | Rcvd: 1 (28B)

Unspecified ping method of host outside the subnet. Uses default ICMP + TCP pings (as expected), and host is deemed to be up (as expected).

$ sudo nmap -vvv -d -4 -n -sn -e eth0 8.8.8.8
Starting Nmap 7.70SVN ( https://nmap.org ) at 2018-10-24 23:19 BST
--------------- Timing report ---------------
  hostgroups: min 1, max 100000
  rtt-timeouts: init 1000, min 100, max 10000
  max-scan-delay: TCP 1000, UDP 1000, SCTP 1000
  parallelism: min 0, max 0
  max-retries: 10, host-timeout: 0
  min-rate: 0, max-rate: 0
---------------------------------------------
Initiating Ping Scan at 23:19
Scanning 8.8.8.8 [4 ports]
Packet capture filter (device eth0): dst host 192.168.1.2 and (icmp or icmp6 or ((tcp or udp or sctp) and (src host 8.8.8.8)))
We got a TCP ping packet back from 8.8.8.8 port 443 (trynum = 0)
Completed Ping Scan at 23:19, 0.05s elapsed (1 total hosts)
Overall sending rates: 76.93 packets / s, 2923.53 bytes / s.
Nmap scan report for 8.8.8.8
Host is up, received syn-ack ttl 117 (0.00096s latency).
Final times for host: srtt: 957 rttvar: 5000  to: 100000
Read from /usr/local/bin/../share/nmap: nmap-payloads.
Nmap done: 1 IP address (1 host up) scanned in 0.09 seconds
           Raw packets sent: 4 (152B) | Rcvd: 1 (44B)

Unspecified ping method for host within the subnet, using a spoofed source address which is not in the current subnet, nor the subnet of the target host. Uses ARP (as you said it would), and host is deemed to be up (as expected, despite not controlling the source IP address, due to the ARP response being a layer 2 broadcast).

$ sudo nmap -vvv -d -4 -n -sn -e eth0 -S 10.99.99.99 192.168.1.1
WARNING: If -S is being used to fake your source address, you may also have to use -e <interface> and -Pn .  If you are using it to specify your real source address, you can ignore this warning.
Starting Nmap 7.70SVN ( https://nmap.org ) at 2018-10-24 23:19 BST
--------------- Timing report ---------------
  hostgroups: min 1, max 100000
  rtt-timeouts: init 1000, min 100, max 10000
  max-scan-delay: TCP 1000, UDP 1000, SCTP 1000
  parallelism: min 0, max 0
  max-retries: 10, host-timeout: 0
  min-rate: 0, max-rate: 0
---------------------------------------------
Initiating ARP Ping Scan at 23:19
Scanning 192.168.1.1 [1 port]
Packet capture filter (device eth0): arp and arp[18:4] = 0xXXXXXXXX and arp[22:2] = 0xXXXX
Completed ARP Ping Scan at 23:19, 0.06s elapsed (1 total hosts)
Overall sending rates: 16.50 packets / s, 692.94 bytes / s.
Nmap scan report for 192.168.1.1
Host is up, received arp-response (0.00042s latency).
MAC Address: XX:XX:XX:XX:XX:XX (XXX XXX)
Final times for host: srtt: 419 rttvar: 5000  to: 100000
Read from /usr/local/bin/../share/nmap: nmap-mac-prefixes nmap-payloads.
Nmap done: 1 IP address (1 host up) scanned in 0.15 seconds
           Raw packets sent: 1 (28B) | Rcvd: 1 (28B)

Unspecified ping method of host outside the subnet, using a spoofed source address which is not in the current subnet, nor the subnet of the target host. Uses ARP (as you said it would), and host is deemed to be down (as expected).

$ sudo nmap -vvv -d -4 -n -sn -e eth0 -S 10.99.99.99 8.8.8.8
WARNING: If -S is being used to fake your source address, you may also have to use -e <interface> and -Pn .  If you are using it to specify your real source address, you can ignore this warning.
Starting Nmap 7.70SVN ( https://nmap.org ) at 2018-10-24 23:19 BST
--------------- Timing report ---------------
  hostgroups: min 1, max 100000
  rtt-timeouts: init 1000, min 100, max 10000
  max-scan-delay: TCP 1000, UDP 1000, SCTP 1000
  parallelism: min 0, max 0
  max-retries: 10, host-timeout: 0
  min-rate: 0, max-rate: 0
---------------------------------------------
Initiating ARP Ping Scan at 23:19
Scanning 8.8.8.8 [1 port]
Packet capture filter (device eth0): arp and arp[18:4] = 0xXXXXXXXX and arp[22:2] = 0xXXXX
Completed ARP Ping Scan at 23:19, 0.46s elapsed (1 total hosts)
Overall sending rates: 4.35 packets / s, 182.51 bytes / s.
Nmap scan report for 8.8.8.8 [host down, received no-response]
Read from /usr/local/bin/../share/nmap: nmap-payloads.
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 0.52 seconds
           Raw packets sent: 2 (56B) | Rcvd: 0 (0B)

Unspecified ping method of host within the current subnet, setting the source IP address as our actual IP address on eth0. Uses ARP (as expected), and host is deemed to be up (as expected).

$ sudo nmap -vvv -d -4 -n -sn -e eth0 -S 192.168.1.2 192.168.1.1
WARNING: If -S is being used to fake your source address, you may also have to use -e <interface> and -Pn .  If you are using it to specify your real source address, you can ignore this warning.
Starting Nmap 7.70SVN ( https://nmap.org ) at 2018-10-24 23:19 BST
--------------- Timing report ---------------
  hostgroups: min 1, max 100000
  rtt-timeouts: init 1000, min 100, max 10000
  max-scan-delay: TCP 1000, UDP 1000, SCTP 1000
  parallelism: min 0, max 0
  max-retries: 10, host-timeout: 0
  min-rate: 0, max-rate: 0
---------------------------------------------
Initiating ARP Ping Scan at 23:19
Scanning 192.168.1.1 [1 port]
Packet capture filter (device eth0): arp and arp[18:4] = 0xXXXXXXXX and arp[22:2] = 0xXXXX
Completed ARP Ping Scan at 23:19, 0.07s elapsed (1 total hosts)
Overall sending rates: 14.87 packets / s, 624.51 bytes / s.
Nmap scan report for 192.168.1.1
Host is up, received arp-response (0.00028s latency).
MAC Address: XX:XX:XX:XX:XX:XX (XXX XXX)
Final times for host: srtt: 285 rttvar: 5000  to: 100000
Read from /usr/local/bin/../share/nmap: nmap-mac-prefixes nmap-payloads.
Nmap done: 1 IP address (1 host up) scanned in 0.15 seconds
           Raw packets sent: 1 (28B) | Rcvd: 1 (28B)

Unspecified ping method of host outside the current subnet, setting the source IP address to be our actual IP address on eth0. Uses ICMP + TCP pings (unexpected), and host is deemed to be up (expected given the actual ping method used, but not the intention of the scan).

$ sudo nmap -vvv -d -4 -n -sn -e eth0 -S 192.168.1.2 8.8.8.8
WARNING: If -S is being used to fake your source address, you may also have to use -e <interface> and -Pn .  If you are using it to specify your real source address, you can ignore this warning.
Starting Nmap 7.70SVN ( https://nmap.org ) at 2018-10-24 23:19 BST
--------------- Timing report ---------------
  hostgroups: min 1, max 100000
  rtt-timeouts: init 1000, min 100, max 10000
  max-scan-delay: TCP 1000, UDP 1000, SCTP 1000
  parallelism: min 0, max 0
  max-retries: 10, host-timeout: 0
  min-rate: 0, max-rate: 0
---------------------------------------------
Initiating Ping Scan at 23:19
Scanning 8.8.8.8 [4 ports]
Packet capture filter (device eth0): dst host 192.168.1.2 and (icmp or icmp6 or ((tcp or udp or sctp) and (src host 8.8.8.8)))
We got a TCP ping packet back from 8.8.8.8 port 443 (trynum = 0)
Completed Ping Scan at 23:19, 0.06s elapsed (1 total hosts)
Overall sending rates: 64.09 packets / s, 2435.31 bytes / s.
Nmap scan report for 8.8.8.8
Host is up, received syn-ack ttl 118 (0.0015s latency).
Final times for host: srtt: 1535 rttvar: 5000  to: 100000
Read from /usr/local/bin/../share/nmap: nmap-payloads.
Nmap done: 1 IP address (1 host up) scanned in 0.11 seconds
           Raw packets sent: 4 (152B) | Rcvd: 1 (44B)

Is that last example expected? My intention is to have a command syntax I can use in a script where I can force ARP to be used as the ping method, but ideally using my actual IP address as the source IP address in the ARP packets, rather than having to spoof it to a random address.

@djcater

This comment has been minimized.

djcater commented Oct 27, 2018

@dmiller-nmap To clarify, this is to cover the case of when the target list contains a mix of IP addresses which are on the local subnet and IP addresses which aren't. Currently with Nmap 7.70 a -PR ARP scan finishes quickly, by using ARP for the local subnet, and no probes at all for other hosts.

Now, the scan takes longer, by doing an actual ICMP / TCP ping for the other hosts, making it slower and more difficult to determine whether a particular target responds to ARP or not (rather than just seeing if the host is "Up" in a -PR scan, you now have to dig into the reason field in the output).

The use case of this is categorising and grouping hosts by the type of pings that they respond to. I'd prefer to keep using Nmap for this if possible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment