Nmap stuck in ping scan phase

[cnotin]

I use the latest Nmap version compiled from source with these scanning options:

ports=80,8080,8081,8090,8000,9080,81,443,8443,4343
"$main_folder/nmap/nmap" \
-PE -PS"$ports" -PP \
-sSV -p "$ports" \
-iL "$input_file" \
-v --open \
--script http-title,http-server-header,ssl-cert,http-headers \
-T5 --script-timeout 5m --host-timeout 5m \
-oA web_recon

(yes my ping options are particular)

From time to time Nmap tends to be stuck in the ping scan phase. It seems to be blocked indefinitely because I discover it in this situation after a few hours with no progress. I kill it and resume the scan. After the resume, the problem does not appear immediately so it is very hard to diagnose and thus I do not think it happens with particular hosts either...

As you can see I already use --host-timeout as a safety against such cases but this does not seem to apply here.

It is still responsive though when I press ENTER or increase/decrease verbosity/debug.
Here is what I get:

Initiating Ping Scan at 14:12
Scanning 4096 hosts [11 ports/host]
Ping Scan Timing: About 16.93% done; ETC: 14:15 (0:02:32 remaining)
Ping Scan Timing: About 30.54% done; ETC: 14:16 (0:02:53 remaining)
Increasing send delay for 10.180.43.38 from 0 to 5 due to 11 out of 12 dropped probes since last increase.
Ping Scan Timing: About 30.65% done; ETC: 14:18 (0:04:00 remaining)
Ping Scan Timing: About 30.74% done; ETC: 14:19 (0:05:06 remaining)
Ping Scan Timing: About 30.84% done; ETC: 14:21 (0:06:12 remaining)
Ping Scan Timing: About 30.93% done; ETC: 14:22 (0:07:18 remaining)
Ping Scan Timing: About 31.03% done; ETC: 14:24 (0:08:22 remaining)
Ping Scan Timing: About 31.12% done; ETC: 14:26 (0:09:27 remaining)
Ping Scan Timing: About 31.21% done; ETC: 14:27 (0:10:30 remaining)
Ping Scan Timing: About 65.81% done; ETC: 14:20 (0:02:44 remaining)
Ping Scan Timing: About 98.93% done; ETC: 14:58 (0:00:30 remaining)
Ping Scan Timing: About 98.93% done; ETC: 14:59 (0:00:30 remaining)
Ping Scan Timing: About 98.93% done; ETC: 14:59 (0:00:31 remaining)
Ping Scan Timing: About 98.93% done; ETC: 15:00 (0:00:31 remaining)
Ping Scan Timing: About 98.93% done; ETC: 15:01 (0:00:31 remaining)
Ping Scan Timing: About 98.93% done; ETC: 15:01 (0:00:32 remaining)
Ping Scan Timing: About 98.93% done; ETC: 15:02 (0:00:32 remaining)
Ping Scan Timing: About 98.93% done; ETC: 15:02 (0:00:32 remaining)
Ping Scan Timing: About 98.93% done; ETC: 15:03 (0:00:33 remaining)
Ping Scan Timing: About 98.93% done; ETC: 15:03 (0:00:33 remaining)
[...]
Ping Scan Timing: About 98.93% done; ETC: 18:30 (0:02:46 remaining)
Ping Scan Timing: About 98.93% done; ETC: 18:33 (0:02:48 remaining)
Ping Scan Timing: About 98.93% done; ETC: 18:36 (0:02:50 remaining)
Stats: 78:24:08 elapsed; 3346473 hosts completed (62423 up), 4096 undergoing Ping Scan
Ping Scan Timing: About 98.93% done; ETC: 18:37 (0:02:51 remaining)
Stats: 78:24:09 elapsed; 3346473 hosts completed (62423 up), 4096 undergoing Ping Scan
Ping Scan Timing: About 98.93% done; ETC: 18:37 (0:02:51 remaining)
Debugging Increased to 1.
Debugging Increased to 2.
Verbosity Increased to 2.

Stats: 78:35:04 elapsed; 3346473 hosts completed (62423 up), 4096 undergoing Ping Scan
Ping Scan Timing: About 98.93% done; ETC: 18:48 (0:02:58 remaining)
Current sending rates: 0.00 packets / s, 0.00 bytes / s.
**TIMING STATS** (282967.9062s): IP, probes active/freshportsleft/retry_stack/outstanding/retranwait/onbench, cwnd/ssthresh/delay, timeout/srtt/rttvar/
   Groupstats (44/4096 incomplete): 1/*/*/*/*/* 1.00/2/* 300000/124671/153714
   a.b.c.d: 0/11/0/0/0/0 10.00/75/0 300000/-1/-1
   w.x.y.z: 0/11/0/0/0/0 10.00/75/0 300000/-1/-1
[44 lines in total with different IPs but same values]
Current sending rates: 0.00 packets / s, 0.00 bytes / s.
Overall sending rates: 1.64 packets / s, 64.48 bytes / s.

I am not aware of the internals of the scanning engine so I have troubles interpreting this output.
Do you have any suggestion for me? Could I be helpful in providing more information or performing some action to help in debugging this?

[nnposter]

Does setting up --initial-rtt-timeout and --max-rtt-timeout make a difference?

[cnotin]

I will try and let you know, thank you!

[dmiller-nmap]

One thing stands out to me: The group scan stats show there is 1 probe still active, but the 44 hosts each show 0 active probes. So either there's a probe that the scan engine is tracking that doesn't belong to any active target, or there's a bug in the accounting of how many outstanding probes there are.

What version are you using? If SVN, what revision or change are you on? I'd like to figure out if it's due to one of the changes since 7.80 or if the bug existed before that.

[cnotin]

Hello @dmiller-nmap!
Thank you for your analysis.

For the results above, I used the SVN version from commit 0b7e02c (January 27 2020)
I had the issue in December too with SVN from commit 04a3eff (November 22 2019)

It does not happen for my other scans where I do not use that many ports in -PS

[cnotin]

Hello @dmiller-nmap
FYI I had again this issue with the latest 5b6645e version (from April 6)
Here is an output extract:

Debugging Increased to 1.
Ping Scan Timing: About 97.61% done; ETC: 11:15 (0:09:22 remaining)
Stats: 64:14:26 elapsed; 3637276 hosts completed (62301 up), 4096 undergoing Ping Scan
Ping Scan Timing: About 97.61% done; ETC: 11:16 (0:09:22 remaining)
Current sending rates: 3.86 packets / s, 0.00 bytes / s.
Debugging Increased to 2.
Ping Scan Timing: About 97.61% done; ETC: 11:16 (0:09:23 remaining)
Verbosity Increased to 2.
Stats: 64:15:22 elapsed; 3637276 hosts completed (62301 up), 4096 undergoing Ping Scan
Ping Scan Timing: About 97.61% done; ETC: 11:17 (0:09:24 remaining)
Current sending rates: 3.86 packets / s, 0.00 bytes / s.
Debugging Increased to 3.
**TIMING STATS** (231327.9062s): IP, probes active/freshportsleft/retry_stack/outstanding/retranwait/onbench, cwnd/ssthresh/delay, timeout/srtt/rttvar/
   Groupstats (98/4096 incomplete): 1/*/*/*/*/* 1.00/2/* 300000/342069/171737
Current sending rates: 0.00 packets / s, 0.00 bytes / s.
Overall sending rates: 1.07 packets / s, 41.34 bytes / s.
[...]
Debugging Decreased to 2.
Ping Scan Timing: About 97.61% done; ETC: 11:20 (0:09:28 remaining)
Ping Scan Timing: About 97.61% done; ETC: 11:20 (0:09:29 remaining)
Debugging Increased to 3.
**TIMING STATS** (231553.0312s): IP, probes active/freshportsleft/retry_stack/outstanding/retranwait/onbench, cwnd/ssthresh/delay, timeout/srtt/rttvar/
   Groupstats (98/4096 incomplete): 1/*/*/*/*/* 1.00/2/* 300000/342069/171737
Current sending rates: 0.00 packets / s, 0.00 bytes / s.
Overall sending rates: 1.06 packets / s, 40.94 bytes / s.
[...]
Debugging Decreased to 2.
Ping Scan Timing: About 97.61% done; ETC: 11:21 (0:09:29 remaining)

I've resumed the scan now but I'll try to add the RTT timeout options if that happens again

[cnotin]

I added "--max-rtt-timeout 1000" in my .xml file and resumed the scan with it. If that allows to add an option in a resumed scan (I guess?) then I can tell you that it doesn't fix the issue as the scan got stuck again.

[cnotin]

I can add, if that helps, that it seems to happen usually when I'm running other network intensive activity on the same server in parallel. Potentially packet loss which isn't properly managed or disturbs some state counters?

[dmiller-nmap]

Definitely the same issue as #2153. In your case, it's not a ICMP Time Exceeded problem causing the targets to timeout during host discovery, but more likely because you have a lot of host discovery probes defined. Either way, the tagged commits here ought to address the issue. I know it can be hard to reproduce bugs like this, but I'd love positive confirmation that it's fixed before I close this issue out.

[cnotin]

Thank you very much @dmiller-nmap!
I will run a new scan in a couple of days and I'll tell you!

[cnotin]

Hi @dmiller-nmap! Good news: no more scan getting stuck under the same conditions! Thank you very much!
I'm closing this one.