Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nmap stuck in ping scan phase #1922

Closed
cnotin opened this issue Feb 14, 2020 · 10 comments
Closed

Nmap stuck in ping scan phase #1922

cnotin opened this issue Feb 14, 2020 · 10 comments
Assignees
Labels

Comments

@cnotin
Copy link

@cnotin cnotin commented Feb 14, 2020

I use the latest Nmap version compiled from source with these scanning options:

ports=80,8080,8081,8090,8000,9080,81,443,8443,4343
"$main_folder/nmap/nmap" \
-PE -PS"$ports" -PP \
-sSV -p "$ports" \
-iL "$input_file" \
-v --open \
--script http-title,http-server-header,ssl-cert,http-headers \
-T5 --script-timeout 5m --host-timeout 5m \
-oA web_recon

(yes my ping options are particular)

From time to time Nmap tends to be stuck in the ping scan phase. It seems to be blocked indefinitely because I discover it in this situation after a few hours with no progress. I kill it and resume the scan. After the resume, the problem does not appear immediately so it is very hard to diagnose and thus I do not think it happens with particular hosts either...

As you can see I already use --host-timeout as a safety against such cases but this does not seem to apply here.

It is still responsive though when I press ENTER or increase/decrease verbosity/debug.
Here is what I get:

Initiating Ping Scan at 14:12
Scanning 4096 hosts [11 ports/host]
Ping Scan Timing: About 16.93% done; ETC: 14:15 (0:02:32 remaining)
Ping Scan Timing: About 30.54% done; ETC: 14:16 (0:02:53 remaining)
Increasing send delay for 10.180.43.38 from 0 to 5 due to 11 out of 12 dropped probes since last increase.
Ping Scan Timing: About 30.65% done; ETC: 14:18 (0:04:00 remaining)
Ping Scan Timing: About 30.74% done; ETC: 14:19 (0:05:06 remaining)
Ping Scan Timing: About 30.84% done; ETC: 14:21 (0:06:12 remaining)
Ping Scan Timing: About 30.93% done; ETC: 14:22 (0:07:18 remaining)
Ping Scan Timing: About 31.03% done; ETC: 14:24 (0:08:22 remaining)
Ping Scan Timing: About 31.12% done; ETC: 14:26 (0:09:27 remaining)
Ping Scan Timing: About 31.21% done; ETC: 14:27 (0:10:30 remaining)
Ping Scan Timing: About 65.81% done; ETC: 14:20 (0:02:44 remaining)
Ping Scan Timing: About 98.93% done; ETC: 14:58 (0:00:30 remaining)
Ping Scan Timing: About 98.93% done; ETC: 14:59 (0:00:30 remaining)
Ping Scan Timing: About 98.93% done; ETC: 14:59 (0:00:31 remaining)
Ping Scan Timing: About 98.93% done; ETC: 15:00 (0:00:31 remaining)
Ping Scan Timing: About 98.93% done; ETC: 15:01 (0:00:31 remaining)
Ping Scan Timing: About 98.93% done; ETC: 15:01 (0:00:32 remaining)
Ping Scan Timing: About 98.93% done; ETC: 15:02 (0:00:32 remaining)
Ping Scan Timing: About 98.93% done; ETC: 15:02 (0:00:32 remaining)
Ping Scan Timing: About 98.93% done; ETC: 15:03 (0:00:33 remaining)
Ping Scan Timing: About 98.93% done; ETC: 15:03 (0:00:33 remaining)
[...]
Ping Scan Timing: About 98.93% done; ETC: 18:30 (0:02:46 remaining)
Ping Scan Timing: About 98.93% done; ETC: 18:33 (0:02:48 remaining)
Ping Scan Timing: About 98.93% done; ETC: 18:36 (0:02:50 remaining)
Stats: 78:24:08 elapsed; 3346473 hosts completed (62423 up), 4096 undergoing Ping Scan
Ping Scan Timing: About 98.93% done; ETC: 18:37 (0:02:51 remaining)
Stats: 78:24:09 elapsed; 3346473 hosts completed (62423 up), 4096 undergoing Ping Scan
Ping Scan Timing: About 98.93% done; ETC: 18:37 (0:02:51 remaining)
Debugging Increased to 1.
Debugging Increased to 2.
Verbosity Increased to 2.

Stats: 78:35:04 elapsed; 3346473 hosts completed (62423 up), 4096 undergoing Ping Scan
Ping Scan Timing: About 98.93% done; ETC: 18:48 (0:02:58 remaining)
Current sending rates: 0.00 packets / s, 0.00 bytes / s.
**TIMING STATS** (282967.9062s): IP, probes active/freshportsleft/retry_stack/outstanding/retranwait/onbench, cwnd/ssthresh/delay, timeout/srtt/rttvar/
   Groupstats (44/4096 incomplete): 1/*/*/*/*/* 1.00/2/* 300000/124671/153714
   a.b.c.d: 0/11/0/0/0/0 10.00/75/0 300000/-1/-1
   w.x.y.z: 0/11/0/0/0/0 10.00/75/0 300000/-1/-1
[44 lines in total with different IPs but same values]
Current sending rates: 0.00 packets / s, 0.00 bytes / s.
Overall sending rates: 1.64 packets / s, 64.48 bytes / s.

I am not aware of the internals of the scanning engine so I have troubles interpreting this output.
Do you have any suggestion for me? Could I be helpful in providing more information or performing some action to help in debugging this?

@nnposter
Copy link

@nnposter nnposter commented Feb 14, 2020

Does setting up --initial-rtt-timeout and --max-rtt-timeout make a difference?

@cnotin
Copy link
Author

@cnotin cnotin commented Feb 14, 2020

I will try and let you know, thank you!

@dmiller-nmap
Copy link

@dmiller-nmap dmiller-nmap commented Feb 18, 2020

One thing stands out to me: The group scan stats show there is 1 probe still active, but the 44 hosts each show 0 active probes. So either there's a probe that the scan engine is tracking that doesn't belong to any active target, or there's a bug in the accounting of how many outstanding probes there are.

What version are you using? If SVN, what revision or change are you on? I'd like to figure out if it's due to one of the changes since 7.80 or if the bug existed before that.

@cnotin
Copy link
Author

@cnotin cnotin commented Feb 18, 2020

Hello @dmiller-nmap!
Thank you for your analysis.

For the results above, I used the SVN version from commit 0b7e02c (January 27 2020)
I had the issue in December too with SVN from commit 04a3eff (November 22 2019)

It does not happen for my other scans where I do not use that many ports in -PS

@cnotin
Copy link
Author

@cnotin cnotin commented Apr 17, 2020

Hello @dmiller-nmap
FYI I had again this issue with the latest 5b6645e version (from April 6)
Here is an output extract:

Debugging Increased to 1.
Ping Scan Timing: About 97.61% done; ETC: 11:15 (0:09:22 remaining)
Stats: 64:14:26 elapsed; 3637276 hosts completed (62301 up), 4096 undergoing Ping Scan
Ping Scan Timing: About 97.61% done; ETC: 11:16 (0:09:22 remaining)
Current sending rates: 3.86 packets / s, 0.00 bytes / s.
Debugging Increased to 2.
Ping Scan Timing: About 97.61% done; ETC: 11:16 (0:09:23 remaining)
Verbosity Increased to 2.
Stats: 64:15:22 elapsed; 3637276 hosts completed (62301 up), 4096 undergoing Ping Scan
Ping Scan Timing: About 97.61% done; ETC: 11:17 (0:09:24 remaining)
Current sending rates: 3.86 packets / s, 0.00 bytes / s.
Debugging Increased to 3.
**TIMING STATS** (231327.9062s): IP, probes active/freshportsleft/retry_stack/outstanding/retranwait/onbench, cwnd/ssthresh/delay, timeout/srtt/rttvar/
   Groupstats (98/4096 incomplete): 1/*/*/*/*/* 1.00/2/* 300000/342069/171737
Current sending rates: 0.00 packets / s, 0.00 bytes / s.
Overall sending rates: 1.07 packets / s, 41.34 bytes / s.
[...]
Debugging Decreased to 2.
Ping Scan Timing: About 97.61% done; ETC: 11:20 (0:09:28 remaining)
Ping Scan Timing: About 97.61% done; ETC: 11:20 (0:09:29 remaining)
Debugging Increased to 3.
**TIMING STATS** (231553.0312s): IP, probes active/freshportsleft/retry_stack/outstanding/retranwait/onbench, cwnd/ssthresh/delay, timeout/srtt/rttvar/
   Groupstats (98/4096 incomplete): 1/*/*/*/*/* 1.00/2/* 300000/342069/171737
Current sending rates: 0.00 packets / s, 0.00 bytes / s.
Overall sending rates: 1.06 packets / s, 40.94 bytes / s.
[...]
Debugging Decreased to 2.
Ping Scan Timing: About 97.61% done; ETC: 11:21 (0:09:29 remaining)

I've resumed the scan now but I'll try to add the RTT timeout options if that happens again

@cnotin
Copy link
Author

@cnotin cnotin commented Apr 24, 2020

I added "--max-rtt-timeout 1000" in my .xml file and resumed the scan with it. If that allows to add an option in a resumed scan (I guess?) then I can tell you that it doesn't fix the issue as the scan got stuck again.

@cnotin
Copy link
Author

@cnotin cnotin commented Jul 3, 2020

I can add, if that helps, that it seems to happen usually when I'm running other network intensive activity on the same server in parallel. Potentially packet loss which isn't properly managed or disturbs some state counters?

@dmiller-nmap
Copy link

@dmiller-nmap dmiller-nmap commented Oct 23, 2020

Definitely the same issue as #2153. In your case, it's not a ICMP Time Exceeded problem causing the targets to timeout during host discovery, but more likely because you have a lot of host discovery probes defined. Either way, the tagged commits here ought to address the issue. I know it can be hard to reproduce bugs like this, but I'd love positive confirmation that it's fixed before I close this issue out.

@cnotin
Copy link
Author

@cnotin cnotin commented Oct 23, 2020

Thank you very much @dmiller-nmap!
I will run a new scan in a couple of days and I'll tell you!

@cnotin
Copy link
Author

@cnotin cnotin commented Nov 4, 2020

Hi @dmiller-nmap! Good news: no more scan getting stuck under the same conditions! Thank you very much!
I'm closing this one.

@cnotin cnotin closed this Nov 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants