Add --defeat-icmp-ratelimit option for UDP scanning. [Issue #216] #353

Closed
wants to merge 7 commits into
from

Conversation

Projects
None yet
2 participants

This PR patches Issue 216.

As was discussed:

More UDP payloads will definitely improve scan times against open
services, but the real time-killer is the closed ports. If a system is
using a firewall to drop probes, then Nmap will detect those drops
pretty quickly, especially if the few open services are quick to
respond (giving an accurate RTT and packet loss estimate). But if the
host is rate-limiting the ICMP responses, then Nmap knows it ought to
wait for them, and so it just keeps slowing down its probes until it
matches the (slow) rate limit that the target is using for responses.

For TCP scans, most hosts will not rate-limit RST packets (closed port
responses), but some do. In this case, we have the
--defeate-rst-ratelimit option which abandons accuracy in
distinguishing closed ports from filtered ones. This allows it to
focus only on open ports (which respond with SYN/ACK). We don't do the
same for UDP because a port may be open even if it doesn't respond at
all (because we haven't sent the proper payload).

I can conceive of a --defeat-icmp-ratelimit option that would treat
open|filtered UDP ports as closed, and only report the definitely-open
ones as up. This would mean a greater loss of accuracy, since some
open services would be missed. This could be partially remedied by
expanding the nmap-payloads file to include the UDP probes from the
nmap-service-probes file. As you saw in your output, service scan will
often change the status of a port from open|filtered to open because
of a received response.

I figured out that Nmap initializes all ports as open|filtered. A port's state is changed only when
some kind of response is received, this also triggers timing adjustment.

One might skim through this email I sent to dev@nmap.org earlier:

My first approach was to:

  1. add --defeat-icmp-ratelimit option for UDP scan only
  2. modify ultrascan_port_probe_update(...) in scan_engine.cc analogous to --defeat-rst-ratelimit

/* Do not slow down if
1) we are in --defeat-icmp-ratelimit mode
2) the new state is open|filtered
3) this is a UDP scan */
if (rcvdtime != NULL
&& o.defeat_icmp_ratelimit && newstate == PORT_OPENFILTERED
&& USI->udp_scan) {
if (probe->tryno > 0)
adjust_timing = false;
adjust_ping = false;
}

I figured out that this is flawed, as ultrascan_port_probe_update(...) is never called with
newstate == PORT_OPENFILTERED during UDP scan. Instead the default port state is
initialized as open|filtered. When nmap sends a probe and does not get a response it just
mark the probe as timeout'ed, stages the probe to retransmit (if other parameters and
timers allow) and does not change the port's state, which is logical. So the port's state is
changed only when nmap gets any kind of response from server.

As I understand, the option --defeat-icmp-ratelimit should let nmap send only one probe
per a port. If there is no response, then the port is treated as open|filtered and we do not
attempt any retransmissions for that port to save time. Would it be reasonable to delete
all probes that are marked as timeout'ed if --defeat-icmp-ratelimit enabled, in other words
just prevent any retransmissions?

Is it a good approach overall? We already lose accuracy since some services might
be missed because we send wrong payload. Furthermore, UDP does not guarantee
delivery, so response might just be dropped on the way back to our machine due to
congestion (and nmap would treat corresponding port as open|filtered).

I realize that discarding timeout'ed probes is a bad idea. Since UDP does not guarantee
delivery packets might be lost because of congestion.

If we want to scan faster, what we should not do is to adjust timing when Nmap changes
port's state because of ICMP response. Otherwise Nmap will behave as described:

If the host is rate-limiting the ICMP responses, then Nmap knows it ought to
wait for them, and so it just keeps slowing down its probes until it
matches the (slow) rate limit that the target is using for responses.

We do not want such behavior. On the other hand we still want Nmap to address
network congestion problems, so we adjust timing when Nmap changes a port's
state to open (a UDP response was received).

@ghost

ghost commented Jun 27, 2016

Hello @sergeykhegay, I have successfully reviewed, built and tested your modifications and everything worked. I will speak to my mentor about it and we'll keep you updated, but I am positive that everything is ready to commit.
Cheers,
Vincent

@ghost Unknown commented on the diff Jul 8, 2016

scan_engine.cc
@@ -2107,6 +2107,25 @@ void ultrascan_port_probe_update(UltraScanInfo *USI, HostScanStats *hss,
adjust_timing = false;
adjust_ping = false;
}
+ /* Do not slow down if
+ 1) we are in --defeat-icmp-ratelimit mode
+ 2) the new state is closed or filtered
+ 3) this is a UDP scan
+ We don't want to adjust timing when we get ICMP response, as the host might
+ be ratelimiting them. E.g. the port is actually closed, but the host ratelimiting
+ ICMP responses so we had to retransmit the probe several times in order to
+ match the (slow) rate limit that the target is using for responses. We
+ do not want to waste time on such ports.
+ On the other hand if the port is detected to be open it is a good idea to
+ adjust timing as we could have done retransmissions due to conjested network */
+ if (rcvdtime != NULL
+ && o.defeat_icmp_ratelimit
+ && (newstate == PORT_CLOSED || newstate == PORT_FILTERED)
@ghost

ghost Jul 8, 2016

According to @dmiller-nmap in issue #216, we should:

treat open|filtered UDP ports as closed, and only report the definitely-open ones as up

Thus I would suggest to replace this line (2123) by:
&& (newstate == PORT_CLOSED || (newstate == PORT_OPEN && newstate == PORT_FILTERED))

That way, if the port is closed or if the port is in state open|filtered, we do not want to slow down and thus don't adjust timing and ping.

Waiting for your feedback,
Vincent

@sergeykhegay

sergeykhegay Jul 8, 2016

Hello Vincent,

This change would not make any sense because (newstate == PORT_OPEN && newstate == PORT_FILTERED) will always be false, as new state cannot be both in PORT_OPEN and PORT_FILTERED states.

There is another option to change the condition as following:
&& (newstate == PORT_CLOSED || (newstate == PORT_OPENFILTERED))
but, as I have already pointed out, this will not work. By default if we use UDP scan, all ports are initialized with PORT_OPENFILTERED state. This makes sense because we do not know yet if the port is open/closed/filtered, we can draw our conclusion only when we get an ICMP message (probably meaning that the port is closed or rate limited) or a UDP response (port open) or a timeout (contested network, firewall, rogue server).

Check this code out to understand how a port's state is changed. Basically, when we get an ICMP response during a UDP scan, the port's state can be changed to either PORT_CLOSED or PORT_FILTERED. But those ICMP responses might be rate limited, thus do not represent adequate information about probe timings, hence we do not adjust timings in the --defeat-icmp-ratelimit mode.

Makes sense?

@ghost

ghost Jul 10, 2016

Hello @sergeykhegay,

Thanks for your reply with explanation. I understand why it cannot work like that. The thing is that with your code, it will show some ports as open|filtered which should be closed if we took the time to slow down and wait for an ICMP response.

After more investigation, it seems very hard to differentiate them, especially because of firewalls and ACL filtering the ICMP responses of the target, leading to even more inaccuracy... I will have to talk to my mentor about it tomorrow.

Talk to you soon

@sergeykhegay

sergeykhegay Jul 11, 2016

Yes, we exactly sacrifice accuracy to get a speed up.
Thanks for your work, and I'm waiting for updates.

@sergeykhegay I've reviewed this and I think it is nearly ready to commit. My only remaining concern is that the default state for timed-out ports is "open|filtered", which means that service scan will try to probe them and NSE will run against them, but they are more likely closed than open.

The easiest fix to this seems to be changing set_default_port_state in scan_engine.cc to make the default for UDP scans PORT_CLOSEDFILTERED if o.defeat_icmp_ratelimit is set. I would have preferred PORT_UNKNOWN but that has the side effect of not being collapsible, so you would get thousands of lines of output on most scans. Example patch:

--- a/scan_engine.cc
+++ b/scan_engine.cc
@@ -851,7 +851,7 @@ static void set_default_port_state(std::vector<Target *> &targets, stype scantyp
       (*target)->ports.setDefaultPortState(IPPROTO_TCP, PORT_OPENFILTERED);
       break;
     case UDP_SCAN:
-      (*target)->ports.setDefaultPortState(IPPROTO_UDP, PORT_OPENFILTERED);
+      (*target)->ports.setDefaultPortState(IPPROTO_UDP, o.defeat_icmp_ratelimit ? PORT_CLOSEDFILTERED : PORT_OPENFILTERED);
       break;
     case IPPROT_SCAN:
       (*target)->ports.setDefaultPortState(IPPROTO_IP, PORT_OPENFILTERED);

I think that maybe adding a warning at the end of the scan would be helpful if we did this. Something like "WARNING: Some ports marked closed|filtered may actually be open. For more accurate results, do not use --defeat-icmp-ratelimit"

@dmiller-nmap dmiller-nmap commented on an outdated diff Jul 20, 2016

scan_engine.h
@@ -206,7 +206,7 @@ int determineScanGroupSize(int hosts_scanned_so_far,
class UltraScanInfo;
-struct ppkt { /* Beginning of ICMP Echo/Timestamp header */
+struct ppkt { /* Beginning of ICMP Echo/Timestamp header */
@dmiller-nmap

dmiller-nmap Jul 20, 2016

This change isn't necessary for this patch. If you want to make a separate whitespace-only commit to clean up stuff like this, feel free to do so elsewhere.

sergeykhegay added some commits Jul 25, 2016

@sergeykhegay sergeykhegay Restore whitespace 806561c
@sergeykhegay sergeykhegay Set the default UDP port state as PORT_CLOSEDFILTERED instead of
PORT_OPENFILTERED _if_ o.defeat_icmp_ratelimit is set.

This will prevent service scan probing and NSE running against
supposedly closed ports.
ae43744
@sergeykhegay sergeykhegay If o.defeat_icmp_ratelimit option is used, add a notice about
possible inaccuracy of the results at the end of the scan.

Some ports marked closed|filtered may actually be open.
1ce373e
@sergeykhegay sergeykhegay Change "note" to "WARNING" d31fc0f

@sergeykhegay sergeykhegay commented on the diff Jul 25, 2016

output.cc
@@ -2526,6 +2526,9 @@ void printfinaloutput() {
log_write(LOG_PLAIN, "OS detection performed. Please report any incorrect results at https://nmap.org/submit/ .\n");
else if (o.servicescan)
log_write(LOG_PLAIN, "Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .\n");
+ else if (o.udpscan && o.defeat_icmp_ratelimit)
+ log_write(LOG_PLAIN, "WARNING: Some ports marked closed|filtered may actually be open. For more accurate results, do not use --defeat-icmp-ratelimit .\n");
@sergeykhegay

sergeykhegay Jul 25, 2016

I tried to use "Note: Some ports marked ...". It did not catch attention as it should have done, so I changed to the initial Daniel's suggestion, warning.

Should I have used error("WARNING: Some ports marked...") or current version is ok?

@sergeykhegay

sergeykhegay Jul 25, 2016

@dmiller-nmap, @w0nan0w I have made changes as was suggested. Whitespace commit was removed.

nmap-bot closed this in 3f1ad07 Dec 9, 2016

@suraj51k suraj51k added a commit to suraj51k/nmap that referenced this pull request Jan 31, 2017

@bonsaiviking @suraj51k bonsaiviking + suraj51k New option --defeat-icmp-ratelimit. Closes #353, Fixes #216 e59e89d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment