Version detection: version.bind / fallbacks

[TomSellers]

The goal of this PR is to use data from a Project Sonar Internet wide survey of DNS responses to a version.bind query on both TCP and UDP to improve Nmap's version detection coverage. As part of this effort a couple of quality and consistency issues were addressed. Full disclosure, I work on the Rapid7 team that runs Project Sonar.

Note: Core version detection fallback logic was changed.

Summary

The DNS query response packet over TCP and UDP only differ by one field. The TCP version contains a two byte length field at the start of the response data. This means that we can use the same match lines for both probes if the regex is constructed with this in mind and fingerprint fallbacks work cross protocol. This PR implements cross protocol fallbacks by making changes to *AllProbes::getProbeByName which is only used in the fallback process. This should not break any existing functionality since fallbacks are only currently used to fall back to GetRequest.

Prior to the above, match lines were implemented separately in the TCP and UDP DNSVersionBindReq probe sections. Given the organic growth of these sections over time they were inconsistent in coverage and had many match quality issues. Many of the fingerprints could have never fired due to over broad fingerprints that occurred earlier in the match process. Additionally there were many fingerprints that generically matched a DNS query response but that had no service specific data in the regex.

To address this I have:

• Moved all version.bind match lines to the UDP DNSVersionBindReq probe section
• Renamed the TCP DNSVersionBindReq to TCP DNSVersionBindReqTCP and configured it to fall back to UDP DNSVersionBindReq Matchlines for this response should now only occur in one section, simplifying maintenance. I have made similar but very limited changes to the UDP and TCP DNSStatusRequest probe.
• Created new matchlines based on a Project Sonar Internet wide survey of DNS responses to a version.bind query on both TCP and UDP. The new match lines have been grouped by product/os where possible and roughly ordered by occurrence on the public Internet. We will be publishing some of the metrics in the near future.
• Improved the accuracy and coverage of cpe elements in the matches.
• Removed many matchlines that were overbroad or simply didn't match on any product specific data.
• Tweaked the existing matches and wrote the new matches to just match on the query and answer section of the DNS query response packet.
• Implemented softmatches for generic DNS errors commonly seen in responses such as Not Implemented, Refused, etc.

Building

I've tossed some notes on cloning the repo and pr as well as building Nmap here:
https://gist.github.com/TomSellers/9ad59588ca0436f2192a90437a660dcf

Example output

PORT   STATE SERVICE REASON               VERSION
53/tcp open  domain  syn-ack ttl 223      Microsoft DNS 6.0.6001 (17714650) (Windows Server 2008 SP1)
53/udp open  domain  udp-response ttl 223 Microsoft DNS 6.0.6001 (17714650) (Windows Server 2008 SP1)
Service Info: OS: Windows; CPE: cpe:/o:microsoft:windows_server_2008:-:sp1

PORT   STATE SERVICE REASON               VERSION
53/tcp open  domain  syn-ack ttl 128      Microsoft DNS 6.3.9600 (25804287) (Windows Server 2012 R2)
53/udp open  domain  udp-response ttl 128 Microsoft DNS 6.3.9600 (25804287) (Windows Server 2012 R2)
Service Info: OS: Windows; CPE: cpe:/o:microsoft:windows_server_2012:r2

PORT   STATE SERVICE REASON               VERSION
53/tcp open  domain  syn-ack ttl 128      Microsoft DNS 10.0.14393 (383905FC) (Windows Server 2016)
53/udp open  domain  udp-response ttl 128 Microsoft DNS 10.0.14393 (383905FC) (Windows Server 2016)
Service Info: OS: Windows; CPE: cpe:/o:microsoft:windows_server_2016

PORT   STATE SERVICE REASON              VERSION
53/tcp open  domain  syn-ack ttl 51      ISC BIND 9.8.2rc1 (RedHat Enterprise Linux 6)
53/udp open  domain  udp-response ttl 48 ISC BIND 9.8.2rc1 (RedHat Enterprise Linux 6)
Service Info: OS: Linux; CPE: cpe:/o:redhat:enterprise_linux:6

PORT   STATE SERVICE REASON              VERSION
53/tcp open  domain  syn-ack ttl 50      ISC BIND 9.9.5 (Debian Linux 7.0 (Wheezy))
53/udp open  domain  udp-response ttl 50 ISC BIND 9.9.5 (Debian Linux 7.0 (Wheezy))
Service Info: OS: Linux; CPE: cpe:/o:debian:debian_linux:7.0

PORT   STATE SERVICE REASON              VERSION
53/tcp open  domain  syn-ack ttl 46      ISC BIND 9.9.5 (Raspbian Linux 8.0 (Jessie based))
53/udp open  domain  udp-response ttl 46 ISC BIND 9.9.5 (Raspbian Linux 8.0 (Jessie based))
Service Info: OS: Linux; CPE: cpe:/o:debian:debian_linux:8.0

PORT   STATE SERVICE REASON              VERSION
53/tcp open  domain  syn-ack ttl 56      UltraDNS Resolver
53/udp open  domain  udp-response ttl 56 UltraDNS Resolver

PORT   STATE SERVICE REASON              VERSION
53/tcp open  domain  syn-ack ttl 52      OzymanDNS DNS tunnel
53/udp open  domain  udp-response ttl 52 OzymanDNS DNS tunnel

PORT   STATE  SERVICE REASON       VERSION
53/tcp closed domain  reset ttl 46
53/udp open   domain  udp-response Michael Tokarev rbldnsd 0.997a

PORT   STATE SERVICE REASON              VERSION
53/tcp open  domain  syn-ack ttl 41      dnsmasq 2.62
53/udp open  domain  udp-response ttl 41 Draytek DNS 1.2.3006

Example output - No match

Custom version.bind response of get lost

PORT   STATE SERVICE REASON              VERSION
53/tcp open  domain  syn-ack ttl 40      (unknown banner: get lost)
53/udp open  domain  udp-response ttl 40 (unknown banner: get lost)
2 services unrecognized despite returning data. If you know the service/version, please submit the following fingerprints at https://nmap.org/cgi-bin/submit.cgi?new-service :
==============NEXT SERVICE FINGERPRINT (SUBMIT INDIVIDUALLY)==============
SF-Port53-TCP:V=7.60SVN%I=7%D=9/11%Time=59B63A0E%P=x86_64-unknown-linux-gn
SF:u%r(DNSVersionBindReqTCP,43,"\0A\0\x06\x85\0\0\x01\0\x01\0\x01\0\0\x07v
SF:ersion\x04bind\0\0\x10\0\x03\xc0\x0c\0\x10\0\x03\0\0\0\0\0\t\x08get\x20
SF:lost\xc0\x0c\0\x02\0\x03\0\0\0\0\0\x02\xc0\x0c")%r(DNSStatusRequestTCP,
SF:E,"\0\x0c\0\0\x90\x04\0\0\0\0\0\0\0\0");
==============NEXT SERVICE FINGERPRINT (SUBMIT INDIVIDUALLY)==============
SF-Port53-UDP:V=7.60SVN%I=7%D=9/11%Time=59B63A08%P=x86_64-unknown-linux-gn
SF:u%r(DNSVersionBindReq,41,"\0\x06\x85\0\0\x01\0\x01\0\x01\0\0\x07version
SF:\x04bind\0\0\x10\0\x03\xc0\x0c\0\x10\0\x03\0\0\0\0\0\t\x08get\x20lost\x
SF:c0\x0c\0\x02\0\x03\0\0\0\0\0\x02\xc0\x0c")%r(DNSStatusRequest,C,"\0\0\x
SF:90\x04\0\0\0\0\0\0\0\0")%r(NBTStat,32,"\x80\xf0\x80\x15\0\x01\0\0\0\0\0
SF:\0\x20CKAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\0\0!\0\x01");

A "not implemented" response which has been softmatched in order to allow the chance for other probe/matchlines the opportunity to determine more information about the service. There are similar for Format Error, Server Fail, etc. responses.

PORT   STATE SERVICE REASON              VERSION
53/tcp open  domain  syn-ack ttl 45      (generic dns response: NOTIMP)
53/udp open  domain  udp-response ttl 45 (generic dns response: NOTIMP)
2 services unrecognized despite returning data. If you know the service/version, please submit the following fingerprints at https://nmap.org/cgi-bin/submit.cgi?new-service :
==============NEXT SERVICE FINGERPRINT (SUBMIT INDIVIDUALLY)==============
SF-Port53-TCP:V=7.60SVN%I=7%D=9/11%Time=59B676C4%P=x86_64-unknown-linux-gn
SF:u%r(DNSVersionBindReqTCP,20,"\0\x1e\0\x06\x81\x05\0\x01\0\0\0\0\0\0\x07
SF:version\x04bind\0\0\x10\0\x03")%r(DNSStatusRequestTCP,E,"\0\x0c\0\0\x90
SF:\x04\0\0\0\0\0\0\0\0");
==============NEXT SERVICE FINGERPRINT (SUBMIT INDIVIDUALLY)==============
SF-Port53-UDP:V=7.60SVN%I=7%D=9/11%Time=59B676BE%P=x86_64-unknown-linux-gn
SF:u%r(DNSVersionBindReq,1E,"\0\x06\x81\x05\0\x01\0\0\0\0\0\0\x07version\x
SF:04bind\0\0\x10\0\x03")%r(DNSStatusRequest,C,"\0\0\x90\x04\0\0\0\0\0\0\0
SF:\0")%r(NBTStat,32,"\x80\xf0\x80\x15\0\x01\0\0\0\0\0\0\x20CKAAAAAAAAAAAA
SF:AAAAAAAAAAAAAAAAAA\0\0!\0\x01");

[jhart-r7]

With BIND as configured on some older OS X 10.6 systems I was unable to fingerprint the service/OS over TCP or UDP with or without this PR.

Here is the base64 encoded version of the single-packet PCAP:

Cg0NCogAAABNPCsaAQAAAP//////////AwAvAE1hYyBPUyBYIDEwLjExLjYsIGJ1aWxkIDE1RzE1MTAgKERhcndpbiAxNS42LjApAAQALQBEdW1wY2FwIChXaXJlc2hhcmspIDIuMi43ICh2Mi4yLjctMC1nMTg2MWE5NikAAAAAAAAAiAAAAAEAAACcAAAAAAAAAAAABAACAAUAdXR1bjAAAAAJAAEABgAAAAsANgAAcG9ydCA1MyBhbmQgKG5ldCAxOTIuMTY4LjMyLjAvMjMgb3IgbmV0IDEwLjQuMTYuMC8yMCkAAAwALwBNYWMgT1MgWCAxMC4xMS42LCBidWlsZCAxNUcxNTEwIChEYXJ3aW4gMTUuNi4wKQAAAAAAnAAAAAYAAACIAAAAAAAAAF1XBQAGRixyZwAAAGcAAAACAAAARQAAY+/EAAA/EfMxwKghBKwQCtcANfaKAE9U2QAGhQAAAQABAAEAAAd2ZXJzaW9uBGJpbmQAABAAA8AMABAAAwAAAAAADw45LjYuMC1BUFBMRS1QMsAMAAIAAwAAAAAAAsAMAIgAAAA=

I've also submitted this as a new fingerprint.

[jhart-r7]

With BIND as configured on an old Ubuntu 7.10 system I was unable to fingerprint the service/OS over TCP or UDP with or without this PR.

Base64 encoded version of the single-packet PCAP:

1MOyoQIABAAAAAAAAAAAAAAABAAAAAAA0pacWWwJBwB9AAAAfQAAAAIAAABFAAB52KJAAD4GkRAKBBHhrBAK1wA1/Z9R0PFPD5Rqi4AYALWXUwAAAQEICpkBqbFZTTbOAEMABoUAAAEAAQABAAAHdmVyc2lvbgRiaW5kAAAQAAPADAAQAAMAAAAAAAsKOS40LjEtUDEuMcAMAAIAAwAAAAAAAsAM

Fingerprint submitted.

[jhart-r7]

Perhaps off topic, but while testing this for @TomSellers I noticed that DNS instances configured to return non-default or non-standard responses for the version.bind query could be handled a bit better. For example, years ago on some of our internal DNS servers, the BIND instance was configured to return a static, humorous phrase instead of the actual BIND version when queried for version.bind by use of a block like:

options {
  version "bind, james bind";
}

In cases like this, as currently written nmap will say the following about the endpoint:

53/tcp open  domain  (unknown dns service)

It will also provide the missing fingerprint data to be submitted to nmap.org, but it seems to imply a bigger problem than there actually is. In reality we know that the endpoint is DNS/domain and we did get a version response, it just doesn't look like any version we've seen before. I'd argue that showing what we did get for the version response is more useful than masking it. In many cases this will be enough to inform the user that some sort of hardening or custom configuration may be in use on the DNS server in question and that perhaps the fingerprint isn't good enough for general distribution with nmap.

[TomSellers]

Thanks @jhart-r7 for the feedback. I've incorporated the matches as well as your feedback about echoing the banner when not matched.

PORT   STATE SERVICE REASON              VERSION
53/tcp open  domain  syn-ack ttl 49      (unknown version: donuts)
53/udp open  domain  udp-response ttl 49 (unknown version: donuts)

==============NEXT SERVICE FINGERPRINT (SUBMIT INDIVIDUALLY)==============
SF-Port53-TCP:V=7.60SVN%I=7%D=8/22%Time=599CBB0E%P=x86_64-unknown-linux-gn
SF:u%r(DNSVersionBindReqTCP,41,"\0\?\0\x06\x85\0\0\x01\0\x01\0\x01\0\0\x07
SF:version\x04bind\0\0\x10\0\x03\xc0\x0c\0\x10\0\x03\0\0\0\0\0\x07\x06donu
SF:ts\xc0\x0c\0\x02\0\x03\0\0\0\0\0\x02\xc0\x0c")%r(DNSStatusRequestTCP,E,
SF:"\0\x0c\0\0\x90\x04\0\0\0\0\0\0\0\0");
==============NEXT SERVICE FINGERPRINT (SUBMIT INDIVIDUALLY)==============
SF-Port53-UDP:V=7.60SVN%I=7%D=8/22%Time=599CBB08%P=x86_64-unknown-linux-gn
SF:u%r(DNSVersionBindReq,3F,"\0\x06\x85\0\0\x01\0\x01\0\x01\0\0\x07version
SF:\x04bind\0\0\x10\0\x03\xc0\x0c\0\x10\0\x03\0\0\0\0\0\x07\x06donuts\xc0\
SF:x0c\0\x02\0\x03\0\0\0\0\0\x02\xc0\x0c")%r(DNSStatusRequest,C,"\0\0\x90\
SF:x04\0\0\0\0\0\0\0\0")%r(RPCCheck,C,"r\xfe\x99\x11\0\0\0\0\0\0\0\0")%r(N
SF:BTStat,32,"\x80\xf0\x80\x15\0\x01\0\0\0\0\0\0\x20CKAAAAAAAAAAAAAAAAAAAA
SF:AAAAAAAAAA\0\0!\0\x01")%r(DNS-SD,2E,"\0\0\x80\x05\0\x01\0\0\0\0\0\0\t_s
SF:ervices\x07_dns-sd\x04_udp\x05local\0\0\x0c\0\x01");

[jhart-r7]

🍩 !