Support for maximum body size and Content-Encoding #1571
Thanks for this important work! I have a quick question before I review: how does this work in conjunction with HTTP pipelining? You said "leaving the open socket in a state unfit for further use," but if we're expecting to receive more responses on the socket, we'd have to continue to discard the rest of the body to get to the next response.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I have been using the size limit for about a month and the gzip support for additional two. That said, I recognize that this is a non-trivial change in code that is underpinning a large variety of scripts so I fully expect that bugs will crop up.
Committed as r37627 (df26932).
7.80: Here is the full list of significant changes: o [Windows] The Npcap Windows packet capturing library (https://npcap.org/) is faster and more stable than ever. Nmap 7.80 updates the bundled Npcap from version 0.99-r2 to 0.9982, including all of these changes from the last 15 Npcap releases: https://nmap.org/npcap/changelog o [NSE] Added 11 NSE scripts, from 8 authors, bringing the total up to 598! They are all listed at https://nmap.org/nsedoc/, and the summaries are below: + broadcast-hid-discoveryd discovers HID devices on a LAN by sending a discoveryd network broadcast probe. + broadcast-jenkins-discover discovers Jenkins servers on a LAN by sending a discovery broadcast probe. + http-hp-ilo-info extracts information from HP Integrated Lights-Out (iLO) servers. + http-sap-netweaver-leak detects SAP Netweaver Portal with the Knowledge Management Unit enabled with anonymous access. + https-redirect detects HTTP servers that redirect to the same port, but with HTTPS. Some nginx servers do this, which made ssl-* scripts not run properly. + lu-enum enumerates Logical Units (LU) of TN3270E servers. + rdp-ntlm-info extracts Windows domain information from RDP services. + smb-vuln-webexec checks whether the WebExService is installed and allows code execution. + smb-webexec-exploit exploits the WebExService to run arbitrary commands with SYSTEM privileges. + ubiquiti-discovery extracts information from the Ubiquiti Discovery service and assists version detection. + vulners queries the Vulners CVE database API using CPE information from Nmap's service and application version detection. o Use pcap_create instead of pcap_live_open in Nmap, and set immediate mode on the pcap descriptor. This solves packet loss problems on Linux and may improve performance on other platforms. o [NSE] Collected utility functions for string processing into a new library, stringaux.lua. o [NSE] New rand.lua library uses the best sources of random available on the system to generate random strings. o [NSE] New library, oops.lua, makes reporting errors easy, with plenty of debugging detail when needed, and no clutter when not. o [NSE] Collected utility functions for manipulating and searching tables into a new library, tableaux.lua. o [NSE] New knx.lua library holds common functions and definitions for communicating with KNX/Konnex devices. o [NSE] The HTTP library now provides transparent support for gzip- encoded response body. (See nmap/nmap#1571 for an overview.) o [Nsock][Ncat] Add AF_VSOCK (Linux VM sockets) functionality to Nsock and Ncat. VM sockets are used for communication between virtual machines and the hypervisor. o [Security][Windows] Address CVE-2019-1552 in OpenSSL by building with the prefix "C:\Program Files (x86)\Nmap\OpenSSL". This should prevent unauthorized users from modifying OpenSSL defaults by writing configuration to this directory. o [Security] Reduced LibPCRE resource limits so that version detection can't use as much of the stack. Previously Nmap could crash when run on low-memory systems against target services which are intentionally or accidentally difficult to match. Someone assigned CVE-2018-15173 for this issue. o Deprecate and disable the -PR (ARP ping) host discovery option. ARP ping is already used whenever possible, and the -PR option would not force it to be used in any other case. o [NSE] bin.lua is officially deprecated. Lua 5.3, added 2 years ago in Nmap 7.25BETA2, has native support for binary data packing via string.pack and string.unpack. All existing scripts and libraries have been updated. o [NSE] Completely removed the bit.lua NSE library. All of its functions are replaced by native Lua bitwise operations, except for `arshift` (arithmetic shift) which has been moved to the bits.lua library. [Daniel Miller] o [NSE] The HTTP library is now enforcing a size limit on the received response body. The default limit can be adjusted with a script argument, which applies to all scripts, and can be overridden case-by-case with an HTTP request option. (See nmap/nmap#1571 for details.) o [NSE] CR characters are no longer treated as illegal in script XML output. o Allow resuming nmap scan with lengthy command line [Clément Notin] o [NSE] Add TLS support to rdp-enum-encryption. Enables determining protocol version against servers that require TLS and lays ground work for some NLA/CredSSP information collection. o [NSE] Address two protocol parsing issues in rdp-enum-encryption and the RDP nse library which broke scanning of Windows XP. Clarify protocol types o [NSE] Script http-fileupload-exploiter failed to locate its resource file unless executed from a specific working directory. o [NSE] Avoid clobbering the "severity" and "ignore_404" values of fingerprints in http-enum. None of the standard fingerprints uses these fields. o [NSE] Fix a crash caused by a double-free of libssh2 session data when running SSH NSE scripts against non-SSH services. o [NSE] Updates the execution rule of the mongodb scripts to be able to run on alternate ports. o [Ncat] Allow Ncat to connect to servers on port 0, provided that the socket implementation allows this. o Update the included libpcap to 1.9.0. o [NSE] Fix a logic error that resulted in scripts not honoring the smbdomain script-arg when the target provided a domain in the NTLM challenge. o [Nsock] Avoid a crash (Protocol not supported) caused by trying to reconnect with SSLv2 when an error occurs during DTLS connect. [Daniel Miller] o [NSE] Removed OSVDB references from scripts and replaced them with BID references where possible. o [NSE] Updates TN3270.lua and adds argument to disable TN3270E o RMI parser could crash when encountering invalid input [Clément Notin] o Avoid reporting negative latencies due to matching an ARP or ND response to a probe sent after it was recieved. o [Ncat] To avoid confusion and to support non-default proxy ports, option --proxy now requires a literal IPv6 address to be specified using square-bracket notation, such as --proxy o [Ncat] New ncat option provides control over whether proxy destinations are resolved by the remote proxy server or locally, by Ncat itself. See option --proxy-dns. o [NSE] Updated script ftp-syst to prevent potential endless looping. o New service probes and match lines for v1 and v2 of the Ubiquiti Discovery protocol. Devices often leave the related service open and it exposes significant amounts of information as well as the risk of being used as part of a DDoS. New nmap-payload entry for v1 of the protocol. o [NSE] Removed hostmap-ip2hosts.nse as the API has been broken for a while and the service was completely shutdown on Feb 17th, 2019. [Paulino Calderon] o [NSE] Adds TN3270E support and additional improvements to tn3270.lua and updates tn3270-screen.nse to display the new setting. o [NSE] Updates product codes and adds a check for response length in enip-info.nse. The script now uses string.unpack. o [Ncat] Temporary RSA keys are now 2048-bit to resolve a compatibility issue with OpenSSL library configured with security level 2, as seen on current Debian or Kali. o [NSE] Fix a crash (double-free) when using SSH scripts against non-SSH services. o [Zenmap] Fix a crash when Nmap executable cannot be found and the system PATH contains non-UTF-8 bytes, such as on Windows. o [Zenmap] Fix a crash in results search when using the dir: operator: AttributeError: 'SearchDB' object has no attribute 'match_dir' [Daniel Miller] o [Ncat] Fixed an issue with Ncat -e on Windows that caused early termination of connections. o [NSE] Fix a false-positive in http-phpmyadmin-dir-traversal when the server responds with 200 status to a POST request to any URI. o [NSE] New vulnerability state in vulns.lua, UNKNOWN, is used to indicate that testing could not rule out vulnerability. o When searching for Lua header files, actually use them where they are found instead of forcing /usr/include. [Fabrice Fontaine, Daniel Miller] o [NSE] Script traceroute-geolocation no longer crashes when www.GeoPlugin.net returns null coordinates o Limit verbose -v and debugging -d levels to a maximum of 10. Nmap does not use higher levels internally. o [NSE] tls.lua when creating a client_hello message will now only use a SSLv3 record layer if the protocol version is SSLv3. Some TLS implementations will not handshake with a client offering less than TLSv1.0. Scripts will have to manually fall back to SSLv3 to talk to SSLv3-only servers. o [NSE] Fix a few false-positive conditions in ssl-ccs-injection. TLS implementations that responded with fatal alerts other than "unexpected message" had been falsely marked as vulnerable. o Emergency fix to Nmap's birthday announcement so Nmap wishes itself a "Happy 21st Birthday" rather than "Happy 21th" in verbose mode (-v) on September 1, 2018. o Start host timeout clocks when the first probe is sent to a host, not when the hostgroup is started. Sometimes a host doesn't get probes until late in the hostgroup, increasing the chance it will time out. o [NSE] Support for edns-client-subnet (ECS) in dns.lua has been improved by: - - Properly trimming ECS address, as mandated by RFC 7871 - Fixing a bug that prevented using the same ECS option table more than once o [Ncat] Fixed communication with commands launched with -e or -c on Windows, especially when --ssl is used. o [NSE] Script http-default-accounts can now select more than one fingerprint category. It now also possible to select fingerprints by name to support very specific scanning. o [NSE] Script http-default-accounts was not able to run against more than one target host/port. o [NSE] New script-arg `http.host` allows users to force a particular value for the Host header in all HTTP requests. o [NSE] Use smtp.domain script arg or target's domain name instead of "example.com" in EHLO command used for STARTTLS. o [NSE] Fix brute.lua's BruteSocket wrapper, which was crashing Nmap with an assertion failure due to socket mixup [Daniel Miller]: nmap: nse_nsock.cc:672: int receive_buf(lua_State*, int, lua_KContext): Assertion `lua_gettop(L) == 7' failed. o [NSE] Handle an error condition in smb-vuln-ms17-010 caused by IPS closing the connection. o [Ncat] Fixed literal IPv6 URL format for connecting through HTTP proxies. o [NSE] Updates vendors from ODVA list for enip-info. [NothinRandom] o [NSE] Add two common error strings that improve MySQL detection by the script http-sql-injection. o [NSE] Fix bug in http-vuln-cve2006-3392 that prevented the script to generate the vulnerability report correctly. o [NSE] Fix bug related to screen rendering in NSE library tn3270. This patch also improves the brute force script tso-brute. o [NSE] Fix SIP, SASL, and HTTP Digest authentication when the algorithm contains lowercase characters. o Nmap could be fooled into ignoring TCP response packets if they used an unknown TCP Option, which would misalign the validation, causing it to fail. o [NSE]The HTTP response parser now tolerates status lines without a reason phrase, which improves compatibility with some HTTP servers. o [NSE]] Parser for HTTP Set-Cookie header is now more compliant with RFC 6265: - empty attributes are tolerated - double quotes in cookie and/or attribute values are treated literally - attributes with empty values and value-less attributes are parsed equally - attributes named "name" or "value" are ignored o [NSE] Fix parsing http-grep.match script-arg. [Hans van den Bogert] o [Zenmap] Avoid a crash when recent_scans.txt cannot be written to. o Fixed --resume when the path to Nmap contains spaces. o New service probe and match lines for adb, the Android Debug Bridge, which allows remote code execution and is left enabled by default on many devices.