Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for maximum body size and Content-Encoding #1571

Closed
wants to merge 5 commits into from

Conversation

@nnposter
Copy link

@nnposter nnposter commented Apr 24, 2019

This patch combines the implementation for both #473 and #1493. Due to circumstances, it was not practical to break the two features apart (but it can be done if absolutely necessary).

Content-Encoding

  • The library now transparently handles the Content-Encoding header, recognizing encodings identity and gzip.
  • The body member of the HTTP response object now contains the processed (decoded) body.
  • The original body, as received from the server, is preserved in a new member, rawbody.
  • Corrupted encoded body is treated like any other response error, including the response status being nil.
  • Unsupported encoding is not triggering an error, instead simply resulting in undecoded body. Note that this body might not be the same as rawbody because other encodings might have been already processed.
  • New response member decoded contains a list of content encodings that were successfully processed.
  • New response member undecoded contains a list of encodings that could not be processed, either because they are not currently supported or the body is corrupt. In other words, a body was successfully decoded if this list is empty (or nil, if no encodings were used in the first place).
  • Returned content-encoding and content-length headers are adjusted to remain consistent with body. (If all encodings got processed then the content-encoding header is removed altogether, which can serve an alternate test for decoding problems.)

Body Size Limit

  • Newly implemented library parameter http.max-body-size defines the maximum response body size (in bytes) the library is willing to receive. The default value is 2 MB. The limit can be completely disabled by setting the value to -1. Note that the limit is enforced by default.
  • This behavior can be overridden case-by-case with request option max_body_size, with its values having the same meaning.
  • This enforcement applies to raw bodies as they are received, regardless of method (Content-Length, Transfer-Encoding, or connection termination)
  • This enforcement also applies when bodies are getting decoded according to Content-Encoding. This provides protection against the so-called zip bomb.
  • Bodies that exceed the limit are still made available in truncated form through the returned response object. (See below for details.) The truncated body is guaranteed to match the size limit exactly.
  • The response object would have status member equal to nil. In other words, the response is composed as a failure, so scripts would not use it unless they explicitly want to.
  • Just as with the status, member body would be nil so scripts would not accidentally use the truncated body.
  • In case of any response processing errors, not just the oversize body, a newly implemented member incomplete contains the partially built response object. This way scripts can for example inspect response status code (preserved as response.incomplete.status) even if the header or body parsing failed.
  • In summary, the truncated body would be available as response.incomplete.body and scripts that do not care about the truncation could obtain the body by sourcing expression response.body or response.incomplete.body.
  • By setting a newly implemented library parameter http.truncated-ok to true, it is possible to suppress the oversized body error, treating the response object as a success, and return the truncated body in the body member as usual. In other words, member incomplete becomes the response itself and a newly implemented response member truncated is set to true.
  • Library parameter http.truncated-ok can be overridden case-by-case with request option truncated_ok.
  • Truncated responses are never cached.
  • By enforcing the body size limit, the library would stop receiving socket data when this limit is reached, leaving the open socket in a state unfit for further use. In other words, the library would not try to "suck up" and discard the remainder of the body.
  • When using a pipeline, the connection is restarted, beginning with the next request. This means that one or more requests following a truncated response could be submitted more than once. To compensate for this behavior, either do not use pipeline for state-changing requests or set parameter http.max-pipeline to 1, which still uses persistent connections but processes only one request at at time.

Please review and comment.

@dmiller-nmap
Copy link

@dmiller-nmap dmiller-nmap commented Apr 27, 2019

Thanks for this important work! I have a quick question before I review: how does this work in conjunction with HTTP pipelining? You said "leaving the open socket in a state unfit for further use," but if we're expecting to receive more responses on the socket, we'd have to continue to discard the rest of the body to get to the next response.

@nnposter
Copy link
Author

@nnposter nnposter commented Apr 27, 2019

Discarding the oversized body just to get to the next response would IMHO defeat the purpose of the size limit. I am instead restarting the connection. You can see it on line 2010:

 if connsent >= connlimit or resp.truncated or not socket:get_info() then
Copy link

@dmiller-nmap dmiller-nmap left a comment

This looks like really great stuff. I left a few comments, but only minor things need to change. Looking forward to seeing this committed!

nselib/http.lua Outdated Show resolved Hide resolved
maxlen = maxlen - #part
if maxlen < 0 then
table.insert(parts, part:sub(1, maxlen - 1))
return nil, ERR_OVERSIZED_BODY, table.concat(parts)

This comment has been minimized.

@dmiller-nmap

dmiller-nmap May 17, 2019

In a future revision, we might consider simply receiving and discarding the rest of the body in this case, at least while pipelining. It would take more time and bandwidth than just dropping the connection, but it would avoid missing later response bodies in the pipeline. I'm not going to suggest you hold up this PR just to make this change, but consider it for future improvement.

This comment has been minimized.

@nnposter

nnposter May 19, 2019
Author

At the moment I am leaning against this alternative. In my opinion we should prevent getting stuck with potentially infinite responses.

If somebody wants to take advantage of the persistent connection but to avoid resubmitted requests then it is always possible to set http.max-pipeline=1.

nselib/http.lua Show resolved Hide resolved
nnposter added 2 commits May 19, 2019
Resolves #1571 (comment)
@nnposter
Copy link
Author

@nnposter nnposter commented May 19, 2019

@dmiller-nmap Thank you for reviewing the PR. Please let me know if you have any other questions or concerns.

@dmiller-nmap
Copy link

@dmiller-nmap dmiller-nmap commented May 20, 2019

No other questions. Go ahead with this and we'll see how the community likes it! I'm guessing you're already using it in your own scans.

@nnposter
Copy link
Author

@nnposter nnposter commented May 21, 2019

Yes, I have been using the size limit for about a month and the gzip support for additional two. That said, I recognize that this is a non-trivial change in code that is underpinning a large variety of scripts so I fully expect that bugs will crop up.

Committed as r37627 (df26932).

@nnposter
Copy link
Author

@nnposter nnposter commented Jun 10, 2019

Commit r37627 (df26932) considered sufficiently stable.

@nnposter nnposter closed this Jun 10, 2019
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Aug 28, 2019
7.80:
Here is the full list of significant changes:

o [Windows] The Npcap Windows packet capturing library (https://npcap.org/)
  is faster and more stable than ever. Nmap 7.80 updates the bundled Npcap
  from version 0.99-r2 to 0.9982, including all of these changes from the
  last 15 Npcap releases: https://nmap.org/npcap/changelog

o [NSE] Added 11 NSE scripts, from 8 authors, bringing the total up to 598!
  They are all listed at https://nmap.org/nsedoc/, and the summaries are
  below:

  +  broadcast-hid-discoveryd discovers HID devices on a LAN by
    sending a discoveryd network broadcast probe.

  +  broadcast-jenkins-discover discovers Jenkins servers on a LAN
    by sending a discovery broadcast probe.

  +  http-hp-ilo-info extracts information from HP
    Integrated Lights-Out (iLO) servers.

  +  http-sap-netweaver-leak detects SAP Netweaver Portal with the
    Knowledge Management Unit enabled with anonymous access.

  + https-redirect detects HTTP servers that redirect to the same port, but
    with HTTPS. Some nginx servers do this, which made ssl-* scripts not run
    properly.

  +  lu-enum enumerates Logical Units (LU) of TN3270E servers.


  +  rdp-ntlm-info extracts Windows domain information from RDP
    services.

  + smb-vuln-webexec checks whether the WebExService is installed and allows
    code execution.

  + smb-webexec-exploit exploits the WebExService to run arbitrary commands
    with SYSTEM privileges.

  +  ubiquiti-discovery extracts information from the Ubiquiti
    Discovery service and assists version detection.

  +  vulners queries the Vulners CVE database API using CPE
    information from Nmap's service and application version detection.


o Use pcap_create instead of pcap_live_open in
  Nmap, and set immediate mode on the pcap descriptor. This solves packet
  loss problems on Linux and may improve performance on other platforms.


o [NSE] Collected utility functions for string processing into a new
  library, stringaux.lua.

o [NSE] New rand.lua library uses the best sources of random available on
  the system to generate random strings.

o [NSE] New library, oops.lua, makes reporting errors easy, with plenty of
  debugging detail when needed, and no clutter when not.

o [NSE] Collected utility functions for manipulating and searching tables
  into a new library, tableaux.lua.

o [NSE] New knx.lua library holds common functions and definitions for
  communicating with KNX/Konnex devices.

o [NSE] The HTTP library now provides transparent support for gzip-
  encoded response body. (See nmap/nmap#1571 for an
  overview.)

o [Nsock][Ncat] Add AF_VSOCK (Linux VM sockets) functionality to
  Nsock and Ncat. VM sockets are used for communication between virtual
  machines and the hypervisor.

o [Security][Windows] Address CVE-2019-1552 in OpenSSL by building with the
  prefix "C:\Program Files (x86)\Nmap\OpenSSL". This should prevent
  unauthorized users from modifying OpenSSL defaults by writing
  configuration to this directory.

o [Security] Reduced LibPCRE resource limits so that
  version detection can't use as much of the stack. Previously Nmap could
  crash when run on low-memory systems against target services which are
  intentionally or accidentally difficult to match. Someone assigned
  CVE-2018-15173 for this issue.

o Deprecate and disable the -PR (ARP ping) host discovery
  option. ARP ping is already used whenever possible, and the -PR option
  would not force it to be used in any other case.

o [NSE] bin.lua is officially deprecated. Lua 5.3, added 2 years ago in Nmap
  7.25BETA2, has native support for binary data packing via string.pack and
  string.unpack. All existing scripts and libraries have been updated.


o [NSE] Completely removed the bit.lua NSE library. All of its functions are
  replaced by native Lua bitwise operations, except for `arshift`
  (arithmetic shift) which has been moved to the bits.lua library. [Daniel
  Miller]

o [NSE] The HTTP library is now enforcing a size limit on the
  received response body. The default limit can be adjusted with a script
  argument, which applies to all scripts, and can be overridden case-by-case
  with an HTTP request option. (See nmap/nmap#1571
  for details.)

o [NSE] CR characters are no longer treated as illegal in script
  XML output.

o Allow resuming nmap scan with lengthy command line [Clément
  Notin]

o [NSE] Add TLS support to rdp-enum-encryption. Enables determining
  protocol version against servers that require TLS and lays ground work for
  some NLA/CredSSP information collection.

o [NSE] Address two protocol parsing issues in rdp-enum-encryption
  and the RDP nse library which broke scanning of Windows XP. Clarify
  protocol types

o [NSE] Script http-fileupload-exploiter failed to locate its
  resource file unless executed from a specific working
  directory.

o [NSE] Avoid clobbering the "severity" and "ignore_404" values of
  fingerprints in http-enum. None of the standard fingerprints uses these
  fields.

o [NSE] Fix a crash caused by a double-free of libssh2 session data
  when running SSH NSE scripts against non-SSH services.

o [NSE] Updates the execution rule of the mongodb scripts to be
  able to run on alternate ports.

o [Ncat] Allow Ncat to connect to servers on port 0, provided that
  the socket implementation allows this.

o Update the included libpcap to 1.9.0.

o [NSE] Fix a logic error that resulted in scripts not honoring the
  smbdomain script-arg when the target provided a domain in the NTLM
  challenge.

o [Nsock] Avoid a crash (Protocol not supported) caused by trying
  to reconnect with SSLv2 when an error occurs during DTLS connect. [Daniel
  Miller]

o [NSE] Removed OSVDB references from scripts and replaced them
  with BID references where possible.

o [NSE] Updates TN3270.lua and adds argument to disable TN3270E


o RMI parser could crash when encountering invalid input [Clément
  Notin]

o Avoid reporting negative latencies due to matching an ARP or ND
  response to a probe sent after it was recieved.

o [Ncat] To avoid confusion and to support non-default proxy ports,
  option --proxy now requires a literal IPv6 address to be specified using
  square-bracket notation, such as --proxy

o [Ncat] New ncat option provides control over
  whether proxy destinations are resolved by the remote proxy server or
  locally, by Ncat itself. See option --proxy-dns.

o [NSE] Updated script ftp-syst to prevent potential endless
  looping.

o New service probes and match lines for v1 and v2 of the Ubiquiti
  Discovery protocol. Devices often leave the related service open and it
  exposes significant amounts of information as well as the risk of being
  used as part of a DDoS. New nmap-payload entry for v1 of the
  protocol.

o [NSE] Removed hostmap-ip2hosts.nse as the API has been broken for a while
  and the service was completely shutdown on Feb 17th, 2019. [Paulino
  Calderon]

o [NSE] Adds TN3270E support and additional improvements to
  tn3270.lua and updates tn3270-screen.nse to display the new
  setting.

o [NSE] Updates product codes and adds a check for response length
  in enip-info.nse. The script now uses string.unpack.

o [Ncat] Temporary RSA keys are now 2048-bit to resolve a
  compatibility issue with OpenSSL library configured with security level 2,
  as seen on current Debian or Kali.

o [NSE] Fix a crash (double-free) when using SSH scripts against
  non-SSH services.

o [Zenmap] Fix a crash when Nmap executable cannot be found and the system
  PATH contains non-UTF-8 bytes, such as on Windows.

o [Zenmap] Fix a crash in results search when using the dir: operator:
    AttributeError: 'SearchDB' object has no attribute 'match_dir' [Daniel
    Miller]

o [Ncat] Fixed an issue with Ncat -e on Windows that caused early
  termination of connections.

o [NSE] Fix a false-positive in http-phpmyadmin-dir-traversal when
  the server responds with 200 status to a POST request to any
  URI.

o [NSE] New vulnerability state in vulns.lua, UNKNOWN, is used to indicate
  that testing could not rule out vulnerability.

o When searching for Lua header files, actually use them where
  they are found instead of forcing /usr/include. [Fabrice Fontaine, Daniel
  Miller]

o [NSE] Script traceroute-geolocation no longer crashes when
  www.GeoPlugin.net returns null coordinates

o Limit verbose -v and debugging -d levels to a maximum of 10. Nmap does not
  use higher levels internally.

o [NSE] tls.lua when creating a client_hello message will now only use a
  SSLv3 record layer if the protocol version is SSLv3. Some TLS
  implementations will not handshake with a client offering less than
  TLSv1.0. Scripts will have to manually fall back to SSLv3 to talk to
  SSLv3-only servers.

o [NSE] Fix a few false-positive conditions in
  ssl-ccs-injection. TLS implementations that responded with fatal alerts
  other than "unexpected message" had been falsely marked as
  vulnerable.

o Emergency fix to Nmap's birthday announcement so Nmap wishes itself a
  "Happy 21st Birthday" rather than "Happy 21th" in verbose mode (-v) on
  September 1, 2018.

o Start host timeout clocks when the first probe is sent to a
  host, not when the hostgroup is started. Sometimes a host doesn't get
  probes until late in the hostgroup, increasing the chance it will time
  out.

o [NSE] Support for edns-client-subnet (ECS) in dns.lua has been improved
by:
  -
  - Properly trimming ECS address, as mandated by RFC 7871
  - Fixing a bug that prevented using the same ECS option table more than
    once

o [Ncat] Fixed communication with commands launched with -e or -c
  on Windows, especially when --ssl is used.

o [NSE] Script http-default-accounts can now select more than one
  fingerprint category. It now also possible to select fingerprints by name
  to support very specific scanning.

o [NSE] Script http-default-accounts was not able to run against more than
  one target host/port.

o [NSE] New script-arg `http.host` allows users to force a
  particular value for the Host header in all HTTP requests.

o [NSE] Use smtp.domain script arg or target's domain name instead
  of "example.com" in EHLO command used for STARTTLS.

o [NSE] Fix brute.lua's BruteSocket wrapper, which was crashing
  Nmap with an assertion failure due to socket mixup [Daniel Miller]: nmap:
  nse_nsock.cc:672: int receive_buf(lua_State*, int, lua_KContext):
  Assertion `lua_gettop(L) == 7' failed.

o [NSE] Handle an error condition in smb-vuln-ms17-010 caused by
  IPS closing the connection.

o [Ncat] Fixed literal IPv6 URL format for connecting through HTTP
  proxies.

o [NSE] Updates vendors from ODVA list for enip-info.
[NothinRandom]

o [NSE] Add two common error strings that improve MySQL detection
  by the script http-sql-injection.

o [NSE] Fix bug in http-vuln-cve2006-3392 that prevented the script
  to generate the vulnerability report correctly.

o [NSE] Fix bug related to screen rendering in NSE library
  tn3270. This patch also improves the brute force script
  tso-brute.

o [NSE] Fix SIP, SASL, and HTTP Digest authentication when the
  algorithm contains lowercase characters.

o Nmap could be fooled into ignoring TCP response packets if they
  used an unknown TCP Option, which would misalign the validation, causing
  it to fail.

o [NSE]The HTTP response parser now tolerates status lines without a reason
  phrase, which improves compatibility with some HTTP servers.

o [NSE]] Parser for HTTP Set-Cookie header
  is now more compliant with RFC 6265:
  - empty attributes are tolerated
  - double quotes in cookie and/or attribute values are treated literally
  - attributes with empty values and value-less attributes are parsed
equally
  - attributes named "name" or "value" are ignored

o [NSE] Fix parsing http-grep.match script-arg. [Hans van den
  Bogert]

o [Zenmap] Avoid a crash when recent_scans.txt cannot be written
  to.

o Fixed --resume when the path to Nmap contains spaces.

o New service probe and match lines for adb, the Android Debug Bridge, which
  allows remote code execution and is left enabled by default on many
  devices.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.