RFC: Improved HTTP API

[adfoster-r7]

Problem

Web applications are becoming increasingly complex over time, and the need for more powerful HTTP support is required.

The current Metasploit Framework HTTP Client is particularly low-level particularly in contrast to modern HTTP libraries such as Python's requests, or Ruby's Faraday or Mechanize. This leads to dealing with a high level of complexity for 'simple' tasks such as session tracking, attempting to upload files with Rex::MIME::Message, or using brittle regex to extract information from HTTP response bodies.

Solution

The Metasploit HTTP Client would be improved to provide first class support for 'modern' workflows:

• Adding DSL for making web requests, i.e. client.post(...), client.get(...)
• Simplified API for uploading files
  • Add uploading of files to HTTP Client #16442
• Improved cookies/session tracking and support
  • Initial support added Add CookieJar support to http_client #14831
• First class support for response assertion - providing a consistent way of raising errors if certain status codes are not returned, or response bodies are missing etc. This should reduce the if res && res.body boilerplate in module code, ensure consistency across modules - both at the code level, as well as for end users may otherwise run into undefined method body for nil:NilClass exceptions
• Improved logging capabilities
• Inbuilt support for retrying HTTP requests on failures, or polling for results
• First class support JSON/CSS/XPath selection
• Websocket support
  • Added as part of Add Module For Kubernetes Pod Authenticated Code Execution #15733

Some of the above functionality would be implemented via the middleware pattern, which would additionally allow for future extensibility.

Further Considerations

The above solution may improve HTTP support initially, but still lacks the required functionality for interacting with complex web applications written in newer web frameworks such as Angular/React/etc.

These improvements to the HTTP client might also might also influence improvements to the FTP client functionality.

[adfoster-r7]

Similar requests/issues:

• Issue As a module developer, it would be nice to have HttpClient 2.0 #6279 - HttpClient should remember my cookie
• Pull Request Allow HttpClient to remember my cookie #2697 - HttpClient should remember my cookie
• Issue HttpClient should remember my cookie #3877 - As a module developer, it would be nice to have HttpClient 2.0

[smcintyre-r7]

I know we haven't mentioned websockets, but if that's something we want to pursue I have a partial implementation using Rex. It might be useful as a starting point if we want websocket support as well.

https://github.com/zeroSteiner/metasploit-framework/blob/feat/mod/jupyter-exec/lib/rex/proto/http/websocket.rb
https://github.com/zeroSteiner/metasploit-framework/blob/b5845f16d95295e4fcda35188ee1cf2dc07cafa2/lib/msf/core/exploit/http/client.rb#L217

[gwillcox-r7]

I've actually been doing some own personal research and there are a few courses like AWAE that mention WebSockets but what strikes me the most is the fact that they mention that many common tools such as BurpSuite don't actually have support for WebSockets in all of their facets (in the case of Burp they recently added support for WebSockets in the proxy but there is still no support in Intruder or their other plugins).

Considering there are likely clients and pentests that are missing out on this testing area as a result of missing tooling, I'd be strongly in favor of adding support for WebSockets to provide users with this missing capability they might need for their engagements, especially given we all know that untested areas of an application tend to initially be ripe pickings for new bugs.

[adfoster-r7]

Added to the feature list, as well as a providing a link to the Kubernetes PR which introduces the Websocket functionality #15733

[adfoster-r7]

This RFC didn't take into consideration the work of adding HTTP/3 support:
https://tools.ietf.org/html/draft-ietf-quic-http-34

I don't think the implementation is officially finalised just yet. But firefox supports it now for instance: https://hacks.mozilla.org/2021/04/quic-and-http-3-support-now-in-firefox-nightly-and-beta/

And from the wiki page:

As of April 2021, the HTTP/3 protocol is an Internet Draft and has multiple implementations. Google Chrome supports HTTP/3 by default since April 2020 (including Chrome for Android, and the new Microsoft Edge, which is based on it).[2] Due to Chrome's large market share, this means that currently 66% of web browsers support HTTP/3.[3] Mozilla Firefox and Safari 14 (on macOS Big Sur) have support behind a feature flag, with Firefox slated to enable support by default in Firefox 88.[4] According to W3Techs, 18.0% of the top 10 million websites support HTTP/3.[5]

From a quick glance it doesn't seem like there's too many popular webservers that support it just yet either