Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

respect ssl_version in crawler #16353

Merged
merged 2 commits into from Mar 21, 2022

Conversation

jmartin-tech
Copy link
Contributor

When utilizing Anemone to crawl pages using Rex sockets
Framework common SSL settings can pull from standardized options.
This change enables more fine grained user control and avoids issues
with missing or deprecated SSL versions in newer Ruby versions.

Prior to this change SSL attempts fail on Ruby 3:

msf6 auxiliary(scanner/http/crawler) > run
[*] Running module against 99.84.77.68

[*] Crawling https://metasploit.com:443/...
[-] [00001/00500]    ERR - metasploit.com - https://metasploit.com/
[*] Crawl of https://metasploit.com:443/ complete
[*] Running module against 99.84.77.91
[*] Crawling https://metasploit.com:443/...
[-] [00001/00500]    ERR - metasploit.com - https://metasploit.com/
[*] Crawl of https://metasploit.com:443/ complete
[*] Running module against 99.84.77.17
[*] Crawling https://metasploit.com:443/...
[-] [00001/00500]    ERR - metasploit.com - https://metasploit.com/
[*] Crawl of https://metasploit.com:443/ complete
[*] Running module against 99.84.77.81
[*] Crawling https://metasploit.com:443/...
[-] [00001/00500]    ERR - metasploit.com - https://metasploit.com/
[*] Crawl of https://metasploit.com:443/ complete
[*] Auxiliary module execution completed

This is due to an error that is no noted or logged:

[-] Error accessing page This version of Ruby does not support the requested SSL/TLS version SSLv23

Verification

List the steps needed to make sure this thing works

  • Start msfconsole
  • use http/crawler
  • set RHOST <known http IP or hostname> such as http.com
  • set RPORT <known http port> typically 80
  • Verify the crawler enumerates endpoints
  • set SSL true
  • set RHOST <known https IP or hostname> such as metasploit.com
  • set RPORT <known https port> typically 443
  • Verify the crawler enumerates endpoints

When utilizing `Anemone` to crawl pages using `Rex` sockets
Framework common `SSL` settings can pull from standardized options.
This change enables more fine grained user control and avoids issues
with missing or deprecated SSL versions in newer Ruby versions.
@@ -110,7 +110,7 @@ def connect
opts[:target].port,
{},
opts[:target].ssl,
'SSLv23',
'Auto',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason this is not @opts[:ssl_version] like in lib/anemone/rex_http.rb?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notice this object expects opts[:target] to contain the targeting details, lib/msf/core/auxiliary/web/target.rb does not have an ssl_version method to access, by using Auto this provide the most compatibility with whatever OpenSSL version Ruby provides.

Copy link
Contributor

@gwillcox-r7 gwillcox-r7 Mar 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me, considering this resolved, though we probably aught to come back to this at some point and look at how we can make this fall in line with the ssl_version standard code other modules and libraries are using. I'll leave this open for passerby's in case we do decide to make a run through this at some point, might be easier for them to find this when searching for code cases.

For those interested, @jmartin-r7 also noted that Auxiliary::Web::Target is not actually used in the framework directly and is actually used in Pro code which might be part of the reason for this discrepancy.

@gwillcox-r7
Copy link
Contributor

Before patch:

[+] [00296/00500]    403 - http.com - http://http.com/stuff/basilic/docs/CHANGELOG
[*] Crawl of http://http.com:80/ complete
[*] Running module against 5.22.145.16
[*] Crawling http://http.com:80/...
[+] [00001/00500]    403 - http.com - http://http.com/
[*] Crawl of http://http.com:80/ complete
[*] Running module against 2a00:18e0:5:2:7577:4e4d:413b:f507
[*] Crawling http://http.com:80/...
[+] [00001/00500]    403 - http.com - http://http.com/
[*] Crawl of http://http.com:80/ complete
[*] Running module against 2a00:18e0:5:3:dfe2:c743:85aa:61bc
[*] Crawling http://http.com:80/...
[+] [00001/00500]    403 - http.com - http://http.com/
[*] Crawl of http://http.com:80/ complete
[*] Auxiliary module execution completed
msf6 auxiliary(scanner/http/crawler) > set SSL true
[!] Changing the SSL option's value may require changing RPORT!
SSL => true
msf6 auxiliary(scanner/http/crawler) > set RHOSTS metasploit.com
RHOSTS => metasploit.com
msf6 auxiliary(scanner/http/crawler) > set RPORT 443
RPORT => 443
msf6 auxiliary(scanner/http/crawler) > run
[*] Running module against 13.249.48.109

[*] Crawling https://metasploit.com:443/...
[-] [00001/00500]    ERR - metasploit.com - https://metasploit.com/
[*] Crawl of https://metasploit.com:443/ complete
[*] Running module against 13.249.48.92
[*] Crawling https://metasploit.com:443/...
[-] [00001/00500]    ERR - metasploit.com - https://metasploit.com/
[*] Crawl of https://metasploit.com:443/ complete
[*] Running module against 13.249.48.107
[*] Crawling https://metasploit.com:443/...
[-] [00001/00500]    ERR - metasploit.com - https://metasploit.com/
[*] Crawl of https://metasploit.com:443/ complete
[*] Running module against 13.249.48.60
[*] Crawling https://metasploit.com:443/...
[-] [00001/00500]    ERR - metasploit.com - https://metasploit.com/
[*] Crawl of https://metasploit.com:443/ complete
[*] Auxiliary module execution completed
msf6 auxiliary(scanner/http/crawler) > 

@gwillcox-r7
Copy link
Contributor

After patch:

msf6 payload(windows/x64/meterpreter/reverse_tcp) > use http/crawler

Matching Modules
================

   #  Name                            Disclosure Date  Rank    Check  Description
   -  ----                            ---------------  ----    -----  -----------
   0  auxiliary/scanner/http/crawler                   normal  No     Web Site Crawler


Interact with a module by name or index. For example info 0, use 0 or use auxiliary/scanner/http/crawler

[*] Using auxiliary/scanner/http/crawler
msf6 auxiliary(scanner/http/crawler) > set SSL true
[!] Changing the SSL option's value may require changing RPORT!
SSL => true
msf6 auxiliary(scanner/http/crawler) > set RPORT 443
RPORT => 443
msf6 auxiliary(scanner/http/crawler) > set RHOSTS metasploit.com
RHOSTS => metasploit.com
msf6 auxiliary(scanner/http/crawler) > run
[*] Running module against 13.226.190.123

[*] Crawling https://metasploit.com:443/...
[*] [00001/00500]    200 - metasploit.com - https://metasploit.com/
[*] [00002/00500]    200 - metasploit.com - https://metasploit.com/includes/images/favicon.ico
[*] [00003/00500]    200 - metasploit.com - https://metasploit.com/get-started
[*] [00004/00500]    200 - metasploit.com - https://metasploit.com/includes/css/all.min.css
[*] [00005/00500]    200 - metasploit.com - https://metasploit.com/includes/css/foundation.min.css
[*] [00006/00500]    200 - metasploit.com - https://metasploit.com/contribute
[*] [00007/00500]    200 - metasploit.com - https://metasploit.com/help
[-] [00008/00500]    404 - metasploit.com - https://metasploit.com/test/
[*] [00009/00500]    200 - metasploit.com - https://metasploit.com/download
[-] [00010/00500]    404 - metasploit.com - https://metasploit.com/awstats/
[-] [00011/00500]    404 - metasploit.com - https://metasploit.com/awstats/awstats/
[-] [00012/00500]    404 - metasploit.com - https://metasploit.com/tmp/
[-] [00013/00500]    404 - metasploit.com - https://metasploit.com/stuff/
[-] [00014/00500]    404 - metasploit.com - https://metasploit.com/docs/text/manual.txt
[-] [00015/00500]    404 - metasploit.com - https://metasploit.com/cacti/
[-] [00016/00500]    404 - metasploit.com - https://metasploit.com/basilic/
[-] [00017/00500]    404 - metasploit.com - https://metasploit.com/docs/CHANGELOG
[*] [00018/00500]    200 - metasploit.com - https://metasploit.com/includes/css/hjs-monokai.min.css
[-] [00019/00500]    404 - metasploit.com - https://metasploit.com/docs/html/php_script_server.html
[-] [00020/00500]    404 - metasploit.com - https://metasploit.com/awstats/download
[-] [00021/00500]    404 - metasploit.com - https://metasploit.com/test/download
[-] [00022/00500]    404 - metasploit.com - https://metasploit.com/awstats/awstats/download
[-] [00023/00500]    404 - metasploit.com - https://metasploit.com/tmp/download
[-] [00024/00500]    404 - metasploit.com - https://metasploit.com/stuff/download
[-] [00025/00500]    404 - metasploit.com - https://metasploit.com/cacti/download
[-] [00026/00500]    404 - metasploit.com - https://metasploit.com/docs/text/download
[-] [00027/00500]    404 - metasploit.com - https://metasploit.com/docs/download
[-] [00028/00500]    404 - metasploit.com - https://metasploit.com/basilic/download
[-] [00029/00500]    404 - metasploit.com - https://metasploit.com/docs/html/download
[*] Crawl of https://metasploit.com:443/ complete
[*] Running module against 13.226.190.55
[*] Crawling https://metasploit.com:443/...
[*] [00001/00500]    200 - metasploit.com - https://metasploit.com/
[*] [00002/00500]    200 - metasploit.com - https://metasploit.com/includes/images/favicon.ico
[*] [00003/00500]    200 - metasploit.com - https://metasploit.com/get-started
[*] [00004/00500]    200 - metasploit.com - https://metasploit.com/includes/css/all.min.css
[*] [00005/00500]    200 - metasploit.com - https://metasploit.com/includes/css/foundation.min.css
[*] [00006/00500]    200 - metasploit.com - https://metasploit.com/help
[*] [00007/00500]    200 - metasploit.com - https://metasploit.com/contribute
[*] [00008/00500]    200 - metasploit.com - https://metasploit.com/download
[-] [00009/00500]    404 - metasploit.com - https://metasploit.com/awstats/
[-] [00010/00500]    404 - metasploit.com - https://metasploit.com/test/
[-] [00011/00500]    404 - metasploit.com - https://metasploit.com/tmp/
[-] [00012/00500]    404 - metasploit.com - https://metasploit.com/stuff/
[-] [00013/00500]    404 - metasploit.com - https://metasploit.com/awstats/awstats/
[-] [00014/00500]    404 - metasploit.com - https://metasploit.com/basilic/
[-] [00015/00500]    404 - metasploit.com - https://metasploit.com/cacti/
[-] [00016/00500]    404 - metasploit.com - https://metasploit.com/docs/text/manual.txt
[-] [00017/00500]    404 - metasploit.com - https://metasploit.com/docs/CHANGELOG
[-] [00018/00500]    404 - metasploit.com - https://metasploit.com/docs/html/php_script_server.html
[-] [00019/00500]    404 - metasploit.com - https://metasploit.com/awstats/download
[*] [00020/00500]    200 - metasploit.com - https://metasploit.com/includes/css/hjs-monokai.min.css
[-] [00021/00500]    404 - metasploit.com - https://metasploit.com/tmp/download
[-] [00022/00500]    404 - metasploit.com - https://metasploit.com/test/download
[-] [00023/00500]    404 - metasploit.com - https://metasploit.com/awstats/awstats/download
[-] [00024/00500]    404 - metasploit.com - https://metasploit.com/basilic/download
[-] [00025/00500]    404 - metasploit.com - https://metasploit.com/cacti/download
[-] [00026/00500]    404 - metasploit.com - https://metasploit.com/docs/text/download
[-] [00027/00500]    404 - metasploit.com - https://metasploit.com/stuff/download
[-] [00028/00500]    404 - metasploit.com - https://metasploit.com/docs/download
[-] [00029/00500]    404 - metasploit.com - https://metasploit.com/docs/html/download
[*] Crawl of https://metasploit.com:443/ complete
[*] Running module against 13.226.190.21
[*] Crawling https://metasploit.com:443/...
[*] [00001/00500]    200 - metasploit.com - https://metasploit.com/
[*] [00002/00500]    200 - metasploit.com - https://metasploit.com/get-started
[*] [00003/00500]    200 - metasploit.com - https://metasploit.com/includes/css/all.min.css
[*] [00004/00500]    200 - metasploit.com - https://metasploit.com/includes/images/favicon.ico
[*] [00005/00500]    200 - metasploit.com - https://metasploit.com/contribute
[*] [00006/00500]    200 - metasploit.com - https://metasploit.com/includes/css/foundation.min.css
[*] [00007/00500]    200 - metasploit.com - https://metasploit.com/help
[-] [00008/00500]    404 - metasploit.com - https://metasploit.com/test/
[*] [00009/00500]    200 - metasploit.com - https://metasploit.com/download
[-] [00010/00500]    404 - metasploit.com - https://metasploit.com/stuff/
[-] [00011/00500]    404 - metasploit.com - https://metasploit.com/awstats/
[-] [00012/00500]    404 - metasploit.com - https://metasploit.com/tmp/
[-] [00013/00500]    404 - metasploit.com - https://metasploit.com/basilic/
[-] [00014/00500]    404 - metasploit.com - https://metasploit.com/awstats/awstats/
[-] [00015/00500]    404 - metasploit.com - https://metasploit.com/cacti/
[-] [00016/00500]    404 - metasploit.com - https://metasploit.com/docs/CHANGELOG
[-] [00017/00500]    404 - metasploit.com - https://metasploit.com/docs/html/php_script_server.html
[-] [00018/00500]    404 - metasploit.com - https://metasploit.com/docs/text/manual.txt
[-] [00019/00500]    404 - metasploit.com - https://metasploit.com/stuff/download
[-] [00020/00500]    404 - metasploit.com - https://metasploit.com/test/download
[*] [00021/00500]    200 - metasploit.com - https://metasploit.com/includes/css/hjs-monokai.min.css
[-] [00022/00500]    404 - metasploit.com - https://metasploit.com/basilic/download
[-] [00023/00500]    404 - metasploit.com - https://metasploit.com/tmp/download
[-] [00024/00500]    404 - metasploit.com - https://metasploit.com/awstats/awstats/download
[-] [00025/00500]    404 - metasploit.com - https://metasploit.com/awstats/download
[-] [00026/00500]    404 - metasploit.com - https://metasploit.com/docs/download
[-] [00027/00500]    404 - metasploit.com - https://metasploit.com/docs/html/download
[-] [00028/00500]    404 - metasploit.com - https://metasploit.com/docs/text/download
[-] [00029/00500]    404 - metasploit.com - https://metasploit.com/cacti/download
[*] Crawl of https://metasploit.com:443/ complete
[*] Running module against 13.226.190.119
[*] Crawling https://metasploit.com:443/...
[*] [00001/00500]    200 - metasploit.com - https://metasploit.com/
[*] [00002/00500]    200 - metasploit.com - https://metasploit.com/get-started
[*] [00003/00500]    200 - metasploit.com - https://metasploit.com/includes/images/favicon.ico
[*] [00004/00500]    200 - metasploit.com - https://metasploit.com/includes/css/all.min.css
[*] [00005/00500]    200 - metasploit.com - https://metasploit.com/contribute
[*] [00006/00500]    200 - metasploit.com - https://metasploit.com/help
[*] [00007/00500]    200 - metasploit.com - https://metasploit.com/download
[*] [00008/00500]    200 - metasploit.com - https://metasploit.com/includes/css/foundation.min.css
[-] [00009/00500]    404 - metasploit.com - https://metasploit.com/tmp/
[-] [00010/00500]    404 - metasploit.com - https://metasploit.com/test/
[-] [00011/00500]    404 - metasploit.com - https://metasploit.com/basilic/
[-] [00012/00500]    404 - metasploit.com - https://metasploit.com/awstats/awstats/
[-] [00013/00500]    404 - metasploit.com - https://metasploit.com/stuff/
[-] [00014/00500]    404 - metasploit.com - https://metasploit.com/docs/text/manual.txt
[-] [00015/00500]    404 - metasploit.com - https://metasploit.com/docs/CHANGELOG
[-] [00016/00500]    404 - metasploit.com - https://metasploit.com/cacti/
[*] [00017/00500]    200 - metasploit.com - https://metasploit.com/includes/css/hjs-monokai.min.css
[-] [00018/00500]    404 - metasploit.com - https://metasploit.com/docs/html/php_script_server.html
[-] [00019/00500]    404 - metasploit.com - https://metasploit.com/tmp/download
[-] [00020/00500]    404 - metasploit.com - https://metasploit.com/test/download
[-] [00021/00500]    404 - metasploit.com - https://metasploit.com/awstats/awstats/download
[-] [00022/00500]    404 - metasploit.com - https://metasploit.com/basilic/download
[-] [00023/00500]    404 - metasploit.com - https://metasploit.com/stuff/download
[-] [00024/00500]    404 - metasploit.com - https://metasploit.com/docs/text/download
[-] [00025/00500]    404 - metasploit.com - https://metasploit.com/awstats/
[-] [00026/00500]    404 - metasploit.com - https://metasploit.com/docs/html/download
[-] [00027/00500]    404 - metasploit.com - https://metasploit.com/cacti/download
[-] [00028/00500]    404 - metasploit.com - https://metasploit.com/docs/download
[-] [00029/00500]    404 - metasploit.com - https://metasploit.com/awstats/download
[*] Crawl of https://metasploit.com:443/ complete
[*] Auxiliary module execution completed
msf6 auxiliary(scanner/http/crawler) > 

Seems to be working well :)

@gwillcox-r7 gwillcox-r7 merged commit 024da20 into rapid7:master Mar 21, 2022
@gwillcox-r7
Copy link
Contributor

Release Notes

A bug has been fixed in the Anemone library and in the HTTP crawler libraries and related module to allow pulling and setting of ssl_version from standardized options. This allows finder grained user control and avoids issues related to missing or depreciated SSL versions in newer Ruby versions, which were at times preventing Metasploit from making successful connections to targets.

@jmartin-tech jmartin-tech deleted the http-use-ssl-version branch March 25, 2022 15:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants