add analyze command

[jmartin-tech]

This is the initial start on an idea of enabling the framework to suggest modules base on what a user has already learned and stored about a host. Currently functionality is limited to linking vulnerabilities already stored about a host to exploit modules that may take advantage of that vulnerability.

For now this enables the analyze command only when a user has a database attached to the console.

msf5 > help database

Database Backend Commands
=========================

    Command           Description
    -------           -----------
    analyze           Analyze database information about a specific address or address range
    db_connect        Connect to an existing data service
    db_disconnect     Disconnect from the current data service
    db_export         Export a file containing the contents of the database
    db_import         Import a scan result file (filetype will be auto-detected)
    db_nmap           Executes nmap and records the output automatically
    db_rebuild_cache  Rebuilds the database-stored module cache
    db_remove         Remove the saved data service entry
    db_save           Save the current data service connection as the default to reconnect on startup
    db_status         Show the current data service status
    hosts             List all hosts in the database
    loot              List all loot in the database
    notes             List all notes in the database
    services          List all services in the database
    vulns             List all vulnerabilities in the database
    workspace         Switch between database workspaces

Example usage against metasploitable3 based on a vulnerability scan imported from nexpose:

$ ./msfconsole -q
msf5 > db_import m3_report.xml
[*] Importing 'NeXpose XML Report' data
[*] Import: Parsing with 'Nokogiri v1.8.5'
[*] Importing host 192.168.18.118
[*] Importing vulnerability 'NEXPOSE-apache-httpd-cve-2012-0053' (1 instance)
[*] Importing vulnerability 'NEXPOSE-certificate-common-name-mismatch' (1 instance)
[*] Importing vulnerability 'NEXPOSE-cifs-smb-signing-disabled' (2 instances)
[*] Importing vulnerability 'NEXPOSE-cifs-smb-signing-not-required' (2 instances)
[*] Importing vulnerability 'NEXPOSE-cifs-smb2-signing-not-required' (1 instance)
[*] Importing vulnerability 'NEXPOSE-database-open-access' (1 instance)
[*] Importing vulnerability 'NEXPOSE-generic-icmp-timestamp' (1 instance)
[*] Importing vulnerability 'NEXPOSE-msft-cve-2017-0146' (1 instance)
[*] Importing vulnerability 'NEXPOSE-netbios-nbstat-amplification' (1 instance)
...
[*] Successfully imported /m3_report.xml
msf5 > hosts

Hosts
=====

address         mac                name            os_name                            os_flavor  os_sp  purpose  info  comments
-------         ---                ----            -------                            ---------  -----  -------  ----  --------
192.168.18.118  00:0c:29:6e:e8:7f  VAGRANT-2008R2  Windows 2008 R2, Standard Edition             SP1    server

msf5 > analyze 192.168.18.118
[*] Analyzing 192.168.18.118...
[*] exploit/windows/smb/ms17_010_eternalblue
[*] exploit/windows/smb/ms17_010_eternalblue_win8
[*] exploit/windows/smb/ms17_010_psexec
msf5 >

Looking Forward

At some point things like the Multi Recon Local Exploit Suggester module could be integrated into a single location to get hints from metasploit framework about next steps a user may consider taking.

Anyone taking a gander at this please provide input on the user experience and how the initial functionality could be tweaked before this lands.

Verification

List the steps needed to make sure this thing works

• Start msfconsole
• db_import <test_file_path>.xml
• Verify modules reported are valid for what is know about services and vulns on imported host

m3_report.xml.zip

[busterb]

An RPC API for this would also be great, so automation tools can get this data from Metasploit too.

[sempervictus]

Awesome stuff, thank you.

[jmartin-tech]

@bcook, took a stab at RPC support (needs thorough testing), I did couple the method into rpc_db for now to match with the requirement for a db on the command dispatcher side. If you think that is a poor location or we should support a better location to avoid breaking API change later happy to reconsider.

I suspect Travis may have something to say about the changes.

[mkienow-r7]

I'm seeing an error when attempting to use the RPC method rpc.call('db.analyze_host', {:host => '127.0.0.1'}):

~/.msf4/logs/framework.log:

[01/25/2019 11:35:13] [e(0)] core: Problem retrieving host: opts must include a valid :workspace
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/metasploit-framework/lib/msf/util/db_manager.rb:37:in `process_opts_workspace'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/metasploit-framework/lib/msf/core/db_manager/host.rb:113:in `block in get_host'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activerecord-4.2.11/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/metasploit-framework/lib/msf/core/db_manager/host.rb:112:in `get_host'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/metasploit-framework/lib/metasploit/framework/data_service/proxy/host_data_proxy.rb:34:in `block in get_host'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/metasploit-framework/lib/metasploit/framework/data_service/proxy/core.rb:166:in `data_service_operation'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/metasploit-framework/lib/metasploit/framework/data_service/proxy/host_data_proxy.rb:33:in `get_host'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/metasploit-framework/lib/msf/core/rpc/v10/rpc_db.rb:701:in `block in rpc_analyze_host'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activerecord-4.2.11/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/metasploit-framework/lib/msf/core/rpc/v10/rpc_db.rb:695:in `rpc_analyze_host'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/metasploit-framework/lib/msf/core/rpc/v10/service.rb:153:in `block in process'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/.rbenv/versions/2.5.3/lib/ruby/2.5.0/timeout.rb:93:in `block in timeout'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/.rbenv/versions/2.5.3/lib/ruby/2.5.0/timeout.rb:33:in `block in catch'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/.rbenv/versions/2.5.3/lib/ruby/2.5.0/timeout.rb:33:in `catch'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/.rbenv/versions/2.5.3/lib/ruby/2.5.0/timeout.rb:33:in `catch'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/.rbenv/versions/2.5.3/lib/ruby/2.5.0/timeout.rb:108:in `timeout'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/metasploit-framework/lib/msf/core/rpc/v10/service.rb:153:in `process'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/metasploit-framework/lib/msf/core/rpc/v10/service.rb:91:in `on_request_uri'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/metasploit-framework/lib/msf/core/rpc/v10/service.rb:72:in `block in start'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/metasploit-framework/lib/rex/proto/http/handler/proc.rb:38:in `on_request'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/metasploit-framework/lib/rex/proto/http/server.rb:368:in `dispatch_request'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/metasploit-framework/lib/rex/proto/http/server.rb:302:in `on_client_data'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/metasploit-framework/lib/rex/proto/http/server.rb:161:in `block in start'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/rex-core-0.1.13/lib/rex/io/stream_server.rb:48:in `on_client_data'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/rex-core-0.1.13/lib/rex/io/stream_server.rb:199:in `block in monitor_clients'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/rex-core-0.1.13/lib/rex/io/stream_server.rb:197:in `each'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/rex-core-0.1.13/lib/rex/io/stream_server.rb:197:in `monitor_clients'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/rex-core-0.1.13/lib/rex/io/stream_server.rb:73:in `block in start'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/metasploit-framework/lib/rex/thread_factory.rb:22:in `block in spawn'
[01/25/2019 11:35:13] [e(0)] core: /home/msfdev/metasploit-framework/lib/msf/core/thread_manager.rb:106:in `block in spawn'
[01/25/2019 11:35:13] [e(0)] core: RPC Exception: RuntimeError Problem retrieving host: opts must include a valid :workspace. See log for more details. ["/home/msfdev/metasploit-framework/lib/metasploit/framework/data_service/proxy/core.rb:174:in `log_error'", "/home/msfdev/metasploit-framework/lib/metasploit/framework/data_service/proxy/host_data_proxy.rb:37:in `rescue in get_host'", "/home/msfdev/metasploit-framework/lib/metasploit/framework/data_service/proxy/host_data_proxy.rb:31:in `get_host'", "/home/msfdev/metasploit-framework/lib/msf/core/rpc/v10/rpc_db.rb:701:in `block in rpc_analyze_host'", "/home/msfdev/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activerecord-4.2.11/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'", "/home/msfdev/metasploit-framework/lib/msf/core/rpc/v10/rpc_db.rb:695:in `rpc_analyze_host'", "/home/msfdev/metasploit-framework/lib/msf/core/rpc/v10/service.rb:153:in `block in process'", "/home/msfdev/.rbenv/versions/2.5.3/lib/ruby/2.5.0/timeout.rb:93:in `block in timeout'", "/home/msfdev/.rbenv/versions/2.5.3/lib/ruby/2.5.0/timeout.rb:33:in `block in catch'", "/home/msfdev/.rbenv/versions/2.5.3/lib/ruby/2.5.0/timeout.rb:33:in `catch'", "/home/msfdev/.rbenv/versions/2.5.3/lib/ruby/2.5.0/timeout.rb:33:in `catch'", "/home/msfdev/.rbenv/versions/2.5.3/lib/ruby/2.5.0/timeout.rb:108:in `timeout'", "/home/msfdev/metasploit-framework/lib/msf/core/rpc/v10/service.rb:153:in `process'", "/home/msfdev/metasploit-framework/lib/msf/core/rpc/v10/service.rb:91:in `on_request_uri'", "/home/msfdev/metasploit-framework/lib/msf/core/rpc/v10/service.rb:72:in `block in start'", "/home/msfdev/metasploit-framework/lib/rex/proto/http/handler/proc.rb:38:in `on_request'", "/home/msfdev/metasploit-framework/lib/rex/proto/http/server.rb:368:in `dispatch_request'", "/home/msfdev/metasploit-framework/lib/rex/proto/http/server.rb:302:in `on_client_data'", "/home/msfdev/metasploit-framework/lib/rex/proto/http/server.rb:161:in `block in start'", "/home/msfdev/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/rex-core-0.1.13/lib/rex/io/stream_server.rb:48:in `on_client_data'", "/home/msfdev/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/rex-core-0.1.13/lib/rex/io/stream_server.rb:199:in `block in monitor_clients'", "/home/msfdev/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/rex-core-0.1.13/lib/rex/io/stream_server.rb:197:in `each'", "/home/msfdev/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/rex-core-0.1.13/lib/rex/io/stream_server.rb:197:in `monitor_clients'", "/home/msfdev/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/rex-core-0.1.13/lib/rex/io/stream_server.rb:73:in `block in start'", "/home/msfdev/metasploit-framework/lib/rex/thread_factory.rb:22:in `block in spawn'", "/home/msfdev/metasploit-framework/lib/msf/core/thread_manager.rb:106:in `block in spawn'"] [{"host"=>"1.1.1.1"}] #<Rex::Proto::Http::Request:0x00007fc7b5b39730 @headers={"Host"=>"127.0.0.1:55553", "User-Agent"=>"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)", "Content-Type"=>"binary/message-pack", "Content-Length"=>"65"}, @auto_cl=true, @state=3, @transfer_chunked=false, @inside_chunk=false, @bufq="", @body="\x93\xAFdb.analyze_host\xC4 TEMPfH4i429emle9H5yHEMOpDnGsVq8H\x81\xA4host\xA71.1.1.1", @method="POST", @raw_uri="/api/", @uri_parts={"QueryString"=>{}, "Resource"=>"/api/"}, @proto="1.1", @chunk_min_size=1, @chunk_max_size=10, @uri_encode_mode="hex-normal", @relative_resource="/api/", @body_bytes_left=0>

[mkienow-r7]

Consider having the analyze command abort earlier if no host or host range was provided by the user to provide a more meaningful message to the user. Most of the DB commands have a default behavior to retrieve all of the given object if no host or host range is provided.

msf5 > analyze
[*] No host found for [].

[jmartin-tech]

Updated documentation on all db rpc methods that did not specify when workspace was needed.

Corrected call format is:
rpc.call('db.analyze_host', {:workspace => 'default', :host => '127.0.0.1'})

Also expanded console dispatcher to report on all hosts when no single ip or range is provided.

[mkienow-r7]

Minor: lib/msf/core/rpc/v10/rpc_db.rb: rpc_get_note is missing a :workspace parameter method comment. The parameter is used by way of the get_notes method, which calls init_db_opts_workspace.

[mkienow-r7]

In addition to the import, I ran db_nmap -sV and exploit/unix/irc/unreal_ircd_3281_backdoor against Metasploitable2 (3 should work as well). Once the new host and vuln entry exist (see below), the analyze command with no options fails to operate as one would expect.

msf5 > hosts

Hosts
=====
address         mac                name            os_name                            os_flavor  os_sp  purpose  info  comments
-------         ---                ----            -------                            ---------  -----  -------  ----  --------
172.28.128.3                                       Linux                                                server         
192.168.18.118  00:0c:29:6e:e8:7f  VAGRANT-2008R2  Windows 2008 R2, Standard Edition             SP1    server         

msf5 > vulns
...
2019-02-19 21:06:04 UTC  172.28.128.3    UnrealIRCD 3.2.8.1 Backdoor Command Execution                                            CVE-2010-2075,OSVDB-65445,URL-http://www.unrealircd.com/txt/unrealsecadvisory.20100612.txt

msf5 > analyze
[*] Analyzing 172.28.128.3...
[*] No suggestions for 172.28.128.3.

[mkienow-r7]

Release Notes

This adds the analyze command, an initial start on an idea of enabling Metasploit Framework to suggest modules based on host details stored in the database.