Remote Data Service (Goliath)

[clee-r7]

This patch allows Metasploit to use a data source over HTTP which gives users the ability to easily connect to one or many different endpoints to retrieve and store data. The remote service can also be used to retrieve data through a well defined API (The API is also a work in progress) for users to build other applications atop.

Note: This project is a work in progress
We are attempting to merge this PR early to prevent an even more massive PR while giving the community a chance to provide feedback early.

For more information about this project see: https://github.com/rapid7/metasploit-framework/wiki/Metasploit-Data-Service-Enhancements-(Goliath)

Breakdown

1. Database server to provide data over HTTP
   The image below show help and startup of the new service
   
   WARNING! - This opens your current database, to test ensure your DB is empty

2. Client side changes allowing Metasploit to connect to aforementioned server
   The image below show the help for the client tool as well as how to connect to the server in '1'
   

Goliath Visualization

Work In Progress

1. Data Models left to port:

   • Credentials
   • Event
   • Notes
   • Web Vulns
   • Sessions

2. API V1 Definition

3. DB Import Commands

Verification

Metasploit should be setup to use a DB

Ensuring prior code is not broken:

• Run rspec

Verify proper storage of hosts and services

• Start the data source as a web service ('msfdb_ws') Note the server exposes current data in DB
• Start Metasploit
• Connect to your web service ('data_services -a ')
• Run 'db_nmap '
• Run 'hosts' command (data is now being retrieved via remote data store)
• Run 'services' command (data is not being retrieved via remote data store)

[busterb]

Things to do next:

• 'msfdb_ws' should be called 'msfdb' and have the same API as 'msfdb' in omnibus and Kali
• rename or merge the 'data_services -a' command to work like 'db_connect'. Don't be arbitrarily different if there isn't a need for it.

The help isn't very useful:

msf5 auxiliary(scanner/ssh/ssh_login) > data_services -h
Usage: data_services [ options ] - list data services by default

OPTIONS:
  -h, --help                  Show this help information.
  -s, --set <id>              Set the data service by identifier.
  -a, --add [ options ] host  Adds data service
  Add Data Service Options:
  -p <port>         The port the data service is listening on. Default is 8080.
  -s, --ssl         Enable SSL. Required for HTTPS data services.
  -c, --cert        Certificate file matching the server's certificate. Needed when using self-signed SSL cert.
  --skip-verify     Skip validating authenticity of server's certificate. NOT RECOMMENDED.

msf5 auxiliary(scanner/ssh/ssh_login) > data_services -a 127.0.0.1 8080
Registered data service: remote_data_service: (http://8080:8080)
/Users/bcook/projects/metasploit-framework/lib/metasploit/framework/data_service/proxy/core.rb:123:in `log_error': Problem retrieving workspace: undefined method `first' for nil:NilClass. See log for more details. (Exception)
	from /Users/bcook/projects/metasploit-framework/lib/metasploit/framework/data_service/proxy/workspace_data_proxy.rb:35:in `rescue in workspace'
	from /Users/bcook/projects/metasploit-framework/lib/metasploit/framework/data_service/proxy/workspace_data_proxy.rb:30:in `workspace'
	from /Users/bcook/projects/metasploit-framework/lib/rex/ui/text/shell.rb:186:in `run'
	from /Users/bcook/projects/metasploit-framework/lib/metasploit/framework/command/console.rb:48:in `start'
	from /Users/bcook/projects/metasploit-framework/lib/metasploit/framework/command/base.rb:82:in `start'
	from ./msfconsole:48:in `<main>'

[busterb]

If there is a failure in connecting, framework completely dies.

[busterb]

I'd rather just be able to do:

msf> db_connect http://127.0.0.1:8080

and it does the right thing. I should also be able to say 'https' and get SSL, not need an oddball flag

[busterb]

What is -s, --set <id> Set the data service by identifier. even for?

[busterb]

Chatted with the team, these are all apparently known and will be addressed soon. Thanks!

[mkienow-r7]

@bcook-r7 If you run the data_services command it will list the available data services and their IDs. You can then use the data_services --set <id> command to change which data service is active. This allows you to switch between local and remote data services.

[busterb]

I marked this as no-release-notes since this isn't shipping in the 4.x branch (and we're largely looking for any integration bugs)