Add enhanced msfdb with web service support #10410
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Derived from the msfdb script in the metasploit-omnibus repo.
|
Jenkins test this please. |
jbarnett-r7
reviewed
Aug 1, 2018
msfdb
Outdated
| puts "Please attempt to upgrade the database manually using pg_upgrade." | ||
| end | ||
| if !fixed | ||
| if ask_yn('If your database is corrupt, would you to reinitialize it?') |
There was a problem hiding this comment.
Typo here: would you to reinitialize it?. Missing ... you like to ...?
There was a problem hiding this comment.
Good catch! I overlooked this from the original script.
There was a problem hiding this comment.
Also, after re-reading it, the beginning part is confusing. If your database is corrupt... sounds like they are assuming I know if my database is corrupt or not. Maybe something like Your database may be corrupt...?
|
I somehow got myself in some loop of failures and nothing ever seems to resolve: If I type This also begs the question, should we keep looping like this for users? Or maybe after a few tries, or a specific error, we send them somewhere for more help? |
Also updates authentication method to trust, but verify. Change made to original script in parallel see rapid7/metasploit-omnibus/rapid7#73.
|
I was able to get past the above errors by changing the permissions on Now that the script can start the database I keep getting the following message when it tries to start the web service. We should probably have some sort of help text here, or send the user in some sort of direction on how to resolve the issue. |
|
Getting the following error in the Also get the same error when running |
|
I was able to get the script to finish successfully, but was confused as to what to do next. We should add a line to the script output that gives the user the command to run in msfconsole to connect to the data service. |
Restore the md5 auth-method for the postgres DB, msftest user. Use the default client authentication config to create roles and databases before writing our own config.
|
I have some additional testing to perform. |
jbarnett-r7
reviewed
Aug 6, 2018
msfdb
Outdated
| # daemonize MSF web service | ||
| puts "Starting MSF web service" | ||
| if run_cmd("#{thin_cmd} start") == 0 | ||
| puts "MSF web service started" |
There was a problem hiding this comment.
This message is a little misleading. See the output below:
Starting MSF web service
MSF web service started
MSF web service does not appear to be online; aborting initialize.
It says that the web service has started, but it never successfully did. I had another web server running on the same port so it never started up, but the logging makes it seem like it did.
msf-ws.log:
/home/jbarnett/.rvm/gems/ruby-2.5.1@metasploit-framework/gems/eventmachine-1.2.7/lib/eventmachine.rb:531:in `start_tcp_server': no acceptor (port is in use or requires root privileges) (RuntimeError)
from /home/jbarnett/.rvm/gems/ruby-2.5.1@metasploit-framework/gems/eventmachine-1.2.7/lib/eventmachine.rb:531:in `start_server'
from /home/jbarnett/.rvm/gems/ruby-2.5.1@metasploit-framework/gems/thin-1.7.2/lib/thin/backends/tcp_server.rb:16:in `connect'
from /home/jbarnett/.rvm/gems/ruby-2.5.1@metasploit-framework/gems/thin-1.7.2/lib/thin/backends/base.rb:63:in `block in start'
from /home/jbarnett/.rvm/gems/ruby-2.5.1@metasploit-framework/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in `run_machine'
from /home/jbarnett/.rvm/gems/ruby-2.5.1@metasploit-framework/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in `run'
from /home/jbarnett/.rvm/gems/ruby-2.5.1@metasploit-framework/gems/thin-1.7.2/lib/thin/backends/base.rb:73:in `start'
from /home/jbarnett/.rvm/gems/ruby-2.5.1@metasploit-framework/gems/thin-1.7.2/lib/thin/server.rb:162:in `start'
from /home/jbarnett/.rvm/gems/ruby-2.5.1@metasploit-framework/gems/thin-1.7.2/lib/thin/controllers/controller.rb:87:in `start'
from /home/jbarnett/.rvm/gems/ruby-2.5.1@metasploit-framework/gems/thin-1.7.2/lib/thin/runner.rb:203:in `run_command'
from /home/jbarnett/.rvm/gems/ruby-2.5.1@metasploit-framework/gems/thin-1.7.2/lib/thin/runner.rb:159:in `run!'
from /home/jbarnett/.rvm/gems/ruby-2.5.1@metasploit-framework/gems/thin-1.7.2/bin/thin:6:in `<top (required)>'
from /home/jbarnett/.rvm/gems/ruby-2.5.1@metasploit-framework/bin/thin:23:in `load'
from /home/jbarnett/.rvm/gems/ruby-2.5.1@metasploit-framework/bin/thin:23:in `<main>'
from /home/jbarnett/.rvm/gems/ruby-2.5.1@metasploit-framework/bin/ruby_executable_hooks:15:in `eval'
from /home/jbarnett/.rvm/gems/ruby-2.5.1@metasploit-framework/bin/ruby_executable_hooks:15:in `<main>'
|
If the database is not running the web service still appears to be started, but it cannot connect to the DB so it starts throwing errors in To reproduce:
|
jbarnett-r7
reviewed
Aug 6, 2018
| end | ||
|
|
||
| def reinit_db | ||
| delete_db |
There was a problem hiding this comment.
I see that this is "by design", but running msfdb reinit deletes the user's data without prompting. This caught me off guard when I got back in and was wondering why my data was not present. I feel like we should prompt the user before deleting even if it is the reinit command.
jbarnett@nexpose-dev:~/goliath$ ./msfdb reinit
Deleting all data at /home/jbarnett/.msf4/db
Stopping database at /home/jbarnett/.msf4/db
Delete database configuration at /home/jbarnett/.msf4/database.yml?: yes
Creating database at /home/jbarnett/.msf4/db
Starting database at /home/jbarnett/.msf4/db...success
Creating database users
Writing client authentication configuration file /home/jbarnett/.msf4/db/pg_hba.conf
Stopping database at /home/jbarnett/.msf4/db
Starting database at /home/jbarnett/.msf4/db...success
Creating initial database schema
MSF web service is no longer running
|
Thanks for the great testing notes! Much of the control flow for the database component was left untouched from the original |
Remove PID check code since thin will stop and cleanup the PID under certain circumstances after it has started and a PID file is written. Reuse web service online check for this purpose.
mkienow-r7
added
database
and removed
blocked
|
I feel like the messaging could be simpler here for the webservice status. What if I'm a noob that doesn't know what a PID file is? |
|
@jbarnett-r7 I’m trying to provide some context and details similar to the feedback for the database.
Does "MSF web service is not running: No PID file found at /home/jbarnett/.msf4/msf-ws.pid" sound better? |
jbarnett-r7
added a commit
that referenced
this pull request
Aug 7, 2018
The msfdb script allows you to create, delete, start, and stop local and remote data services. The database backend requires PostgreSQL and the webservice provides a REST API to interact with it.
Release NotesAdds a script for configuring and managing the database backend and web service for REST API. See |
Merged
15 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds enhanced
msfdbscript to manage a Metasploit Framework database and web service. This is derived from themsfdbscript in the rapid7/metasploit-omnibus repo. The script provides command line options to control a number of default values used in the originalmsfdbwhen creating and initializing the database. If none of the database related options are modified the script should operate as it did before.If the
--componentoption is not provided then the command will be executed for the database followed by the web service. This default mode of operation is useful when first setting up the database and web service. The component may be specified if the user wishes to make changes to a given component independent of the other.By default SSL is enabled and the script will generate a fake "snakeoil" SSL certificate using
Rex::Socket::Ssl.ssl_generate_certificateif one is not provided. The generated SSL certificate uses a random common name (CN) which will not match your hostname and thus you will need to make appropriate accommodations when operating with such a certificate. Please generate your own SSL certificate and key instead and supply those tomsfdbusing the--ssl-cert-fileand--ssl-key-fileoptions, and enable SSL verification by passing the option--no-ssl-disable-verify.Verification
Note, manually backup your database configuration and DB directory before testing since the script will be making changing or deleting files.
msfdb reinit, otherwise,msfdb initand follow prompts to initialize both a database and web servicemsfdb statuscurl --insecure https://localhost:8080/api/v1/msf/version | python -m json.toolcurl --insecure -H "Accept: application/json" -H "Authorization: Bearer <token>" https://localhost:8080/api/v1/msf/version | python -m json.toolhttps://localhost:8080/api/v1/auth/account. You will need to add an exception in the browser for the fake SSL certificate to get the page to load.msfdbto log in to the web interface, log out and close the browser.msfdbwith various options and commands and verify that they operate as expectedmsfconsolemsfdbscript. The command will look something like the following:data_services --add --token <token> --ssl --cert ~/.msf4/msf-ws-cert.pem --skip-verify localhosthosts,loot,vulns,services, etc.) that create, read, update and delete data work as expected