Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] DB Manager for Payloads #10675

Merged
merged 22 commits into from Mar 4, 2019

Conversation

Projects
None yet
4 participants
@ebleiweiss-r7
Copy link
Contributor

ebleiweiss-r7 commented Sep 19, 2018

Along with a new Payload Metasploit Data Model (PR here), this change replaces the JSON hash file payloads.json with psql database entries for payloads with a uuid.

This also includes REST endpoints with CRUD operations related to the payload data model

Verification

This PR is dependent on changes from metasploit_data_models PR #172

  • git checkout the above branch from metasploit_data_models
  • Update local Gemfile to point to your local metasploit_data_models:
    In Gemfile, add:
    group :development do
    ...
      gem 'metasploit_data_models', path: '/path/to/metasploit_data_models'
    end
    
  • bundle install
  • rake db:migrate

  • Verify that you can run an exploit like eternalblue
    • ./msfconsole
    • use exploit/windows/smb/ms17_010_eternalblue
    • set rhosts <address>
    • run

  • ./msfdb_ws
  • Obtain a session token
export MSFDB_USER="postgres"
export MSFDB_PASS="J3uqvJ9nRadtIaSCJeU"
export MSFDB_TOKEN=`curl --silent -k -X POST "https://127.0.0.1:5443/api/v1/auth/generate-token" \
  -H  "accept: application/json" \
  -H  "Content-Type: application/json" \
  -H  "Authorization: Bearer undefined" \
  -d "{  \"username\": \"$MSFDB_USER\",  \"password\": \"$MSFDB_PASS\"}" \
  | jq -r '.["data"]["token"]'`

  • Verify that POST /payloads works as expected
    • Verify that you can post a new payload
    curl --silent -k -X POST \
      -H 'Accept: application/json' \
      -H 'Content-Type: application/json' \
      -H "Authorization: Bearer $MSFDB_TOKEN" \
      'https://0.0.0.0:5443/api/v1/payloads?workspace=default' \
      --data '{ "name": "Super duper payload", "uuid": "8675309", "timestamp": "1537306670", "arch": "x64", "platform": "windows", "urls": [], "workspace_id": 1 }' \
      | jq
    
    • Verify that you cannot post another new payload with a duplicate UUID
    curl --silent -k -X POST \
      -H 'Accept: application/json' \
      -H 'Content-Type: application/json' \
      -H "Authorization: Bearer $MSFDB_TOKEN" \
      'https://0.0.0.0:5443/api/v1/payloads?workspace=default' \
      --data '{ "name": "Duplicate payload", "uuid": "8675309", "timestamp": "1537306670", "arch": "x64", "platform": "windows", "urls": [], "workspace_id": 1 }' \
      | jq
    

  • Verify that GET /payloads works as expected
    • Verify that you can query all payloads
    curl --silent -k -X GET \
      -H 'Accept: application/json' \
      -H 'Content-Type: application/json' \
      -H "Authorization: Bearer $MSFDB_TOKEN" \
      'https://0.0.0.0:5443/api/v1/payloads?workspace=default' \
      | jq
    
    • Verify that you can search payloads by id
    curl --silent -k -X GET \
      -H 'Accept: application/json' \
      -H 'Content-Type: application/json' \
      -H "Authorization: Bearer $MSFDB_TOKEN" \
      'https://0.0.0.0:5443/api/v1/payloads/1' \
      | jq
    
    • Verify that you can search payloads by uuid
    curl --silent -k -X GET \
      -H 'Accept: application/json' \
      -H 'Content-Type: application/json' \
      -H "Authorization: Bearer $MSFDB_TOKEN" \
      'https://0.0.0.0:5443/api/v1/payloads?workspace=default&uuid=8675309' \
      | jq
    

  • Verify that PUT /payloads works as expected
    • Verify that you can update a payload
    curl --silent -k -X PUT \
      -H 'Accept: application/json' \
      -H 'Content-Type: application/json' \
      -H "Authorization: Bearer $MSFDB_TOKEN" \
      'https://0.0.0.0:5443/api/v1/payloads/1?workspace=default' \
      --data '{ "name": "Payload numero uno"}' \
      | jq
    
    • Verify that you cannot update a payload that doesn't exist
    curl --silent -k -X PUT \
      -H 'Accept: application/json' \
      -H 'Content-Type: application/json' \
      -H "Authorization: Bearer $MSFDB_TOKEN" \
      'https://0.0.0.0:5443/api/v1/payloads/999?workspace=default' \
      --data '{ "name": "Payload numero dos"}' \
      | jq
    

  • Verify that DELETE /payloads works as expected
    • Verify that you can delete a payload
    curl --silent -k -X DELETE \
      -H 'Accept: application/json' \
      -H 'Content-Type: application/json' \
      -H "Authorization: Bearer $MSFDB_TOKEN" \
      'https://0.0.0.0:5443/api/v1/payloads?workspace=default' \
      --data '{ "ids": [ 1 ] }' \
      | jq
    
    • Verify that you cannot delete a payload that doesn't exist
    curl --silent -k -X DELETE \
      -H 'Accept: application/json' \
      -H 'Content-Type: application/json' \
      -H "Authorization: Bearer $MSFDB_TOKEN" \
      'https://0.0.0.0:5443/api/v1/payloads?workspace=default' \
      --data '{ "ids": [ 999 ] }' \
      | jq
    

  • Verify that the Swagger docs display documentation for payloads as expected
    • ./msfdb_ws
    • Go to https://0.0.0.0:5443/api/v1/api-docs
    • Verify that the payload section displays correctly
    • Verify that the content is reasonably correct and typo-free
    • Verify that the "Try it out" buttons work correctly. (Functionality should be the same as test cases above).

ebleiweiss-r7 added some commits Sep 4, 2018

ebleiweiss-r7 added some commits Sep 20, 2018

@ebleiweiss-r7 ebleiweiss-r7 changed the title [WIP] DB Manager for Payloads DB Manager for Payloads Sep 20, 2018

@ebleiweiss-r7 ebleiweiss-r7 removed the delayed label Sep 20, 2018

arch: uuid.arch,
platform: uuid.platform,
timestamp: uuid.timestamp,
payload: self.fullname,
datastore: self.datastore
# payload: self.fullname,

This comment has been minimized.

@jbarnett-r7

jbarnett-r7 Sep 20, 2018

Contributor

Can you add a comment here about why these are commented out?

@jbarnett-r7

This comment has been minimized.

Copy link
Contributor

jbarnett-r7 commented Sep 21, 2018

Creating a payload using PayloadUUIDTracking doesn't set the workspace value:

meterpreter > background
[*] Backgrounding session 1...
msf5 exploit(windows/smb/ms17_010_eternalblue) > irb
[*] Starting IRB shell...

Mcannot load such file -- awesome_print
>> Mdm::Payload.first
=> #<Mdm::Payload id: 1, name: nil, uuid: "705aaf33d4073e98", timestamp: 1537549500, arch: "x64", platform: "windows", urls: ["/cFqvM9QHPphkdGV2P9FAyA0czC6QaxSXfnVStuXKHA8fVcKbc6xGf-HH6AxNWFZkiceKc_-CdmpTrDJddPaywztIGGHGz2wHJlvFIfWygW6EH-oj_bPiqNmbZfkai0k_JLMENAb_JKZdSor0-xd"], description: nil, workspace_id: nil, raw_payload: nil, raw_payload_hash: nil, build_opts: nil, created_at: "2018-09-21 17:05:00", updated_at: "2018-09-21 17:05:00">
>> Mdm::Payload.first.workspace
=> nil
return Array.wrap(Mdm::Payload.find(opts[:id]))
end

opts.delete(:workspace)

This comment has been minimized.

@jbarnett-r7

jbarnett-r7 Sep 21, 2018

Contributor

Other models use Msf::Util::DBManager.process_opts_workspace to ensure that :workspace is present. I'm not really sure that it's necessary here, but it does make it inconsistent with other models.

include Swagger::Blocks

NAME_DESC = 'A name for the payload'
UUID_DESC = 'A payload\'s unique identifier'

This comment has been minimized.

@jbarnett-r7

jbarnett-r7 Sep 21, 2018

Contributor

Can you please add periods to the end of these sentences so it's consistent across each of the descriptions?

key :description, 'Return payloads that are stored in the database.'
key :tags, [ 'payload' ]

parameter :workspace

This comment has been minimized.

@jbarnett-r7

jbarnett-r7 Sep 21, 2018

Contributor

This saved parameter has :workspace marked as required. :workspace isn't actually required for this request based on the other comment. If we decide not to make it required, we should update the docs.

ebleiweiss-r7 added some commits Sep 24, 2018

ebleiweiss-r7 added some commits Sep 24, 2018

@ebleiweiss-r7 ebleiweiss-r7 changed the title DB Manager for Payloads [WIP] DB Manager for Payloads Oct 3, 2018

@asoto-r7

This comment has been minimized.

Copy link
Contributor

asoto-r7 commented Feb 28, 2019

Hey @ebleiweiss-r7,

@bcook-r7 asked me to land this PR and the corresponding PR in metasploit_data_models, so expect some activity here today as I take some testing notes. Feel free to ignore me, or chime in if I'm headed in the wrong direction. :-)

Thanks!

@asoto-r7

This comment has been minimized.

Copy link
Contributor

asoto-r7 commented Feb 28, 2019

Alright, here's my notes so far:

I've cloned rapid7/metasploit_data_models#172 and merged it to metasploit_data_models:master successfully. I've also pulled this PR and merged it to metasploit-framework:master after resolving two conflicts:

  • lib/msf/core/framework.rb:82: uuid_db and analyze were both added, so I just kept both lines.
  • lib/msf/core/web_services/metasploit_api_app.rb:27: It looks like the require statements changed namespaces from msf/core/db_manager/http/servlet/* to msf/core/web_services/servlet/*. I went ahead and changed the following payload_servet line to point to the new web_services namespace:

https://github.com/rapid7/metasploit-framework/pull/10675/files#diff-330aa4d90551e343121aa149827c7a91R27

That said, upon launching msfconsole, I have the following fatal error:

~/git/r7/metasploit-framework # msfconsole
Starting git version of msfconsole...
Traceback (most recent call last):
        8: from ./msfconsole:49:in `<main>'
        7: from /home/administrator/git/r7/metasploit-framework/lib/metasploit/framework/command/base.rb:82:in `start'
        6: from /home/administrator/git/r7/metasploit-framework/lib/metasploit/framework/command/console.rb:48:in `start'
        5: from /home/administrator/git/r7/metasploit-framework/lib/metasploit/framework/command/console.rb:62:in `driver'
        4: from /home/administrator/git/r7/metasploit-framework/lib/metasploit/framework/command/console.rb:62:in `new'
        3: from /home/administrator/git/r7/metasploit-framework/lib/msf/ui/console/driver.rb:82:in `initialize'
        2: from /home/administrator/git/r7/metasploit-framework/lib/msf/base/simple/framework.rb:73:in `create'
        1: from /home/administrator/git/r7/metasploit-framework/lib/msf/base/simple/framework.rb:73:in `new'
/home/administrator/git/r7/metasploit-framework/lib/msf/core/framework.rb:82:in `initialize': undefined method `uuid_db=' for #<Msf::Framework:0x000055869a351cb0> (NoMethodError)                                                                                            

Strangely (and perhaps relatedly), Github reports a conflict with lib/msf/core/db_manager/http/servlet/api_docs_servlet.rb, but Git on the command line didn't mention it. In fact, the entire http folder is missing. 🤔

@ebleiweiss-r7

This comment has been minimized.

Copy link
Contributor Author

ebleiweiss-r7 commented Feb 28, 2019

Thanks for reviving this one!
uuid_db shouldn't exist anywhere in the codebase after this change, so my uneducated guess is that some other PR added a reference to it and got merged in the interim.

The servlets have since been moved into lib/msf/core/web_services/servlet.

I've already forgotten most of what I did here, but if you need me I can jump back in it and do some digging

@asoto-r7

This comment has been minimized.

Copy link
Contributor

asoto-r7 commented Feb 28, 2019

uuid_db shouldn't exist anywhere in the codebase after this change...

Yep! That was it. Thanks for the sanity check! 😃

asoto-r7 added a commit that referenced this pull request Mar 4, 2019

@asoto-r7 asoto-r7 merged commit 02eeaaf into rapid7:master Mar 4, 2019

3 checks passed

Metasploit Automation - Sanity Test Execution Successfully completed all tests.
Details
Metasploit Automation - Test Execution Successfully completed all tests.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

asoto-r7 added a commit that referenced this pull request Mar 4, 2019

@asoto-r7 asoto-r7 referenced this pull request Mar 5, 2019

Closed

Fix https payload db exception #11526

0 of 9 tasks complete
@busterb

This comment has been minimized.

Copy link
Contributor

busterb commented Mar 6, 2019

Release Notes

This changes Metasploit's payload UUID tracking mechanism to use the Metasploit database mechanism instead of a local file in ~/.msf4/payloads.json. If you are using payloads.json currently, you may need to stay on a previous release of Metasploit 5, the Metasploit 4.x branch, or regenerate your payloads to use the new mechanism.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.