New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Miscellaneous Loot Bugs #11270

Merged
merged 15 commits into from Jan 26, 2019

Conversation

Projects
None yet
3 participants
@jbarnett-r7
Copy link
Contributor

jbarnett-r7 commented Jan 16, 2019

This PR makes a number of fixes to bugs that existed around the loot API endpoint. They were originally reported in MS-3755. The following fixes are included:

  • Update the docs to indicate the data field should be Base64 encoded.
  • Updated the docs to indicate that the path field on create and update is just the filename.
  • If a host with the same id existed in a local database, that information would be displayed instead of the data returned from the remote data service.
  • Remove the ability to update the data field. This is to ensure integrity for the original looted files.
  • Some endpoints didn't properly Base64 encode the data field.
  • Handle return of empty data fields.
  • Create and update operations on the remote data service will now prepend a unique string to the filename to prevent accidentally overwriting files when they have the same name.
  • Remove the nested loot JSON from host objects. It wasn't actually being used anywhere.

Verification

List the steps needed to make sure this thing works

  • Start msfdb on a true remote system such as a VM.
    • Testing through `msfconsole
    • Start msfconsole
    • Connect to the remote data service using db_connect
    • Load up metasploitable3 and run the following resource script:
    use exploit/unix/irc/unreal_ircd_3281_backdoor
    set RHOST 172.28.128.3
    set RPORT 6697
    exploit -z
    sleep 5
    sessions -u 1
    sleep 5
    use exploit/linux/local/docker_daemon_privilege_escalation
    set SESSION 2
    set LHOST 172.28.128.1
    set LPORT 4445
    exploit
    
    • use post/linux/gather/enum_configs set SESSION 3 run
    • Run the loot command. Verify the output is correct. Ensure the host value is correct.
    • Open ~/.msf4/loot on the server and verify the loot files are present with the correct contents
  • Test through the API
    • Open the API docs <server address>:<port>/api/v1/api-docs
    • Verify the loot data models under create and update look correct.
    • Create a loot object. Verify that it is successfully created and the file is created on the remote server
    • Update that loot object. Verify that the operation completes successfully. Ensure you are changing the path attribute` and that it gets a new unique string.
    • Create a loot object with an empty data field. Run a GET and verify it displays correctly.

jbarnett-r7 added some commits Jan 10, 2019

Dont do a separate lookup for loot.host, use the included JSON
This is just a temporary change. Eventually we should be doing separate
lookups for associated objects as that is the RESTful way of doing it.
Implementing this now to prevent extra load on the server until we can
put a better system in place of doing multiple lookups with a single call.
Remove nested loot object from host JSON
The code on the framework side that was utilizing this was removed
a while ago. It was never actually being used anywhere, and was causing
issues with getting host objects back when the loot contained
non-UTF-8 characters
Prepend loot filenames with unique string
This should help prevent accidentally overwriting files with the same name

@jmartin-r7 jmartin-r7 added the msf5 label Jan 18, 2019

@mkienow-r7 mkienow-r7 self-assigned this Jan 24, 2019

@mkienow-r7

This comment has been minimized.

Copy link
Contributor

mkienow-r7 commented Jan 24, 2019

I have my local msfconsole connected to a Metasploit web service running on a VM (msfdb --address 0.0.0.0 start). I created 2 new loot entries via POSTs to the API endpoint /api/v1/loots.

Request Body 1:

{
  "workspace": "default",
  "host": "172.28.128.3",
  "ltype": "file",
  "path": "/opt/test/no-data.txt",
  "ctype": "text/plain",
  "name": "no-data.txt",
  "info": "no data"
}

Request Body 2:

{
  "workspace": "default",
  "host": "172.28.128.3",
  "ltype": "file",
  "path": "/opt/test/no-data2.txt",
  "data": null,
  "ctype": "text/plain",
  "name": "no-data2.txt",
  "info": "no data"
}

The response for both of these has that data stored as: "data": {}. When I perform an operation inside msfconsole that results in a loot query it fails.

msf5 > workspace -v
[-] Error while running command workspace: Problem retrieving loot: undefined method `tr' for {}:Hash
Did you mean?  try. See log for more details.

Call stack:
/home/msfdev/metasploit-framework/lib/metasploit/framework/data_service/proxy/core.rb:174:in `log_error'
/home/msfdev/metasploit-framework/lib/metasploit/framework/data_service/proxy/loot_data_proxy.rb:46:in `rescue in loots'
/home/msfdev/metasploit-framework/lib/metasploit/framework/data_service/proxy/loot_data_proxy.rb:39:in `loots'
/home/msfdev/metasploit-framework/lib/msf/ui/console/command_dispatcher/db.rb:242:in `block in cmd_workspace'
/home/msfdev/metasploit-framework/lib/msf/ui/console/command_dispatcher/db.rb:234:in `each'
/home/msfdev/metasploit-framework/lib/msf/ui/console/command_dispatcher/db.rb:234:in `cmd_workspace'
/home/msfdev/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:502:in `run_command'
/home/msfdev/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:453:in `block in run_single'
/home/msfdev/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:447:in `each'
/home/msfdev/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:447:in `run_single'
/home/msfdev/metasploit-framework/lib/rex/ui/text/shell.rb:151:in `run'
/home/msfdev/metasploit-framework/lib/metasploit/framework/command/console.rb:48:in `start'
/home/msfdev/metasploit-framework/lib/metasploit/framework/command/base.rb:82:in `start'
./msfconsole:49:in `<main>'
msf5 > loot
[-] Error while running command loot: Problem retrieving loot: undefined method `tr' for {}:Hash
Did you mean?  try. See log for more details.

Call stack:
/home/msfdev/metasploit-framework/lib/metasploit/framework/data_service/proxy/core.rb:174:in `log_error'
/home/msfdev/metasploit-framework/lib/metasploit/framework/data_service/proxy/loot_data_proxy.rb:46:in `rescue in loots'
/home/msfdev/metasploit-framework/lib/metasploit/framework/data_service/proxy/loot_data_proxy.rb:39:in `loots'
/home/msfdev/metasploit-framework/lib/msf/ui/console/command_dispatcher/db.rb:1303:in `cmd_loot'
/home/msfdev/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:502:in `run_command'
/home/msfdev/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:453:in `block in run_single'
/home/msfdev/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:447:in `each'
/home/msfdev/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:447:in `run_single'
/home/msfdev/metasploit-framework/lib/rex/ui/text/shell.rb:151:in `run'
/home/msfdev/metasploit-framework/lib/metasploit/framework/command/console.rb:48:in `start'
/home/msfdev/metasploit-framework/lib/metasploit/framework/command/base.rb:82:in `start'
./msfconsole:49:in `<main>'
@mkienow-r7

This comment has been minimized.

Copy link
Contributor

mkienow-r7 commented Jan 24, 2019

Updating a loot object's path does create a "new unique string" as a part of the file path, however, that is not reflected on disk.

Loot object before update:

{
  "data": {
    "id": 22,
    "workspace_id": 1,
    "host_id": 1,
    "service_id": null,
    "ltype": "file",
    "path": "/home/msfdev/.msf4/loot/391ba5ba4a18b8abfbe6-no-data3.txt",
    "data": "",
    "created_at": "2019-01-24T19:09:06.391Z",
    "updated_at": "2019-01-24T19:09:06.391Z",
    "content_type": "text/plain",
    "name": "no-data3.txt",
    "info": "no data",
    "module_run_id": null
  }
}

File on disk:

-rw-r--r-- 1 msfdev msfdev 0 Jan 24 14:09 391ba5ba4a18b8abfbe6-no-data3.txt

Update loot object, PUT to /api/v1/loots/{id} request body:

{
  "path": "/home/msfdev/.msf4/loot/no-data3.txt"
}

Response body:

{
  "data": {
    "id": 22,
    "workspace_id": 1,
    "host_id": 1,
    "service_id": null,
    "ltype": "file",
    "path": "/home/msfdev/.msf4/loot/1ff8ee7e5b91e9bed1ad-no-data3.txt",
    "data": "",
    "created_at": "2019-01-24T19:09:06.391Z",
    "updated_at": "2019-01-24T23:44:01.735Z",
    "content_type": "text/plain",
    "name": "no-data3-up1.txt",
    "info": "update2",
    "module_run_id": null
  }
}

File on disk is unchanged:

-rw-r--r-- 1 msfdev msfdev 0 Jan 24 14:09 391ba5ba4a18b8abfbe6-no-data3.txt
CONTENT_TYPE_DESC = 'The mime/content type of the file at {#path}. Used to server the file correctly so browsers understand whether to render or download the file.'
CONTENT_TYPE_EXAMPLE = 'text/plain'
NAME_DESC = 'The name of the loot.'
NAME_EXAMPLE = 'password_file.txt'
INFO_DESC = 'Information about the loot.'
MODULE_RUN_ID_DESC = 'The ID of the module run record this loot is associated with.'

# Some of the attributes expect different data when doing a create.
CREATE_PATH_DESC = 'The name to give the file on the server. Note that this will prepend a unique string to the given value.'

This comment has been minimized.

@mkienow-r7

mkienow-r7 Jan 25, 2019

Contributor

Minor: I found the following a little confusing, consider rewording. "Note that this will prepend a unique string to the given value." Suggestion: "A unique string will be prepended to the filename portion of the full path."

This comment has been minimized.

@jbarnett-r7

jbarnett-r7 Jan 25, 2019

Author Contributor

I wanted to somehow note that when you're creating a loot object, the full path isn't actually honored. We save everything in the ~/.msf4/loot directory when the Base64 is processed. If you have a full file path in there, we just chop off everything but the filename.

Your suggested edit makes it seem like we'll still honor the full path value that the user specifies, and just prepend the unique string after the last / but before the actual file name. I'll use a version of that suggestion that makes it more obvious only the file name is needed on a create operation.

rv[data.index(loot)].path = process_file(loot[:data], local_path)
end
if loot[:host]
host_object = to_ar(RemoteHostDataService::HOST_MDM_CLASS.constantize, loot[:host])

This comment has been minimized.

@mkienow-r7

mkienow-r7 Jan 25, 2019

Contributor

What is the purpose of calling constantize on HOST_MDM_CLASS when this isn't performed earlier on LOOT_MDM_CLASS or in other locations?

This comment has been minimized.

@jbarnett-r7

jbarnett-r7 Jan 25, 2019

Author Contributor

The other methods are calling json_to_mdm_object, this is calling to_ar directly. json_to_mdm_object does the constantize call within it when it calls to_ar.

@jbarnett-r7

This comment has been minimized.

Copy link
Contributor Author

jbarnett-r7 commented Jan 25, 2019

Updating a loot object's path does create a "new unique string" as a part of the file path, however, that is not reflected on disk.

Loot object before update:

{
  "data": {
    "id": 22,
    "workspace_id": 1,
    "host_id": 1,
    "service_id": null,
    "ltype": "file",
    "path": "/home/msfdev/.msf4/loot/391ba5ba4a18b8abfbe6-no-data3.txt",
    "data": "",
    "created_at": "2019-01-24T19:09:06.391Z",
    "updated_at": "2019-01-24T19:09:06.391Z",
    "content_type": "text/plain",
    "name": "no-data3.txt",
    "info": "no data",
    "module_run_id": null
  }
}

File on disk:

-rw-r--r-- 1 msfdev msfdev 0 Jan 24 14:09 391ba5ba4a18b8abfbe6-no-data3.txt

Update loot object, PUT to /api/v1/loots/{id} request body:

{
  "path": "/home/msfdev/.msf4/loot/no-data3.txt"
}

Response body:

{
  "data": {
    "id": 22,
    "workspace_id": 1,
    "host_id": 1,
    "service_id": null,
    "ltype": "file",
    "path": "/home/msfdev/.msf4/loot/1ff8ee7e5b91e9bed1ad-no-data3.txt",
    "data": "",
    "created_at": "2019-01-24T19:09:06.391Z",
    "updated_at": "2019-01-24T23:44:01.735Z",
    "content_type": "text/plain",
    "name": "no-data3-up1.txt",
    "info": "update2",
    "module_run_id": null
  }
}

File on disk is unchanged:

-rw-r--r-- 1 msfdev msfdev 0 Jan 24 14:09 391ba5ba4a18b8abfbe6-no-data3.txt

I made a change locally to make it so we did actually update the file on disk. One thing to note I found is if you have a loot object with no data, the path attribute doesn't really mean much. By that I mean that since we never process any data, we don't reassign the path attribute to the ~/.msf4/loot/ location; it will be whatever the user set it as. If you then perform an update, we perform the action where we prepend it with a unique string.

Example:

Create loot with blank data:

    {
      "id": 2,
      "workspace_id": 1,
      "host_id": 2,
      "service_id": null,
      "ltype": "file",
      "path": "/opt/test/no-data2.txt",
      "data": {},
      "created_at": "2019-01-24T23:17:27.192Z",
      "updated_at": "2019-01-24T23:17:27.192Z",
      "content_type": "text/plain",
      "name": "no-data2.txt",
      "info": "no data",
      "module_run_id": null,
      "host": {
        "id": 2,
        "created_at": "2019-01-24T23:17:27.186Z",
        "address": "172.28.128.3",
        "mac": null,
        "comm": "",
        "name": null,
        "state": "alive",
        "os_name": null,
        "os_flavor": null,
        "os_sp": null,
        "os_lang": null,
        "arch": null,
        "workspace_id": 1,
        "updated_at": "2019-01-24T23:17:27.186Z",
        "purpose": null,
        "info": null,
        "comments": null,
        "scope": null,
        "virtual_host": null,
        "note_count": 0,
        "vuln_count": 0,
        "service_count": 0,
        "host_detail_count": 0,
        "exploit_attempt_count": 0,
        "cred_count": 0,
        "detected_arch": null,
        "os_family": null
      }
    }

Update the path attribute on that object to no-data3.txt:

{
  "data": {
    "id": 2,
    "workspace_id": 1,
    "host_id": 2,
    "service_id": null,
    "ltype": "file",
    "path": "/Users/jbarnett/.msf4/loot/6babee331783d85cf388-no-data3.txt",
    "data": {},
    "created_at": "2019-01-24T23:17:27.192Z",
    "updated_at": "2019-01-25T19:30:50.460Z",
    "content_type": "text/plain",
    "name": "no-data2.txt",
    "info": "no data",
    "module_run_id": null
  }
}

I'm not sure how we should handle this. Do we just ignore it since there isn't actually any file on disk associated with this object anyway?

jbarnett-r7 added some commits Jan 25, 2019

@mkienow-r7

This comment has been minimized.

Copy link
Contributor

mkienow-r7 commented Jan 26, 2019

I observed another odd behavior that I previously overlooked, but verified it currently exists in master so I'm going to land this PR as-is and we can assess this and open another ticket if necessary.

I have my local msfconsole connected to a Metasploit web service running on a VM (msfdb --address 0.0.0.0 start). I created a new loot entry with data via a POST to the API endpoint /api/v1/loots.

Request body:

{
  "workspace": "default",
  "host": "172.28.128.3",
  "ltype": "file",
  "path": "/opt/test/test-1b.txt",
  "data": "cXVpY2sgYnJvd24gZm94Cg==",
  "ctype": "text/plain",
  "name": "test-1b.txt",
  "info": "original"
}

Response body:

{
  "data": {
    "id": 61,
    "workspace_id": 1,
    "host_id": 1,
    "service_id": null,
    "ltype": "file",
    "path": "/home/msfdev/.msf4/loot/0f98128c676cd538cbe9-test-1b.txt",
    "data": "cXVpY2sgYnJvd24gZm94Cg==",
    "created_at": "2019-01-25T23:55:05.146Z",
    "updated_at": "2019-01-25T23:55:05.146Z",
    "content_type": "text/plain",
    "name": "test-1b.txt",
    "info": "original",
    "module_run_id": null
  }
}

The response from the API is what I would expect, however, the output of the loot command in msfconsole outputs another a different path than what exists in the DB.

msf5 > loot

Loot
====

host          service  type  name             content     info      path
----          -------  ----  ----             -------     ----      ----
172.28.128.3           file  test-1b.txt      text/plain  original  /Users/mkienow/.msf4/loot/0f98128c676cd538cbe9-test-1b.txt

@mkienow-r7 mkienow-r7 merged commit e55f459 into rapid7:master Jan 26, 2019

3 checks passed

Metasploit Automation - Sanity Test Execution Successfully completed all tests.
Details
Metasploit Automation - Test Execution Successfully completed all tests.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

mkienow-r7 added a commit that referenced this pull request Jan 26, 2019

@mkienow-r7 mkienow-r7 added the bug label Jan 26, 2019

@mkienow-r7

This comment has been minimized.

Copy link
Contributor

mkienow-r7 commented Jan 26, 2019

Release Notes

Fixes miscellaneous issues that existed around the loot API endpoint.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment