Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement query operation for Event on DBManager and Remote Data Service #11163

Merged
merged 9 commits into from
Jan 8, 2019
Merged

Implement query operation for Event on DBManager and Remote Data Service #11163

merged 9 commits into from
Jan 8, 2019

Conversation

mkienow-r7
Copy link
Contributor

@mkienow-r7 mkienow-r7 commented Dec 22, 2018 •
edited by jbarnett-r7
Loading

Implements a query (read) operation for Event on both the DBManager and Remote Data Service. The existing Msf::DBManager::Event#events method on both master (MSF5) and 4.x branches fails to execute without issue.

Ticket: MS-2818

Changes

  • Functional events method for querying Event database records from both the local and remote data services
  • Pagination and sort options for events query:
    • :order - The event created_at sort order. Valid values: :asc, :desc, asc or desc. Default: :desc
    • :limit - The maximum number of events that will be retrieved from the query. Default: 100
    • :offset - The number of events the query will begin reading from the start of the set. Default: 0
  • OpenAPI documentation reflects the events query changes

Verification

  • Restart the database and MSF web service (data services) msfdb restart, and init/reinit if necessary.
  • Start msfconsole and connect to the data service started above if you didn't select the option to connect automatically during initialization. See Metasploit Web Service for more information.
  • Verify db_status reports Connection type: http. Connected to remote_data_service: (https://localhost:8080)
  • Run commands, pop shells, interact with sessions - all of these actions create event data
  • Verify that events are being created for your actions by using the event section of the interactive documentation at https://localhost:8080/api/v1/api-docs
  • Quit msfconsole
  • Test local DB mode
    • Temporarily disable the data service auto-connect by renaming the config file: mv ~/.msf4/config ~/.msf4/config.disabled
    • Start msfconsole
    • Verify db_status reports Connected to msf. Connection type: postgresql.
    • Repeat the remote data service verification steps above
    • Quit msfconsole
    • Restore the config file: mv ~/.msf4/config.disabled ~/.msf4/config

@jbarnett-r7 jbarnett-r7 self-assigned this Jan 3, 2019
@jbarnett-r7
Copy link
Contributor

jbarnett-r7 commented Jan 7, 2019 •
edited
Loading

I'm seeing weird behavior when setting the offset value and sorting by desc (which is the default sort order. It looks like sorting descending starts the count at 1 and counts up to the offset, instead of returning the results from the offset and then just ordering in descending order. It probably makes more sense with examples:

Background: I have 58 events in my DB, with id starting at 1 and going up to 58.

With this command: curl -X GET "https://localhost:8080/api/v1/events?workspace=default&limit=5&offset=50" -H "accept: application/json" -H "Authorization: Bearer REDACTED"

Outputs:

{
  "data": [
    {
      "id": 8,
      "workspace_id": 1,
      "host_id": null,
      "created_at": "2019-01-07T20:23:58.572Z",
      "name": "ui_start",
      "updated_at": "2019-01-07T20:23:58.572Z",
      "critical": null,
      "seen": null,
      "username": null,
      "info": {
        "revision": "$Revision$"
      }
    },
    {
      "id": 7,
      "workspace_id": 1,
      "host_id": null,
      "created_at": "2019-01-07T20:23:55.235Z",
      "name": "ui_command",
      "updated_at": "2019-01-07T20:23:55.235Z",
      "critical": null,
      "seen": null,
      "username": null,
      "info": {
        "command": "db_connect local-https-data-service"
      }
    },
    {
      "id": 6,
      "workspace_id": 1,
      "host_id": null,
      "created_at": "2019-01-07T20:23:31.655Z",
      "name": "ui_stop",
      "updated_at": "2019-01-07T20:23:31.655Z",
      "critical": null,
      "seen": null,
      "username": null,
      "info": {}
    },
    {
      "id": 5,
      "workspace_id": 1,
      "host_id": null,
      "created_at": "2019-01-07T20:23:31.652Z",
      "name": "ui_command",
      "updated_at": "2019-01-07T20:23:31.652Z",
      "critical": null,
      "seen": null,
      "username": null,
      "info": {
        "command": "exit"
      }
    },
    {
      "id": 4,
      "workspace_id": 1,
      "host_id": null,
      "created_at": "2019-01-07T20:23:31.648Z",
      "name": "ui_command",
      "updated_at": "2019-01-07T20:23:31.648Z",
      "critical": null,
      "seen": null,
      "username": null,
      "info": {
        "command": "db_save"
      }
    }
  ]
}

As you can see, it looks like it starts the data at MaxID - 50 and shows data up to my limit from there. I would expect that it would still start the set at ID=50 and show me 5 results from there, so IDs 50-55.

An interesting note is that if I switch the ordering to asc, it works as I expected.

Command: curl -X GET "https://localhost:8080/api/v1/events?workspace=default&limit=5&offset=50&order=asc" -H "accept: application/json" -H "Authorization: Bearer REDACTED"

Output:

{
  "data": [
    {
      "id": 51,
      "workspace_id": 1,
      "host_id": null,
      "created_at": "2019-01-07T21:02:10.031Z",
      "name": "module_run",
      "updated_at": "2019-01-07T21:02:10.031Z",
      "critical": null,
      "seen": null,
      "username": "jbarnett",
      "info": {
        "module_name": "exploit/linux/local/docker_daemon_privilege_escalation",
        "module_uuid": "rok8bd3j",
        "datastore": {
          "PrependFork": "true",
          "WfsDelay": "60",
          "WORKSPACE": "",
          "VERBOSE": "false",
          "EnableContextEncoding": "false",
          "ContextInformationFile": "",
          "DisablePayloadHandler": "false",
          "SESSION": "2",
          "EXE::EICAR": "false",
          "EXE::Custom": "",
          "EXE::Path": "",
          "EXE::Template": "",
          "EXE::Inject": "false",
          "EXE::OldMethod": "false",
          "EXE::FallBack": "false",
          "MSI::EICAR": "false",
          "MSI::Custom": "",
          "MSI::Path": "",
          "MSI::Template": "",
          "MSI::UAC": "false",
          "FileDropperDelay": "",
          "WritableDir": "/tmp",
          "LHOST": "10.6.2.70",
          "LPORT": "4445",
          "PAYLOAD": "linux/x86/meterpreter/reverse_tcp",
          "ReverseListenerBindPort": "",
          "ReverseAllowProxy": "false",
          "ReverseListenerComm": "",
          "ReverseListenerBindAddress": "",
          "ReverseListenerThreaded": "false",
          "StagerRetryCount": "10",
          "StagerRetryWait": "5",
          "AutoLoadStdapi": "true",
          "AutoVerifySession": "true",
          "AutoVerifySessionTimeout": "30",
          "InitialAutoRunScript": "",
          "AutoRunScript": "",
          "AutoSystemInfo": "true",
          "EnableUnicodeEncoding": "false",
          "HandlerSSLCert": "",
          "SessionRetryTotal": "3600",
          "SessionRetryWait": "10",
          "SessionExpirationTimeout": "604800",
          "SessionCommunicationTimeout": "300",
          "PayloadProcessCommandLine": "",
          "AutoUnhookProcess": "false",
          "PayloadUUIDSeed": "",
          "PayloadUUIDRaw": "",
          "PayloadUUIDName": "",
          "PayloadUUIDTracking": "false",
          "EnableStageEncoding": "false",
          "StageEncoder": "",
          "StageEncoderSaveRegisters": "",
          "StageEncodingFallback": "true",
          "PrependSetresuid": "false",
          "PrependSetreuid": "false",
          "PrependSetuid": "false",
          "PrependSetresgid": "false",
          "PrependSetregid": "false",
          "PrependSetgid": "false",
          "PrependChrootBreak": "false",
          "AppendExit": "false",
          "TARGET": "0"
        }
      }
    },
    {
      "id": 52,
      "workspace_id": 1,
      "host_id": 1,
      "created_at": "2019-01-07T21:02:11.926Z",
      "name": "session_open",
      "updated_at": "2019-01-07T21:02:11.926Z",
      "critical": null,
      "seen": null,
      "username": "jbarnett",
      "info": {
        "session_id": 3,
        "session_info": null,
        "session_uuid": "wyedcrys",
        "session_type": "meterpreter",
        "username": "jbarnett",
        "target_host": "172.28.128.3",
        "via_exploit": "exploit/linux/local/docker_daemon_privilege_escalation",
        "via_payload": "payload/linux/x86/meterpreter/reverse_tcp",
        "tunnel_peer": "10.6.2.70:49880",
        "exploit_uuid": "rok8bd3j",
        "datastore": {
          "PrependFork": "true",
          "WfsDelay": "60",
          "WORKSPACE": "",
          "VERBOSE": "false",
          "EnableContextEncoding": "false",
          "ContextInformationFile": "",
          "DisablePayloadHandler": "false",
          "SESSION": "2",
          "EXE::EICAR": "false",
          "EXE::Custom": "",
          "EXE::Path": "",
          "EXE::Template": "",
          "EXE::Inject": "false",
          "EXE::OldMethod": "false",
          "EXE::FallBack": "false",
          "MSI::EICAR": "false",
          "MSI::Custom": "",
          "MSI::Path": "",
          "MSI::Template": "",
          "MSI::UAC": "false",
          "FileDropperDelay": "",
          "WritableDir": "/tmp",
          "LHOST": "10.6.2.70",
          "LPORT": "4445",
          "PAYLOAD": "linux/x86/meterpreter/reverse_tcp",
          "ReverseListenerBindPort": "",
          "ReverseAllowProxy": "false",
          "ReverseListenerComm": "",
          "ReverseListenerBindAddress": "",
          "ReverseListenerThreaded": "false",
          "StagerRetryCount": "10",
          "StagerRetryWait": "5",
          "AutoLoadStdapi": "true",
          "AutoVerifySession": "true",
          "AutoVerifySessionTimeout": "30",
          "InitialAutoRunScript": "",
          "AutoRunScript": "",
          "AutoSystemInfo": "true",
          "EnableUnicodeEncoding": "false",
          "HandlerSSLCert": "",
          "SessionRetryTotal": "3600",
          "SessionRetryWait": "10",
          "SessionExpirationTimeout": "604800",
          "SessionCommunicationTimeout": "300",
          "PayloadProcessCommandLine": "",
          "AutoUnhookProcess": "false",
          "PayloadUUIDSeed": "",
          "PayloadUUIDRaw": "",
          "PayloadUUIDName": "",
          "PayloadUUIDTracking": "false",
          "EnableStageEncoding": "false",
          "StageEncoder": "",
          "StageEncoderSaveRegisters": "",
          "StageEncodingFallback": "true",
          "PrependSetresuid": "false",
          "PrependSetreuid": "false",
          "PrependSetuid": "false",
          "PrependSetresgid": "false",
          "PrependSetregid": "false",
          "PrependSetgid": "false",
          "PrependChrootBreak": "false",
          "AppendExit": "false",
          "TARGET": "0"
        },
        "critical": true
      }
    },
    {
      "id": 53,
      "workspace_id": 1,
      "host_id": 1,
      "created_at": "2019-01-07T21:02:12.227Z",
      "name": "session_command",
      "updated_at": "2019-01-07T21:02:12.227Z",
      "critical": null,
      "seen": null,
      "username": "jbarnett",
      "info": {
        "session_id": 3,
        "session_info": null,
        "session_uuid": "wyedcrys",
        "session_type": "meterpreter",
        "username": "jbarnett",
        "target_host": "172.28.128.3",
        "via_exploit": "exploit/linux/local/docker_daemon_privilege_escalation",
        "via_payload": "payload/linux/x86/meterpreter/reverse_tcp",
        "tunnel_peer": "10.6.2.70:49880",
        "exploit_uuid": "rok8bd3j",
        "command": "load stdapi"
      }
    },
    {
      "id": 54,
      "workspace_id": 1,
      "host_id": null,
      "created_at": "2019-01-07T21:02:12.292Z",
      "name": "module_complete",
      "updated_at": "2019-01-07T21:02:12.292Z",
      "critical": null,
      "seen": null,
      "username": "jbarnett",
      "info": {
        "module_name": "exploit/linux/local/docker_daemon_privilege_escalation",
        "module_uuid": "rok8bd3j"
      }
    },
    {
      "id": 55,
      "workspace_id": 1,
      "host_id": null,
      "created_at": "2019-01-07T21:02:12.299Z",
      "name": "ui_command",
      "updated_at": "2019-01-07T21:02:12.299Z",
      "critical": null,
      "seen": null,
      "username": null,
      "info": {
        "command": "sessions -q -i 3"
      }
    }
  ]
}

@mkienow-r7
Copy link
Contributor Author

mkienow-r7 commented Jan 8, 2019 •
edited
Loading

As you can see, it looks like it starts the data at MaxID - 50 and shows data up to my limit from there. I would expect that it would still start the set at ID=50 and show me 5 results from there, so IDs 50-55.

The order parameter is used to control the order of the event records by their created_at time. In the default, descending order your events would start at your max ID of 58 and end with the first record ID 1. In ascending order your events would start at your first record ID 1 and end with the max ID of 58.

The offset parameter is not the record ID the result set begins with, rather, the offset is the number of records that will be skipped over before records will be returned (See ActiveRecord::QueryMethods offset(value)). In the first example, descending order is specified so your events would start with the max ID first and end with the first record ID 1. An offset of 50 is used, therefore, after skipping over 50 records in descending order the first record returned will be that with the ID of 8 (58-50 = 8). The limit or maximum number of events specified was 5, therefore, it will have records with the IDs: 8, 7, 6, 5 and 4.

An interesting note is that if I switch the ordering to asc, it works as I expected.

In the second example, ascending order is specified so your events would start with your first record ID 1 and end with the max ID of 58. An offset of 50 is used, therefore, after skipping over 50 records the first record returned will be that with the ID of 51 (1+50 = 51). The limit or maximum number of events specified was 5, therefore, it will have records with the IDs: 51, 52, 53, 54 and 55.

@jbarnett-r7 jbarnett-r7 merged commit 8361dab into rapid7:master Jan 8, 2019
jbarnett-r7 added a commit that referenced this pull request Jan 8, 2019
@jbarnett-r7
Land #11163, add API endpoint for retrieving Mdm::Events
466b000
@mkienow-r7 mkienow-r7 deleted the MS-2818-enhance-event-query branch January 8, 2019 17:07
@jbarnett-r7
Copy link
Contributor

Release Notes

Add a /api/v1/events endpoint for retrieving Mdm::Event data from the API.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@jbarnett-r7 jbarnett-r7

Labels
database enhancement msf5
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mkienow-r7 @jbarnett-r7