Update to v2 based process and event searches #241

crahan · 2020-07-03T12:53:18Z

Pull request checklist

Please check if your PR fulfills the following requirements:

Docs have been reviewed and added / updated if needed (for bug fixes / features)
Tests have been added that prove the fix is effective or that the feature works.
New and existing tests pass locally with the changes.
Code follows the style guidelines of this project (PEP8, clean code).
Linter has passed locally and any fixes were made for failures.
A self-review of the code has been done.

Pull request type

Please check the type of change your PR introduces:

Bugfix
Feature
Code style update (formatting, renaming)
Refactoring (no functional changes, no api changes)
Build related changes
Documentation content changes (not tied to bugs/features)
Other (please describe):

What is the ticket or issue number?

No ticket number available.

Pull Request Description

This pull request updates Cb ThreatHunter process and event searches to use the new v2 style API. Tests (see below) indicate that process search results now contain aggregated lists of TTP and enriched_event_type data. We are also able to retrieve more or even all events for a particular Process object (versus only a handful of events with the v1 event search API). Retrieval of event data now correctly uses batches of 100 events per iteration and it will loop until the value of num_available has been reached (fixing the issues reported in #239).

Does this introduce a breaking change?

Yes (maybe? see below)
No

As the change from Process Search v1 to v2 splits out enriched Process search results from the regular process search results (and moves these solely to the enriched_event search) this PR will result in procecss search results not including enriched events. An additional update will be required to support explicitly searching for enriched process events. Retrieving events from a process GUID are not affected by this and function the same as before.

Validation for event and process searches had to be disabled as the v2 search API does not supply a validation endpoint and the query parameter json format changed between v1 and v2. As a result the v1 validation API endpoints can not be used anymore. Any lines in query.py where _validate() is called have been commented out and the _validate() has been left in place. (see 'Update 2' below)

Update 1: It also still uses the v1 process search status API endpoint (/api/investigate/v1/orgs/{org_key}/processes/search_jobs/{query_id}) as v2 does not provide that functionality yet (it only has /api/investigate/v2/orgs/{org_key}/processes/search_jobs/{job_id}/results). However, using a v2 job-id with the v1 status endpoint returns the contacted vs completed information as expected. If this is not desired, then the v2 process search results endpoint (/api/investigate/v2/orgs/{org_key}/processes/search_jobs/{job_id}/results) can be used instead. It will simply return the first set of results for each status check instead of just search completion status (i.e. only the counts of the search nodes that were contacted and have returned results).

Update 2: the validate() function calls have been enabled again and the validate() function was updated to send the query as a q parameter instead of the now default query argument used in the v2 search API.

How Has This Been Tested?

The following script was used to test the changes:

from cbapi.psc.threathunter import CbThreatHunterAPI
from cbapi.psc.threathunter.models import Event, Process

cb = CbThreatHunterAPI(profile='redlab2')
guid = 'ORGKEY-01319b03-00000cc4-00000000-1d64c634925f39c'

# Process search test
query_result = cb.select(Process).where(process_guid=guid)

for process in query_result:
    print(process)

# Event search test
query_result = cb.select(Event).where(process_guid=guid).and_(event_type="childproc")

for event in query_result:
    print(event)

The below alternative code works as well (using the events() function on a Process object):

from cbapi.psc.threathunter import CbThreatHunterAPI
from cbapi.psc.threathunter.models import Event, Process

cb = CbThreatHunterAPI(profile='redlab2')
guid = 'N5DFVQXD-01319b03-00000cc4-00000000-1d64c634925f39c'

query_result = cb.select(Process).where(process_guid=guid)

for process in query_result:
    print(process)

    for event in process.events(event_type='childproc'):
        print(event)

Either of these test scripts results in the below output:

❯ python test.py
Process object, bound to https://defense-prod05.conferdeploy.net.
-------------------------------------------------------------------------------

       backend_timestamp: 2020-07-03T12:27:16.961Z
         childproc_count: 11
         crossproc_count: 7550
      device_external_ip: REDACTED
               device_id: 20028163
      device_internal_ip: REDACTED
             device_name: redlab-win10-2
               device_os: WINDOWS
        device_policy_id: 100934
        device_timestamp: 2020-07-03T12:25:45.297Z
                enriched: True
     enriched_event_type: ['CREATE_PROCESS', 'FILE_CREATE', 'REGISTRY_ACC...
              event_type: ['childproc', 'crossproc', 'filemod', 'regmod']
           filemod_count: 616
            ingress_time: 1593779216128
                  legacy: True
           modload_count: 107
           netconn_count: 0
                  org_id: ORGKEY
             parent_guid: ORGKEY-01319b03-000002b4-00000000-1d64c60d26a...
             parent_hash: ['41e785ff78c698c860be93f6f1f0a2369f8932d186d7d...
              parent_pid: 692
    process_effective_reputation: TRUSTED_WHITE_LIST
            process_guid: ORGKEY-01319b03-00000cc4-00000000-1d64c634925...
            process_hash: ['0308289d3bd5c9476bcfb96acf530343', '0a8fb7ed0...
            process_name: c:\programdata\microsoft\windows defender\platf...
             process_pid: [3268]
      process_reputation: TRUSTED_WHITE_LIST
        process_username: ['NT AUTHORITY\\SYSTEM']
            regmod_count: 401
        scriptload_count: 0
                     ttp: ['ENUMERATE_PROCESSES', 'MITRE_T1003_CREDENTIAL...
           watchlist_hit: ['wJVUBB44RdmWs2w4IexFxQ:ENRn6bsERDytLA1fk8GjzQ...
Event object, bound to https://defense-prod05.conferdeploy.net.
-------------------------------------------------------------------------------

       backend_timestamp: 2020-07-03T09:26:59.326Z
       childproc_cmdline: "C:\ProgramData\Microsoft\Windows Defender\plat...
    childproc_effective_reputation: REP_WHITE
           childproc_md5: c9ecd001c9c6d38837dab6d16b79143b
          childproc_name: c:\programdata\microsoft\windows defender\platf...
    childproc_process_guid: ORGKEY-01319b03-00000770-00000000-1d6511bbc38...
        childproc_sha256: 5bb7a21fc68c78218b713999bd478580a1a9efcc4846f1a...
      childproc_username: NT AUTHORITY\SYSTEM
       created_timestamp: 2020-07-03T12:30:49.774Z
              event_guid: aedGYDbRSb6olNQm1uWQDw
              event_hash: 7cc4809e83ebed0a6f174474669fd9af
         event_timestamp: 2020-07-03T09:24:26.975Z
              event_type: childproc
                  legacy: False
            process_guid: ORGKEY-01319b03-00000cc4-00000000-1d64c634925...
             process_pid: 3268
Event object, bound to https://defense-prod05.conferdeploy.net.
-------------------------------------------------------------------------------

       backend_timestamp: 2020-07-03T09:26:59.326Z
       childproc_cmdline: "C:\ProgramData\Microsoft\Windows Defender\plat...
    childproc_effective_reputation: REP_WHITE
           childproc_md5: c9ecd001c9c6d38837dab6d16b79143b
          childproc_name: c:\programdata\microsoft\windows defender\platf...
    childproc_process_guid: ORGKEY-01319b03-0000134c-00000000-1d6511bbc3d...
        childproc_sha256: 5bb7a21fc68c78218b713999bd478580a1a9efcc4846f1a...
      childproc_username: NT AUTHORITY\SYSTEM
       created_timestamp: 2020-07-03T12:30:49.774Z
              event_guid: uEcQi43oSpau7HMvDXKmyA
              event_hash: f00b3364df8706fb5962020bc25bee43
         event_timestamp: 2020-07-03T09:24:27.022Z
              event_type: childproc
                  legacy: False
            process_guid: ORGKEY-01319b03-00000cc4-00000000-1d64c634925...
             process_pid: 3268
...

jrhackett

This is great stuff, thanks for the help here @crahan. I'm not the one who should be approving this PR as I'm not the most familiar with this project but I left some comments in here for some context on the underlying APIs. I'm sure @avanbrunt-cb would like to look at this at some point, too.

jrhackett · 2020-07-05T21:15:56Z

src/cbapi/psc/threathunter/query.py

@@ -285,28 +303,31 @@ def _validate(self, args):
    def _search(self, start=0, rows=0):
        # iterate over total result set, 100 at a time
        args = self._get_query_parameters()
-        self._validate(args)
+        #self._validate(args)


You can use the v1 validation endpoint for this, I believe the _validate function just needs to pull the query from query instead of q.

You're correct, I've updated the _validate() function in the PR by grabbing the query parameter from args and submitting it as the value for q to the v1 validation API endpoint. Validation now passes as expected. I've updated the PR description to reflect this.

src/cbapi/psc/threathunter/query.py

src/cbapi/psc/threathunter/models.py

crahan · 2020-07-13T14:01:40Z

@avanbrunt-cb sort_by() has now been updated to support the new v2 search sort request parameter format (list of dicts vs string). Because of this I removed the sort arg from the items sent to the API v1 query validation endpoint (as validation would error). The implementation supports stacking sort_by() function calls so you can do things like:

query_result = cb.select(Process).where('process_name:powershell.exe').sort_by('device_timestamp', direction='DESC').sort_by('backend_timestamp', direction='ASC')

Which will translate into:

"sort": [
    {
        "field": "device_timestamp",
        "order": "DESC"
    },
    {
        "field": "backend_timestamp",
        "order": "ASC"
    }
]

avanbrunt-cb

LGTM

Update to v2 based process and event searches

fa396df

crahan mentioned this pull request Jul 3, 2020

WIP: starting updates to TH APIs to support v2 search_jobs #235

Closed

15 tasks

jrhackett reviewed Jul 5, 2020

View reviewed changes

Thomas Bouve added 3 commits July 6, 2020 09:31

Enabling _validate based again.

6f31adc

Update _validate logic

c63d6a9

Merge branch 'master' into search_v2

ed19f5d

crahan mentioned this pull request Jul 10, 2020

Cb ThreatHunter _search() function does not yield the correct num_available results #239

Closed

avanbrunt-cb reviewed Jul 10, 2020

View reviewed changes

src/cbapi/psc/threathunter/models.py Show resolved Hide resolved

Update sort_by() to support search v2

7d6eac0

avanbrunt-cb changed the base branch from master to develop July 13, 2020 22:11

Merge branch 'develop' into search_v2

97a9933

abowersox-cb approved these changes Jul 13, 2020

View reviewed changes

avanbrunt-cb approved these changes Jul 13, 2020

View reviewed changes

abowersox-cb merged commit 3d81184 into carbonblack:develop Jul 13, 2020

llyon-cb approved these changes Jul 13, 2020

View reviewed changes

crahan deleted the search_v2 branch July 14, 2020 12:21

crahan mentioned this pull request Jul 23, 2020

Fix typo in query field parameter and added process_start_time and device_policy fields #256

Merged

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to v2 based process and event searches #241

Update to v2 based process and event searches #241

crahan commented Jul 3, 2020 •

edited

Loading

jrhackett left a comment

jrhackett Jul 5, 2020

crahan Jul 6, 2020 •

edited

Loading

crahan commented Jul 13, 2020

avanbrunt-cb left a comment

Update to v2 based process and event searches #241

Update to v2 based process and event searches #241

Conversation

crahan commented Jul 3, 2020 • edited Loading

Pull request checklist

Pull request type

What is the ticket or issue number?

Pull Request Description

Does this introduce a breaking change?

How Has This Been Tested?

jrhackett left a comment

Choose a reason for hiding this comment

jrhackett Jul 5, 2020

Choose a reason for hiding this comment

crahan Jul 6, 2020 • edited Loading

Choose a reason for hiding this comment

crahan commented Jul 13, 2020

avanbrunt-cb left a comment

Choose a reason for hiding this comment

crahan commented Jul 3, 2020 •

edited

Loading

crahan Jul 6, 2020 •

edited

Loading