Feature/v2 file event apis (#413)

* init commit

* add v2 query filters

* v2 apis

* Add saved search v2 compatibility

CHANGELOG.md

@@ -10,6 +10,19 @@ how a consumer would use the library (e.g. adding unit tests, updating documenta
 
 ## 1.23.0 - 2022-05-12
 
+## Added
+- Support for V2 file event data.
+  - Use queries built with V2 filters by importing the appropriate modules with `from py42.sdk.queries.fileevents.v2 import *`. Documentation is available for all V2 filter terms.
+  - The following functions will now use V2 apis for searching file events if sent a V2 query object:
+    - `securitydata.search_file_events()`
+    - `securitydata.search_all_file_events()`
+  - All saved search methods now have an optional `use_v2=False` argument.  Set `use_v2=True` to opt into using the V2 saved search APIs. The following methods now accept this arg:
+    - `securitydata.savedsearches.get()`
+    - `securitydata.savedsearches.get_by_id()`
+    - `securitydata.savedsearches.get_query()`
+    - `securitydata.savedsearches.execute()`
+    - `securitydata.savedsearches.search_file_events()`
+
 ### Added
 
 - `Watchlists` and `UserRiskProfile` clients

CONTRIBUTING.md

@@ -4,11 +4,11 @@
 - [Installation](#installation)
 - [Run a full build](#run-a-full-build)
 - [Coding Style](#coding-style)
-  - [General](#general)
   - [Wrapping web APIs](#wrapping-web-apis)
   - [Style linter](#style-linter)
 - [Tests](#tests)
   - [Writing tests](#writing-tests)
+    - [Integration tests](#integration-tests)
 - [Documentation](#documentation)
   - [Generating documentation](#generating-documentation)
     - [Performing a test build](#performing-a-test-build)
@@ -86,7 +86,7 @@ Next, with your virtual environment activated, install py42 and its development
 ["editable mode"](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs).
 
 ```bash
-pip install -e '.[dev]'
+pip install -e .'[dev]'
 ```
 
 Open the project in your IDE of choice and change the python environment to

docs/guides.md

@@ -18,6 +18,7 @@
     Backup Sets <userguides/backupsets>
     userguides/cases
     userguides/trustedactivities
+    V2 File Events <userguides/v2apis>
     userguides/userriskprofile
     userguides/watchlists
 ```
@@ -33,5 +34,6 @@
 * [Backup Sets](userguides/backupsets.md)
 * [Cases](userguides/cases.md)
 * [Trust Settings](userguides/trustedactivities.md)
+* [V2 File Events](userguides/v2apis.md)
 * [User Risk Profiles](userguides/userriskprofile.md)
 * [Watchlists](userguides/watchlists.md)

docs/methoddocs/fileeventqueries.md

@@ -1,4 +1,9 @@
-# File Event Queries
+# File Event Queries - V1 (DEPRECATED)
+
+```{eval-rst}
+.. warning:: V1 file events, saved searches, and queries are **deprecated**.
+```
+For details on using the new file event data model, see the [V2 File Events User Guide](../userguides/v2apis.md).
 
 ```{eval-rst}
 .. autoclass:: py42.sdk.queries.fileevents.file_event_query.FileEventQuery
@@ -35,19 +40,19 @@ See [Executing Searches](../userguides/searches.md) for more on building search
 ### Event Filters
 
 ```{eval-rst}
-.. automethod:: py42.sdk.queries.fileevents.file_event_query.create_exists_filter_group
+.. automethod:: py42.sdk.queries.fileevents.util.create_exists_filter_group
 ```
 
 ```{eval-rst}
-.. automethod:: py42.sdk.queries.fileevents.file_event_query.create_not_exists_filter_group
+.. automethod:: py42.sdk.queries.fileevents.util.create_not_exists_filter_group
 ```
 
 ```{eval-rst}
-.. automethod:: py42.sdk.queries.fileevents.file_event_query.create_greater_than_filter_group
+.. automethod:: py42.sdk.queries.fileevents.util.create_greater_than_filter_group
 ```
 
 ```{eval-rst}
-.. automethod:: py42.sdk.queries.fileevents.file_event_query.create_less_than_filter_group
+.. automethod:: py42.sdk.queries.fileevents.util.create_less_than_filter_group
 ```
 
 ```{eval-rst}

docs/methoddocs/fileeventqueriesv2.md

@@ -0,0 +1,117 @@
+# File Event Queries - V2
+
+For details on using the new file event data model, see the [V2 File Events User Guide](../userguides/v2apis.md).
+
+```{eval-rst}
+.. autoclass:: py42.sdk.queries.fileevents.v2.file_event_query.FileEventQuery
+    :members:
+    :show-inheritance:
+    :noindex:
+```
+
+## Saved Searches
+
+```{eval-rst}
+.. important::
+    Make sure to set the optional argument `use_v2=True` on saved search functions to get V2 file event data and queries.
+
+```
+
+```{eval-rst}
+.. autoclass:: py42.services.savedsearch.SavedSearchService
+    :members:
+    :show-inheritance:
+    :noindex:
+```
+
+
+## Filter Classes
+
+The following classes construct filters for file event queries. Each filter class corresponds to a file event detail.
+Call the appropriate class method on your desired filter class with the `value` you want to match and it will return a
+`FilterGroup` object that can be passed to `FileEventQuery`'s `all()` or `any()` methods to create complex queries
+that match multiple filter rules.
+
+Example:
+
+To search for events observed for certain set of documents, you can use the `file.Name` and `file.MD5` filter classes to
+construct `FilterGroup`s that will search for matching filenames or (in case someone renamed the sensitive file) the
+known MD5 hashes of the files:
+
+    from py42.sdk.queries.fileevents.v2 import *
+    filename_filter = File.Name.is_in(['confidential_plans.docx', 'confidential_plan_projections.xlsx'])
+    md5_filter = File.MD5.is_in(['133765f4fff5e3038b9352a4d14e1532', 'ea16f0cbfc76f6eba292871f8a8c794b'])
+
+See [Executing Searches](../userguides/searches.md) for more on building search queries.
+
+### Destination Filters
+
+```{eval-rst}
+.. automodule:: py42.sdk.queries.fileevents.v2.filters.destination
+    :members:
+    :inherited-members:
+    :show-inheritance:
+```
+
+### Event Filters
+
+```{eval-rst}
+.. automodule:: py42.sdk.queries.fileevents.v2.filters.event
+    :members:
+    :inherited-members:
+    :show-inheritance:
+```
+
+### File Filters
+
+```{eval-rst}
+.. automodule:: py42.sdk.queries.fileevents.v2.filters.file
+    :members:
+    :inherited-members:
+    :show-inheritance:
+```
+
+### Process Filters
+
+```{eval-rst}
+.. automodule:: py42.sdk.queries.fileevents.v2.filters.process
+    :members:
+    :inherited-members:
+    :show-inheritance:
+```
+
+### Risk Filters
+
+```{eval-rst}
+.. automodule:: py42.sdk.queries.fileevents.v2.filters.risk
+    :members:
+    :inherited-members:
+    :show-inheritance:
+```
+
+### Source Filters
+
+```{eval-rst}
+.. automodule:: py42.sdk.queries.fileevents.v2.filters.source
+    :members:
+    :inherited-members:
+    :show-inheritance:
+```
+
+### Timestamp Filters
+
+```{eval-rst}
+.. automodule:: py42.sdk.queries.fileevents.v2.filters.timestamp
+    :members:
+    :inherited-members:
+    :show-inheritance:
+```
+
+### User Filters
+
+```{eval-rst}
+.. automodule:: py42.sdk.queries.fileevents.v2.filters.user
+    :members:
+    :inherited-members:
+    :show-inheritance:
+```

docs/methoddocs/securitydata.md

@@ -1,5 +1,10 @@
 # Security Data
 
+```{eval-rst}
+.. warning:: V1 file events, saved searches, and queries are **deprecated**.
+```
+For details on using the new file event data model, see the [V2 File Events User Guide](../userguides/v2apis.md).
+
 ```{eval-rst}
 .. autoclass:: py42.clients.securitydata.SecurityDataClient
     :members:

docs/methods.md

@@ -4,7 +4,7 @@
 ```{eval-rst}
 .. toctree::
     :hidden:
-    :maxdepth: 2
+    :maxdepth: 4
     :glob:
 
     methoddocs/*
@@ -38,7 +38,8 @@ Explore the complete public documentation for `py42` below.
 * [Devices](methoddocs/devices.md)
 * [Device Settings](methoddocs/devicesettings.md)
 * [Exceptions](methoddocs/exceptions.md)
-* [File Event Queries](methoddocs/fileeventqueries.md)
+* [(DEPRECATED) File Event Queries - V1](methoddocs/fileeventqueries.md)
+* [File Event Queries - V2](methoddocs/fileeventqueriesv2.md)
 * [Legal Hold](methoddocs/legalhold.md)
 * [Orgs](methoddocs/orgs.md)
 * [Org Settings](methoddocs/orgsettings.md)

docs/userguides/searches.md

@@ -8,24 +8,30 @@ This guide explains the syntax for building queries and executing searches.
 ```
 ## Search File Events
 
-First, import the required modules and classes and create the SDK:
-
+To query for V2 file events, import the required modules and classes and create the SDK:
 ```python
 import py42.sdk
-from py42.sdk.queries.fileevents.filters import *
-from py42.sdk.queries.fileevents.file_event_query import FileEventQuery
-
+from py42.sdk.queries.fileevents.v2 import *
 sdk = py42.sdk.from_local_account("https://console.us.code42.com", "my_username", "my_password")
 ```
 
+V1 events are **DEPRECATED**.  If you need to build V1 queries, the corresponding V1 modules can be imported in a similar manner.
+
+```python
+from py42.sdk.queries.fileevents.v1 import *
+```
+
+**For more details on updating to V2 file events, see the [V2 File Events Guide](v2apis.md)**
+
 You must create `query_filter.FilterGroup` objects to conduct searches. Filter groups have a type
-(in the form of a class), such as `EmailSender`, and an operator (in the form of a function), such as `is_in()`.
+(in the form of a class), such as `email.Sender`, and an operator (in the form of a function), such as `is_in()`.
 Some example filter groups look like this:
 
 ```python
-email_filter = EmailSender.is_in(["test.user@example.com", "test.sender@example.com"])
-exposure_filter = ExposureType.exists()
-ip_filter = PrivateIPAddress.eq("127.0.0.1")
+from py42.sdk.queries.fileevents.v2 import *
+source_email_filter = source.EmailSender.is_in(["test.user@example.com", "test.sender@example.com"])
+event_action_filter = event.Action.exists()
+destination_ip_filter = destination.PrivateIpAddress.eq("127.0.0.1")
 ```
 
 It is also possible to create `query_filter.FilterGroups` from raw JSON. For example:
@@ -36,19 +42,24 @@ json_dict = json.loads(raw_json)
 filter_group = FilterGroup.from_dict(json_dict)
 ```
 
+```{eval-rst}
+.. important::
+    The filter terms and query objects for file events have changed for V2.  Make sure you're using the appropriate modules to construct your queries.
+```
+
 There are two operators when building `file_event_query.FileEventQuery` objects: `any()` and `all()`.
 
 `any()` gets results where at least one of the filters is true and `all()` gets results where all of the filters are true.
 
 ```python
-any_query = FileEventQuery.any(email_filter, exposure_filter)
-all_query = FileEventQuery.all(exposure_filter, ip_filter)
+any_query = FileEventQuery.any(source_email_filter, event_action_filter)
+all_query = FileEventQuery.all(event_action_filter, destination_ip_filter)
 ```
 
 For convenience, the `FileEventQuery` constructor works the same way as `all()`:
 
 ```python
-all_query = FileEventQuery(exposure_filter, ip_filter)
+all_query = FileEventQuery(event_action_filter, destination_ip_filter)
 ```
 
 You can put filters in an iterable and unpack them (using the `*` operator) in a `FileEventQuery`. This is a common
@@ -58,41 +69,40 @@ use case for programs that need to conditionally build up filters:
 # Conditionally appends filters to a list for crafting a query
 
 filter_list = []
-if need_shared:
-    filter_list.append(Shared.is_true())
-elif need_actors:
-    actor_filter = Actor.is_in(["foo@example.com", "baz@example.com"])
-    filter_list.append(actor_filter)
+if need_trusted:
+    filter_list.append(risk.Trusted.is_true())
+elif need_user_emails:
+    user_email_filter = user.Email.is_in(["foo@example.com", "baz@example.com"])
+    filter_list.append(user_email_filter)
 # Notice the use of the '*' operator to unpack filter_list
 query = FileEventQuery(*filter_list)
 ```
 
 To execute the search, use `securitydata.SecurityModule.search_file_events()`:
 
 ```python
-# Prints the MD5 hashes of all the files that caused exposure events where files were moved to an external drive.
+# Prints the MD5 hashes of all the events where files were read by browser or other app.
 
-query = FileEventQuery(ExposureType.eq(ExposureType.REMOVABLE_MEDIA))
+query = FileEventQuery(event.Action.eq(event.Action.APPLICATION_READ))
 response = sdk.securitydata.search_file_events(query)
 file_events = response["fileEvents"]
 for event in file_events:
-    print(event["md5Checksum"])
+    print(event["file"]["hash"]["md5"])
 ```
 
 If the number of events exceeds 10,000 against a query, use `securitydata.SecurityModule.search_all_file_events()`:
 
 ```python
-
-query = FileEventQuery(ExposureType.eq(ExposureType.REMOVABLE_MEDIA))
+query = FileEventQuery(event.Action.eq(event.Action.APPLICATION_READ))
 response = sdk.securitydata.search_all_file_events(query)
 file_events = response["fileEvents"]
 for event in file_events:
-    print(event["md5Checksum"])
+    print(event["file"]["hash"]["md5"])
 while response["nextPgToken"] is not None:
     response = sdk.securitydata.search_all_file_events(query, page_token=response["nextPgToken"])
     file_events = response["fileEvents"]
     for event in file_events:
-        print(event["md5Checksum"])
+        print(event["file"]["hash"]["md5"])
 ```
 
 ```{eval-rst}

docs/userguides/v2apis.md

@@ -0,0 +1,31 @@
+# V2 File Events
+
+```{eval-rst}
+.. warning:: V1 file events, saved searches, and queries are **deprecated**.
+```
+
+For details on the updated File Event Model, see the V2 File Events API documentation on the [Developer Portal](https://developer.code42.com/api/#tag/File-Events).
+
+## Querying file events
+
+To query for V2 file events, import the V2 filter modules and `FileEventQuery` class with:
+```python
+from py42.sdk.queries.fileevents.v2 import *
+```
+
+Using the `FileEventQuery` and filter classes, construct a query and search for file events as detailed in the [Executing Searches Guide](searches.md).
+
+## Saved Searches
+
+All saved search methods functions have an additional optional `use_v2=False` argument.  If set to `True`, the saved search module will ingest from the V2 saved search APIs.  The `use_v2` argument defaults to `False` and the V1 saved searches are still available.
+
+For example, use the following to view all saved searches with the new V2 apis:
+
+```python
+import py42.sdk
+
+sdk = py42.sdk.from_local_account("https://console.us.code42.com", "my_username", "my_password")
+sdk.securitydata.savedsearches.get(use_v2=True)
+```
+
+Retrieving saved searches with V2 settings enabled will retrieve existing V1 saved search queries translated to the V2 model.  Existing V1 queries that cannot be properly converted to V2 will be omitted from the response.