Netskope event collector rewrite (demisto#28941)

* Temporarily added the following packs to the update_core_packs_list:
Core, DemistoRESTAPI, FiltersAndTransformers, Palo_Alto_Networks_WildFire, rasterize

* Added all packs to update core list

* Added al core packs to update_core_packs_list

* Added new API endpoint

* Added testing copies

* changes from testing

* changes from testing

* changes from testing

* Changed default first fecth

* Added slipping for no wait time

* First code change

* Fixed description and log

* UT fixes + mypy

* UT fixes + mypy

* Bumped Docker image and added rn

* Formatting and typos

* Fixed honor_rate_limit and added ut

* Flake 8 fix

* Added more UT

* revert core list change

* Enhanced docs

* Small UT fixes

* Removed is_command variable

* Added docs
Fixed UT

* changes rn version

* - Removed first_fetch param
- Added types to perform_data_export parameters
- Removed unused code

* lint fixes

* lint fixes

Packs/Netskope/Integrations/NetskopeEventCollector/NetskopeEventCollector.yml

@@ -17,28 +17,12 @@ configuration:
   required: true
   type: 9
   section: Connect
-- additionalinfo: The API version to use (v1 or v2).
-  defaultvalue: v2
-  display: API Version
-  name: api_version
-  type: 15
-  section: Connect
-  options:
-  - v1
-  - v2
-  required: false
-- additionalinfo: 'The first fetch is limited to the last 50,000 events (for each event type).'
-  defaultvalue: 3 days
-  display: First fetch timestamp (<number> <time unit>, e.g., 12 hours, 7 days)
-  name: first_fetch
-  section: Collect
-  type: 0
-  required: false
-- additionalinfo: The maximum amount of events to retrieve for each event type (up to 30000 events). For more information about event types see the help section.
-  defaultvalue: '1000'
+  hidden: true
+- additionalinfo: 'The maximum amount of events to retrieve PER EACH EVENT TYPE type. For more information about event types see the help section.'
+  defaultvalue: '50000'
   display: Max events per fetch
-  section: Collect
   name: max_fetch
+  section: Collect
   type: 0
   required: false
 - display: Trust any certificate (not secure)
@@ -67,9 +51,10 @@ script:
       required: true
     - description: The maximum number of alerts to return (maximum value - 10000).
       name: limit
+      defaultValue: 10000
     description: Returns events extracted from SaaS traffic and or logs.
     name: netskope-get-events
-  dockerimage: demisto/python3:3.10.12.66339
+  dockerimage: demisto/python3:3.10.12.68714
   runonce: false
   script: '-'
   subtype: python3

Packs/Netskope/Integrations/NetskopeEventCollector/NetskopeEventCollector_description.md

@@ -1,15 +1,18 @@
-### Netskope Event Collector
+## Netskope Event Collector
 
+### General Info
 - Collects events extracted from SaaS traffic and logs.
 - The collector collects 5 types of events: 
    - Audit
    - Application
    - Network
    - Alert
    - Page
-- To generate the API token, in your Netskope UI go to **Settings** > **Tools** > **Rest API v1 or v2**
-- Please make sure to choose the appropriate **API token** according to the chosen **API Version**.
-- Visit the [Netskope API Overview](https://docs.netskope.com/en/rest-api-v2-overview-312207.html) for more information.
-
-
+- Note: The collector can handle 10K events per minute on average per each event type.  
 
+### API Key
+- To generate the API token, in your Netskope UI go to **Settings** > **Tools** > **Rest API v2**
+- The KEY requires the following permissions:
+  - /api/v2/events/dataexport/events/*
+  - /api/v2/events/dataexport/alerts/*
+- Visit the [Netskope API Overview](https://docs.netskope.com/en/rest-api-v2-overview-312207.html) for more information.

Packs/Netskope/Integrations/NetskopeEventCollector/NetskopeEventCollector_test.py

@@ -1,46 +1,28 @@
 import io
 import json
+import re
+import time
 
-from NetskopeEventCollector import Client
+import dateparser
+import pytest
+
+from NetskopeEventCollector import Client, ALL_SUPPORTED_EVENT_TYPES, RATE_LIMIT_REMAINING, RATE_LIMIT_RESET
 
 
 def util_load_json(path):
     with io.open(path, mode='r', encoding='utf-8') as f:
         return json.loads(f.read())
 
 
-MOCK_ENTRY = util_load_json('test_data/mock_events.json')
-EVENTS_RAW_V2 = util_load_json('test_data/events_raw_v2.json')
-EVENTS_RAW_V2_MULTI = util_load_json('test_data/events_raw_v2_2_results.json')
-EVENTS_PAGE_RAW_V1 = util_load_json('test_data/page_raw_v1.json')
+MOCK_ENTRY = util_load_json('test_data/mock_events_entry.json')
+EVENTS_RAW = util_load_json('test_data/events_raw.json')
+EVENTS_PAGE_RAW = util_load_json('test_data/multiple_events_raw.json')
 BASE_URL = 'https://netskope.example.com'
-FIRST_LAST_RUN = {'alert': 1680182467, 'alert-ids': [], 'application': 1680182467, 'application-ids': [],
-                  'audit': 1680182467, 'audit-ids': [], 'network': 1680182467, 'network-ids': [],
-                  'page': 1680182467, 'page-ids': []}
-
-
-def test_dedup_by_id():
-    """
-    Given:
-        - Results from the API
-    When:
-        - Running the dedup_by_id command
-    Then:
-        - Make sure only the limited number of events return.
-        - Make sure that first comes the event that with the earlier timestamp
-        - Make sure that the last_run timestamp has been updated
-        - Make sure that the correct last_run_ids returned.
-    """
-    from NetskopeEventCollector import dedup_by_id
-    results = EVENTS_PAGE_RAW_V1.get('data')
-    events, new_last_run = dedup_by_id(last_run=FIRST_LAST_RUN, event_type='page', limit=4, results=results)
-    assert events[0].get('timestamp') == 1684751415
-    assert len(events) == 4
-    assert new_last_run == {'page': 1684751416, 'page-ids': ['3757761212778242bfda29cd', '9e99b72b957416a43222fa7a',
-                                                             '66544bf5fda515f229592644', '98938eb19b4f9bea24ef9a8c']}
+FIRST_LAST_RUN = {'alert': {'operation': 1680182467}, 'application': {'operation': 1680182467},
+                  'audit': {'operation': 1680182467}, 'network': {'operation': 1680182467}, 'page': {'operation': 1680182467}}
 
 
-def test_test_module_v2(mocker):
+def test_test_module(mocker):
     """
     Given:
         - raw_response of an event (as it returns from the api)
@@ -50,28 +32,30 @@ def test_test_module_v2(mocker):
         - Verify that 'ok' is returned.
     """
     from NetskopeEventCollector import test_module
-    client = Client(BASE_URL, 'dummy_token', 'v2', False, False)
-    mocker.patch.object(client, 'get_events_request_v2', return_value=EVENTS_RAW_V2)
-    results = test_module(client, api_version='v2', last_run=FIRST_LAST_RUN, max_fetch=1)
+    client = Client(BASE_URL, 'dummy_token', False, False)
+    mocker.patch.object(client, 'perform_data_export', return_value=EVENTS_RAW)
+    results = test_module(client, last_run=FIRST_LAST_RUN, max_fetch=1)
     assert results == 'ok'
 
 
-def test_populate_parsing_rule_fields():
+def test_populate_prepare_events():
     """
     Given:
         - Event from the API of type audit
     When:
         - Running the command
     Then:
-        - Make sure the field _time is populated properly.
+        - Make sure the _time, evnet_id, and source_log_event fields are populated properly.
     """
-    from NetskopeEventCollector import populate_parsing_rule_fields
-    event = EVENTS_RAW_V2.get('result')[0]
-    populate_parsing_rule_fields(event, event_type='audit')
+    from NetskopeEventCollector import prepare_events
+    event = EVENTS_RAW.get('result')[0]
+    prepare_events([event], event_type='audit')
     assert event.get('_time') == '2022-01-18T19:58:07.000Z'
+    assert event.get('source_log_event') == 'audit'
+    assert event.get('event_id') == 'f0e9b2cadd17402b59b3938b'
 
 
-def test_get_all_events(mocker):
+def test_get_all_events(requests_mock):
     """
     Given:
         - netskope-get-events call
@@ -82,18 +66,20 @@ def test_get_all_events(mocker):
         - Make sure that the _time and event_id fields are populated as expected
         - Make sure the new_last_run is set.
     """
+
+    def json_callback(request, _):
+        endpoint = request.path.split('/')[-1]
+        return EVENTS_PAGE_RAW[endpoint]
+
     from NetskopeEventCollector import get_all_events
-    client = Client(BASE_URL, 'netskope_token', 'v1', validate_certificate=False, proxy=False)
-    mocker.patch.object(client, 'get_alerts_request_v1', return_value=EVENTS_PAGE_RAW_V1)
-    mocker.patch.object(client, 'get_events_request_v1', return_value=EVENTS_PAGE_RAW_V1)
-    events, new_last_run = get_all_events(client, FIRST_LAST_RUN, api_version='v1', limit=6, is_command=False)
+    client = Client(BASE_URL, 'netskope_token', validate_certificate=False, proxy=False)
+    url_matcher = re.compile('https://netskope[.]example[.]com/events/dataexport/events')
+    requests_mock.get(url_matcher, json=json_callback)
+    events, new_last_run = get_all_events(client, FIRST_LAST_RUN)
     assert len(events) == 25
-    assert events[0].get('event_id') == '3757761212778242bfda29cd'
-    assert events[0].get('_time') == '2023-05-22T10:30:15.000Z'
-    assert new_last_run['page'] == 1684751416
-    assert new_last_run['page-ids'] == ['3757761212778242bfda29cd', '9e99b72b957416a43222fa7a',
-                                        '66544bf5fda515f229592644', '98938eb19b4f9bea24ef9a8c',
-                                        'fe6d7f3a9a1d4e1abce21713']
+    assert events[0].get('event_id') == '1'
+    assert events[0].get('_time') == '2023-05-22T10:30:16.000Z'
+    assert all([new_last_run[event_type]['operation'] == 'next' for event_type in ALL_SUPPORTED_EVENT_TYPES])
 
 
 def test_get_events_command(mocker):
@@ -108,11 +94,72 @@ def test_get_events_command(mocker):
         - Make sure the outputs are set correctly.
     """
     from NetskopeEventCollector import get_events_command
-    client = Client(BASE_URL, 'dummy_token', 'v2', False, False)
+    client = Client(BASE_URL, 'dummy_token', False, False)
     mocker.patch('NetskopeEventCollector.get_all_events', return_value=[MOCK_ENTRY, {}])
-    results, events = get_events_command(client, args={}, last_run=FIRST_LAST_RUN, api_version='v2',
-                                         is_command=True)
+    mocker.patch.object(time, "sleep")
+    results, events = get_events_command(client, args={}, last_run=FIRST_LAST_RUN)
     assert 'Events List' in results.readable_output
     assert len(events) == 9
     assert results.outputs_prefix == 'Netskope.Event'
     assert results.outputs == MOCK_ENTRY
+
+
+@pytest.mark.parametrize('headers, endpoint, expected_sleep', [
+    ({RATE_LIMIT_REMAINING: 1}, 'test_endpoint', None),
+    ({}, 'test_endpoint', None),
+    ({RATE_LIMIT_REMAINING: 0, RATE_LIMIT_RESET: 2}, 'test_endpoint', 2),
+    ({RATE_LIMIT_REMAINING: 0}, 'test_endpoint', 1),
+])
+def test_honor_rate_limiting(mocker, headers, endpoint, expected_sleep):
+    """
+    Given:
+        Case a: Netskope response headers with RATE_LIMIT_REMAINING = 1
+        Case b: Netskope with response headers
+        Case c: Netskope with response headers RATE_LIMIT_REMAINING = 1 and RATE_LIMIT_RESET = 2
+        Case c: Netskope with response headers RATE_LIMIT_REMAINING = 0
+
+    When:
+        Checking if sleeping is required
+
+    Then:
+        Case a: validate that there is no sleeping
+        Case b: validate that there is no sleeping
+        Case c: validate that we sleep for 2 secs (which is the reset time)
+        Case c: validate that we sleep for 1 sec (which is the default in case not rest time is given)
+    """
+    time_mock = mocker.patch.object(time, "sleep")
+    from NetskopeEventCollector import honor_rate_limiting
+    honor_rate_limiting(headers=headers, endpoint=endpoint)
+    if expected_sleep:
+        time_mock.assert_called_once_with(expected_sleep)
+    else:
+        time_mock.assert_not_called()
+
+
+@pytest.mark.parametrize('last_run_dict, expected_operation_value', [
+    ({}, 1672567200),
+    ({'application': {'operation': 'next'},
+      'alert': {'operation': 'next'},
+      'page': {'operation': 'next'},
+      'audit': {'operation': 'next'},
+      'network': {'operation': 'next'}}, 'next'),
+])
+def test_setup_last_run(mocker, last_run_dict, expected_operation_value):
+    """
+    Given:
+        Case a: previous empty last run
+        Case a: previous last run with operation= 'next' for all event types
+
+    When:
+        Setting the last run values for the current run
+
+    Then:
+        Case a: make sure all event types in last run are saved with operation= 1672567200
+        Case b: make sure all event types in last run are saved with operation= 'next'
+
+    """
+    from NetskopeEventCollector import setup_last_run
+    first_fetch = dateparser.parse('2023-01-01T10:00:00Z')
+    mocker.patch.object(dateparser, "parse", return_value=first_fetch)
+    last_run = setup_last_run(last_run_dict)
+    assert all([val.get('operation') == expected_operation_value for key, val in last_run.items()])

Packs/Netskope/Integrations/NetskopeEventCollector/README.md

@@ -8,29 +8,31 @@
     | --- | --- | --- |
     | Server URL |  | True |
     | API token |  | True |
-    | API Version | The API version to use \(v1 or v2\). | False |
     | Trust any certificate (not secure) |  | False |
     | Use system proxy settings |  | False |
-    | First fetch timestamp (<number> <time unit>, e.g., 12 hours, 7 days) |  | False |
-    | Max events per fetch | The maximum amount of events to retrieve \(up to 30000 events\). | False |
+    | Max events per fetch | The maximum amount of events to retrieve per each event type. For more information about event types see the help section. | False |
 
 4. Click **Test** to validate the URLs, token, and connection.
 
 ## Fetch Events Limitation
-The first fetch is limited to the last 50,000 events (for each event type) due to API limitation.
 
+The collector can handle 10K events per minute on average per each event type  
 
 ## Commands
+
 You can execute these commands from the Cortex XSIAM CLI, as part of an automation, or in a playbook.
 After you successfully execute a command, a DBot message appears in the War Room with the command details.
+
 ### netskope-get-events
+
 ***
 Returns events extracted from SaaS traffic and or logs.
 
 
 #### Base Command
 
 `netskope-get-events`
+
 #### Input
 
 | **Argument Name** | **Description** | **Required** |
@@ -42,9 +44,13 @@ Returns events extracted from SaaS traffic and or logs.
 #### Context Output
 
 There is no context output for this command.
+
 #### Command example
+
 ```!netskope-get-events limit=1```
+
 #### Context Example
+
 ```json
 {
     "Netskope": {
@@ -265,6 +271,7 @@ There is no context output for this command.
 #### Human Readable Output
 
 >### Events List:
+
 >|Id|Timestamp|Type|Access Method|App|Traffic Type|
 >|---|---|---|---|---|---|
 >| 23a372c433381a6a11798123 | 2022-07-17T23:48:52.000Z | nspolicy | API Connector | Microsoft Office 365 Sharepoint Online | CloudApp |