Private Upload Mode - ThreatExchange v2 (#28249)

* ThreatExchange integration * ThreatExchange updates * Added param to instance configuration * pre-commit * updated RN * RN test * CR updates * Removed Threat_Crowd * Update Packs/ThreatExchange/ReleaseNotes/2_0_12.md Co-authored-by: ShirleyDenkberg <62508050+ShirleyDenkberg@users.noreply.github.com> * docker * format * skip tests since theres no instance * no testing instance --------- Co-authored-by: ShirleyDenkberg <62508050+ShirleyDenkberg@users.noreply.github.com> Co-authored-by: Yehuda Rosenberg <90599084+RosenbergYehuda@users.noreply.github.com> Co-authored-by: Yehuda <yrosenberg@paloaltonetworks.com>
demisto · Sep 21, 2023 · 416b6b2 · 416b6b2
1 parent 14822f8
commit 416b6b2
Show file tree

Hide file tree

Showing 6 changed files with 90 additions and 28 deletions.
diff --git a/Packs/ThreatExchange/Integrations/ThreatExchangeV2/README.md b/Packs/ThreatExchange/Integrations/ThreatExchangeV2/README.md
@@ -21,10 +21,11 @@ For Cortex XSOAR versions 6.0 and below, the App Secret should be set in the *pa
 3. Click **Add instance** to create and configure a new integration instance.
 
     | **Parameter** | **Description** | **Required** |
-    | --- | --- | --- |
+    | --- | -- | --- |
     | App ID |  | True |
     | App Secret | | True |
     | Source Reliability | Reliability of the source providing the intelligence data | True |
+    | Share Level Type | A designation of how the indicator may be shared based on the US-CERT's Traffic Light Protocol | False |
     | Use system proxy settings |  | False |
     | Trust any certificate (not secure) |  | False |
     | Malicious Threshold | If the percentage of 'Malicious' reported statuses is above this threshold the indicator will be defined as malicious, otherwise suspicious. | False |
@@ -284,13 +285,14 @@ Checks URL Reputation
 `url`
 #### Input
 
-| **Argument Name** | **Description** | **Required** |
-| --- | --- | --- |
-| url | URL to be checked. | Required | 
-| limit | The maximum number of results per page. The maximum is 1000. Default is 20. | Optional | 
-| headers | A comma-separated list of headers to display in human-readable format. For example: header1,header2,header3. | Optional | 
-| since | The start timestamp for collecting malware. Supported time formats: epoch time (e.g., 1619870400), ISO 8601 (e.g., 2021-05-01T12:00:00), and free text (e.g., 24 hours ago). | Optional | 
-| until | The end timestamp for collecting malware. Supported time formats: epoch time (e.g., 1619870400), ISO 8601 (e.g., 2021-05-01T12:00:00), and free text (e.g., 24 hours ago). | Optional | 
+| **Argument Name** | **Description**                                                                                                                                                              | **Required** |
+|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| --- |
+| url               | URL to be checked.                                                                                                                                                           | Required | 
+| limit             | The maximum number of results per page. The maximum is 1000. Default is 20.                                                                                                  | Optional | 
+| headers           | A comma-separated list of headers to display in human-readable format. For example: header1,header2,header3.                                                                 | Optional | 
+| since             | The start timestamp for collecting malware. Supported time formats: epoch time (e.g., 1619870400), ISO 8601 (e.g., 2021-05-01T12:00:00), and free text (e.g., 24 hours ago). | Optional | 
+| until             | The end timestamp for collecting malware. Supported time formats: epoch time (e.g., 1619870400), ISO 8601 (e.g., 2021-05-01T12:00:00), and free text (e.g., 24 hours ago).   | Optional | 
+| share_level       | A designation of how the indicator may be shared, based on the US-CERT's Traffic Light Protocol. Default is RED.                                                             | Optional | 
 
 
 #### Context Output
@@ -421,7 +423,7 @@ Checks domain reputation.
 | headers | A comma-separated list of headers to display in human-readable format. For example: header1,header2,header3. | Optional | 
 | since | The start timestamp for collecting malware. Supported time formats: epoch time (e.g., 1619870400), ISO 8601 (e.g., 2021-05-01T12:00:00), and free text (e.g., 24 hours ago). | Optional | 
 | until | The end timestamp for collecting malware. Supported time formats: epoch time (e.g., 1619870400), ISO 8601 (e.g., 2021-05-01T12:00:00), and free text (e.g., 24 hours ago). | Optional | 
-
+| share_level       | A designation of how the indicator may be shared, based on the US-CERT's Traffic Light Protocol. Default is RED.                                                             | Optional | 
 
 #### Context Output
 

diff --git a/Packs/ThreatExchange/Integrations/ThreatExchangeV2/ThreatExchangeV2.py b/Packs/ThreatExchange/Integrations/ThreatExchangeV2/ThreatExchangeV2.py
@@ -5,7 +5,6 @@
 """
 
 import collections
-from typing import Tuple
 import urllib3
 from CommonServerUserPython import *  # noqa
 from CommonServerPython import *  # noqa # pylint: disable=unused-wildcard-import
@@ -102,7 +101,7 @@ def file(self, file: str, since: Optional[int], until: Optional[int], limit: Opt
         )
         return response
 
-    def domain(self, domain: str, since: Optional[int], until: Optional[int],
+    def domain(self, domain: str, since: Optional[int], until: Optional[int], share_level: str,
                limit: Optional[int] = DEFAULT_LIMIT) -> Dict:
         """
         See Also:
@@ -111,6 +110,7 @@ def domain(self, domain: str, since: Optional[int], until: Optional[int],
             domain: Domain
             since: Returns malware collected after a timestamp
             until: Returns malware collected before a timestamp
+            share_level: A designation of how the indicator may be shared, based on the US-CERT's Traffic Light Protocol.
             limit: Defines the maximum size of a page of results. The maximum is 1,000
 
         Returns: The API call response
@@ -126,19 +126,22 @@ def domain(self, domain: str, since: Optional[int], until: Optional[int],
                 'strict_text': True,
                 'since': since,
                 'until': until,
-                'limit': limit
+                'limit': limit,
+                'share_level': share_level
             })
         )
         return response
 
-    def url(self, url: str, since: Optional[int], until: Optional[int], limit: Optional[int] = DEFAULT_LIMIT) -> Dict:
+    def url(self, url: str, since: Optional[int], until: Optional[int], share_level: str,
+            limit: Optional[int] = DEFAULT_LIMIT) -> Dict:
         """
         See Also:
             https://developers.facebook.com/docs/threat-exchange/reference/apis/threat-descriptors
         Args:
             url: URL
             since: Returns malware collected after a timestamp
             until: Returns malware collected before a timestamp
+            share_level: A designation of how the indicator may be shared, based on the US-CERT's Traffic Light Protocol.
             limit: Defines the maximum size of a page of results. The maximum is 1,000
 
         Returns: The API call response
@@ -154,7 +157,8 @@ def url(self, url: str, since: Optional[int], until: Optional[int], limit: Optio
                 'strict_text': True,
                 'since': since,
                 'until': until,
-                'limit': limit
+                'limit': limit,
+                'share_level': share_level
             })
         )
         return response
@@ -348,7 +352,7 @@ def calculate_dbot_score(reputation_data: List, params: Dict[str, Any]) -> int:
     return score
 
 
-def calculate_engines(reputation_data: List) -> Tuple[int, int]:
+def calculate_engines(reputation_data: List) -> tuple[int, int]:
     """
     Calculates the number of engines that scanned the indicator, and how many of them are positive
      - i.e returned malicious status.
@@ -431,7 +435,7 @@ def convert_string_to_epoch_time(date: Optional[str], arg_name: Optional[str] =
             return int(epoch_time)
         else:  # date was given in a wrong format
             if arg_name:
-                raise ValueError('Invalid date: "{}"="{}"'.format(arg_name, date))
+                raise ValueError(f'Invalid date: "{arg_name}"="{date}"')
 
     return None
 
@@ -464,7 +468,7 @@ def ip_command(client: Client, args: Dict[str, Any], params: Dict[str, Any]) ->
     limit = arg_to_number(args.get('limit'), arg_name='limit')
     headers = argToList(args.get('headers'))
     reliability = params.get('feedReliability')
-    results: List[CommandResults] = list()
+    results: List[CommandResults] = []
 
     for ip in ips:
         if not is_ip_valid(ip, accept_v6_ips=True):  # check IP's validity
@@ -536,7 +540,7 @@ def file_command(client: Client, args: Dict[str, Any], params: Dict[str, Any]) -
     limit = arg_to_number(args.get('limit'), arg_name='limit')
     headers = argToList(args.get('headers'))
     reliability = params.get('feedReliability')
-    results: List[CommandResults] = list()
+    results: List[CommandResults] = []
 
     for file in files:
         if get_hash_type(file) not in ('sha256', 'sha1', 'md5'):  # check file's validity
@@ -612,16 +616,26 @@ def domain_command(client: Client, args: Dict[str, Any], params: Dict[str, Any])
     limit = arg_to_number(args.get('limit'), arg_name='limit')
     headers = argToList(args.get('headers'))
     reliability = params.get('feedReliability')
-    results: List[CommandResults] = list()
+    share_level = args.get('share_level', params.get('share_level', 'RED'))
+    demisto.debug(f'Setting share level to {share_level}')
+    results: List[CommandResults] = []
 
     for domain in domains:
         try:
-            raw_response = client.domain(domain, since, until, limit)
+            raw_response = client.domain(domain, since, until, share_level, limit)
         except Exception as exception:
             # If anything happens, handle like there are no results
             err_msg = f'Could not process domain: "{domain}"\n {str(exception)}'
             demisto.debug(err_msg)
             raw_response = {}
+            readable_output = f'Processing domain "{domain}" resulted in an exception. See logs for the exact error.'
+            result = CommandResults(
+                outputs={},
+                readable_output=readable_output,
+                raw_response=raw_response
+            )
+            results.append(result)
+            continue
         if data := raw_response.get('data'):
             score = calculate_dbot_score(reputation_data=data, params=params)
             num_of_engines, num_of_positive_engines = calculate_engines(reputation_data=data)
@@ -681,15 +695,25 @@ def url_command(client: Client, args: Dict[str, Any], params: Dict[str, Any]) ->
     limit = arg_to_number(args.get('limit'), arg_name='limit')
     headers = argToList(args.get('headers'))
     reliability = params.get('feedReliability')
-    results: List[CommandResults] = list()
+    share_level = args.get('share_level', params.get('share_level', 'RED'))
+    demisto.debug(f'Setting share level to {share_level}')
+    results: List[CommandResults] = []
     for url in urls:
         try:
-            raw_response = client.url(url, since, until, limit)
+            raw_response = client.url(url, since, until, share_level, limit)
         except Exception as exception:
             # If anything happens, handle like there are no results
             err_msg = f'Could not process URL: "{url}"\n {str(exception)}'
             demisto.debug(err_msg)
             raw_response = {}
+            readable_output = f'Processing URL "{url}" resulted in an exception. See logs for the exact error.'
+            result = CommandResults(
+                outputs={},
+                readable_output=readable_output,
+                raw_response=raw_response
+            )
+            results.append(result)
+            continue
         if data := raw_response.get('data'):
             score = calculate_dbot_score(reputation_data=data, params=params)
             num_of_engines, num_of_positive_engines = calculate_engines(reputation_data=data)

diff --git a/Packs/ThreatExchange/Integrations/ThreatExchangeV2/ThreatExchangeV2.yml b/Packs/ThreatExchange/Integrations/ThreatExchangeV2/ThreatExchangeV2.yml
@@ -38,6 +38,17 @@ configuration:
   - F - Reliability cannot be judged
   required: true
   type: 15
+- defaultvalue: 'RED'
+  display: Share Level Type
+  name: share_level
+  type: 15
+  required: false
+  additionalinfo: A designation of how the indicator may be shared based on the US-CERT's Traffic Light Protocol.
+  options:
+  - RED
+  - AMBER
+  - GREEN
+  - WHITE
 - defaultvalue: 'false'
   display: Use system proxy settings
   name: proxy
@@ -225,7 +236,7 @@ script:
       description: The ID of the ThreatExchange member that submitted the descriptor. Non-editable.
       type: String
     - contextPath: ThreatExchange.IP.owner.name
-      description: The name of the ThreatExchange member that submitted the descriptor. Non-editable
+      description: The name of the ThreatExchange member that submitted the descriptor. Non-editable.
       type: String
     - contextPath: ThreatExchange.IP.raw_indicator
       description: A raw, unsanitized string of the indicator being described.
@@ -263,6 +274,14 @@ script:
       name: since
     - description: 'The end timestamp for collecting malware. Supported time formats: epoch time (e.g., 1619870400), ISO 8601 (e.g., 2021-05-01T12:00:00), and free text (e.g., 24 hours ago).'
       name: until
+    - auto: PREDEFINED
+      description: A designation of how the indicator may be shared, based on the US-CERT's Traffic Light Protocol.
+      name: share_level
+      predefined:
+      - RED
+      - AMBER
+      - GREEN
+      - WHITE
     description: Checks the URL reputation.
     name: url
     outputs:
@@ -321,7 +340,7 @@ script:
       description: The ID of the ThreatExchange member that submitted the descriptor. Non-editable.
       type: String
     - contextPath: ThreatExchange.URL.owner.name
-      description: The name of the ThreatExchange member that submitted the descriptor. Non-editable
+      description: The name of the ThreatExchange member that submitted the descriptor. Non-editable.
       type: String
     - contextPath: ThreatExchange.URL.raw_indicator
       description: A raw, unsanitized string of the indicator being described.
@@ -359,6 +378,14 @@ script:
       name: since
     - description: 'The end timestamp for collecting malware. Supported time formats: epoch time (e.g., 1619870400), ISO 8601 (e.g., 2021-05-01T12:00:00), and free text (e.g., 24 hours ago).'
       name: until
+    - auto: PREDEFINED
+      description: A designation of how the indicator may be shared, based on the US-CERT's Traffic Light Protocol.
+      name: share_level
+      predefined:
+      - RED
+      - AMBER
+      - GREEN
+      - WHITE
     description: Checks a domain reputation.
     name: domain
     outputs:
@@ -414,7 +441,7 @@ script:
       description: The ID of the ThreatExchange member that submitted the descriptor. Non-editable.
       type: String
     - contextPath: ThreatExchange.Domain.owner.name
-      description: The name of the ThreatExchange member that submitted the descriptor. Non-editable
+      description: The name of the ThreatExchange member that submitted the descriptor. Non-editable.
       type: String
     - contextPath: ThreatExchange.Domain.raw_indicator
       description: A raw, unsanitized string of the indicator being described.
@@ -659,7 +686,7 @@ script:
     - contextPath: ThreatExchange.Object.id
       description: ID of a ThreatExchange object.
       type: String
-  dockerimage: demisto/python3:3.10.12.63474
+  dockerimage: demisto/python3:3.10.13.74666
   runonce: false
   script: '-'
   subtype: python3

diff --git a/Packs/ThreatExchange/ReleaseNotes/2_0_12.md b/Packs/ThreatExchange/ReleaseNotes/2_0_12.md
@@ -0,0 +1,8 @@
+
+#### Integrations
+
+##### ThreatExchange v2
+
+- Added support for the *share_level* argument in the ***!url*** and ***!domain*** commands.
+- Added support for the *Share Level Type* parameter in the instance configuration.
+- Updated the Docker image to: *demisto/python3:3.10.13.74666*.
diff --git a/Packs/ThreatExchange/pack_metadata.json b/Packs/ThreatExchange/pack_metadata.json
@@ -2,7 +2,7 @@
     "name": "ThreatExchange",
     "description": "Receive threat intelligence about applications, IP addresses, URLs and hashes, a service by Facebook",
     "support": "xsoar",
-    "currentVersion": "2.0.11",
+    "currentVersion": "2.0.12",
     "author": "Cortex XSOAR",
     "url": "https://www.paloaltonetworks.com/cortex",
     "email": "",

diff --git a/Tests/conf.json b/Tests/conf.json
@@ -5863,7 +5863,8 @@
         "ThreatGridv2": "No instance - developed by Qmasters",
         "SentinelOne V2": "No instance - developed by partner",
         "CheckPhish": "Issue CRTX-86562",
-        "SecurityAndComplianceV2": "Can only be authenticated via MFA and requires user interaction to configure."
+        "SecurityAndComplianceV2": "Can only be authenticated via MFA and requires user interaction to configure.",
+        "ThreatExchange v2": "No instance"
     },
     "nightly_packs": [
         "CommonScripts",