Ciac 7986 (#29137)

* added functionality and test

* rn

* lint fix

* Apply suggestions from code review

Co-authored-by: Dan Tavori <38749041+dantavori@users.noreply.github.com>

* unit test fixes

* bug fix

* validations

* Bump pack from version CommonScripts to 1.12.18.

* pic

* readme

* CR fix

* Bump pack from version CommonScripts to 1.12.19.

* Bump pack from version CommonScripts to 1.12.20.

* changed datadir

* docker

* rn

* changed dataridr

* fix

* fix

* pre commit

* fixes

---------

Co-authored-by: Dan Tavori <38749041+dantavori@users.noreply.github.com>
Co-authored-by: Content Bot <bot@demisto.com>

Packs/CommonScripts/ReleaseNotes/1_12_20.md

@@ -0,0 +1,7 @@
+
+#### Scripts
+
+##### SetGridField
+- Updated the Docker image to: *demisto/pandas:1.0.0.72335*.
+
+- Added the *keys_from_nested* argument to support dictionary and array fields under the provided context path.

Packs/CommonScripts/Scripts/SetGridField/README.md

@@ -11,14 +11,16 @@ Update Grid Table from items or key value pairs.
 ## Inputs
 ---
 
-| **Argument Name** | **Description** |
-| --- | --- |
-| context_path | Context path to list of items with similar properties or key value pairs. |
-| grid_id | Grid ID to modify. This argument can be either: 1) Grid name as it appears in the layout. 2) Grid "Machine name", as can be found in the grid incident field editor under Settings->Advanced->Fields (Incidents). |
-| overwrite | True if to overwrite Grid Data, False otherwise. |
-| columns | Comma-separated list of grid columns to populate (as appear in the original Grid), for example: (col1,col2,..,coln). |
-| keys | Keys to retrieve from items or "\*" for max keys \(limited when item list to columns amount\) \- Key will not be columns correlated. If you want to leave an empty column, please provide a place holder name that should not be in the context data such as "PLACE_HOLDER" |
-| sort_by | Columns names by which to sort the rows. |
+| **Argument Name** | **Description**                                                                                                                                                                                                                                                                                       |
+| --- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| context_path | Context path to list of items with similar properties or key value pairs.                                                                                                                                                                                                                             |
+| grid_id | Grid ID to modify. This argument can be either: 1) Grid name as it appears in the layout. 2) Grid "Machine name", as can be found in the grid incident field editor under Settings->Advanced->Fields (Incidents).                                                                                     |
+| overwrite | True if to overwrite Grid Data, False otherwise.                                                                                                                                                                                                                                                      |
+| columns | Comma-separated list of grid columns to populate (as appear in the original Grid), for example: (col1,col2,..,coln).                                                                                                                                                                                  |
+| keys | Keys to retrieve from items or "\*" for max keys \(limited when item list to columns amount\) \- Key will not be columns correlated. If you want to leave an empty column, please provide a place holder name that should not be in the context data such as "PLACE_HOLDER"                 |
+| sort_by | Columns names by which to sort the rows.                                                                                                                                                                                                                                                              |
+| unpack_nested_elements | Set to 'true' to unpack nested elements.                                                                                                                                                                                                                                                              |
+| keys_from_nested | Keys to retrieve from nested dictionaries. Can be used only when the unpack_nested_elements argument is set to false. Keys will not be columns correlated. Default is all keys. **Note**: when the number of values exceeds the number of columns, it truncates the last values that are outside the range for table. |
 
 ## Command Example
 Assume the following:
@@ -78,6 +80,45 @@ keys="name, value"
 Grid after update: \
 ![Grid](https://github.com/demisto/content/raw/4510eafaf6cfeb48a42d9032dd0e71200b288ad5/Packs/Legacy/Scripts/SetGridField/doc_files/grid_list_update.png) 
 
+Entry Context:
+```json
+{
+    "PaloAltoNetworksXDR": {
+        "RiskyUser": [
+            {
+                "email": null,
+                "id": "1",
+                "norm_risk_score": 1000,
+                "reasons": [
+                    {
+                        "date created": "2023-08-20",
+                        "description": "test",
+                        "points": 90,
+                        "severity": "test",
+                        "status": "test"
+                    },
+                    {
+                        "date created": "2023-08-20",
+                        "description": "test",
+                        "points": 90,
+                        "severity": "test",
+                        "status": "test"
+                    }
+                ],
+                "risk_level": "HIGH",
+                "score": 244,
+                "type": "user"
+            }
+        ]
+    }
+}
+```
+
+```script
+!SetGridField_CopyForInvestigation columns=`User id,Risk level,Score,Reasons` grid_id=xdrriskyusers context_path=`PaloAltoNetworksXDR.RiskyUser` keys=`id,risk_level,score,reasons` keys_from_nested=description,points,severity
+```
+![nested_dict_grid](https://github.com/demisto/content/raw/a1fbdccd9bf97d8d27f6b5bf1d08802a7bdcf475/Packs/CommonScripts/Scripts/SetGridField/doc_files/nested_dict_grid.png)
+
 ## Troubleshooting
 
 The first time you run `SetGridField` on a newly created grid field, you may see an error similar to the following:

Packs/CommonScripts/Scripts/SetGridField/SetGridField.py

@@ -77,6 +77,37 @@ def filter_dict(dict_obj: Dict[Any, Any], keys: List[str], max_keys: Optional[in
     return new_dict
 
 
+def entry_dicts_to_string(dict_obj: Dict[Any, Any], keys_to_choose: list[str]):
+    """
+
+    Args:
+        dict_obj: context entry to iterate on
+        keys_to_choose: specific keys to filter from the nested dictionaries
+
+    Returns:
+        string contains all selected values from the nested dictionary of the context entry.
+    """
+    new_dict = {key: '' for key in dict_obj.keys()}
+    for (key, value) in dict_obj.items():
+        if isinstance(value, dict):
+            value = filter_dict(value, keys_to_choose)
+            new_dict[key] = "\n".join(f'{dict_key}: {dict_value}' for dict_key, dict_value in value.items())
+        elif isinstance(value, list):
+            array_to_join = []
+            for list_value in value:
+                if isinstance(list_value, dict):
+                    list_value = filter_dict(list_value, keys_to_choose)
+                    array_to_join.append("\n".join(f'{dict_key}: {dict_value}' for dict_key, dict_value in list_value.items()))
+                else:
+                    array_to_join.append(f"\n{list_value}")
+            final_value = "\n\n".join(array_to_join)
+            new_dict[key] = final_value
+        else:
+            new_dict[key] = value
+
+    return new_dict
+
+
 def unpack_all_data_from_dict(entry_context: Dict[Any, Any], keys: List[str], columns: List[str]):
     """ Unpacks lists and dicts to flatten the object for the grid.
 
@@ -143,10 +174,10 @@ def get_current_table(grid_id: str) -> pd.DataFrame:
 
 
 @logger
-def validate_entry_context(context_path: str, entry_context: Any, keys: List[str], unpack_nested_elements: bool):
+def validate_entry_context(context_path: str, entry_context: Any, unpack_nested_elements: bool):
     """ Validate entry context structure is valid, should be:
         - For unpack_nested_elements==False:
-            1. List[Dict[str, str/bool/int/float]]
+            1. List[Dict[str, Any]]
             2. List[str/bool/int/float]
             3. Dict[str, str/bool/int/float] - for developer it will be in first index of a list.
         - For unpack_nested_elements==True:
@@ -155,7 +186,6 @@ def validate_entry_context(context_path: str, entry_context: Any, keys: List[str
     Args:
         context_path: Path of entry context
         entry_context: Entry context to validate
-        keys: Keys to collect data from
         unpack_nested_elements: True for unpacking nested elements, False otherwise.
 
     Raises:
@@ -191,16 +221,6 @@ def validate_entry_context(context_path: str, entry_context: Any, keys: List[str
                 break
 
         has_seen_dict = True
-        for key, value in item.items():
-            if key not in keys:
-                continue
-            if value is not None and not isinstance(value, (str, int, float, bool)):
-                demisto.error(f'expected list of dictionaries with simple values, found a complex item with '
-                              f'key {type(key)} type:\t {key}\nproblematic item: {item}')
-                raise ValueError(
-                    f'The context path {context_path} in index {index} - key {key} contains a dict item with a '
-                    f'complex value.\n'
-                    f'The value must be of type: string, number, boolean.')
 
     if not has_seen_dict:
         data_type = 'list'
@@ -214,7 +234,8 @@ def validate_entry_context(context_path: str, entry_context: Any, keys: List[str
     return data_type
 
 
-def build_grid(context_path: str, keys: List[str], columns: List[str], unpack_nested_elements: bool) -> pd.DataFrame:
+def build_grid(context_path: str, keys: List[str], columns: List[str], unpack_nested_elements: bool,
+               keys_from_nested: List[str]) -> pd.DataFrame:
     """ Build new DateFrame from current context retrieved by DT.
         There are 3 cases:
             1. DT returns dict - In this case we will insert it in the table as key, value in each row.
@@ -226,18 +247,20 @@ def build_grid(context_path: str, keys: List[str], columns: List[str], unpack_ne
         keys: Keys to be included
         columns: Grid columns name.
         unpack_nested_elements: True for unpacking nested elements, False otherwise.
+        keys_from_nested: Keys to extract from nested dictionaries.
 
     Returns:
         pd.DataFrame: New Table include data from Entry Context
     """
     # Retrieve entry context data
     entry_context_data = demisto.dt(demisto.context(), context_path)
     # Validate entry context structure
-    data_type = validate_entry_context(context_path, entry_context_data, keys, unpack_nested_elements)
+    data_type = validate_entry_context(context_path, entry_context_data, unpack_nested_elements)
 
     demisto.debug('context object is valid. starting to build the grid.')
     # Building new Grid
     if unpack_nested_elements:
+
         # Handle entry context as dict, with unpacking of nested elements
         table = pd.DataFrame(unpack_all_data_from_dict(entry_context_data, keys, columns))
         table.rename(columns=dict(zip(table.columns, columns)), inplace=True)
@@ -247,17 +270,21 @@ def build_grid(context_path: str, keys: List[str], columns: List[str], unpack_ne
         table.rename(columns=dict(zip(table.columns, columns)), inplace=True)
     elif isinstance(entry_context_data, list):
         # Handle entry context as list of dicts
-        entry_context_data = [filter_dict(item, keys, len(columns)) for item in entry_context_data]
+        entry_context_data = [entry_dicts_to_string(dict_obj=filter_dict(item, keys, len(columns)),
+                                                    keys_to_choose=keys_from_nested)
+                              for item in entry_context_data]
         table = pd.DataFrame(entry_context_data)
         table.rename(columns=dict(zip(table.columns, columns)), inplace=True)
     elif isinstance(entry_context_data, dict):
-        # Handle entry context key-value of primitive types option
+        # Handle entry context key-value
         # If the keys arg is * it means we don't know which keys we have in the context - Will create key-value table.
+        entry_context_data = entry_dicts_to_string(dict_obj=filter_dict(entry_context_data, keys),
+                                                   keys_to_choose=keys_from_nested)
         if keys == ['*']:
-            entry_context_data = filter_dict(entry_context_data, keys).items()
+            entry_context_data = entry_context_data.items()
             table = pd.DataFrame(entry_context_data, columns=columns[:2])
         else:
-            entry_context_data = filter_dict(entry_context_data, keys)
+            entry_context_data = entry_context_data
             table = pd.DataFrame([entry_context_data])
             table.rename(columns=dict(zip(table.columns, columns)), inplace=True)
 
@@ -269,7 +296,7 @@ def build_grid(context_path: str, keys: List[str], columns: List[str], unpack_ne
 
 @logger
 def build_grid_command(grid_id: str, context_path: str, keys: List[str], columns: List[str], overwrite: bool,
-                       sort_by: List[str], unpack_nested_elements: bool) \
+                       sort_by: List[str], unpack_nested_elements: bool, keys_from_nested: List[str]) \
         -> List[Dict[Any, Any]]:
     """ Build Grid in one of the 3 options:
             1. Context_path contains list of dicts where values are of primitive types (str, int, float, bool),
@@ -289,6 +316,7 @@ def build_grid_command(grid_id: str, context_path: str, keys: List[str], columns
             overwrite: True if to overwrite existing data else False.
             sort_by: Name(s) of the columns to sort by.
             unpack_nested_elements: True for unpacking nested elements, False otherwise.
+            keys_from_nested: Keys to extract from nested dictionaries.
 
         Returns:
             list: Table representation for the Grid.
@@ -304,7 +332,8 @@ def build_grid_command(grid_id: str, context_path: str, keys: List[str], columns
     new_table: pd.DataFrame = build_grid(context_path=context_path,
                                          keys=keys,
                                          columns=columns,
-                                         unpack_nested_elements=unpack_nested_elements)
+                                         unpack_nested_elements=unpack_nested_elements,
+                                         keys_from_nested=keys_from_nested)
 
     # Merge tables if not specified to overwrite.
     if not overwrite:
@@ -336,6 +365,7 @@ def main():
                                    columns=argToList(args.get('columns')),
                                    sort_by=argToList(args.get('sort_by')),
                                    unpack_nested_elements=argToBoolean(args.get('unpack_nested_elements')),
+                                   keys_from_nested=argToList(args.get('keys_from_nested'))
                                    )
         # Execute automation 'setIncident` which change the Context data in the incident
         res = demisto.executeCommand("setIncident", {

Packs/CommonScripts/Scripts/SetGridField/SetGridField.yml

@@ -30,6 +30,10 @@ args:
   predefined:
   - 'false'
   - 'true'
+- defaultValue: '*'
+  description: 'Keys to retrieve from nested dictionaries. Can be used only when the unpack_nested_elements argument is set to false. Keys will not be columns correlated. Default is all keys.'
+  isArray: true
+  name: keys_from_nested
 comment: Creates a Grid table from items or key-value pairs.
 commonfields:
   id: SetGridField
@@ -39,7 +43,7 @@ script: '-'
 subtype: python3
 timeout: '0'
 type: python
-dockerimage: demisto/pandas:1.0.0.26289
+dockerimage: demisto/pandas:1.0.0.72335
 fromversion: 5.0.0
 tests:
 - No tests