New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CreateHash, DeduplicateValuesbyKey, JsonUnescape [Common Scripts Utility Consolidations] #23672
Merged
Merged
Changes from 23 commits
Commits
Show all changes
31 commits
Select commit
Hold shift + click to select a range
ec0906d
Moved CreateHash script to CommonScripts
samuelFain 4e561d2
Standardized CreateHash code
samuelFain e0a13f4
Added createHash UT
samuelFain ec2138f
Moved DeduplicateValuesbyKey to CommonScripts
samuelFain e58a564
Added DeduplicateValuesbyKey UT
samuelFain a61e1e0
Updated release notes, Moved JsonUnescape
samuelFain 86c7e91
Added JsonUnescape UT
samuelFain 057696a
Merge branch 'master' into sf_CreateHash
samuelFain b7217f1
fixed DeduplicateValuesbyKey UT
samuelFain 6bb438e
Merge branch 'master' into sf_CreateHash
samuelFain c4d9d6b
Updated CommonScripts RN
samuelFain a62c72a
Merge branch 'master' into sf_CreateHash
samuelFain 8e6ded4
Removed nosec from hashlib functions
samuelFain 6a46a7c
Added to known_words.txt
samuelFain f5f3888
Fixed linting issue, Updated RN
samuelFain a347154
Merge branch 'master' into sf_CreateHash
samuelFain 6fc6b9a
added noseq to ignore hash Bandit error
samuelFain a04ef42
Updated RN
samuelFain 3037ad7
Merge branch 'master' into sf_CreateHash
samuelFain 1e57bbc
Updated RN
samuelFain ce333a2
Updated RN
samuelFain d6cde16
Updated RN
samuelFain 91409e5
Merge branch 'master' into sf_CreateHash
samuelFain c23c351
Updated RN
samuelFain 3daf991
Updated RN
samuelFain 7a4fa57
Merge branch 'master' into sf_CreateHash
samuelFain 8b9937f
Updated RN
samuelFain c16cfe7
Merge branch 'master' into sf_CreateHash
samuelFain 6fd242d
Merge branch 'master' into sf_CreateHash
samuelFain 5b481a7
Updated docker images & RN
samuelFain 06c9392
Merge branch 'master' into sf_CreateHash
samuelFain File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
|
||
#### Scripts | ||
##### New: JsonUnescape | ||
- Recursively un-escapes JSON data if escaped JSON is found (Available from Cortex XSOAR 6.0.0). | ||
##### New: DeduplicateValuesbyKey | ||
- Given a list of objects and a key found in each of those objects, return a unique list of values associated with that key. Returns error if the objects provided do not contain the key of interest. (Available from Cortex XSOAR 6.0.0). | ||
##### New: CreateHash | ||
- Creating a hash of a given input, support sha1, sha256, sha512, md5 and blake. Wrapper for https://docs.python.org/3/library/hashlib.html. | ||
(Available from Cortex XSOAR 6.0.0). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,52 @@ | ||
|
||
import hashlib | ||
from hashlib import blake2b | ||
|
||
import demistomock as demisto # noqa: F401 | ||
from CommonServerPython import * # noqa: F401 | ||
|
||
|
||
def create_hash(text, hashtype): | ||
"""Create a hash from a given input and return it as a context outputs | ||
|
||
Args: | ||
text (str): input to hash | ||
hashtype (str): hash type | ||
|
||
Returns: | ||
Dict[str,str]: Dictionary representing the command results context | ||
""" | ||
if hashtype == "sha512": | ||
h = hashlib.sha512() | ||
h.update(text.encode('utf-8')) | ||
elif hashtype == "sha256": | ||
h = hashlib.sha256() | ||
h.update(text.encode('utf-8')) | ||
elif hashtype == 'sha1': | ||
h = hashlib.sha1() # nosec | ||
h.update(text.encode('utf-8')) | ||
elif hashtype == 'md5': | ||
h = hashlib.md5() # nosec | ||
h.update(text.encode('utf-8')) | ||
else: | ||
h = blake2b() | ||
h.update(text.encode('utf-8')) | ||
|
||
context = { | ||
"CreateHash": str(h.hexdigest()) | ||
} | ||
|
||
return context | ||
|
||
|
||
def main(): # pragma: no cover | ||
args = demisto.args() | ||
text = args.get('text') | ||
hashtype = args.get('type') | ||
|
||
context = create_hash(text, hashtype) | ||
return_results(CommandResults(outputs=context)) | ||
|
||
|
||
if __name__ in ('__builtin__', 'builtins', '__main__'): | ||
main() |
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
import pytest | ||
|
||
|
||
@pytest.mark.parametrize("hash_method", ['sha512', 'sha256', 'sha1', 'md5', '']) | ||
def test_create_hash(hash_method): | ||
""" | ||
Given: | ||
- A string | ||
When: | ||
- Running the script | ||
Then: | ||
- Ensure the expected hash is returned | ||
""" | ||
from CreateHash import create_hash | ||
context = create_hash('test', hash_method) | ||
|
||
assert isinstance(context, dict) | ||
assert isinstance(context.get('CreateHash'), str) |
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
71 changes: 71 additions & 0 deletions
71
Packs/CommonScripts/Scripts/DeduplicateValuesbyKey/DeduplicateValuesbyKey_test.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,71 @@ | ||
|
||
import pytest | ||
|
||
|
||
def test_generate_unique_values_from_objects(): | ||
from DeduplicateValuesbyKey import generate_unique_values_from_objects | ||
objects = [ | ||
{ | ||
"key": "value1", | ||
"value": "value1" | ||
}, | ||
{ | ||
"key": "value1", | ||
"value": "value2" | ||
}, | ||
{ | ||
"key": "value2", | ||
"value": "value3" | ||
}, | ||
{ | ||
"key": "value2", | ||
"value": "value4" | ||
}, | ||
{ | ||
"key": "value3", | ||
"value": "value5" | ||
}, | ||
{ | ||
"key": "value3", | ||
"value": "value6" | ||
}] | ||
values = generate_unique_values_from_objects(objects, "key", False) | ||
assert set(values) == set(["value1", "value2", "value3"]) | ||
|
||
|
||
def test_generate_unique_values_from_objects_with_none(): | ||
from DeduplicateValuesbyKey import generate_unique_values_from_objects | ||
objects = [ | ||
{ | ||
"key": "value1", | ||
"value": "value1" | ||
}, | ||
{ | ||
"key": "value1", | ||
"value": "value2" | ||
}, | ||
{ | ||
"key": "value2", | ||
"value": "value3" | ||
}, | ||
{ | ||
"key": "value2", | ||
"value": "value4" | ||
}, | ||
{ | ||
"key": "value3", | ||
"value": "value5" | ||
}, | ||
{ | ||
"key": "None_value", | ||
"value": None | ||
}] | ||
values = generate_unique_values_from_objects(objects, "key", True) | ||
assert set(values) == set(["None_value", "value1", "value2", "value3"]) | ||
|
||
|
||
def test_generate_unique_values_from_objects_fail(): | ||
from DeduplicateValuesbyKey import generate_unique_values_from_objects | ||
|
||
with pytest.raises(SystemExit): | ||
generate_unique_values_from_objects([], "key", True) |
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
26 changes: 26 additions & 0 deletions
26
Packs/CommonScripts/Scripts/JsonUnescape/JsonUnescape_test.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
def test_hook(): | ||
""" | ||
Given | ||
- A string value representing a json | ||
When | ||
- Running the hook function | ||
Then | ||
- Ensure the string value is converted to json | ||
""" | ||
from JsonUnescape import hook | ||
assert hook({"key": "value"}) == {"key": "value"} | ||
assert hook({"key": "{'key': 'value'}"}) == {'key': "{'key': 'value'}"} | ||
|
||
|
||
def test_unescape(): | ||
""" | ||
Given | ||
- A dictionary with a string value | ||
When | ||
- Running the unescape function | ||
Then | ||
- Ensure the string value is converted to json | ||
""" | ||
from JsonUnescape import unescape | ||
assert unescape({"value": "value"}) == 'value' | ||
assert unescape({"value": "{'key': 'value'}"}) == "{'key': 'value'}" |
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
{"breakingChanges":true,"breakingChangesNotes":"The script: CreateHash moved to *CommonScripts* pack. Make sure to update the *CommonScripts* pack to the latest version in order to use this script."} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
##### CreateHash | ||
- Deprecated. **CreateHash** script moved to *Common Scripts* pack. |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
{"breakingChanges":true,"breakingChangesNotes":"The script: DeduplicateValuesbyKey moved to *CommonScripts* pack. Make sure to update the *CommonScripts* pack to the latest version in order to use this script."} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
##### DeduplicateValuesbyKey | ||
- Deprecated. **DeduplicateValuesbyKey** script moved to *Common Scripts* pack. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,11 +1,12 @@ | ||
{ | ||
"name": "DeduplicateValuesbyKey", | ||
"description": "Given a list of objects (dictionaries) and a key (string), return a list of unique values that appear for that key in the list of objects. \n\nThe current implementation only works on the first-level of keys. That is, given a dictionary like {key1: {key2: value}}, you can return a list of unique values for key1, but not key2. ", | ||
"name": "DeduplicateValuesbyKey (Deprecated)", | ||
"description": "Deprecated. DeduplicateValuesbyKey script moved to CommonScripts pack.", | ||
"support": "community", | ||
"currentVersion": "1.0.0", | ||
"currentVersion": "1.0.1", | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Add hidden: true There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed. |
||
"author": "Laura Farvour", | ||
"url": "", | ||
"email": "", | ||
"hidden": true, | ||
"created": "2021-12-07T17:00:09Z", | ||
"categories": [ | ||
"Utilities" | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
{"breakingChanges":true,"breakingChangesNotes":"The script: JsonUnescape moved to *CommonScripts* pack. Make sure to update the *CommonScripts* pack to the latest version in order to use this script."} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
##### JsonUnescape | ||
- Deprecated. **JsonUnescape** script moved to *Common Scripts* pack. |
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Where is the deprecating notes in the files? + hidden: true
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added in later commit. all deprecating notes now exist for CreateHash.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@samuelFain I still don't see it in here. Where is the deprecated in the name and description in this file?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed now :)