Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

StixParser #31643

Merged
merged 44 commits into from Dec 27, 2023
Merged

StixParser #31643

merged 44 commits into from Dec 27, 2023

Conversation

sapirshuker
Copy link
Contributor

Contributing to Cortex XSOAR Content

Make sure to register your contribution by filling the contribution registration form

The Pull Request will be reviewed only after the contribution registration form is filled.

Status

  • In Progress
  • Ready
  • In Hold - (Reason for hold)

Related Issues

fixes: https://jira-dc.paloaltonetworks.com/browse/XSUP-31411

Description

Fixed an issue where STIX indicators were not parsed when type information was missing.

Must have

  • Tests
  • Documentation

Copy link

github-actions bot commented Dec 21, 2023

Coverage

Coverage Report
FileStmtsMissCoverMissing
Packs/Base/Scripts/CommonServerPython
   CommonServerPython.py428183680%164–167, 178, 180, 227–228, 337, 403, 406, 445, 448, 538–539, 563–568, 571, 593, 596, 608, 748–749, 758, 792, 799–800, 818, 864, 932, 935–936, 951–959, 1003–1008, 1042–1044, 1174–1178, 1181–1182, 1184–1186, 1200–1202, 1204, 1270, 1283, 1296, 1304–1306, 1308, 1321, 1329, 1351, 1416–1417, 1472, 1494–1495, 1498, 1504, 1506, 1508, 1602, 1625–1626, 1647, 1717–1720, 1728, 1730, 1737, 1758–1759, 1773–1778, 1780, 1827, 1848, 1931, 1933, 1953, 1983, 1987, 2026, 2202, 2226, 2265–2266, 2346–2347, 2349, 2362–2369, 2371, 2448–2450, 2464, 2477, 2508, 2514–2518, 2520–2522, 2524, 2528–2529, 2531, 2544–2550, 2554, 2558–2560, 2566–2567, 2579, 2607, 2610, 2615, 2635, 2641, 2643, 2645, 2665, 2668, 2680, 2695, 2726–2734, 2736, 2795, 2827–2828, 2831, 2886, 2898–2899, 2908, 2917, 3011, 3013, 3188, 3198, 3204, 3213, 3219, 3233, 3289, 3304, 3306, 3337–3342, 3345, 3396–3397, 3400, 3566–3567, 3570, 3677, 3680, 3906, 3909, 3912, 3915, 3920, 3929, 3955, 3961, 3964, 3973, 3975, 4091–4092, 4116, 4118, 4137, 4140, 4143, 4211, 4232, 4234, 4370, 4388, 4400, 4436, 4438, 4612–4613, 4691, 4693, 4718, 4733–4749, 4752, 4756–4757, 4759–4760, 4762–4763, 4765–4766, 4768–4769, 4771–4772, 4774–4775, 4777–4778, 4780–4781, 4783–4784, 4786–4787, 4789–4790, 4792–4793, 4795, 4797–4798, 4800–4803, 4805–4807, 4809, 4811, 4815, 4886, 4918, 4924, 4926, 4964, 4969–4970, 4975, 4979–4980, 4982, 5064, 5070, 5138, 5162, 5183, 5186–5191, 5199–5205, 5207–5208, 5236, 5253, 5266, 5361–5364, 5369, 5372, 5375, 5409, 5412, 5456, 5458, 5460, 5555, 5591, 5606, 5647, 5688, 5700, 5718, 5758–5759, 5763, 5765, 5799, 5803, 5805, 5807, 5809, 5855, 5859, 5958–5961, 5963, 5976, 5995, 6005, 6011, 6020–6022, 6025–6028, 6043, 6085, 6114, 6117, 6166–6169, 6172–6176, 6179–6180, 6183, 6188, 6194, 6222, 6251, 6257–6258, 6309, 6329–6330, 6332, 6367–6370, 6372–6373, 6375, 6377–6378, 6380, 6382–6383, 6385–6386, 6390–6391, 6393, 6395, 6397–6398, 6400, 6402, 6404–6405, 6407, 6465, 6482, 6643, 6654, 6678–6680, 6699, 6712, 6724, 6752, 6912, 6927, 6992, 7022, 7060–7061, 7073, 7088–7089, 7091–7092, 7094–7095, 7097–7098, 7100–7103, 7105, 7107–7108, 7110, 7189–7190, 7202, 7242–7245, 7247, 7254–7255, 7315, 7321, 7325, 7330, 7334, 7339–7340, 7343, 7348–7349, 7352, 7357–7358, 7361, 7366–7367, 7370, 7375–7376, 7379, 7384–7385, 7388, 7393–7394, 7397, 7402, 7457, 7492, 7504, 7512, 7514, 7584, 7615, 7623, 7776–7779, 7781–7783, 7842, 7861–7862, 7874, 7890, 7911–7912, 7914, 7931, 8018, 8032, 8035–8038, 8041, 8044, 8066–8069, 8166, 8185–8186, 8314, 8333, 8335–8338, 8362–8364, 8366–8369, 8383, 8386, 8399–8401, 8404, 8476, 8617, 8621, 8624, 8631, 8633, 8635–8639, 8641–8643, 8645–8647, 8703, 8721, 8723–8725, 8769–8771, 8774, 8785, 8791–8794, 8796, 8798, 8800–8801, 8914, 8918, 8937, 8945, 8956, 8959, 8988–8993, 9090–9091, 9095, 9156–9159, 9161, 9183, 9195, 9291, 9327–9328, 9341, 9354–9359, 9375–9376, 9384–9385, 9398, 9406–9407, 9424–9425, 9439, 9449, 9465, 9476, 9484–9486, 9488, 9513–9514, 9529, 9540, 9558–9559, 9562, 9580, 9583, 9598, 9612, 9618, 9633, 9650, 9662–9670, 9673–9674, 9676, 9691–9693, 9704, 9707, 9794–9795, 9798, 9802, 9806–9807, 9821, 9828, 9891–9892, 9935, 9937–9940, 10011, 10020, 10141, 10159, 10205, 10209–10210, 10301–10306, 10401–10404, 10494–10496, 10522, 10555, 10605–10606, 10611–10612, 10629, 10683–10688, 10690, 10692–10694, 10696–10697, 10710–10716, 10734, 10761–10762, 10764, 10766, 10768–10770, 10772, 10774, 10938, 10992, 11054, 11078, 11101, 11126, 11133–11139, 11141, 11143, 11230–11243, 11278, 11375–11384, 11386–11387, 11389–11390, 11394, 11396–11398, 11400–11402, 11404–11405, 11407–11410, 11421–11423, 11425, 11427, 11431–11436, 11438–11440, 11442, 11444, 11452–11453, 11455–11461, 11466
Packs/Base/Scripts/StixParser
   StixParser.py73714380%133–137, 204–211, 224, 484, 486, 492, 501–502, 590, 592, 594, 607, 633, 643–644, 650, 699, 879–880, 884–886, 888, 972–976, 978–980, 982–984, 986–988, 991–993, 996–998, 1001–1003, 1006–1012, 1014, 1094–1097, 1099–1102, 1104–1107, 1112–1113, 1127–1128, 1138–1139, 1166, 1173, 1183–1185, 1187, 1189–1190, 1194–1195, 1211, 1236, 1240, 1244, 1275, 1294–1295, 1297, 1318–1321, 1332–1345, 1359–1360, 1362–1370, 1376–1377, 1407, 1409–1412, 1433, 1464, 1499, 1522, 1613–1614
TOTAL501897980% 

Tests Skipped Failures Errors Time
684 15 💤 0 ❌ 0 🔥 26.568s ⏱️

@sapirshuker sapirshuker marked this pull request as ready for review December 21, 2023 13:56
@@ -1221,6 +1221,21 @@ def decode(props, **kwargs):
return domains


def detect_indicator_type_file(indicator_value: str):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't we have something similar in common server python?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks that it can be useful to write this one in CommonServerPython maybe

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does, I also thought we have it (I even remember writing something like it somewhere), but it should be as a different PR as CSP is the worst.

v = v.strip()
if type := detect_indicator_type_file(v):
result.append({
'indicator': v.strip(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
'indicator': v.strip(),
'indicator': v,

looks like v is already striped.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

@@ -1221,6 +1221,21 @@ def decode(props, **kwargs):
return domains


def detect_indicator_type_file(indicator_value: str):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does, I also thought we have it (I even remember writing something like it somewhere), but it should be as a different PR as CSP is the worst.

Copy link
Contributor

@eyalpalo eyalpalo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@@ -709,6 +709,30 @@ def get_schedule_metadata(context):
return schedule_metadata


def detect_file_indicator_type(indicator_value):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets add unit test for this method.

@sapirshuker
Copy link
Contributor Author

Pre commit failed due to a known issue CIAC-9294

@dantavori dantavori merged commit dde7cd2 into master Dec 27, 2023
19 of 20 checks passed
@dantavori dantavori deleted the XSUP31411/StixParser branch December 27, 2023 08:33
maimorag pushed a commit that referenced this pull request Dec 28, 2023
* fix stix parser

* fix docker image and ruff

* ruff

* ruff

* fix CR

* Update 1_33_6.md

* fix CR

* csp add rn

* csp add rn

* tests

* update RN conflicts

* cr review

* add use case

* Update CommonServerPython_test.py

* add use case

* fix coverge report

* fix coverge report

* fix coverge report

* fix tests

* Update CommonServerPython_test.py

* fix_tests

* Update CommonServerPython_test.py

* Update CommonServerPython_test.py

* add tests

* add tests

* add tests

* add tests

* add tests

* remove tests

* remove tests

* Update 1_33_7.md
maimorag pushed a commit that referenced this pull request Dec 31, 2023
* fix stix parser

* fix docker image and ruff

* ruff

* ruff

* fix CR

* Update 1_33_6.md

* fix CR

* csp add rn

* csp add rn

* tests

* update RN conflicts

* cr review

* add use case

* Update CommonServerPython_test.py

* add use case

* fix coverge report

* fix coverge report

* fix coverge report

* fix tests

* Update CommonServerPython_test.py

* fix_tests

* Update CommonServerPython_test.py

* Update CommonServerPython_test.py

* add tests

* add tests

* add tests

* add tests

* add tests

* remove tests

* remove tests

* Update 1_33_7.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants