Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Digital Shadows Mapping #32493

Merged
merged 9 commits into from Jan 30, 2024
Merged

Digital Shadows Mapping #32493

merged 9 commits into from Jan 30, 2024

Conversation

eepstain
Copy link
Contributor

Contributing to Cortex XSOAR Content

Make sure to register your contribution by filling the contribution registration form

The Pull Request will be reviewed only after the contribution registration form is filled.

Status

  • In Progress
  • Ready
  • In Hold - (Reason for hold)

Related Issues

fixes: link to the issue

Description

A few sentences describing the overall goals of the pull request's commits.

Must have

  • Tests
  • Documentation

Copy link

Your contributed DigitalShadows pack has been modified on files:

Packs/DigitalShadows/pack_metadata.json
Packs/DigitalShadows/ModelingRules/DigitalShadows/DigitalShadows.xif
Packs/DigitalShadows/ReleaseNotes/1_0_10.md
Packs/DigitalShadows/ModelingRules/DigitalShadows/DigitalShadows.yml
Packs/DigitalShadows/ModelingRules/DigitalShadows/DigitalShadows_schema.json
Please review the changes here

@eepstain
Copy link
Contributor Author

Query to run the mapping:

config timeframe = 5y | datamodel dataset = reliaquest_greymatter_drp_raw
| fields xdm.event.id, xdm.event.is_completed, xdm.observer.action, xdm.event.type, xdm.alert.severity, xdm.observer.type, xdm.source.host.device_category, xdm.event.outcome, xdm.event.outcome_reason, xdm.source.host.hostname, xdm.alert.subcategory, xdm.event.operation_sub_type, xdm.observer.unique_identifier, xdm.alert.risks, xdm.event.original_event_type, xdm.event.description, xdm.alert.description, xdm.alert.mitre_tactics, xdm.alert.mitre_techniques, xdm.alert.original_threat_id

@eepstain eepstain requested a review from cweltPA January 30, 2024 10:36
cweltPA
cweltPA previously approved these changes Jan 30, 2024
@cweltPA cweltPA dismissed their stale review January 30, 2024 11:03

_time assignment

@GuyAfik GuyAfik changed the base branch from master to ReliaQuest_Collector January 30, 2024 12:56
@GuyAfik GuyAfik merged commit d244e54 into ReliaQuest_Collector Jan 30, 2024
9 of 12 checks passed
@GuyAfik GuyAfik deleted the digital_shadows_mapping branch January 30, 2024 13:02
Copy link

Coverage

Coverage Report
FileStmtsMissCoverMissing
Packs/Base/Scripts/CommonServerPython
   CommonServerPython.py430765984%167–170, 181, 183, 406, 448, 542, 567–571, 596, 611, 751–752, 761, 802–803, 821, 867, 954–962, 1006–1011, 1045–1047, 1177–1181, 1184–1185, 1187–1189, 1203–1205, 1207, 1273, 1286, 1299, 1307–1309, 1311, 1324, 1332, 1419–1420, 1475, 1497–1498, 1501, 1507, 1509, 1511, 1605, 1650, 1724–1727, 1765–1766, 1780–1785, 1787, 1855, 1938, 1940, 1960, 1990, 1994, 2033, 2209, 2233, 2272–2273, 2353–2354, 2356, 2369–2376, 2378, 2455–2457, 2471, 2484, 2515, 2521–2525, 2527–2529, 2531, 2535–2536, 2538, 2551–2557, 2561, 2565–2567, 2573–2574, 2586, 2614, 2617, 2622, 2642, 2648, 2650, 2652, 2672, 2675, 2687, 2733–2741, 2743, 2802, 2824, 2844, 2902, 2914–2915, 3027, 3029, 3204, 3214, 3220, 3229, 3235, 3249, 3305, 3320, 3322, 3353–3358, 3361, 3412–3413, 3416, 3582–3583, 3586, 3693, 3696, 3921, 3924, 3927, 3930, 3935, 3944, 3970, 3976, 3979, 3988, 3990, 4106–4107, 4131, 4133, 4152, 4155, 4158, 4226, 4247, 4249, 4385, 4403, 4415, 4451, 4453, 4627–4628, 4706, 4708, 4733, 4748–4764, 4767, 4771–4772, 4774–4775, 4777–4778, 4780–4781, 4783–4784, 4786–4787, 4789–4790, 4792–4793, 4795–4796, 4798–4799, 4801–4802, 4804–4805, 4807–4808, 4810, 4812–4813, 4815–4818, 4820–4822, 4824, 4826, 4830, 4901, 4933, 4939, 4941, 4979, 4984–4985, 4990, 4994–4995, 4997, 5079, 5085, 5177, 5198, 5201–5206, 5214–5220, 5222–5223, 5268, 5281, 5376–5379, 5384, 5387, 5390, 5424, 5427, 5471, 5473, 5475, 5570, 5621, 5703, 5715, 5733, 5773–5774, 5778, 5780, 5814, 5818, 5820, 5822, 5824, 5870, 5874, 5973–5976, 5978, 5991, 6010, 6020, 6026, 6035–6037, 6040–6043, 6058, 6100, 6129, 6132, 6181–6184, 6187–6191, 6195, 6203, 6209, 6237, 6266, 6272–6273, 6324, 6344–6345, 6347, 6387–6388, 6395, 6400, 6406, 6419–6420, 6422, 6693–6695, 6714, 6727, 6739, 6767, 6927, 6942, 7007, 7037, 7075–7076, 7088, 7103–7104, 7106–7107, 7109–7110, 7112–7113, 7115–7118, 7120, 7122–7123, 7125, 7204–7205, 7217, 7257–7260, 7262, 7269–7270, 7355, 7364, 7373, 7382, 7391, 7400, 7409, 7417, 7472, 7507, 7527, 7529, 7630, 7791–7794, 7796–7798, 7857, 7876–7877, 7889, 7905, 7926–7927, 7929, 7946, 8033, 8047, 8050–8053, 8056, 8059, 8081–8084, 8181, 8200–8201, 8329, 8379, 8383–8384, 8398, 8401, 8414–8416, 8419, 8491, 8661–8662, 8736, 8738–8740, 8784–8785, 8787, 8790, 8801, 8807–8810, 8812, 8814, 8816–8817, 8930, 8961, 8972, 8975, 9004–9009, 9106–9107, 9111, 9172–9175, 9177, 9199, 9211, 9307, 9343–9344, 9357, 9370–9375, 9391–9392, 9400–9401, 9414, 9422–9423, 9440–9441, 9455, 9465, 9481, 9492, 9500–9502, 9504, 9530, 9545, 9556, 9574–9575, 9578, 9596, 9599, 9614, 9628, 9634, 9649, 9666, 9678–9686, 9689–9690, 9692, 9707–9709, 9720, 9723, 9811, 9837, 9844, 9888, 9914–9915, 9958, 9960–9962, 9964, 10035, 10044, 10165, 10229, 10233–10234, 10325–10330, 10425–10428, 10518–10520, 10546, 10579, 10629–10630, 10635–10636, 10653, 10739–10741, 10760, 10968, 11084, 11108, 11131, 11156, 11163–11169, 11171, 11173, 11309, 11347–11348, 11350–11357, 11362–11364, 11366, 11368, 11461–11462, 11480, 11513–11514, 11518
Packs/DigitalShadows/Integrations/ReliaQuestGreyMatterDRPEventCollector
   ReliaQuestGreyMatterDRPEventCollector.py2383884%76, 78–79, 98, 100, 204, 214, 243, 260–265, 273–275, 303, 395, 401, 404, 410–411, 429–432, 435–438, 481, 484–485, 487, 490–492
TOTAL454569784% 

Tests Skipped Failures Errors Time
687 12 💤 0 ❌ 0 🔥 32.990s ⏱️

GuyAfik added a commit that referenced this pull request Feb 11, 2024
* update pack-name and add keywords

* add integration params

* init clients

* prepare api endpoints

* remove template stuff

* start implementation of endpoints

* update yml

* update readme

* update yml

* implement test-module

* implement retry decorator

* enhance retry in relia

* handle pagination

* update pagination params

* pagination fix

* handle test module + traiged-itesm

* incident and alert captures

* continue fetch

* remove first fetch

* working version

* update readme and ID of the collector

* logic into functions

* fix bug of endless incident/alert

* set last run

* pre-commit

* dedup

* pre-commit

* bump csp rn

* bump rn

* test module tests

* fix bug when latest created time is none

* revert test changes

* create test infra

* add sanity fetch test

* fix issue with dates returned without miliseconds

* date format with Z

* log last run

* rate limit fixes

* handle rate limits

* sleep for 60 sec

* fix pagination

* events[:limit]

* refactor get_triage_item_ids_to_events

* refactor get_events_with_latest_created_time

* add latest-created-time

* store event nums as UUID for next run

* dedup according to event-num

* fix duplicate events bug

* get_events_command implementation

* graph no mv

* smart recover of api limits

* change log for rate-limit

* fix context and add documentation

* fix the debug-command

* update unit-test + fix bug with pagination

* update rn

* Bump pack from version Base to 1.33.20.

* handle rate-limit errors

* add logs

* logs

* add finally clause

* add sanity test

* pre-commit

* auto aligment

* rate-limit sanity test

* add generator unit-test

* update tests

* fix long lines

* fix validations

* fix csp test

* test empty response from api

* fix validations / linting

* fix syntax csp

* mypy fixes

* 120x50 photo

* update pic to correct format

* fix an issue of endless loop when there are no events

* add test_fetch_events_no_events_with_last_run

* doc-strings and documentations

* csp docs

* catch internal server errors

* pagination with event-num

* remove marketplaces for now

* fetch by event-num pagination

* fix unit-tests

* pre-commit

* fix fetch command

* marketplaceV2

* doc review fixes

* doc-review

* code enhancments and test additions

* pre-commit

* finish

* Bump pack from version Base to 1.33.21.

* add mitre_tactic names

* add mitre-attack-mappings

* update docker image

* fix wrong ids

* pre-commit

* Bump pack from version Base to 1.33.22.

* implement _time

* Update Packs/DigitalShadows/Integrations/ReliaQuestGreyMatterDRPEventCollector/README.md

Co-authored-by: ShirleyDenkberg <62508050+ShirleyDenkberg@users.noreply.github.com>

* Update Packs/DigitalShadows/Integrations/ReliaQuestGreyMatterDRPEventCollector/README.md

Co-authored-by: ShirleyDenkberg <62508050+ShirleyDenkberg@users.noreply.github.com>

* Update Packs/DigitalShadows/Integrations/ReliaQuestGreyMatterDRPEventCollector/ReliaQuestGreyMatterDRPEventCollector_description.md

Co-authored-by: ShirleyDenkberg <62508050+ShirleyDenkberg@users.noreply.github.com>

* Update Packs/DigitalShadows/ReleaseNotes/2_0_0.md

Co-authored-by: ShirleyDenkberg <62508050+ShirleyDenkberg@users.noreply.github.com>

* Digital Shadows Mapping (#32493)

* Created ModelingRules

* Updated ReleaseNotes

* Updated ReleaseNotes

* Updated ModelingRules directory

* Updated ModelingRules

* Updated ModelingRules

* Updated ModelingRules

* solve c onflict

---------

Co-authored-by: GuyAfik <guyafik11@gmail.com>

* update rn

* remove old rn

* Bump pack from version Base to 1.33.23.

* bump rn

* Bump pack from version Base to 1.33.26.

* for demo - revert after demo

* handle health module + fix descriptions + ids

* some more documentation

* implement entry status

* ruff

* take back marketplaces field

* fix csp not good

* review fixes

* bump rn

* fix health-module

* update default max fetch to 1000

* change _time logic

* fix test-module

* docker update

* mypy fix

---------

Co-authored-by: Content Bot <bot@demisto.com>
Co-authored-by: ShirleyDenkberg <62508050+ShirleyDenkberg@users.noreply.github.com>
Co-authored-by: eepstain <116078117+eepstain@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants