Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rsa netwitness mirroring #29220

Conversation

Winultimatum
Copy link
Contributor

Contributing to Cortex XSOAR Content

Make sure to register your contribution by filling the contribution registration form

The Pull Request will be reviewed only after the contribution registration form is filled.

Status

  • In Progress
  • Ready
  • In Hold - (Reason for hold)

Description

Add mirroring to RSA Netwitness 11.5.
In order to do so and because the RSA API don't tell us when a new alerts are aggregated to an incident, I store inside the context data of the incident when the incident was last pull and how many alerts + events was retrieved. After a 24 days (time fixed by RSA), another alert cannot be aggregated and a new incident is created on RSA. A subroutine clean every incident >24 days inside the context data in order to not saturate the context data of the integration.

Improve SIEM RSA layout in investigate the incident with data:
image

Must have

  • Tests
  • Documentation

support mirror in and out

Add a dedicated layout to view alerts, raws logs and events

use RSA Netwitness Packet & Logs to get raw log
patch encoding issue and add mirroring function in the yml list
@content-bot content-bot added Contribution Thank you! Contributions are always welcome! External PR Xsoar Support Level Indicates that the contribution is for XSOAR supported pack labels Aug 25, 2023
@content-bot content-bot changed the base branch from master to contrib/Winultimatum_RSANetwitness_mirroring August 25, 2023 15:59
@content-bot
Copy link
Collaborator

Thank you for your contribution. Your generosity and caring are unrivaled! Make sure to register your contribution by filling the Contribution Registration form, so our content wizard @MichaelYochpaz will know the proposed changes are ready to be reviewed.
For your convenience, here is a link to the contributions SLAs document.

@content-bot content-bot added Community Contribution Form Filled Whether contribution form filled or not. labels Aug 25, 2023
@Winultimatum
Copy link
Contributor Author

Winultimatum commented Aug 25, 2023

@MichaelYochpaz the format of the "old" field may cause some issue, I don't know how to refresh them so I just add the line script inside the commit 5954e03

For example if I import the field in the UI I got this error:
image

@maimorag maimorag requested review from MichaelYochpaz and melamedbn and removed request for MichaelYochpaz and melamedbn August 27, 2023 08:48
@RotemAmit RotemAmit requested review from RotemAmit and removed request for MichaelYochpaz August 28, 2023 07:47
@melamedbn
Copy link
Contributor

Dear @Winultimatum,

I hope this message finds you well. Thank you for your content contribution.
After reviewing, everything looks great! Just complete @MichaelYochpaz review changes request and we are good to go.

@Winultimatum
Copy link
Contributor Author

Dear @Winultimatum,

I hope this message finds you well. Thank you for your content contribution. After reviewing, everything looks great! Just complete @MichaelYochpaz review changes request and we are good to go.

Hi @melamedbn,
Thank you, I'm guessing that I'm waiting for @RotemAmit now :)

Copy link
Contributor

@RotemAmit RotemAmit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great Job with the tests! They look very good.
They are almost done, but please add the checks that I requested before.

@RotemAmit RotemAmit added ready-for-instance-test In contribution PRs, this label will cause a trigger of a build with a modified pack from the PR. and removed ready-for-instance-test In contribution PRs, this label will cause a trigger of a build with a modified pack from the PR. labels Sep 18, 2023
Copy link
Contributor

@RotemAmit RotemAmit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job on the mirroring feature and it's test!
Thank you very much for your contribution! It is highly appreciated!

@RotemAmit RotemAmit merged commit e784d2c into demisto:contrib/Winultimatum_RSANetwitness_mirroring Sep 18, 2023
19 of 20 checks passed
@content-bot content-bot mentioned this pull request Sep 18, 2023
5 tasks
RotemAmit added a commit that referenced this pull request Sep 18, 2023
* Implement mirror in and out for RSANetwitness 11.5

support mirror in and out

Add a dedicated layout to view alerts, raws logs and events

use RSA Netwitness Packet & Logs to get raw log

* change version

patch encoding issue and add mirroring function in the yml list

* forgot to modify a field

* remove wrong modification

* missing required parameters during merge

* patch CI/CD error

* add mirroring in README + change name parameter + add mirrorid in test data

* implement correction asked by palo

* generated README files for the scripts

* remove strange ' inside the file

improve readme

add some define to better set the mirror date range

* add parameter to set mirroring duration time

* Add return after return_result

* Create test units for RSA Get Raw logs and display scripts

* Add return after return_result of RSAGetRawLog.py, Create unit tests for RSA scripts

* Apply suggestions from code review




* remove duplicate instructions in README

* Implement mirror in and out for RSANetwitness 11.5

support mirror in and out

Add a dedicated layout to view alerts, raws logs and events

use RSA Netwitness Packet & Logs to get raw log

* change version

patch encoding issue and add mirroring function in the yml list

* forgot to modify a field

* remove wrong modification

* missing required parameters during merge

* patch CI/CD error

* add mirroring in README + change name parameter + add mirrorid in test data

* implement correction asked by palo

* generated README files for the scripts

* remove strange ' inside the file

improve readme

add some define to better set the mirror date range

* add parameter to set mirroring duration time

* Add return after return_result

* Create test units for RSA Get Raw logs and display scripts

* Apply suggestions from code review




* remove duplicate instructions in README

* line too long... + error in README

* line too long...

* patch Ci/cd error

* patch ci/cd error

* patch ci/cd error

* Add first version of tests units for RSA Netwitness Mirroring

* patch flake8 error script RSADisplayMetasEvents

* patch flake8 error script RSAGetRawLog

* patch flake8 error integration RSANetWitness 11_5

* Add get_modified_remote_data_command unit test

* Add unit test for command get_modified_remote_data_command

* Fix function name

* Fix flake8 issue

* Remove unused class

* docker version + ignore secret fake ip

* Delete get-pip.py

* Remove unused test files

* format updates

* pre-commit and format updates

* Apply suggestions from code review



* add doc string to get_mapping_fields_command

* Fix bracket error

* Add 2 test functions to check update_remote_system_command

* remove possible duplicate changed inc id

* Fix minor error in return tests functions

* Add units tests to get remote data command

* rename metasevents to rsametasevents

* flake8 space missing

* default config value not set correctly

* Update unit test of get_modified_remote_data_command

* Add unit test for expired and non expired incident

* remove unused line

* update release note

* update release note

* update release note

* change comparaison for pulling new alert

* Update Packs/RSANetWitness_v11_1/Integrations/RSANetWitnessv115/RSANetWitnessv115_test.py



* remove comment in the release note

* Apply suggestions from code review



* Resolv issue of unit tests

* updated the release notes and pack-ignore

* pre-commit updates

* Fix all suggestions

* Fix flake8 errors

* Add mirrored object in test_get_modified_remote_data_command_from_timestamp

* patch flake8 error

* added = in get_remote_data_command

* patch unit test

* Fix token string in incident old clean test

* Fix indent

* Fix indent

* patch flake8

* updated the release notes

* pre-commit changes

* added the file CONTRIBUTORS.json

---------

Co-authored-by: Pierre <Winultimatum@users.noreply.github.com>
Co-authored-by: Pierre SOLER <9917674+Winultimatum@users.noreply.github.com>
Co-authored-by: RotemAmit <ramit@paloaltonetworks.com>
Co-authored-by: Sébastien Guisnet <sebastien.guisnet@nucleon-security.com>
Co-authored-by: Sébastien Guisnet <sebastien.guisnet@gmail.com>
Co-authored-by: ShirleyDenkberg <62508050+ShirleyDenkberg@users.noreply.github.com>
wolyslager pushed a commit to wolyslager/content that referenced this pull request Oct 2, 2023
* Implement mirror in and out for RSANetwitness 11.5

support mirror in and out

Add a dedicated layout to view alerts, raws logs and events

use RSA Netwitness Packet & Logs to get raw log

* change version

patch encoding issue and add mirroring function in the yml list

* forgot to modify a field

* remove wrong modification

* missing required parameters during merge

* patch CI/CD error

* add mirroring in README + change name parameter + add mirrorid in test data

* implement correction asked by palo

* generated README files for the scripts

* remove strange ' inside the file

improve readme

add some define to better set the mirror date range

* add parameter to set mirroring duration time

* Add return after return_result

* Create test units for RSA Get Raw logs and display scripts

* Add return after return_result of RSAGetRawLog.py, Create unit tests for RSA scripts

* Apply suggestions from code review




* remove duplicate instructions in README

* Implement mirror in and out for RSANetwitness 11.5

support mirror in and out

Add a dedicated layout to view alerts, raws logs and events

use RSA Netwitness Packet & Logs to get raw log

* change version

patch encoding issue and add mirroring function in the yml list

* forgot to modify a field

* remove wrong modification

* missing required parameters during merge

* patch CI/CD error

* add mirroring in README + change name parameter + add mirrorid in test data

* implement correction asked by palo

* generated README files for the scripts

* remove strange ' inside the file

improve readme

add some define to better set the mirror date range

* add parameter to set mirroring duration time

* Add return after return_result

* Create test units for RSA Get Raw logs and display scripts

* Apply suggestions from code review




* remove duplicate instructions in README

* line too long... + error in README

* line too long...

* patch Ci/cd error

* patch ci/cd error

* patch ci/cd error

* Add first version of tests units for RSA Netwitness Mirroring

* patch flake8 error script RSADisplayMetasEvents

* patch flake8 error script RSAGetRawLog

* patch flake8 error integration RSANetWitness 11_5

* Add get_modified_remote_data_command unit test

* Add unit test for command get_modified_remote_data_command

* Fix function name

* Fix flake8 issue

* Remove unused class

* docker version + ignore secret fake ip

* Delete get-pip.py

* Remove unused test files

* format updates

* pre-commit and format updates

* Apply suggestions from code review



* add doc string to get_mapping_fields_command

* Fix bracket error

* Add 2 test functions to check update_remote_system_command

* remove possible duplicate changed inc id

* Fix minor error in return tests functions

* Add units tests to get remote data command

* rename metasevents to rsametasevents

* flake8 space missing

* default config value not set correctly

* Update unit test of get_modified_remote_data_command

* Add unit test for expired and non expired incident

* remove unused line

* update release note

* update release note

* update release note

* change comparaison for pulling new alert

* Update Packs/RSANetWitness_v11_1/Integrations/RSANetWitnessv115/RSANetWitnessv115_test.py



* remove comment in the release note

* Apply suggestions from code review



* Resolv issue of unit tests

* updated the release notes and pack-ignore

* pre-commit updates

* Fix all suggestions

* Fix flake8 errors

* Add mirrored object in test_get_modified_remote_data_command_from_timestamp

* patch flake8 error

* added = in get_remote_data_command

* patch unit test

* Fix token string in incident old clean test

* Fix indent

* Fix indent

* patch flake8

* updated the release notes

* pre-commit changes

* added the file CONTRIBUTORS.json

---------

Co-authored-by: Pierre <Winultimatum@users.noreply.github.com>
Co-authored-by: Pierre SOLER <9917674+Winultimatum@users.noreply.github.com>
Co-authored-by: RotemAmit <ramit@paloaltonetworks.com>
Co-authored-by: Sébastien Guisnet <sebastien.guisnet@nucleon-security.com>
Co-authored-by: Sébastien Guisnet <sebastien.guisnet@gmail.com>
Co-authored-by: ShirleyDenkberg <62508050+ShirleyDenkberg@users.noreply.github.com>
xsoar-bot pushed a commit to xsoar-contrib/content that referenced this pull request Oct 5, 2023
* Implement mirror in and out for RSANetwitness 11.5

support mirror in and out

Add a dedicated layout to view alerts, raws logs and events

use RSA Netwitness Packet & Logs to get raw log

* change version

patch encoding issue and add mirroring function in the yml list

* forgot to modify a field

* remove wrong modification

* missing required parameters during merge

* patch CI/CD error

* add mirroring in README + change name parameter + add mirrorid in test data

* implement correction asked by palo

* generated README files for the scripts

* remove strange ' inside the file

improve readme

add some define to better set the mirror date range

* add parameter to set mirroring duration time

* Add return after return_result

* Create test units for RSA Get Raw logs and display scripts

* Add return after return_result of RSAGetRawLog.py, Create unit tests for RSA scripts

* Apply suggestions from code review




* remove duplicate instructions in README

* Implement mirror in and out for RSANetwitness 11.5

support mirror in and out

Add a dedicated layout to view alerts, raws logs and events

use RSA Netwitness Packet & Logs to get raw log

* change version

patch encoding issue and add mirroring function in the yml list

* forgot to modify a field

* remove wrong modification

* missing required parameters during merge

* patch CI/CD error

* add mirroring in README + change name parameter + add mirrorid in test data

* implement correction asked by palo

* generated README files for the scripts

* remove strange ' inside the file

improve readme

add some define to better set the mirror date range

* add parameter to set mirroring duration time

* Add return after return_result

* Create test units for RSA Get Raw logs and display scripts

* Apply suggestions from code review




* remove duplicate instructions in README

* line too long... + error in README

* line too long...

* patch Ci/cd error

* patch ci/cd error

* patch ci/cd error

* Add first version of tests units for RSA Netwitness Mirroring

* patch flake8 error script RSADisplayMetasEvents

* patch flake8 error script RSAGetRawLog

* patch flake8 error integration RSANetWitness 11_5

* Add get_modified_remote_data_command unit test

* Add unit test for command get_modified_remote_data_command

* Fix function name

* Fix flake8 issue

* Remove unused class

* docker version + ignore secret fake ip

* Delete get-pip.py

* Remove unused test files

* format updates

* pre-commit and format updates

* Apply suggestions from code review



* add doc string to get_mapping_fields_command

* Fix bracket error

* Add 2 test functions to check update_remote_system_command

* remove possible duplicate changed inc id

* Fix minor error in return tests functions

* Add units tests to get remote data command

* rename metasevents to rsametasevents

* flake8 space missing

* default config value not set correctly

* Update unit test of get_modified_remote_data_command

* Add unit test for expired and non expired incident

* remove unused line

* update release note

* update release note

* update release note

* change comparaison for pulling new alert

* Update Packs/RSANetWitness_v11_1/Integrations/RSANetWitnessv115/RSANetWitnessv115_test.py



* remove comment in the release note

* Apply suggestions from code review



* Resolv issue of unit tests

* updated the release notes and pack-ignore

* pre-commit updates

* Fix all suggestions

* Fix flake8 errors

* Add mirrored object in test_get_modified_remote_data_command_from_timestamp

* patch flake8 error

* added = in get_remote_data_command

* patch unit test

* Fix token string in incident old clean test

* Fix indent

* Fix indent

* patch flake8

* updated the release notes

* pre-commit changes

* added the file CONTRIBUTORS.json

---------

Co-authored-by: Pierre <Winultimatum@users.noreply.github.com>
Co-authored-by: Pierre SOLER <9917674+Winultimatum@users.noreply.github.com>
Co-authored-by: RotemAmit <ramit@paloaltonetworks.com>
Co-authored-by: Sébastien Guisnet <sebastien.guisnet@nucleon-security.com>
Co-authored-by: Sébastien Guisnet <sebastien.guisnet@gmail.com>
Co-authored-by: ShirleyDenkberg <62508050+ShirleyDenkberg@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Community Contribution Form Filled Whether contribution form filled or not. Contribution Thank you! Contributions are always welcome! docs-approved External PR ready-for-instance-test In contribution PRs, this label will cause a trigger of a build with a modified pack from the PR. Security Review Xsoar Support Level Indicates that the contribution is for XSOAR supported pack
Projects
None yet
8 participants