[Rapid7] Initial Release for the Rapid7 #4904

rajvi-ptl · 2022-12-28T09:57:49Z

What does this PR do?

Generated the skeleton of the Rapid7 integration package.
Added data streams.
Added data collection logic for all the data streams.
Added the ingest pipeline for all the data streams.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files.
Added dashboards and visualizations.
Added detection rules for correlation.
Added test for pipeline for all the data streams.
Added system test cases for all the data streams.

Integration release checklist

This checklist is intended for integrations maintainers to ensure consistency
when creating or updating a Package, Module or Dataset for an Integration.

All changes

Change follows the contributing guidelines
Supported versions of the monitoring target is documented
Supported operating systems are documented (if applicable)
Integration or System tests exist
Documentation exists
Fields follow ECS and naming conventions
At least a manual test with ES / Kibana / Agent has been performed.
Required Kibana version set to: ^8.4.0

New Package

Screenshot of the "Add Integration" page on Fleet added

Dashboards changes

Dashboards exists
Screenshots added or updated
Datastream filters added to visualizations

Log dataset changes

Pipeline tests exist (if applicable)
Generated output for at least 1 log file exists
Sample event (sample_event.json) exists

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/ti_rapid7 directory.
Run the following command to run tests.

elastic-package test

Screenshots

cla-checker-service · 2022-12-28T09:57:52Z

💚 CLA has been signed

elasticmachine · 2022-12-28T10:01:07Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-02-08T08:01:27.525+0000
Duration: 17 min 34 sec

Test stats 🧪

Test	Results
Failed	0
Passed	23
Skipped	0
Total	23

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

CyberTaoFlow · 2023-01-17T21:52:11Z

👍

elasticmachine · 2023-01-18T08:53:21Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

jamiehynds · 2023-01-18T08:55:59Z

@rajvi-crest can you confirm which Rapid7 product this integration relates to? I'm guessing Threat Command? I'd recommend adjusting the name of the integration to include the product name. We're about to work on an InsightVM integration too, and want to ensure there's no confusion.

If the integration is focused on Threat Intel, we should also consider populating the log-ti-* data stream, to ensure the IOCs can be leveraged by out out-of-the-box threat intel capabilities.

rajvi-ptl · 2023-01-18T13:30:38Z

@rajvi-crest can you confirm which Rapid7 product this integration relates to? I'm guessing Threat Command? I'd recommend adjusting the name of the integration to include the product name. We're about to work on an InsightVM integration too, and want to ensure there's no confusion.

If the integration is focused on Threat Intel, we should also consider populating the log-ti-* data stream, to ensure the IOCs can be leveraged by out out-of-the-box threat intel capabilities.

Yes @jamiehynds, This integration package is created for the Rapid7 Threat Command product which collects IOCs, Alerts and Vulnerabilities.

I understand your concern regarding the package name and we are planning to change ti_rapid7 with the ti_rapid7_threat_command. What do you think? We will also change the README and titles with the same wherever applicable.

This integration package is intended to collect threat intel data and will store collected data to the logs-ti_rapid7.<data-stream>-default data-stream. Also, we have tried to map collected data as per the ECS schema and followed all the best practices as per the other threat intel integrations. Hence, I think IOCs collected from the Rapid7 platform will be leveraged by the threat intel capabilities.

We will accommodate all the suggested changes once the entire PR gets reviewed by the team. I hope that would be fine.

efd6 · 2023-01-19T00:06:51Z

/test

elasticmachine · 2023-01-19T00:22:49Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`3/3`)	💚
Files	100.0% (`3/3`)	💚
Classes	100.0% (`3/3`)	💚
Methods	88.372% (`38/43`)	👍 55.039
Lines	94.245% (`868/921`)	👎 -5.755
Conditionals	100.0% (`0/0`)	💚

efd6 · 2023-01-23T05:50:19Z

packages/ti_rapid7/_dev/build/docs/README.md

+
+## Overview
+
+The [Rapid7](https://www.rapid7.com/) integration allows users to retrieve `IOCs (Indicator of Compromises)`, organization-specific `Threat Command alerts`, and `CVEs (Common Vulnerabilities and Exposures)`. Furthermore, the correlation between data collected from the Rapid7 platform (IOCs and CVEs) and the user's environment helps to identify threats. Rapid7 platform gives protectors the tools and clarity they need to assess their attack surface, detect suspicious behavior, and respond and remediate quickly with intelligent automation.


ISTM that the items here that are code formatted should probably rather be either bold or italic.

Also some (but not all) below.

packages/ti_rapid7/_dev/build/docs/README.md

packages/ti_rapid7/_dev/deploy/docker/transform/ti_rapid7_ioc_rule_transform/fields/ecs.yml

...rapid7/_dev/deploy/docker/transform/ti_rapid7_unique_cve_transform/fields/overridden-ecs.yml

efd6 · 2023-01-23T06:17:38Z

packages/ti_rapid7/data_stream/alert/_dev/test/pipeline/test-common-config.yml

+dynamic_fields:
+  event.ingested: ".*"


This is not needed, we don't add event.ingested in the pipeline.

efd6 · 2023-01-23T06:32:40Z

packages/ti_rapid7/data_stream/ioc/elasticsearch/ingest_pipeline/default.yml

+      if: ctx.json?.type == 'Urls'
+  - script:
+      lang: painless
+      if: ctx.json.type != null


Suggested change

if: ctx.json.type != null

if: ctx.json?.type != null

efd6 · 2023-01-23T06:34:13Z

packages/ti_rapid7/data_stream/vulnerability/_dev/test/pipeline/test-common-config.yml

+dynamic_fields:
+  event.ingested: ".*"


efd6 · 2023-01-23T06:34:58Z

packages/ti_rapid7/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml

+      target_field: json
+      ignore_failure: true
+  - drop:
+      if: ctx.json?.content != null && ctx.json?.content?.isEmpty()


Suggested change

if: ctx.json?.content != null && ctx.json?.content?.isEmpty()

if: ctx.json?.content != null && ctx.json.content.isEmpty()

efd6 · 2023-01-23T06:35:15Z

packages/ti_rapid7/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml

+  - set:
+      field: '@timestamp'
+      copy_from: rapid7.vulnerability.update_date
+      on_failure:
+        - set:
+            field: '@timestamp'
+            copy_from: _ingest.timestamp


Move this up.

packages/ti_rapid7/data_stream/vulnerability/fields/overridden-ecs.yml

efd6 · 2023-02-08T00:13:43Z

/test

efd6

Please separate into different commits wholesale renames from line based changes to ease review follow-ups, less important but also helpful if you can separate large scale ~mechanical changes into their own commit (for example the rapid7 -> rapid7.tc changes).

efd6 · 2023-02-08T00:22:40Z

...d/_dev/deploy/docker/transform/ti_rapid7_threat_command_cve_rule_transform/fields/fields.yml

@@ -1,4 +1,4 @@
- name: rapid7.vulnerability
+- name: rapid7.tc.vulnerability


Why is this sub-namespaced rather than rapid7_tc.vulnerability or rapid7_threat_command.vulnerability? (similar throughout)

Will there be future rapid7 packages that are not "Threat Command"?

Yes, the elastic security team is currently working on the InsightVM Integration of Rapid7 as @jamiehynds mentioned here. So we thought to keep the field name as rapid7.tc.vulnerability because both products are of Rapid7 and we can differentiate the field name between their products. Let me know your thoughts on this.

Yeah, OK. Thanks.

efd6 · 2023-02-08T00:38:04Z

.github/CODEOWNERS

@@ -185,6 +185,7 @@
 /packages/ti_cybersixgill @elastic/security-external-integrations
 /packages/ti_misp @elastic/security-external-integrations
 /packages/ti_otx @elastic/security-external-integrations
+/packages/ti_rapid7 @elastic/security-external-integrations


Suggested change

/packages/ti_rapid7 @elastic/security-external-integrations

/packages/ti_rapid7_threat_command @elastic/security-external-integrations

rajvi-ptl · 2023-02-08T06:02:57Z

Please separate into different commits wholesale renames from line based changes to ease review follow-ups, less important but also helpful if you can separate large scale ~mechanical changes into their own commit (for example the rapid7 -> rapid7.tc changes).

Sure @efd6, I see there are only name changes in the fields are the major ones and a few minor changes along with this existing commit. Let me know if you still want to split the existing commit. But definitely going forwards we'll keep this in mind and keep the different code changes in different commits for making the review process easy .

efd6 · 2023-02-08T06:32:13Z

No need to split what is done here, I was just thinking for the future. Thank you.

efd6 · 2023-02-08T08:01:12Z

/test

rajvi-ptl · 2023-02-08T13:22:59Z

@efd6, We have addressed all the review comments. Hence, can you please approve and merge this PR?

efd6

Thanks

elasticmachine · 2023-02-08T23:33:13Z

Package ti_rapid7_threat_command - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=ti_rapid7_threat_command

rajvi-patel-22 added 2 commits December 28, 2022 12:40

Initial release for ti_rapid7

03c32b2

Update ecs version

0d46825

rajvi-ptl changed the title ~~Package ti rapid7~~ [Rapid7] Initial Release for the Rapid7 Dec 28, 2022

rajvi-patel-22 added 2 commits December 28, 2022 16:01

Update the changelog entry and codeowners

c695727

CLA-Check

1709c9d

jamiehynds requested a review from a team January 10, 2023 09:11

jamiehynds added Team:Security-External Integrations New Integration labels Jan 18, 2023

efd6 reviewed Jan 23, 2023

View reviewed changes

jamiehynds mentioned this pull request Jan 24, 2023

[Fleet] Support installing transforms without granting kibana_system index privileges elastic/kibana#137278

Open

Changes as per review comments

5956e30

rajvi-ptl requested review from efd6 February 6, 2023 11:59

efd6 reviewed Feb 8, 2023

View reviewed changes

Change the package name in codeowners

54dbb26

efd6 approved these changes Feb 8, 2023

View reviewed changes

efd6 merged commit 437d57e into elastic:main Feb 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Rapid7] Initial Release for the Rapid7 #4904

[Rapid7] Initial Release for the Rapid7 #4904

rajvi-ptl commented Dec 28, 2022

cla-checker-service bot commented Dec 28, 2022 •

edited

Loading

elasticmachine commented Dec 28, 2022 •

edited

Loading

Build stats

Test stats 🧪

CyberTaoFlow commented Jan 17, 2023

elasticmachine commented Jan 18, 2023

jamiehynds commented Jan 18, 2023

rajvi-ptl commented Jan 18, 2023

efd6 commented Jan 19, 2023

elasticmachine commented Jan 19, 2023 •

edited

Loading

efd6 Jan 23, 2023

efd6 Jan 23, 2023

efd6 Jan 23, 2023

efd6 Jan 23, 2023

efd6 Jan 23, 2023

efd6 Jan 23, 2023

efd6 commented Feb 8, 2023

efd6 left a comment

efd6 Feb 8, 2023

rajvi-ptl Feb 8, 2023

efd6 Feb 8, 2023

efd6 Feb 8, 2023

rajvi-ptl commented Feb 8, 2023

efd6 commented Feb 8, 2023

efd6 commented Feb 8, 2023

rajvi-ptl commented Feb 8, 2023

efd6 left a comment

elasticmachine commented Feb 8, 2023


		## Overview

		The [Rapid7](https://www.rapid7.com/) integration allows users to retrieve `IOCs (Indicator of Compromises)`, organization-specific `Threat Command alerts`, and `CVEs (Common Vulnerabilities and Exposures)`. Furthermore, the correlation between data collected from the Rapid7 platform (IOCs and CVEs) and the user's environment helps to identify threats. Rapid7 platform gives protectors the tools and clarity they need to assess their attack surface, detect suspicious behavior, and respond and remediate quickly with intelligent automation.

	if: ctx.json?.content != null && ctx.json?.content?.isEmpty()
	if: ctx.json?.content != null && ctx.json.content.isEmpty()

		@@ -1,4 +1,4 @@
		- name: rapid7.vulnerability
		- name: rapid7.tc.vulnerability

	/packages/ti_rapid7 @elastic/security-external-integrations
	/packages/ti_rapid7_threat_command @elastic/security-external-integrations

[Rapid7] Initial Release for the Rapid7 #4904

[Rapid7] Initial Release for the Rapid7 #4904

Conversation

rajvi-ptl commented Dec 28, 2022

What does this PR do?

Integration release checklist

All changes

New Package

Dashboards changes

Log dataset changes

How to test this PR locally

Screenshots

cla-checker-service bot commented Dec 28, 2022 • edited Loading

elasticmachine commented Dec 28, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

CyberTaoFlow commented Jan 17, 2023

elasticmachine commented Jan 18, 2023

jamiehynds commented Jan 18, 2023

rajvi-ptl commented Jan 18, 2023

efd6 commented Jan 19, 2023

elasticmachine commented Jan 19, 2023 • edited Loading

🌐 Coverage report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

efd6 commented Feb 8, 2023

efd6 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rajvi-ptl commented Feb 8, 2023

efd6 commented Feb 8, 2023

efd6 commented Feb 8, 2023

rajvi-ptl commented Feb 8, 2023

efd6 left a comment

Choose a reason for hiding this comment

elasticmachine commented Feb 8, 2023

cla-checker-service bot commented Dec 28, 2022 •

edited

Loading

elasticmachine commented Dec 28, 2022 •

edited

Loading

elasticmachine commented Jan 19, 2023 •

edited

Loading