[SentinelOne_Cloud_Funnel] Initial release for the SentinelOne Cloud Funnel #6386

piyush-elastic · 2023-05-31T08:52:57Z

What does this PR do?

Updated data collection logic for the event data stream.
Updated the ingest pipeline for the event data stream.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files.
Added dashboards and visualizations.
Updated test for pipeline for the event data stream.
Updated system test cases for the event data stream.

Integration release checklist

This checklist is intended for integrations maintainers to ensure consistency
when creating or updating a Package, Module or Dataset for an Integration.

All changes

Change follows the contributing guidelines
Supported versions of the monitoring target is documented
Supported operating systems are documented (if applicable)
Integration or System tests exist
Documentation exists
Fields follow ECS and naming conventions
At least a manual test with ES / Kibana / Agent has been performed.
Required Kibana version set to: ^8.3.0

New Package

Screenshot of the "Add Integration" page on Fleet added

Dashboards changes

Dashboards exists
Screenshots added or updated
Datastream filters added to visualizations

Log dataset changes

Pipeline tests exist (if applicable)
Generated output for at least 1 log file exists
Sample event (sample_event.json) exists

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/sentinel_one_cloud_funnel directory.
Run the following command to run tests.

elastic-package test

Related issues

Closes SentinelOne Cloud Funnel #6145

Screenshots

2023/05/29 13:05:25 DEBUG Error: can't check latest release, GET https://api.github.com/repos/elastic/elastic-package/releases/latest: 403 API rate limit exceeded for 117.217.127.213. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.) [rate reset in 3m05s]
Run test suite for the package
Run pipeline tests for the package
--- Test results for package: sentinel_one_cloud_funnel - START ---
╭───────────────────────────┬─────────────┬───────────┬────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE                   │ DATA STREAM │ TEST TYPE │ TEST NAME                              │ RESULT │ TIME ELAPSED │
├───────────────────────────┼─────────────┼───────────┼────────────────────────────────────────┼────────┼──────────────┤
│ sentinel_one_cloud_funnel │ event       │ pipeline  │ test-command-script.log                │ PASS   │ 190.869803ms │
│ sentinel_one_cloud_funnel │ event       │ pipeline  │ test-cross-process.log                 │ PASS   │  27.770119ms │
│ sentinel_one_cloud_funnel │ event       │ pipeline  │ test-dns.log                           │ PASS   │  34.706417ms │
│ sentinel_one_cloud_funnel │ event       │ pipeline  │ test-file.log                          │ PASS   │  25.355196ms │
│ sentinel_one_cloud_funnel │ event       │ pipeline  │ test-indicator.log                     │ PASS   │  27.550508ms │
│ sentinel_one_cloud_funnel │ event       │ pipeline  │ test-login.log                         │ PASS   │  14.970616ms │
│ sentinel_one_cloud_funnel │ event       │ pipeline  │ test-module.log                        │ PASS   │  15.957612ms │
│ sentinel_one_cloud_funnel │ event       │ pipeline  │ test-network-action.log                │ PASS   │  18.613347ms │
│ sentinel_one_cloud_funnel │ event       │ pipeline  │ test-process.log                       │ PASS   │  14.139338ms │
│ sentinel_one_cloud_funnel │ event       │ pipeline  │ test-registry.log                      │ PASS   │  14.955223ms │
│ sentinel_one_cloud_funnel │ event       │ pipeline  │ test-scheduled-task.log                │ PASS   │  12.350956ms │
│ sentinel_one_cloud_funnel │ event       │ pipeline  │ test-threat-intelligence-indicator.log │ PASS   │   16.33078ms │
│ sentinel_one_cloud_funnel │ event       │ pipeline  │ test-url.log                           │ PASS   │  10.554792ms │
╰───────────────────────────┴─────────────┴───────────┴────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: sentinel_one_cloud_funnel - END   ---
Done
Run static tests for the package
--- Test results for package: sentinel_one_cloud_funnel - START ---
╭───────────────────────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE                   │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├───────────────────────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ sentinel_one_cloud_funnel │ event       │ static    │ Verify sample_event.json │ PASS   │ 153.202407ms │
╰───────────────────────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: sentinel_one_cloud_funnel - END   ---
Done
Run system tests for the package
--- Test results for package: sentinel_one_cloud_funnel - START ---
No test results
--- Test results for package: sentinel_one_cloud_funnel - END   ---
Done
Run asset tests for the package
2023/05/29 13:06:57 DEBUG installing package...
2023/05/29 13:06:57 DEBUG POST https://127.0.0.1:5601/api/fleet/epm/packages/sentinel_one_cloud_funnel-0.1.0
--- Test results for package: sentinel_one_cloud_funnel - START ---
╭───────────────────────────┬─────────────┬───────────┬────────────────────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE                   │ DATA STREAM │ TEST TYPE │ TEST NAME                                                                          │ RESULT │ TIME ELAPSED │
├───────────────────────────┼─────────────┼───────────┼────────────────────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ sentinel_one_cloud_funnel │             │ asset     │ dashboard sentinel_one_cloud_funnel-008e94d0-facc-11ed-9768-a973b08e03b4 is loaded │ PASS   │      1.793µs │
│ sentinel_one_cloud_funnel │             │ asset     │ dashboard sentinel_one_cloud_funnel-42ecf8a0-fae3-11ed-a771-5bdf1dfe027b is loaded │ PASS   │        162ns │
│ sentinel_one_cloud_funnel │             │ asset     │ dashboard sentinel_one_cloud_funnel-724f8d70-f966-11ed-b51d-ef5658e04999 is loaded │ PASS   │        168ns │
│ sentinel_one_cloud_funnel │             │ asset     │ dashboard sentinel_one_cloud_funnel-afecde30-f966-11ed-b51d-ef5658e04999 is loaded │ PASS   │        190ns │
│ sentinel_one_cloud_funnel │             │ asset     │ dashboard sentinel_one_cloud_funnel-c90a2bc0-fa2e-11ed-b63a-6f5ce39a3d73 is loaded │ PASS   │        190ns │
│ sentinel_one_cloud_funnel │             │ asset     │ dashboard sentinel_one_cloud_funnel-dd209080-f967-11ed-b51d-ef5658e04999 is loaded │ PASS   │        220ns │
│ sentinel_one_cloud_funnel │             │ asset     │ search sentinel_one_cloud_funnel-6a9c9640-fa2e-11ed-b63a-6f5ce39a3d73 is loaded    │ PASS   │        263ns │
│ sentinel_one_cloud_funnel │ event       │ asset     │ index_template logs-sentinel_one_cloud_funnel.event is loaded                      │ PASS   │        643ns │
│ sentinel_one_cloud_funnel │ event       │ asset     │ ingest_pipeline logs-sentinel_one_cloud_funnel.event-0.1.0 is loaded               │ PASS   │        179ns │
╰───────────────────────────┴─────────────┴───────────┴────────────────────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: sentinel_one_cloud_funnel - END   ---
Done

elasticmachine · 2023-05-31T08:58:25Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-06-20T07:14:55.716+0000
Duration: 13 min 26 sec

Test stats 🧪

Test	Results
Failed	0
Passed	23
Skipped	0
Total	23

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-05-31T09:24:05Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`14/14`)	💚 2.936
Classes	100.0% (`14/14`)	💚 2.936
Methods	93.651% (`59/63`)	👍 0.914
Lines	89.886% (`4008/4459`)	👎 -1.938
Conditionals	100.0% (`0/0`)	💚

jamiehynds · 2023-05-31T09:52:37Z

packages/sentinel_one_cloud_funnel/_dev/build/docs/README.md

@@ -0,0 +1,71 @@
+# SentinelOne Cloud Funnel
+


Reworded slightly: This Sentinel One Cloud Funnel integration enables your security team to securely stream XDR data to Elastic Security, via Amazon S3. When integrated with Elastic Security, this valuable data can be leveraged within Elastic for threat protection, detection and incident response.

jamiehynds · 2023-05-31T09:57:42Z

packages/sentinel_one_cloud_funnel/_dev/build/docs/README.md

+
+## Requirements
+
+Elastic Agent must be installed. For more information, refer to the link [here](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html).


Can we expand on this? Users are sometimes confused on the need for agent, what it does and where it should be deployed. We should make it clearer that Elastic Agent is required to stream data from the S3 bucket and ship the data to Elastic, whereby the events will then be processing via the integration's ingest pipelines. Should also provide some guidance as to where agent should be installed, and state it can run on a host, a container, etc.

packages/sentinel_one_cloud_funnel/_dev/build/docs/README.md

jamiehynds · 2023-05-31T10:01:40Z

packages/sentinel_one_cloud_funnel/manifest.yml

+type: integration
+categories: ["security", "edr_xdr"]
+conditions:
+  kibana.version: ^8.3.0


Can we set the version to ^8.7.1, inline with our current Sentinel One integration?

jamiehynds · 2023-05-31T10:02:18Z

packages/sentinel_one_cloud_funnel/_dev/build/docs/README.md

+## Requirements
+
+Elastic Agent must be installed. For more information, refer to the link [here](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html).
+The minimum **kibana.version** required is **8.3.0**.


Can we adjust to 8.7.1, inline with our current Sentinel One integration?

elasticmachine · 2023-06-06T11:04:47Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

efd6 · 2023-06-08T00:29:40Z

packages/sentinel_one_cloud_funnel/LICENSE.txt

Do we need this? We have the same license in the root of the repo.

@efd6 As per new version of elastic-package license file is getting created at package creation level reference - elastic/elastic-package#898 and before that we had been using old elastic-package version. Had discussion on same before link

Yep, I know mechanically why it's there, but I'm not convinced that it's needed. It would be good to get another view on this.

packages/sentinel_one_cloud_funnel/_dev/build/docs/README.md

...sentinel_one_cloud_funnel/data_stream/event/_dev/test/pipeline/test-module.log-expected.json

..._one_cloud_funnel/data_stream/event/_dev/test/pipeline/test-network-action.log-expected.json

...entinel_one_cloud_funnel/data_stream/event/_dev/test/pipeline/test-process.log-expected.json

jamiehynds

LGTM, thanks for implementing my suggested changes.

...sentinel_one_cloud_funnel/data_stream/event/_dev/test/pipeline/test-module.log-expected.json

…ntinel_one_cloud_funnel-0.1.0

efd6

Please address these unresolved comments:

efd6 · 2023-06-18T21:58:18Z

packages/sentinel_one_cloud_funnel/changelog.yml

+  changes:
+    - description: Initial release.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/1


Suggested change

link: https://github.com/elastic/integrations/pull/1

link: https://github.com/elastic/integrations/pull/6386

elasticmachine · 2023-06-19T07:55:38Z

💚 Build Succeeded

Buildkite Build
Commit: a5d73dc

History

💚 Build #135 succeeded 2cd904c
💚 Build #133 succeeded 99bcffc
💚 Build #28 succeeded 84a9e8b
💔 Build #22 failed 7917c1b
💔 Build #13 failed a3e19d4
💔 Build #12 failed 2df3cec

efd6

Thanks. Only two remaining unresolved queries.

efd6 · 2023-06-19T21:55:29Z

packages/sentinel_one_cloud_funnel/_dev/build/docs/README.md

+You can run Elastic Agent inside a container, either with Fleet Server or standalone. Docker images for all versions of Elastic Agent are available from the Elastic Docker registry, and we provide deployment manifests for running on Kubernetes.
+
+There are some minimum requirements for running Elastic Agent and for more information, refer to the link [here](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html).
+The minimum **kibana.version** required is **8.7.1**.


That's not quite my question; this particular bit of information is quite important and so it may benefit from being in its own paragraph to avoid eye-glide. Adding a newline before this would do that.

packages/sentinel_one_cloud_funnel/data_stream/event/elasticsearch/ingest_pipeline/default.yml

efd6

Thanks

elasticmachine · 2023-06-20T09:58:53Z

Package sentinel_one_cloud_funnel - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=sentinel_one_cloud_funnel

piyush-elastic added 2 commits May 31, 2023 14:07

Initial release

06a930b

Update the changelog entry

f469fa6

Add codeowners

51b3a5d

jamiehynds reviewed May 31, 2023

View reviewed changes

packages/sentinel_one_cloud_funnel/_dev/build/docs/README.md Show resolved Hide resolved

jamiehynds reviewed May 31, 2023

View reviewed changes

Resolve review comments on README.md

709f758

piyush-elastic requested a review from jamiehynds June 2, 2023 07:31

jamiehynds added the Team:Security-External Integrations label Jun 6, 2023

jamiehynds requested a review from a team June 6, 2023 11:04

jamiehynds added the New Integration label Jun 6, 2023

efd6 reviewed Jun 8, 2023

View reviewed changes

Resolve review comments

2814e1b

piyush-elastic requested a review from efd6 June 9, 2023 07:30

jamiehynds approved these changes Jun 9, 2023

View reviewed changes

efd6 requested changes Jun 9, 2023

View reviewed changes

...sentinel_one_cloud_funnel/data_stream/event/_dev/test/pipeline/test-module.log-expected.json Outdated Show resolved Hide resolved

Resolve review comments

a3e19d4

piyush-elastic force-pushed the sentinel_one_cloud_funnel-0.1.0 branch from 2df3cec to a3e19d4 Compare June 13, 2023 07:33

piyush-elastic added 2 commits June 13, 2023 14:55

Resolve review comments

7917c1b

Merge branch 'main' of github.com:piyush-elastic/integrations into se…

84a9e8b

…ntinel_one_cloud_funnel-0.1.0

piyush-elastic requested a review from efd6 June 13, 2023 11:14

Resolve review comments and removed license.txt

2cd904c

piyush-elastic force-pushed the sentinel_one_cloud_funnel-0.1.0 branch from 99bcffc to 2cd904c Compare June 15, 2023 11:31

efd6 reviewed Jun 18, 2023

View reviewed changes

Review comments resolved

a5d73dc

piyush-elastic requested a review from efd6 June 19, 2023 08:24

efd6 reviewed Jun 19, 2023

View reviewed changes

Review comments resolved

8149a49

efd6 approved these changes Jun 20, 2023

View reviewed changes

P1llus merged commit 26a3902 into elastic:main Jun 20, 2023
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SentinelOne_Cloud_Funnel] Initial release for the SentinelOne Cloud Funnel #6386

[SentinelOne_Cloud_Funnel] Initial release for the SentinelOne Cloud Funnel #6386

piyush-elastic commented May 31, 2023 •

edited

elasticmachine commented May 31, 2023 •

edited

Build stats

Test stats 🧪

elasticmachine commented May 31, 2023 •

edited

jamiehynds May 31, 2023

jamiehynds May 31, 2023

jamiehynds May 31, 2023

jamiehynds May 31, 2023

elasticmachine commented Jun 6, 2023

efd6 Jun 8, 2023

piyush-elastic Jun 8, 2023

efd6 Jun 8, 2023

jamiehynds left a comment

efd6 left a comment •

edited

efd6 Jun 18, 2023

elasticmachine commented Jun 19, 2023

efd6 left a comment

efd6 Jun 19, 2023

efd6 left a comment

elasticmachine commented Jun 20, 2023


		## Requirements

		Elastic Agent must be installed. For more information, refer to the link [here](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html).

	link: https://github.com/elastic/integrations/pull/1
	link: https://github.com/elastic/integrations/pull/6386

[SentinelOne_Cloud_Funnel] Initial release for the SentinelOne Cloud Funnel #6386

[SentinelOne_Cloud_Funnel] Initial release for the SentinelOne Cloud Funnel #6386

Conversation

piyush-elastic commented May 31, 2023 • edited

What does this PR do?

Integration release checklist

All changes

New Package

Dashboards changes

Log dataset changes

How to test this PR locally

Related issues

Screenshots

elasticmachine commented May 31, 2023 • edited

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented May 31, 2023 • edited

🌐 Coverage report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Jun 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jamiehynds left a comment

Choose a reason for hiding this comment

efd6 left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Jun 19, 2023

💚 Build Succeeded

History

efd6 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

efd6 left a comment

Choose a reason for hiding this comment

elasticmachine commented Jun 20, 2023

piyush-elastic commented May 31, 2023 •

edited

elasticmachine commented May 31, 2023 •

edited

elasticmachine commented May 31, 2023 •

edited

efd6 left a comment •

edited