[amazon_security_lake] Initial Release for Amazon Security Lake #7176

janvi-elastic · 2023-07-28T09:18:49Z

What does this PR do?

Generated the skeleton of the Amazon Security Lake integration package.
Added data stream.
Added data collection logic for the data stream.
Added the ingest pipeline for the data stream.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files.
Added dashboards and visualizations.
Added test for pipeline for the data stream.

Integration release checklist

This checklist is intended for integrations maintainers to ensure consistency
when creating or updating a Package, Module or Dataset for an Integration.

All changes

Change follows the contributing guidelines
Supported versions of the monitoring target is documented
Supported operating systems are documented (if applicable)
Integration or System tests exist
Documentation exists
Fields follow ECS and naming conventions
At least a manual test with ES / Kibana / Agent has been performed.
Required Kibana version set to: ^8.9.0

New Package

Screenshot of the "Add Integration" page on Fleet added

Dashboards changes

Dashboards exists
Screenshots added or updated
Datastream filters added to visualizations

Log dataset changes

Pipeline tests exist (if applicable)
Generated output for at least 1 log file exists
Sample event (sample_event.json) exists

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/amazon_security_lake directory.
Run the following command to run tests.

elastic-package test

Related issues

Closes AWS Security Lake #5286

Automated Test

Run test suite for the package
Run system tests for the package
--- Test results for package: amazon_security_lake - START ---
No test results
--- Test results for package: amazon_security_lake - END   ---
Done
Run asset tests for the package
2023/07/27 17:25:57 DEBUG installing package...
2023/07/27 17:25:57 DEBUG GET https://127.0.0.1:5601/api/status
2023/07/27 17:25:57 DEBUG Build directory: /root/integrations/build/packages/amazon_security_lake/0.1.0
2023/07/27 17:25:57 DEBUG Clear target directory (path: /root/integrations/build/packages/amazon_security_lake/0.1.0)
2023/07/27 17:25:57 DEBUG Copy package content (source: /root/integrations/packages/amazon_security_lake)
2023/07/27 17:25:57 DEBUG Copy license file if needed
2023/07/27 17:25:57  INFO License text found in "/root/integrations/LICENSE.txt" will be included in package
2023/07/27 17:25:57 DEBUG Encode dashboards
2023/07/27 17:25:57 DEBUG Resolve external fields
2023/07/27 17:25:57 DEBUG Package has external dependencies defined
2023/07/27 17:25:58 DEBUG data_stream/event/fields/base-fields.yml: source file hasn't been changed
2023/07/27 17:25:58 DEBUG data_stream/event/fields/beats.yml: source file hasn't been changed
2023/07/27 17:25:58 DEBUG data_stream/event/fields/ecs.yml: source file hasn't been changed
2023/07/27 17:25:58 DEBUG data_stream/event/fields/fields.yml: source file hasn't been changed
2023/07/27 17:25:58  INFO Import ECS mappings into the built package (technical preview)
2023/07/27 17:25:58 DEBUG Build zipped package
2023/07/27 17:25:58 DEBUG Compress using archiver.Zip (destination: /root/integrations/build/packages/amazon_security_lake-0.1.0.zip)
2023/07/27 17:25:58 DEBUG Create work directory for archiving: /tmp/elastic-package-950251527/amazon_security_lake-0.1.0
2023/07/27 17:25:58 DEBUG Skip validation of the built .zip package
2023/07/27 17:25:58 DEBUG POST https://127.0.0.1:5601/api/fleet/epm/packages
2023/07/27 17:26:00 DEBUG removing package...
2023/07/27 17:26:00 DEBUG DELETE https://127.0.0.1:5601/api/fleet/epm/packages/amazon_security_lake-0.1.0
--- Test results for package: amazon_security_lake - START ---
╭──────────────────────┬─────────────┬───────────┬───────────────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE              │ DATA STREAM │ TEST TYPE │ TEST NAME                                                                     │ RESULT │ TIME ELAPSED │
├──────────────────────┼─────────────┼───────────┼───────────────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ amazon_security_lake │             │ asset     │ dashboard amazon_security_lake-0d2d7a60-2472-11ee-8d80-89e82659e0f1 is loaded │ PASS   │      1.317µs │
│ amazon_security_lake │             │ asset     │ dashboard amazon_security_lake-15b6e140-24a3-11ee-bb84-975fc16e8386 is loaded │ PASS   │        653ns │
│ amazon_security_lake │             │ asset     │ dashboard amazon_security_lake-1bbac7b0-2632-11ee-a94e-bfa24df19b15 is loaded │ PASS   │        680ns │
│ amazon_security_lake │             │ asset     │ dashboard amazon_security_lake-41b73270-25fe-11ee-983a-17fb20a3b25d is loaded │ PASS   │        652ns │
│ amazon_security_lake │             │ asset     │ dashboard amazon_security_lake-ed18e3a0-2565-11ee-be5c-17edc959116c is loaded │ PASS   │        671ns │
│ amazon_security_lake │             │ asset     │ dashboard amazon_security_lake-f21df8e0-249d-11ee-aa05-4dd9349682f3 is loaded │ PASS   │        707ns │
│ amazon_security_lake │             │ asset     │ search amazon_security_lake-16a0aa00-26dd-11ee-a94e-bfa24df19b15 is loaded    │ PASS   │        649ns │
│ amazon_security_lake │             │ asset     │ search amazon_security_lake-81902d50-2538-11ee-9f72-193490b86197 is loaded    │ PASS   │        760ns │
│ amazon_security_lake │             │ asset     │ search amazon_security_lake-93f1c2f0-262e-11ee-abb4-f9698f7e351e is loaded    │ PASS   │        655ns │
│ amazon_security_lake │             │ asset     │ search amazon_security_lake-ab4090f0-2618-11ee-983a-17fb20a3b25d is loaded    │ PASS   │        725ns │
│ amazon_security_lake │             │ asset     │ search amazon_security_lake-c2472e60-262e-11ee-a94e-bfa24df19b15 is loaded    │ PASS   │        729ns │
│ amazon_security_lake │ event       │ asset     │ index_template logs-amazon_security_lake.event is loaded                      │ PASS   │        880ns │
│ amazon_security_lake │ event       │ asset     │ ingest_pipeline logs-amazon_security_lake.event-0.1.0 is loaded               │ PASS   │        731ns │
╰──────────────────────┴─────────────┴───────────┴───────────────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: amazon_security_lake - END   ---
Done
Run pipeline tests for the package
--- Test results for package: amazon_security_lake - START ---
╭──────────────────────┬─────────────┬───────────┬────────────────────────────┬────────┬──────────────╮
│ PACKAGE              │ DATA STREAM │ TEST TYPE │ TEST NAME                  │ RESULT │ TIME ELAPSED │
├──────────────────────┼─────────────┼───────────┼────────────────────────────┼────────┼──────────────┤
│ amazon_security_lake │ event       │ pipeline  │ test-account-change.log    │ PASS   │  67.610136ms │
│ amazon_security_lake │ event       │ pipeline  │ test-api-activity.log      │ PASS   │   9.350505ms │
│ amazon_security_lake │ event       │ pipeline  │ test-authentication.log    │ PASS   │   9.987652ms │
│ amazon_security_lake │ event       │ pipeline  │ test-dns-activity.log      │ PASS   │   8.577234ms │
│ amazon_security_lake │ event       │ pipeline  │ test-network-activity.log  │ PASS   │   4.859739ms │
│ amazon_security_lake │ event       │ pipeline  │ test-security-findings.log │ PASS   │  21.435988ms │
╰──────────────────────┴─────────────┴───────────┴────────────────────────────┴────────┴──────────────╯
--- Test results for package: amazon_security_lake - END   ---
Done
Run static tests for the package
--- Test results for package: amazon_security_lake - START ---
No test results
--- Test results for package: amazon_security_lake - END   ---
Done

Screenshot

elasticmachine · 2023-07-28T09:29:06Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2023-07-28T09:33:50Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-08-03T06:21:19.855+0000
Duration: 16 min 13 sec

Test stats 🧪

Test	Results
Failed	0
Passed	19
Skipped	0
Total	19

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-07-28T09:33:59Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	70.588% (`12/17`)	👎 -26.569
Classes	70.588% (`12/17`)	👎 -26.569
Methods	56.818% (`50/88`)	👎 -35.299
Lines	39.96% (`3841/9612`)	👎 -50.724
Conditionals	100.0% (`0/0`)	💚

...e/data_stream/event/elasticsearch/ingest_pipeline/pipeline_category_application_activity.yml

andrewkroh · 2023-07-28T14:42:28Z

There are 1956 fields in the fields/*.yml files. This does not include the ECS field count because those are dynamically mapped. This means that the index.query.default_field value, which can only hold 1024, field will be incomplete. Also it will not contain ECS fields since import_mappings is used. This might lead to a degraded search experience because none of the ECS fields will be queried by default unless they are explicitly named in the query.

This is not a blocker, but we should discuss this with @P1llus when he is back. IIUC any time we use the ECS import_mappings feature then the index.query.default_field list won't contain most ECS fields.

andrewkroh

Looks good. The dashboards are nice. I did not review the ingest pipelines.

jamiehynds · 2023-07-28T14:56:32Z

packages/amazon_security_lake/_dev/build/docs/README.md

+
+## Compatibility
+
+This module follows the latest OCSF Schema Version **v1.0.0-rc.3**.


Can we highlight which OCSF classes are supported, as we're currently limited to Classes used only by the AWS Services.

Hey @jamiehynds ,
We have highlighted OCSF classes in data stream section which are supported in current integration. Do we need to also add here? Please let us know your thoughts.

jamiehynds · 2023-07-28T14:58:00Z

packages/amazon_security_lake/_dev/build/docs/README.md

+| VPC Flow Logs                       | Network Activity                                    |
+
+### **NOTE**:
+- The Amazon Security Lake integration supports events collected from [AWS services](https://docs.aws.amazon.com/security-lake/latest/userguide/internal-sources.html).


Can we also a note here that we recommend ingesting data from these AWS Services via our out-of-the-box integrations to avail of more comprehensive ECS mappings and dashboards specific to each AWS service?

..._lake/data_stream/event/elasticsearch/ingest_pipeline/pipeline_category_network_activity.yml

packages/amazon_security_lake/data_stream/event/elasticsearch/ingest_pipeline/default.yml

packages/amazon_security_lake/_dev/build/build.yml

packages/amazon_security_lake/data_stream/event/elasticsearch/ingest_pipeline/default.yml

ShourieG

LGTM

kcreddy

LGTM as my review comments are addressed. Please also address other reviews as well. Thanks!

elasticmachine · 2023-08-04T07:54:26Z

Package amazon_security_lake - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=amazon_security_lake

janvi-elastic added 2 commits July 28, 2023 14:41

Initial Release for Amazon Security Lake

b9bdd40

Update the changelog entry

5127eec

narph requested a review from ShourieG July 28, 2023 09:28

narph added the Team:Security-External Integrations label Jul 28, 2023

ShourieG reviewed Jul 28, 2023

View reviewed changes

...e/data_stream/event/elasticsearch/ingest_pipeline/pipeline_category_application_activity.yml Show resolved Hide resolved

ShourieG requested review from kcreddy and andrewkroh July 28, 2023 10:02

andrewkroh reviewed Jul 28, 2023

View reviewed changes

jamiehynds reviewed Jul 28, 2023

View reviewed changes

efd6 reviewed Jul 31, 2023

View reviewed changes

janvi-elastic requested review from jamiehynds and ShourieG July 31, 2023 06:18

kcreddy reviewed Jul 31, 2023

View reviewed changes

packages/amazon_security_lake/_dev/build/build.yml Outdated Show resolved Hide resolved

packages/amazon_security_lake/data_stream/event/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

Resolve review comments

daa1089

janvi-elastic requested review from kcreddy and efd6 August 3, 2023 06:31

ShourieG approved these changes Aug 3, 2023

View reviewed changes

kcreddy approved these changes Aug 3, 2023

View reviewed changes

narph requested a review from andrewkroh August 3, 2023 08:35

andrewkroh approved these changes Aug 3, 2023

View reviewed changes

efd6 approved these changes Aug 3, 2023

View reviewed changes

narph merged commit 90b25b0 into elastic:main Aug 4, 2023
1 check passed

andrewkroh added the Integration:amazon_security_lake Amazon Security Lake label Jul 22, 2024

andrewkroh added Crest New Integration labels Aug 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[amazon_security_lake] Initial Release for Amazon Security Lake #7176

[amazon_security_lake] Initial Release for Amazon Security Lake #7176

janvi-elastic commented Jul 28, 2023

elasticmachine commented Jul 28, 2023

elasticmachine commented Jul 28, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Jul 28, 2023 •

edited

Loading

andrewkroh commented Jul 28, 2023

andrewkroh left a comment

jamiehynds Jul 28, 2023

janvi-elastic Jul 29, 2023

jamiehynds Jul 28, 2023

ShourieG left a comment

kcreddy left a comment

elasticmachine commented Aug 4, 2023


		## Compatibility

		This module follows the latest OCSF Schema Version v1.0.0-rc.3.

[amazon_security_lake] Initial Release for Amazon Security Lake #7176

[amazon_security_lake] Initial Release for Amazon Security Lake #7176

Conversation

janvi-elastic commented Jul 28, 2023

What does this PR do?

Integration release checklist

All changes

New Package

Dashboards changes

Log dataset changes

How to test this PR locally

Related issues

Automated Test

Screenshot

elasticmachine commented Jul 28, 2023

elasticmachine commented Jul 28, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Jul 28, 2023 • edited Loading

🌐 Coverage report

andrewkroh commented Jul 28, 2023

andrewkroh left a comment

Choose a reason for hiding this comment

jamiehynds Jul 28, 2023

Choose a reason for hiding this comment

janvi-elastic Jul 29, 2023

Choose a reason for hiding this comment

jamiehynds Jul 28, 2023

Choose a reason for hiding this comment

ShourieG left a comment

Choose a reason for hiding this comment

kcreddy left a comment

Choose a reason for hiding this comment

elasticmachine commented Aug 4, 2023

elasticmachine commented Jul 28, 2023 •

edited

Loading

elasticmachine commented Jul 28, 2023 •

edited

Loading