[New Integration] Menlo Security #9315

tehbooom · 2024-03-08T16:16:40Z

Enhancement

What does this PR do?

This PR adds the Menlo Security integration for Web and DLP logs

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Screenshots

mrodm

Commenting about general settings, code owner team should also review this package

mrodm · 2024-03-11T11:25:04Z

packages/menlo/manifest.yml

@@ -0,0 +1,81 @@
+format_version: 2.5.1


Could it be used here a newer version of package-spec ? At this moment, latest version is 3.1.2.

This would also ensure there are more validation checks performed.

mrodm · 2024-03-11T11:28:25Z

packages/menlo/manifest.yml

+  elastic:
+    subscription: "basic"


Remember to add the capabilities field if this package requires it for Serverless.
For instance, other security packages define this:

integrations/packages/ded/manifest.yml

Line 18 in 0da0ea5

capabilities:

Suggested change

elastic:

subscription: "basic"

elastic:

subscription: "basic"

capabilities:

- security

This would make this package available in Serverless just for Security projects (and not for Observability projects). cc @elastic/security-service-integrations

mrodm · 2024-03-11T11:32:31Z

.github/CODEOWNERS

@@ -223,6 +223,7 @@
 /packages/m365_defender @elastic/security-service-integrations
 /packages/mattermost @elastic/security-service-integrations
 /packages/memcached @elastic/obs-infraobs-integrations
+/packages/menlo @elastic/security-external-integrations


Just double checking, it has been set as a reviewer of this PR @elastic/security-service-integrations but here it is set another team.

Which team should be used as a code owner?

I believe this was a mistake on my part selecting the wrong team. The CODEOWNERS is correct

Could you please change this to @elastic/security-service-integrations?
Our older handle used to be @elastic/security-external-integrations.

elasticmachine · 2024-03-11T12:19:46Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

kcreddy · 2024-03-15T10:50:43Z

.github/CODEOWNERS

@@ -223,6 +223,7 @@
 /packages/m365_defender @elastic/security-service-integrations
 /packages/mattermost @elastic/security-service-integrations
 /packages/memcached @elastic/obs-infraobs-integrations
+/packages/menlo @elastic/security-external-integrations


Could you please change this to @elastic/security-service-integrations?
Our older handle used to be @elastic/security-external-integrations.

packages/menlo/changelog.yml

packages/menlo/_dev/build/docs/README.md

packages/menlo/_dev/deploy/docker/config.yml

kcreddy · 2024-03-15T11:13:05Z

packages/menlo/data_stream/dlp/_dev/test/pipeline/test-dlp.log-expected.json

+                "version": "8.11.0"
+            },
+            "event": {
+                "action": "block",


Have event.type as denied and event.action as blocked
Reference: https://www.elastic.co/guide/en/ecs/8.11/ecs-allowed-values-event-type.html#ecs-event-type-denied

packages/menlo/data_stream/dlp/elasticsearch/ingest_pipeline/default.yml

packages/menlo/data_stream/web/elasticsearch/ingest_pipeline/default.yml

packages/menlo/manifest.yml

kcreddy

Thanks for clearing all the review comments. Just one last thing and we can merge 😄

kcreddy · 2024-04-11T09:20:41Z

packages/menlo/data_stream/dlp/elasticsearch/ingest_pipeline/default.yml

+  - date:
+      field: "json.event.event_time"
+      formats:
+        - "ISO8601"
+      target_field: "event.created"
+      timezone: "UTC"
+      on_failure:
+        - append:
+            field: error.message
+            value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'


When you are populating error.message with {{{_ingest.on_failure_processor_tag}}}, add tag: <tag-name> to the processor definition and also to render {{{_ingest.on_failure_processor_tag}}} inside error.message.

Example: https://github.com/elastic/integrations/blob/main/packages/ti_threatconnect/data_stream/indicator/elasticsearch/ingest_pipeline/default.yml#L77

elasticmachine · 2024-04-11T12:59:44Z

💚 Build Succeeded

Buildkite Build
Commit: a8db6fd

History

💚 Build #9971 succeeded 94b7771
💔 Build #9968 failed 09c812c
💔 Build #9949 failed b1710e8
💔 Build #9946 failed 8a0181f
💚 Build #9941 succeeded 4abe232
💚 Build #9640 succeeded bb6a3ca

elastic-sonarqube · 2024-04-11T12:59:45Z

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
87.7% 87.7% Coverage on New Code
0.0% 0.0% Duplication on New Code

See analysis details on SonarQube

kcreddy

LGTM 👍🏼
Dashboards will be a nice addition to this package next.

elasticmachine · 2024-04-11T15:35:43Z

Package menlo - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=menlo

tehbooom and others added 30 commits May 24, 2023 14:30

package setup

3900d4e

expected json for web and dlp

c8cae3e

Added title and desc for lint

ad95d42

Updated readme

0269620

ignore dynamic field

3c7d06c

Merge branch 'elastic:main' into main

d21e7f8

Changed field description to unknown

03c4523

Renamed to common

a2769ba

Updated expected events

62c782d

removed test.json for lint

6b4f8d3

Updated change log

2912e3c

added code owner

59874e0

removed license

ab0aa2f

updated git reference

44c14f3

removed dynamic fields set by fleet

54fb3b0

removed extra character

3b74bb2

Removed duplicate logs

4275d09

Unique groups

2b600b5

removed extra group_id

24822cc

Updated pipelines based on recommendations

bd23044

Changed github owner

00b77d9

Added test infrastructure

58f1ef4

Tested with Menlo Security API

be5312b

Merge branch 'elastic:main' into main

4c8ae66

Updated version

3f5080f

Updated version and syntax

ee7a48f

Updated ecs version

c9f2ea5

Syntax

d0cebe2

Added missing fields

53e6898

Removed conflicting agent fields

199e9e7

chore: updated buildkite to main

d4cdaba

mrodm reviewed Mar 11, 2024

View reviewed changes

tehbooom added the Team:Security-External Integrations Label for the Security External Integrations team label Mar 11, 2024

tehbooom removed the Team:Security-Service Integrations Security Service Integrations Team label Mar 11, 2024

jamiehynds added Team:Security-Service Integrations Security Service Integrations Team and removed Team:Security-External Integrations Label for the Security External Integrations team labels Mar 14, 2024

Merge branch 'elastic:main' into main

e028ce6

kcreddy reviewed Mar 15, 2024

View reviewed changes

tehbooom and others added 8 commits March 15, 2024 08:26

Merge branch 'elastic:main' into main

bb6a3ca

Merge branch 'elastic:main' into main

4abe232

resolved comments

2cd3147

Merge branch 'elastic:main' into main

68356c8

Merge branch 'main' of github.com:tehbooom/integrations

8a0181f

updated github owner in manifest

b1710e8

test with package-spec v3.0.0

09c812c

changed kibana version to 8.13.0 and package-spec to v3.0.2

94b7771

tehbooom requested review from kcreddy, elasticitservice and a team and removed request for elasticitservice April 3, 2024 12:07

kcreddy reviewed Apr 11, 2024

View reviewed changes

fix: date tag on failure

a8db6fd

tehbooom requested a review from a team April 11, 2024 13:06

kcreddy approved these changes Apr 11, 2024

View reviewed changes

kcreddy merged commit 8d1d0ad into elastic:main Apr 11, 2024
5 checks passed

chrisberkhout added the Integration:Menlo Security label May 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[New Integration] Menlo Security #9315

[New Integration] Menlo Security #9315

tehbooom commented Mar 8, 2024

mrodm left a comment

mrodm Mar 11, 2024

mrodm Mar 11, 2024

mrodm Mar 11, 2024

tehbooom Mar 11, 2024

kcreddy Mar 15, 2024

elasticmachine commented Mar 11, 2024

kcreddy Mar 15, 2024

kcreddy Mar 15, 2024

kcreddy left a comment

kcreddy Apr 11, 2024

elasticmachine commented Apr 11, 2024

elastic-sonarqube bot commented Apr 11, 2024

kcreddy left a comment •

edited

elasticmachine commented Apr 11, 2024

[New Integration] Menlo Security #9315

[New Integration] Menlo Security #9315

Conversation

tehbooom commented Mar 8, 2024

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

mrodm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Mar 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kcreddy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Apr 11, 2024

💚 Build Succeeded

History

elastic-sonarqube bot commented Apr 11, 2024

Quality Gate passed

kcreddy left a comment • edited

Choose a reason for hiding this comment

elasticmachine commented Apr 11, 2024

kcreddy left a comment •

edited