[cisco_umbrella] Add DLP and Intrusion logs for SIG users #5716

LaZyDK · 2023-03-28T13:55:43Z

What does this PR do?

Added DLP logs, Intrusion logs and Firewall logs for Umbrella SIG users.
https://docs.umbrella.com/umbrella-user-guide/docs/log-format-and-versioning

Fixes audit logs being multiline.

Added event.severity, event.action, event.kind, event.category and event.type.

Proxylogs will now show up in SIEM -> Network -> HTTP as more HTTP ECS fields are used.

User identities is now in user.name.
Host identities is now in host.name.
Network identities is now in network.name.

Added found FQDNs to related.hosts.
Added files hashes from DLP logs to related.hash.

user_agent.original is now processed.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

elasticmachine · 2023-03-28T13:59:16Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-04-04T08:06:53.077+0000
Duration: 14 min 49 sec

Test stats 🧪

Test	Results
Failed	0
Passed	10
Skipped	0
Total	10

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

P1llus

Some small nitpick, can see if I can help out here.

I can also confirm that indeed the CSVs for auditlog are multiline, and in quite a few different format types.
Since we do not have system tests for S3 inputs we would have to test this manually.

P1llus · 2023-03-28T14:33:45Z

packages/cisco_umbrella/data_stream/log/elasticsearch/ingest_pipeline/default.yml

+  - remove:
+      field: user.email
+      if: ctx.user?.email == "null"
+  - remove:
+      field: user.id
+      if: ctx.user?.id == "null"
+  - remove:
+      field: user.name
+      if: ctx.user?.name == "null"
+  # Remove user field if present and empty
+  - remove:
+      field:
+        - user
+      if: ctx.user instanceof Map && ctx.user.size() == 0
+      ignore_failure: true
+


I think we have to go back to where these are populated and create better logic for when not to populate those fields.
We also try to not use ignore_failure (ignore_missing is fine)

These are populated by the audit log - so we should go back to Cisco to fix fields containing the string "null". 🥇
I can remove the ignore_failure real fast.

P1llus · 2023-03-28T14:36:13Z

/test

elasticmachine · 2023-03-28T14:51:55Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`1/1`)	💚
Classes	100.0% (`1/1`)	💚
Methods	95.238% (`20/21`)	👎 -4.762
Lines	94.805% (`511/539`)	👍 9.723
Conditionals	100.0% (`0/0`)	💚

efd6 · 2023-03-28T21:50:24Z

/test

LaZyDK · 2023-03-29T06:36:29Z

I will deliver small edits to event.severity and event.action today.

LaZyDK · 2023-03-29T08:52:21Z

Ready for comments.

packages/cisco_umbrella/data_stream/log/elasticsearch/ingest_pipeline/default.yml

…ipeline/default.yml Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>

elasticmachine · 2023-03-31T12:21:47Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

efd6 · 2023-04-02T23:19:37Z

/test

LaZyDK · 2023-04-03T07:02:11Z

All tests are passing locally on my machine.

efd6 · 2023-04-03T07:13:26Z

Needs a field definition. In "static test: Verify sample_event.json – cisco_umbrella.log"

 one or more errors found in document: [0] field "cisco.umbrella.content_type" is undefined

This isn't being generated anymore, so you removed it from fields.yml, but it's still in the sample event. So you'll need to rerun elastic-package test system -g.

LaZyDK · 2023-04-03T08:02:58Z

By some reason, the sample_event.json is never regenerated. Not even if I fill in the file with {}.
All tests run multiple times, including the exact line you wrote me.

efd6 · 2023-04-03T22:00:58Z

Turns out there are no system tests. This means that the sample event was hand constructed or added. I'm in two (three?) minds; the options are to remove the sample event and update the docs, hand correct the event (not ideal), obtain a real sample event and replace with that (also not ideal but probably better). Writing a system test is not really feasible since s3 is the only input. I think I'd be OK with a combination of 2. and 3. where you've checked that a hand edited sample is valid when compared to a real event.

LaZyDK · 2023-04-04T07:09:38Z

New sample event, but with more information.

efd6 · 2023-04-04T08:06:36Z

/test

efd6

🎉

elasticmachine · 2023-04-04T08:43:29Z

Package cisco_umbrella - 1.9.0 containing this change is available at https://epr.elastic.co/search?package=cisco_umbrella

Initial commit

df7eaae

LaZyDK requested a review from a team as a code owner March 28, 2023 13:55

Remove string null user

106d720

P1llus reviewed Mar 28, 2023

View reviewed changes

LaZyDK added 2 commits March 28, 2023 17:06

Remove ignore_failure

08ddb1d

Add ignore_missing

9e53ad8

LaZyDK added 3 commits March 29, 2023 10:45

Align event fields

574b734

Fix intrusion log action

dbfa7dc

Fix intrusion log action

6eaede9

efd6 reviewed Mar 30, 2023

View reviewed changes

packages/cisco_umbrella/data_stream/log/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

LaZyDK and others added 2 commits March 30, 2023 08:38

Update packages/cisco_umbrella/data_stream/log/elasticsearch/ingest_p…

2ed13fb

…ipeline/default.yml Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>

Check if log.file.path is null

b0149e1

jamiehynds added the Team:Security-External Integrations label Mar 31, 2023

jamiehynds added the Integration:Cisco Umbrella label Mar 31, 2023

LaZyDK added 2 commits March 31, 2023 20:42

Merge branch 'main' into umbrella_identity

f91b0f0

Update manifest accordingly to changelog

898792c

Merge branch 'elastic:main' into umbrella_identity

b0a748b

Fix sample_event.json

8d7978e

Remove duplicates in related fields

6156dbd

efd6 approved these changes Apr 4, 2023

View reviewed changes

efd6 merged commit e1b60a5 into elastic:main Apr 4, 2023

LaZyDK deleted the umbrella_identity branch April 4, 2023 08:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[cisco_umbrella] Add DLP and Intrusion logs for SIG users #5716

[cisco_umbrella] Add DLP and Intrusion logs for SIG users #5716

LaZyDK commented Mar 28, 2023 •

edited

Loading

elasticmachine commented Mar 28, 2023 •

edited

Loading

Build stats

Test stats 🧪

P1llus left a comment

P1llus Mar 28, 2023

LaZyDK Mar 28, 2023

P1llus commented Mar 28, 2023

elasticmachine commented Mar 28, 2023 •

edited

Loading

efd6 commented Mar 28, 2023

LaZyDK commented Mar 29, 2023

LaZyDK commented Mar 29, 2023

elasticmachine commented Mar 31, 2023

efd6 commented Apr 2, 2023

LaZyDK commented Apr 3, 2023

efd6 commented Apr 3, 2023

LaZyDK commented Apr 3, 2023

efd6 commented Apr 3, 2023

LaZyDK commented Apr 4, 2023

efd6 commented Apr 4, 2023

efd6 left a comment

elasticmachine commented Apr 4, 2023

[cisco_umbrella] Add DLP and Intrusion logs for SIG users #5716

[cisco_umbrella] Add DLP and Intrusion logs for SIG users #5716

Conversation

LaZyDK commented Mar 28, 2023 • edited Loading

What does this PR do?

Checklist

elasticmachine commented Mar 28, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

P1llus left a comment

Choose a reason for hiding this comment

P1llus Mar 28, 2023

Choose a reason for hiding this comment

LaZyDK Mar 28, 2023

Choose a reason for hiding this comment

P1llus commented Mar 28, 2023

elasticmachine commented Mar 28, 2023 • edited Loading

🌐 Coverage report

efd6 commented Mar 28, 2023

LaZyDK commented Mar 29, 2023

LaZyDK commented Mar 29, 2023

elasticmachine commented Mar 31, 2023

efd6 commented Apr 2, 2023

LaZyDK commented Apr 3, 2023

efd6 commented Apr 3, 2023

LaZyDK commented Apr 3, 2023

efd6 commented Apr 3, 2023

LaZyDK commented Apr 4, 2023

efd6 commented Apr 4, 2023

efd6 left a comment

Choose a reason for hiding this comment

elasticmachine commented Apr 4, 2023

LaZyDK commented Mar 28, 2023 •

edited

Loading

elasticmachine commented Mar 28, 2023 •

edited

Loading

elasticmachine commented Mar 28, 2023 •

edited

Loading