[Darktrace] Initial Release for the Darktrace #4001

vinit-chauhan · 2022-08-16T07:44:24Z

What does this PR do?

Generated the skeleton of the Darktrace integration package.
Added data streams.
Added data collection logic for all the data streams.
Added the ingest pipeline for all the data streams.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files.
Added dashboards and visualizations.
Added test for pipeline for all the data streams.
Added system test cases for all the data streams.

Integration release checklist

This checklist is intended for integrations maintainers to ensure consistency
when creating or updating a Package, Module or Dataset for an Integration.

All changes

Change follows the contributing guidelines
Supported versions of the monitoring target are documented
Supported operating systems are documented (if applicable)
Integration or System tests exist
Documentation exists
Fields follow ECS and naming conventions
At least a manual test with ES / Kibana / Agent has been performed.
Required Kibana version set to: ^8.2.1

New Package

Screenshot of the "Add Integration" page on Fleet added

Dashboards changes

Dashboards exists
Screenshots added or updated
Datastream filters added to visualizations

Log dataset changes

Pipeline tests exist (if applicable)
Generated output for at least 1 log file exists
Sample event (sample_event.json) exists

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/darktrace directory.
Run the following command to run tests.

elastic-package test

Related issues

Closes Darktrace #2754

Screenshots

elasticmachine · 2022-08-16T07:48:48Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2022-08-16T08:12:51Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-10-03T12:24:09.310+0000
Duration: 25 min 21 sec

Test stats 🧪

Test	Results
Failed	0
Passed	29
Skipped	0
Total	29

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-08-16T08:13:00Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`3/3`)	💚
Files	100.0% (`3/3`)	💚 2.564
Classes	100.0% (`3/3`)	💚 2.564
Methods	100.0% (`46/46`)	💚 9.946
Lines	95.645% (`2350/2457`)	👍 4.116
Conditionals	100.0% (`0/0`)	💚

efd6

Query: There are a lot of object arrays in the documents produced here under the darktrace key path. Is this a concern?

vinit-chauhan · 2022-08-24T07:26:31Z

Hey @efd6, Yes the Darktrace API sends the data in such a way. And We can't flatten in as it would result in conflicts in index mappings. So that doesn't seem to be a good idea to flatten it.

efd6 · 2022-08-24T07:55:27Z

Thanks

jamiehynds · 2022-09-13T10:40:27Z

@andrewkroh @efd6 could we prioritise review on this integration next please? Thanks!

P1llus

I am a bit concerned about the amount of arrays in the resulting data, but I don't feel there is any good way around it, since creating single documents for each would simply be too many.

I presume there has not been any big challenges in still using them for visualizations?

I am also a bit unsure how this impacts possibility to create SIEM rules for certain content, seeing as they are all stored in arrays.

If this is something that has already been discussed/thought about, feel free to ignore that.

Outside of that I added a few small comments, the rest seems good to go, great work! :)

P1llus · 2022-09-28T07:02:25Z

packages/darktrace/_dev/deploy/docker/files/config.yml

@@ -0,0 +1,13 @@
+rules:
+  - path: /modelbreaches
+    methods: ["GET"]


Does this API use any sort of authentication? Any way we could add that in as well?

P1llus · 2022-09-28T07:05:02Z

packages/darktrace/data_stream/ai_analyst_alert/agent/stream/httpjson.yml.hbs

@@ -0,0 +1,48 @@
+config_version: 2
+interval: {{interval}}
+request.timeout: 5m


Could this be a configurable advanced option which defaults to 5min? We usually allow users to override this in some niche usecases.

P1llus · 2022-09-28T07:06:17Z

packages/darktrace/data_stream/model_breach_alert/agent/stream/httpjson.yml.hbs

@@ -0,0 +1,51 @@
+config_version: 2
+interval: {{interval}}
+request.timeout: 5m


Same as above, please make this configurable

P1llus · 2022-09-28T07:18:18Z

I think maybe we have to discuss the duplicate custom fields, as has been brought up in some other PR comments as well

vinit-chauhan · 2022-09-30T18:03:24Z

/test

elasticmachine · 2022-10-03T12:49:24Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

vinit-chauhan added 2 commits August 16, 2022 12:53

Initial Release for the Darktrace

9c534b8

Update changelog file

36a6169

vinit-chauhan added enhancement New feature or request Team:Security-External Integrations New Integration labels Aug 16, 2022

efd6 approved these changes Aug 17, 2022

View reviewed changes

updated the file names for the pipeline test

54e142b

epixa assigned P1llus Sep 20, 2022

P1llus approved these changes Sep 28, 2022

View reviewed changes

Update request.timeout parameter, tag & user guide

8a93f8e

resolved build failed

0c1f837

P1llus merged commit b5e9f7e into elastic:main Oct 4, 2022

andrewkroh mentioned this pull request Oct 4, 2022

[production] Promote packages from snapshot (darktrace-0.1.0, gcp-2.11.10, infoblox_bloxone_ddi-0.1.1, lastpass-0.1.0, ping_one-0.1.0, trend_micro_vision_one-0.1.0) elastic/package-storage#5786

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Darktrace] Initial Release for the Darktrace #4001

[Darktrace] Initial Release for the Darktrace #4001

vinit-chauhan commented Aug 16, 2022 •

edited

Loading

elasticmachine commented Aug 16, 2022

elasticmachine commented Aug 16, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Aug 16, 2022 •

edited

Loading

efd6 left a comment

vinit-chauhan commented Aug 24, 2022

efd6 commented Aug 24, 2022

jamiehynds commented Sep 13, 2022

P1llus left a comment

P1llus Sep 28, 2022

P1llus Sep 28, 2022

P1llus Sep 28, 2022

P1llus commented Sep 28, 2022

vinit-chauhan commented Sep 30, 2022

elasticmachine commented Oct 3, 2022

[Darktrace] Initial Release for the Darktrace #4001

[Darktrace] Initial Release for the Darktrace #4001

Conversation

vinit-chauhan commented Aug 16, 2022 • edited Loading

What does this PR do?

Integration release checklist

All changes

New Package

Dashboards changes

Log dataset changes

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Aug 16, 2022

elasticmachine commented Aug 16, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Aug 16, 2022 • edited Loading

🌐 Coverage report

efd6 left a comment

Choose a reason for hiding this comment

vinit-chauhan commented Aug 24, 2022

efd6 commented Aug 24, 2022

jamiehynds commented Sep 13, 2022

P1llus left a comment

Choose a reason for hiding this comment

P1llus Sep 28, 2022

Choose a reason for hiding this comment

P1llus Sep 28, 2022

Choose a reason for hiding this comment

P1llus Sep 28, 2022

Choose a reason for hiding this comment

P1llus commented Sep 28, 2022

vinit-chauhan commented Sep 30, 2022

elasticmachine commented Oct 3, 2022

🚀 Benchmarks report

vinit-chauhan commented Aug 16, 2022 •

edited

Loading

elasticmachine commented Aug 16, 2022 •

edited

Loading

elasticmachine commented Aug 16, 2022 •

edited

Loading