[ti_maltiverse] Add new Maltiverse threat intelligence integration #6741

jlopezzarza · 2023-06-28T14:31:26Z

What does this PR do?

Add Maltiverse as a source of data for Elastic Intelligence

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

Fields are correct
Tests are in place

How to test this PR locally

The changes can be tested in a local environment by adding the new integration to the stack. You will need to register in Maltiverse to generate an API key in order to request data.

Related issues

Screenshots

cla-checker-service · 2023-06-28T14:31:30Z

💚 CLA has been signed

elasticmachine · 2023-06-28T14:35:00Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-07-26T09:14:38.885+0000
Duration: 15 min 52 sec

Test stats 🧪

Test	Results
Failed	0
Passed	5
Skipped	0
Total	5

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-06-28T14:40:00Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

marc-gr · 2023-06-29T07:07:12Z

/cla

marc-gr · 2023-06-29T07:14:43Z

/cla

kcreddy · 2023-07-20T13:39:46Z

/test

packages/ti_maltiverse/data_stream/indicator/elasticsearch/ingest_pipeline/default.yml

kcreddy · 2023-07-20T13:47:35Z

packages/ti_maltiverse/data_stream/indicator/elasticsearch/ingest_pipeline/default.yml

+      fields: ["threat.indicator"]
+  - set:
+      field: _id
+      value: "{{ event.id }}"


Add if: ctx.event?.id != null. Also please verify this in one of the document in Kibana

packages/ti_maltiverse/data_stream/indicator/_dev/test/system/test-default-config.yml

kcreddy · 2023-07-20T14:02:51Z

Also the CI is failing on tests because tests are not run after the changes are made. So, once you are done with above changes, please run:

eval "$(elastic-package stack shellinit)" && elastic-package test --generate -v

and then

 elastic-package build && elastic-package format && elastic-package lint && elastic-package check && elastic-package build

kcreddy · 2023-07-23T15:20:12Z

/test

kcreddy · 2023-07-24T04:08:41Z

Hey @jlopezzarza The pipeline tests are failing. Are they successful on your end? If so, I think maybe some files are missing to be committed.

kcreddy · 2023-07-24T08:53:04Z

/test

This reverts commit 8d02aae.

kcreddy · 2023-07-24T09:38:17Z

/test

kcreddy · 2023-07-25T10:10:29Z

Hey @jlopezzarza Tests are now successful.

There's field conflicts between source and destination (latest) indices.

It seems like a bug in elastic-package as its not expanding some fields in transform/latest/fields/fields.yml that are defined as ECS external .

To fix the issue, we will need to define their types explicitly. Please follow below steps to fix the mapping issue:

Inside the transform/latest/fields/fields.yml , delete the fields that are copied from ecs.yml file. i.e, defined as - external: ecs
Run elastic-package build
Check the ecs.yml inside build folder. These fields should be expanded: build/packages/ti_...../data_stream/threatstream/fields/ecs.yml
Copy these expanded fields into transform/latest/fields/fields.yml .

kcreddy · 2023-07-26T07:48:21Z

packages/ti_maltiverse/elasticsearch/transform/latest/fields/fields.yml

- name: input.type
-  type: keyword
-  description: Input type.
- name: data_stream.type
-  type: constant_keyword
-  description: Data stream type.
- name: data_stream.dataset
-  type: constant_keyword
-  description: Data stream dataset.
- name: data_stream.namespace
-  type: constant_keyword
-  description: Data stream namespace.
- name: '@timestamp'
-  type: date
-  description: Event timestamp.


You will need to keep these above fields. Its just the fields defined as external are what needs to be replaced with full definition.

kcreddy · 2023-07-26T07:48:46Z

packages/ti_maltiverse/elasticsearch/transform/latest/fields/fields.yml

+- description: |-
+    Custom key/value pairs.
+    Can be used to add meta information to events. Should not contain nested objects. All values are stored as keyword.
+    Example: `docker` and `k8s` labels.
+  name: labels
+  type: object


You can remove labels field since its not going to be present in the destination index.

kcreddy · 2023-07-26T09:00:11Z

/test

kcreddy · 2023-07-26T09:14:24Z

/test

kcreddy

LGTM 👍🏼
Thanks for the contribution @jlopezzarza 😄

elasticmachine · 2023-07-26T10:15:12Z

Package ti_maltiverse - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=ti_maltiverse

jlopezzarza and others added 12 commits May 24, 2023 11:32

Add basic structure

71bbf4c

bump kibana.version condition to 8.7.1

172bf42

add main logo to manifest

691c95a

some fixes

8cd2a7c

Merge branch 'main' into add_maltiverse

d18a6cc

quick fixes

38c3ef6

Add reponse split

2ac5f14

Add base fields of our data

5b4f59b

ingestion working

10f7c5e

some more fixes

4806caa

More documentation

5621e02

Add pipeline tests

b442841

marc-gr added the Team:Security-External Integrations label Jun 28, 2023

marc-gr self-requested a review June 28, 2023 14:33

jlopezzarza marked this pull request as ready for review June 28, 2023 14:39

jlopezzarza marked this pull request as draft June 28, 2023 14:45

some tweaks and fixes

3cb7a4a

andrewkroh changed the title ~~Add Maltiverse to threat intelligence integrations~~ [ti_maltiverse] Add new Maltiverse threat intelligence integration Jun 29, 2023

andrewkroh added Integration:ti_maltiverse Maltiverse New Integration Integration:ti_util Threat Intelligence Utilities and removed Integration:ti_util Threat Intelligence Utilities labels Jun 29, 2023

jlopezzarza added 3 commits July 3, 2023 11:31

set timestamp

37ee65d

ilm wip

458738d

ilm and transform in place

90ee723

jlopezzarza added 7 commits July 20, 2023 14:06

fix changelog

35d0eb2

Fix example

b5bcd2a

Add sample event

a3b6f7a

More PR tweaks

b910fe2

More PR tweaks

df933be

Update README

28153c1

Re-update README

f310fa4

kcreddy reviewed Jul 20, 2023

View reviewed changes

More PR tweaks

8104257

Fix tests

a4f2abb

Fix pipeline test

8d02aae

jlopezzarza requested a review from a team as a code owner July 24, 2023 07:26

jlopezzarza added 2 commits July 24, 2023 11:01

Revert "Fix pipeline test"

f51ffa9

This reverts commit 8d02aae.

Fix test

4a83809

Add expanded ecs fields to transform

f630649

kcreddy reviewed Jul 26, 2023

View reviewed changes

Minor fixes

ff2425d

kcreddy approved these changes Jul 26, 2023

View reviewed changes

kcreddy merged commit 2e27c44 into elastic:main Jul 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ti_maltiverse] Add new Maltiverse threat intelligence integration #6741

[ti_maltiverse] Add new Maltiverse threat intelligence integration #6741

jlopezzarza commented Jun 28, 2023 •

edited

Loading

cla-checker-service bot commented Jun 28, 2023 •

edited

Loading

elasticmachine commented Jun 28, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Jun 28, 2023

marc-gr commented Jun 29, 2023

marc-gr commented Jun 29, 2023

kcreddy commented Jul 20, 2023

kcreddy Jul 20, 2023

kcreddy commented Jul 20, 2023

kcreddy commented Jul 23, 2023

kcreddy commented Jul 24, 2023

kcreddy commented Jul 24, 2023

kcreddy commented Jul 24, 2023

kcreddy commented Jul 25, 2023 •

edited

Loading

kcreddy Jul 26, 2023

kcreddy Jul 26, 2023

kcreddy commented Jul 26, 2023

kcreddy commented Jul 26, 2023

kcreddy left a comment

elasticmachine commented Jul 26, 2023

[ti_maltiverse] Add new Maltiverse threat intelligence integration #6741

[ti_maltiverse] Add new Maltiverse threat intelligence integration #6741

Conversation

jlopezzarza commented Jun 28, 2023 • edited Loading

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

cla-checker-service bot commented Jun 28, 2023 • edited Loading

elasticmachine commented Jun 28, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Jun 28, 2023

marc-gr commented Jun 29, 2023

marc-gr commented Jun 29, 2023

kcreddy commented Jul 20, 2023

kcreddy Jul 20, 2023

Choose a reason for hiding this comment

kcreddy commented Jul 20, 2023

kcreddy commented Jul 23, 2023

kcreddy commented Jul 24, 2023

kcreddy commented Jul 24, 2023

kcreddy commented Jul 24, 2023

kcreddy commented Jul 25, 2023 • edited Loading

kcreddy Jul 26, 2023

Choose a reason for hiding this comment

kcreddy Jul 26, 2023

Choose a reason for hiding this comment

kcreddy commented Jul 26, 2023

kcreddy commented Jul 26, 2023

kcreddy left a comment

Choose a reason for hiding this comment

elasticmachine commented Jul 26, 2023

jlopezzarza commented Jun 28, 2023 •

edited

Loading

cla-checker-service bot commented Jun 28, 2023 •

edited

Loading

elasticmachine commented Jun 28, 2023 •

edited

Loading

kcreddy commented Jul 25, 2023 •

edited

Loading