[Cloud Posture] Findings datastream mapping #3775

kfirpeled · 2022-07-20T19:04:31Z

What does this PR do?

In the following PR we've adjusted the findings mappings.

Turned off dynamic mappings
Imported ECS mappings (auto-generated)
Aligned mappings to ECS conventions (reference)

This means that from now on we opt-in which fields we index in the findings being sent by cloudbeat

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

To test this PR I recommend building the package locally and use it, send data using cloudbeat and check that all fields have the correct mappings

Related issues

Screenshots

elasticmachine · 2022-07-20T19:17:26Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-07-26T16:28:14.907+0000
Duration: 17 min 7 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2
Skipped	0
Total	2

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

kfirpeled · 2022-07-20T21:06:27Z

/test

elasticmachine · 2022-07-21T09:29:17Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`0/0`)	💚
Files	100.0% (`0/0`)	💚 2.879
Classes	100.0% (`0/0`)	💚 2.879
Methods	25.0% (`1/4`)	👎 -64.297
Lines	100.0% (`0/0`)	💚 9.421
Conditionals	100.0% (`0/0`)	💚

eyalkraft

nice, some nits

eyalkraft · 2022-07-21T13:33:39Z

packages/cloud_security_posture/changelog.yml

+    - description: Adjust findings data-stream mappings to fit ECS conventions
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/3775
+    - description: Turned off dynamic mappings of findings data-stream
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/3775
+    - description: Added default pipeline to findings data-stream
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/3775


I'm not sure listing the same PR multiple times is encouraged.

Suggested change

- description: Adjust findings data-stream mappings to fit ECS conventions

type: enhancement

link: https://github.com/elastic/integrations/pull/3775

- description: Turned off dynamic mappings of findings data-stream

type: enhancement

link: https://github.com/elastic/integrations/pull/3775

- description: Added default pipeline to findings data-stream

type: enhancement

link: https://github.com/elastic/integrations/pull/3775

- description: |

Adjust findings data-stream mappings to fit ECS conventions

Turned off dynamic mappings of findings data-stream

Added default pipeline to findings data-stream

type: enhancement

link: https://github.com/elastic/integrations/pull/3775

saw several packages doing the same in their changelog, thought to give it a try

tbh, my previous try with version 0.0.19 combining multiple prs to the same changelog was not a successful one. Because of the post-merge process happens when the version is being create for the first time, it is not being updated afterwards

eyalkraft · 2022-07-21T13:36:41Z

packages/cloud_security_posture/data_stream/findings/fields/ecs.yml

+- name: event.ingested
+  external: ecs
+  type: date
+- name: file.accessed


maybe add a comment and link https://github.com/elastic/cloudbeat/blob/main/resources/fetchers/file_system_fetcher.go#L58-L74

eyalkraft · 2022-07-21T13:37:18Z

packages/cloud_security_posture/data_stream/findings/fields/ecs.yml

+- name: message
+  external: ecs
+  type: match_only_text
+- name: process.args


maybe comment with https://github.com/elastic/cloudbeat/blob/main/resources/fetchers/process_fetcher.go#L57-L98

or was this file purely autogenerated? In this case let's not edit it

semi-auto generated, the ecs generator is half-way through to get this file format
I assume this file is going to be fully auto generated

I saw osquery did their own tool, could be a good idea for us as well going forward
https://github.com/elastic/integrations/blob/main/packages/osquery_manager/data_stream/result/fields/ecs.yml

eyalkraft · 2022-07-21T13:39:28Z

packages/cloud_security_posture/data_stream/findings/fields/rule.yml

+    Examples of data sources that would populate the rule fields include: network
+    admission control platforms, network or host IDS/IPS, network firewalls, web
+    application firewalls, url filters, endpoint detection and response (EDR) systems,


maybe this could be specified for us (if not autogenerated)

So I copy-pasted from the generated code
I didn't try to follow if the documentation being created out of it.

I guess you are correct but I'd leave that out-of-scope and we'll deal with that later on
Assuming the official ECS documentation is good enough for now

eyalkraft · 2022-07-21T13:40:11Z

packages/cloud_security_posture/data_stream/findings/fields/rule.yml

+      default_field: false
+    - name: tags
+      level: extended
+      type: keyword


tags is a list isn't it? Is this the way to map list of keywords?

when I was using type: array the mapping was working as expected.
I was googling and based on the following answer https://stackoverflow.com/questions/43530610/how-to-do-a-mapping-of-array-of-strings-in-elasticsearch
it seems that type: keyword is good enough

And looking on ECS base fields they do the same for the tags field:
https://www.elastic.co/guide/en/ecs/current/ecs-base.html#field-tags

oren-zohar · 2022-07-21T14:36:11Z

@ofiriro3 @tehilashn we may need to do the same for the new EKS fetchers, are they introducing any new fields? besides the raw resource?

ofiriro3 · 2022-07-21T14:38:06Z

@oren-zohar No new fields, only the raw resource

Imported ECS fields (auto-generated)

f0837c2

kfirpeled added enhancement New feature or request v8.4.0 Team:Cloud Security Label for the Cloud Security team labels Jul 20, 2022

indentation fix

db56c25

kfirpeled mentioned this pull request Jul 20, 2022

Automatic index mapping updates #3642

Closed

2 tasks

fix elastic-package checks

8db051e

kfirpeled mentioned this pull request Jul 21, 2022

[Cloud Posture] Adjusted the use of the new findings index mappings elastic/kibana#136766

Merged

9 tasks

kfirpeled added 5 commits July 21, 2022 15:19

added change log

20bb720

updated changelog

8011e2a

Added default pipeline to findings

5f10f89

updated changelogs

df919f3

version bump

82f80d7

kfirpeled marked this pull request as ready for review July 21, 2022 13:28

kfirpeled requested a review from a team as a code owner July 21, 2022 13:28

eyalkraft approved these changes Jul 21, 2022

View reviewed changes

eyalkraft approved these changes Jul 24, 2022

View reviewed changes

Merge branch 'main' into cspm/v840-findings-mapping

3b8ebfb

kfirpeled merged commit 8998871 into elastic:main Jul 26, 2022

kfirpeled deleted the cspm/v840-findings-mapping branch July 26, 2022 16:46

kfirpeled added the breaking change label Jul 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Cloud Posture] Findings datastream mapping #3775

[Cloud Posture] Findings datastream mapping #3775

kfirpeled commented Jul 20, 2022 •

edited

Loading

elasticmachine commented Jul 20, 2022 •

edited

Loading

Build stats

Test stats 🧪

kfirpeled commented Jul 20, 2022

elasticmachine commented Jul 21, 2022 •

edited

Loading

eyalkraft left a comment

eyalkraft Jul 21, 2022

kfirpeled Jul 21, 2022

kfirpeled Jul 21, 2022

eyalkraft Jul 21, 2022

eyalkraft Jul 21, 2022

eyalkraft Jul 21, 2022

kfirpeled Jul 21, 2022

eyalkraft Jul 21, 2022

kfirpeled Jul 21, 2022 •

edited

Loading

eyalkraft Jul 21, 2022

kfirpeled Jul 21, 2022

oren-zohar commented Jul 21, 2022

ofiriro3 commented Jul 21, 2022

[Cloud Posture] Findings datastream mapping #3775

[Cloud Posture] Findings datastream mapping #3775

Conversation

kfirpeled commented Jul 20, 2022 • edited Loading

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Jul 20, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

kfirpeled commented Jul 20, 2022

elasticmachine commented Jul 21, 2022 • edited Loading

🌐 Coverage report

eyalkraft left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kfirpeled Jul 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oren-zohar commented Jul 21, 2022

ofiriro3 commented Jul 21, 2022

kfirpeled commented Jul 20, 2022 •

edited

Loading

elasticmachine commented Jul 20, 2022 •

edited

Loading

elasticmachine commented Jul 21, 2022 •

edited

Loading

kfirpeled Jul 21, 2022 •

edited

Loading