Add criticality fields and risk score fields to alert schema #174626

nkhristinin · 2024-01-10T14:29:36Z

Update alerts fields names for asset criticality, and add risk score field

We want to update
kibana.alert.user.criticality_level to host.asset.criticality
kibana.alert.host.criticality_level to host.asset.criticality

kibana.alert.user.criticality_level and kibana.alert.host.criticality_level will be still present in the schema/mappings, for backward compatibility as it was released to serverless/

Also, we added host.risk.calculated_score_norm, host.risk.calculated_level, user.risk.calculated_score_norm, user.risk.calculated_level.
Those fields enriched alerts from8.5.0, but weren't added to the alert schema

@SourinPaul approved usage of new fields

nkhristinin · 2024-01-10T14:32:44Z

/ci

nkhristinin · 2024-01-24T15:53:59Z

/ci

SourinPaul · 2024-01-24T18:45:41Z

@nkhristinin Since asset criticality has not yet been released (as of 8.12), I recommend we remove the older Kibana fieldmapsets so it does not clutter the alert fields.

With the above internal fields cleaned up, I approve merging this PR so we use the proposed ECS fields instead.

cc: @jaredburgettelastic @paulewing

nkhristinin · 2024-01-25T11:37:18Z

@elasticmachine merge upstream

nkhristinin · 2024-01-25T12:17:20Z

With the above internal fields cleaned up, I approve merging this PR so we use the proposed ECS fields instead.

As there change that it was released in serverless, those fields can be in alert_mappings, but not in the alert document.

I will not remove them from mappings for backwards compatibility

nkhristinin · 2024-01-25T12:25:05Z

@elasticmachine merge upstream

rylnd · 2024-01-25T21:02:26Z

x-pack/plugins/security_solution/common/field_maps/8.13.0/alerts.ts


 export const alertsFieldMap8130 = {
  ...alertsFieldMap840,
+  /**
+   * @deprecated Use ALERT_HOST_CRITICALITY instead.


I wasn't able to see these JSDoc being used anywhere, either on the broader alertsFieldMap8130 type, nor on pulling the value out. Since the key itself is deprecated, perhaps that's good enough?

rylnd · 2024-01-25T21:11:44Z

x-pack/plugins/security_solution/common/field_maps/field_names.ts

-export const ALERT_HOST_CRITICALITY = `${ALERT_NAMESPACE}.host.criticality_level` as const;
-export const ALERT_USER_CRITICALITY = `${ALERT_NAMESPACE}.user.criticality_level` as const;
+/**
+ * @deprecated Use ALERT_HOST_CRITICALITY


This format will actually link to the other type:

Suggested change

* @deprecated Use ALERT_HOST_CRITICALITY

* @deprecated Use {@link ALERT_HOST_CRITICALITY}

rylnd · 2024-01-25T21:18:50Z

x-pack/plugins/security_solution/common/field_maps/field_names.ts

+
+export const ALERT_HOST_CRITICALITY = `host.asset.criticality` as const;
+export const ALERT_USER_CRITICALITY = `user.asset.criticality` as const;
+export const ALERT_HOST_RISK_SCORE_CALCULATED_LEVEL = `host.risk.calculated_level` as const;


Do these need these ALERT_ prefixes if the fields aren't specific to alerts?

Yes, for those fields I want to show that they are related to alerts.

We do have host.risk.calculated_level in other places in our app, but host.asset.criticality unique only for alerts, so I wanted to specify that

...curity_solution/server/lib/detection_engine/rule_types/utils/enrichments/__mocks__/alerts.ts

kqualters-elastic

lgtm, but traditionally it's been @elastic/response-ops that want to keep the alerts as data schemas as light as possible

nkhristinin · 2024-01-30T08:47:58Z

@elasticmachine merge upstream

ymao1 · 2024-01-30T13:51:23Z

packages/kbn-alerts-as-data-utils/src/schemas/generated/security_schema.ts

@@ -122,6 +122,9 @@ const SecurityAlertOptional = rt.partial({
  'ecs.version': schemaString,
  'event.action': schemaString,
  'event.kind': schemaString,
+  'host.asset.criticality': schemaString,
+  'host.risk.calculated_level': schemaString,


host.risk.calculated_level and host.risk.calculated_score_norm are already in the ECS component template, which is referenced by by the security alerts index. Do we need to redefine them in the security alerts component template?

Good catch!

I will check if I can have in alert types, but remove from field_maps, thanks!

Tested locally, you rights, the field already in alert mappings, we probably don't need them here. Thanks!

ymao1 · 2024-01-30T13:52:45Z

packages/kbn-alerts-as-data-utils/src/schemas/generated/security_schema.ts

@@ -204,6 +207,9 @@ const SecurityAlertOptional = rt.partial({
  'kibana.alert.workflow_user': schemaString,
  'kibana.version': schemaString,
  tags: schemaStringArray,
+  'user.asset.criticality': schemaString,
+  'user.risk.calculated_level': schemaString,


same question about user.risk.calculated_score_norm and user.risk.calculated_level. If they are already in the ECS component template, do we need to redefine them in the security alerts component template?

nkhristinin · 2024-01-31T07:21:27Z

@elasticmachine merge upstream

nkhristinin · 2024-01-31T08:32:03Z

@elasticmachine merge upstream

nkhristinin · 2024-01-31T11:41:53Z

@elasticmachine merge upstream

ymao1

Response ops changes LGTM

kibana-ci · 2024-01-31T13:21:23Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 8e52049

Failed CI Steps

Jest Tests #12

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	11.4MB	11.4MB	+76.0B

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

rylnd

I was briefly hung up on this exchange, but it sounds like you and Sourin have come to an agreement there 👍 .

@ymao1 thanks for catching those redundancies!

EA/SecSol changes LGTM.

@SourinPaul

…#174626) ## Update alerts fields names for asset criticality, and add risk score field We want to update `kibana.alert.user.criticality_level` to `host.asset.criticality` `kibana.alert.host.criticality_level` to `host.asset.criticality` `kibana.alert.user.criticality_level` and `kibana.alert.host.criticality_level` will be still present in the schema/mappings, for backward compatibility as it was released to serverless/ Also, we added `host.risk.calculated_score_norm`, `host.risk.calculated_level`, `user.risk.calculated_score_norm`, `user.risk.calculated_level`. Those fields enriched alerts from[8.5.0](elastic#139478), but weren't added to the alert schema @SourinPaul [approved](elastic#174626 (comment)) usage of new fields --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

@SourinPaul

…#174626) ## Update alerts fields names for asset criticality, and add risk score field We want to update `kibana.alert.user.criticality_level` to `host.asset.criticality` `kibana.alert.host.criticality_level` to `host.asset.criticality` `kibana.alert.user.criticality_level` and `kibana.alert.host.criticality_level` will be still present in the schema/mappings, for backward compatibility as it was released to serverless/ Also, we added `host.risk.calculated_score_norm`, `host.risk.calculated_level`, `user.risk.calculated_score_norm`, `user.risk.calculated_level`. Those fields enriched alerts from[8.5.0](elastic#139478), but weren't added to the alert schema @SourinPaul [approved](elastic#174626 (comment)) usage of new fields --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

@SourinPaul

…#174626) ## Update alerts fields names for asset criticality, and add risk score field We want to update `kibana.alert.user.criticality_level` to `host.asset.criticality` `kibana.alert.host.criticality_level` to `host.asset.criticality` `kibana.alert.user.criticality_level` and `kibana.alert.host.criticality_level` will be still present in the schema/mappings, for backward compatibility as it was released to serverless/ Also, we added `host.risk.calculated_score_norm`, `host.risk.calculated_level`, `user.risk.calculated_score_norm`, `user.risk.calculated_level`. Those fields enriched alerts from[8.5.0](elastic#139478), but weren't added to the alert schema @SourinPaul [approved](elastic#174626 (comment)) usage of new fields --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

@SourinPaul

…#174626) ## Update alerts fields names for asset criticality, and add risk score field We want to update `kibana.alert.user.criticality_level` to `host.asset.criticality` `kibana.alert.host.criticality_level` to `host.asset.criticality` `kibana.alert.user.criticality_level` and `kibana.alert.host.criticality_level` will be still present in the schema/mappings, for backward compatibility as it was released to serverless/ Also, we added `host.risk.calculated_score_norm`, `host.risk.calculated_level`, `user.risk.calculated_score_norm`, `user.risk.calculated_level`. Those fields enriched alerts from[8.5.0](elastic#139478), but weren't added to the alert schema @SourinPaul [approved](elastic#174626 (comment)) usage of new fields --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

@SourinPaul

…#174626) ## Update alerts fields names for asset criticality, and add risk score field We want to update `kibana.alert.user.criticality_level` to `host.asset.criticality` `kibana.alert.host.criticality_level` to `host.asset.criticality` `kibana.alert.user.criticality_level` and `kibana.alert.host.criticality_level` will be still present in the schema/mappings, for backward compatibility as it was released to serverless/ Also, we added `host.risk.calculated_score_norm`, `host.risk.calculated_level`, `user.risk.calculated_score_norm`, `user.risk.calculated_level`. Those fields enriched alerts from[8.5.0](elastic#139478), but weren't added to the alert schema @SourinPaul [approved](elastic#174626 (comment)) usage of new fields --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Add criticality fields and risk score fields to alert schema

1c0f6c3

Add fields to shema

6ebdcb1

kibanamachine and others added 2 commits January 25, 2024 06:37

Merge branch 'main' into alert-enrichments-new-fields

0170c0d

update alert_as_data_fields

c2a5b9d

nkhristinin marked this pull request as ready for review January 25, 2024 12:15

nkhristinin requested review from a team as code owners January 25, 2024 12:15

nkhristinin requested a review from rylnd January 25, 2024 12:15

nkhristinin added the release_note:feature Makes this part of the condensed release notes label Jan 25, 2024

kibanamachine and others added 2 commits January 25, 2024 07:25

Merge branch 'main' into alert-enrichments-new-fields

bdb948a

Fix unit tests

7812ee7

nkhristinin requested a review from a team as a code owner January 25, 2024 13:46

rylnd reviewed Jan 25, 2024

View reviewed changes

nkhristinin added 2 commits January 26, 2024 13:25

Merge branch 'main' into alert-enrichments-new-fields

4107538

Fix jsdoc

45ff085

kqualters-elastic approved these changes Jan 26, 2024

View reviewed changes

Merge branch 'main' into alert-enrichments-new-fields

b151997

ymao1 reviewed Jan 30, 2024

View reviewed changes

nkhristinin added 3 commits January 30, 2024 15:30

Remove fields from field_maps, which are present in ECS

76b9292

Remove unused fields

762fd5f

Fix eslint error

694be2a

nkhristinin added 2 commits January 30, 2024 18:47

update snapshot

1e4fa6e

update schema

4244b7b

Merge branch 'main' into alert-enrichments-new-fields

182705c

Merge branch 'main' into alert-enrichments-new-fields

ecbf336

Merge branch 'main' into alert-enrichments-new-fields

8e52049

ymao1 approved these changes Jan 31, 2024

View reviewed changes

nkhristinin requested a review from rylnd January 31, 2024 13:55

rylnd approved these changes Jan 31, 2024

View reviewed changes

nkhristinin merged commit f3a7fd6 into elastic:main Jan 31, 2024
39 checks passed

kibanamachine added v8.13.0 backport:skip This commit does not require backporting labels Jan 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add criticality fields and risk score fields to alert schema #174626

Add criticality fields and risk score fields to alert schema #174626

nkhristinin commented Jan 10, 2024 •

edited

nkhristinin commented Jan 10, 2024

nkhristinin commented Jan 24, 2024

SourinPaul commented Jan 24, 2024

nkhristinin commented Jan 25, 2024

nkhristinin commented Jan 25, 2024

nkhristinin commented Jan 25, 2024

rylnd Jan 25, 2024

rylnd Jan 25, 2024

rylnd Jan 25, 2024

nkhristinin Jan 26, 2024

kqualters-elastic left a comment

nkhristinin commented Jan 30, 2024

ymao1 Jan 30, 2024

nkhristinin Jan 30, 2024

nkhristinin Jan 30, 2024

ymao1 Jan 30, 2024 •

edited

nkhristinin commented Jan 31, 2024

nkhristinin commented Jan 31, 2024

nkhristinin commented Jan 31, 2024

ymao1 left a comment

kibana-ci commented Jan 31, 2024

rylnd left a comment

	* @deprecated Use ALERT_HOST_CRITICALITY
	* @deprecated Use {@link ALERT_HOST_CRITICALITY}

Add criticality fields and risk score fields to alert schema #174626

Add criticality fields and risk score fields to alert schema #174626

Conversation

nkhristinin commented Jan 10, 2024 • edited

Update alerts fields names for asset criticality, and add risk score field

nkhristinin commented Jan 10, 2024

nkhristinin commented Jan 24, 2024

SourinPaul commented Jan 24, 2024

nkhristinin commented Jan 25, 2024

nkhristinin commented Jan 25, 2024

nkhristinin commented Jan 25, 2024

rylnd Jan 25, 2024

Choose a reason for hiding this comment

rylnd Jan 25, 2024

Choose a reason for hiding this comment

rylnd Jan 25, 2024

Choose a reason for hiding this comment

nkhristinin Jan 26, 2024

Choose a reason for hiding this comment

kqualters-elastic left a comment

Choose a reason for hiding this comment

nkhristinin commented Jan 30, 2024

ymao1 Jan 30, 2024

Choose a reason for hiding this comment

nkhristinin Jan 30, 2024

Choose a reason for hiding this comment

nkhristinin Jan 30, 2024

Choose a reason for hiding this comment

ymao1 Jan 30, 2024 • edited

Choose a reason for hiding this comment

nkhristinin commented Jan 31, 2024

nkhristinin commented Jan 31, 2024

nkhristinin commented Jan 31, 2024

ymao1 left a comment

Choose a reason for hiding this comment

kibana-ci commented Jan 31, 2024

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Async chunks

History

rylnd left a comment

Choose a reason for hiding this comment

nkhristinin commented Jan 10, 2024 •

edited

ymao1 Jan 30, 2024 •

edited