[RAC][Rule Registry] BUG: Component template bootstrapping fails on conflicting fields

[xcrzx]

Parent ticket: #101016

Summary

Rule Registry resources bootstrapping fails when component templates update lead to conflicting fields during the upgrade process.

How to reproduce

Let's say we want to update mappings by moving a field from a solution-specific component template to a common one and changing the field's type in the process. For example, the changes could look like this:

diff --git a/x-pack/plugins/rule_registry/common/assets/field_maps/technical_rule_field_map.ts b/x-pack/plugins/rule_registry/common/assets/field_maps/technical_rule_field_map.ts
index b4ae89b7694..73a06d69191 100644
--- a/x-pack/plugins/rule_registry/common/assets/field_maps/technical_rule_field_map.ts
+++ b/x-pack/plugins/rule_registry/common/assets/field_maps/technical_rule_field_map.ts
@@ -229,6 +229,9 @@ export const technicalRuleFieldMap = {
     array: false,
     required: false,
   },
+  'kibana.alert.rule.threshold.cardinality': {
+    type: 'keyword',
+  },
 } as const;
 
 export type TechnicalRuleFieldMap = typeof technicalRuleFieldMap;
diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/field_maps/rules.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/field_maps/rules.ts
index 21405672fdf..3a070361303 100644
--- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/field_maps/rules.ts
+++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/field_maps/rules.ts
@@ -106,21 +106,6 @@ export const rulesFieldMap = {
     array: true,
     required: false,
   },
-  'kibana.alert.rule.threshold.cardinality': {
-    type: 'object',
-    array: true,
-    required: false,
-  },
-  'kibana.alert.rule.threshold.cardinality.field': {
-    type: 'keyword',
-    array: true,
-    required: false,
-  },
-  'kibana.alert.rule.threshold.cardinality.value': {
-    type: 'long',
-    array: true,
-    required: false,
-  },
   'kibana.alert.rule.timeline_id': {
     type: 'keyword',
     array: true,

When we restart Kibana, the bootstrapping fails with the following error:

ResponseError: illegal_argument_exception: [illegal_argument_exception] Reason: updating component template [.alerts-technical-mappings] results in invalid composable template [.alerts-security.alerts-index-template] after templates are merged

It happens because we first try to add the new field to the common template and, after that, remove it from the solution-specific template. So despite, in the end, we would receive a valid template, during the upgrade process, templates could become incompatible.

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[oatkiller]

@banderror Does this need to be addressed in 7.16?

[jasonrhodes]

This is no longer an issue as we have things planned because we are only allowing additive changes for now.

[banderror]

As you can see from the PR description this bug could be reproduced when moving a field between component templates and changing its type. I'd guess that it's not possible to reproduce it with an additive change or refactoring (e.g. moving a field from one template to another without changing its type), but I'm not 100% sure about that.

[xcrzx]

it's not possible to reproduce it with an additive change or refactoring (e.g. moving a field from one template to another without changing its type), but I'm not 100% sure about that.

I confirm that. To reproduce the bug, we should move a field + change its type. With additive changes, it is not an issue.

[banderror]

Hey everyone, I removed this ticket from the backlog of the Detection Rules area. We (@elastic/security-detections-response-rules) are not the owners anymore (however feel free to still ping us if you have any tech questions about the ticket).

Ownership of this ticket and other tickets related to rule_registry (like #101016) now goes to the Detection Alerts area (Team:Detection Alerts label). Please ping @peluja1012 and @marshallmain if you have any questions.

[marshallmain]

Closing as this scenario only occurs when making a non-additive change to the mappings, which is unsupported in the rule registry currently.