[Security Solution] Migrate security rules encrypted saved objects to new schema with ruleSource field (BLOCKED)

[jpdjere]

Epics: https://github.com/elastic/security-team/issues/1974 (internal), #174168
Depends on: #180141
Blocked by: #187651, #50216, #183603 (comment)
Needed for: #180126

Summary

Use a (currently non-existing) rule migration mechanism provided by the Alerting Framework to migrate detection rules to a new schema that contains the new ruleSource field.

In Security Solution, as part of the rule customization epic, we need to change the rule parameters from:

type RuleParams = {
  immutable: boolean;
}

to

type RuleParams = {
  rule_source: {
    type: 'internal' | 'external'; // <- this can be derived from the existing 'immutable' field
    is_customized: boolean;
  }
}

Semantically, the fields have similar meanings; both the old field and the new field will be used to distinguish prebuilt detection rules from custom rules created by users. However, the new field allows for more flexibility and enables us to build rule customization features on top of it.

Proposed solution

Initially, we proposed to use the Model Version API for this migration in a POC, but the proposal wasn't accepted by the ResponseOps team.

At the moment, we don't have an idea what this solution should be. We depend on the ResponseOps team here, the problem is being tracked in #187651 by us and in #50216 by the ResponseOps team. We can contribute to the design of this mechanism, propose any solutions, or open an RFC.

Useful links

• ESO model versioning example: https://github.com/elastic/kibana/tree/main/examples/eso_model_version_example
• SO migrations on serverless docs: https://docs.elastic.dev/kibana-dev-docs/tutorials/saved-objects#saved-object-migrations-on-serverless
• Implementation PRs
  • Implements Encrypted Saved Objects Model Version API #166302
  • Replace Encrypted Saved Object AAD exclude list with include list #167705

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[banderror]

Blocked by ResponseOps in #183603