[ResponseOps] Decouple rule producer/consumer settings from Kibana feature ID

[xcrzx]

Blocker for: https://github.com/elastic/security-team/issues/9533 (internal), https://github.com/elastic/security-team/issues/8799 (internal)

Summary

As part of the RBAC changes in the Security Solution (see this issue for more details), we need to extract rule management into a standalone Kibana product feature.

This involves transferring security rule types from the siem feature (current configuration here) to a new ruleManagement feature. This will require changing the producer and consumer rule settings to a new value, as feature IDs currently serve as the producer/consumer identifiers.

As we discussed previously, migrating rules is not an option (see this comment). Therefore, we need to find out how to make the Alerting RBAC work with the new feature ID while keeping the current rule producer/consumer siem value.

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[marshallmain]

As we discussed previously, migrating rules is not an option.

@xcrzx Can you add more details about why migrating rules is not an option? Either a summary in this ticket or a link to where a reader can find more information.

[xcrzx]

@marshallmain The producer/consumer fields are used for RBAC, and migrating them without downtime is not possible in Serverless. The upgrade process in Serverless works in such a way that there might be several Kibanas of different versions running simultaneously, all consuming data from a single Elasticsearch cluster. When a newer version of Kibana starts up, it begins migrating saved objects, including changing the producer/consumer fields in this case. As a result, older versions of Kibana will encounter access errors when attempting to read the migrated data.

[banderror]

Hey @cnasikas, I was told today in a sync meeting with the ResponseOps team that this is going to be taken into work around the 8.16 release cycle, but without a commitment to ship in 8.16. Just double-checking here to confirm.

[cnasikas]

Hey @banderror, this is correct. I am still working on a POC (#181559) for this. We are confident that is feasible but we need to finalize some edge cases. I will post our findings on the issue soon. The next step is to make it production-ready but as you said without a commitment to ship in 8.16.