[RAC] Rule registry index bootstrapping: implement suggestions for backwards compatibility

[banderror]

Parent ticket: #101016

Summary

We need to fix the indexing implementation in rule_registry plugin in such a way so it can provide backwards compatibility for alerts-as-data indices in the future. Alerts-as-data schema can evolve, and we might need to introduce breaking changes into it. These breaking changes should be handled in one way or another, for example by:

• applying field aliases to old indices
• applying runtime fields to old indices
• reindexing old indices

Background of the problem

The current Alerts as Data implementation uses the same (well, similar) method that Fleet does for upgrading packages for "migrating" the alerts data indices. However, the approach that Fleet packages use for upgrading indices isn't appropriate to use for Alerts as Data because we want to maintain backward compatibility for all alerts data and the Fleet package upgrade approach doesn't provide backward compatibility and it's only focused on allowing data to be ingested in the new format. Additionally, the index-naming scheme without an index alias doesn't leave us with the option of using a reindex operation in the future.

Issues:

• Issue 1: non-additive changes do not trigger rollover. The Fleet package upgrade algorithm relies on Elasticsearch throwing an incompatible mappings error to trigger a rollover into a new index for upgrades. However, this does not occur for all non-additives changes, including renaming fields.
• Issue 2: Field aliases and runtime fields may not provide all backward compatibility. At this time, we don't know all of the types of breaking changes we will need to make to the Alerts data indices. Additionally, the Fleet package upgrade approach doesn't provide any backward compatibility right now using runtime fields or field aliases, so we don't know for certain how far it will get us. As a result, there is the potential that we will need to reindex the Alerts data indices. If we need to perform a reindex operation, we'll want to use an index alias that is updated to point at the indices in the new data format and exclude the data that is in the old format. However, the existing index-naming scheme in the design document should be adjusted to make the reindex possible. Currently, we will be creating indices with the following pattern: .alerts-{registrationContext}.{datasetSuffix}-{namespace}-{leftPaddedIncrementingNumber}. This will likely result in users creating Dashboards, Visualization, etc. against index patterns like the following: .alerts-security.alerts-*, .alerts-*. If we discover that we need to reindex the data and perform a migration, it will require us to create a new index and copy all of the data from the original indices into the new indices while performing a transformation. If we were to add a version to the index-naming pattern to designate that we have a new version, there's a high risk that the index-patterns that users have been using will match both the old and the new indices. For example, if we were to add the version right after .alerts- in the index-name, this would result in us having a .alerts-security.alerts-default-000001 index and a .alerts-v2-security.alerts.default-000001 index, and the .alerts-* index-pattern would match both.

Solution for 7.15

We need to separate naming schemes for index aliases and concrete indices. This will allow us to implement reindexing in the future.

Aliases will be used for reading and writing data:

.alerts-{registrationContext}.{datasetSuffix}-{namespace}

Concrete indices will have a different naming scheme:

.internal.alerts-{registrationContext}.{datasetSuffix}-{namespace}-000001

How this enables reindexing: after reindexing a new (target) index will be created; we will be able to add the alias to it, and remove the alias from the old (source) index.

Also, we need to use versioning for index templates and documents being written into those indices. This will help us apply field aliases, runtime fields etc to the old indices when we need it.

Our decisions previously discussed:

• Use the current Kibana version for the versioning described above.
• Don’t include the version in the concrete index name by default (exception - it will be included after reindexing).
• Naming for concrete backing indices: .internal.alerts-security.alerts-default-000001-r8.0.0
  • prefix: .internal.alerts
  • ilm suffix: 000002
  • reindexing suffix: r8.0.0, not included by default, only included into a target index name after reindexing
• We will include the version into the index template metadata:
  • _meta.kibana.version = 7.16.0
  • will be included into both the index template _meta and mappings._meta
• We will include the version into the document fields:
  • kibana.version = 7.16.0
  • it will use the version field type

Sub-tasks

• Open a PR in Elasticsearch repo for supporting the .internal.alerts prefix.
  • Granting kibana_system reserved role access to "all" privileges to .internal.alerts* index elasticsearch#76624
• Implement the suggested naming scheme and versioning.
  • [RAC][Rule Registry] Implement versioning and backing indices #109276

Brandon's concerns regarding the index upgrade/rollover logic (#109276 (comment)) will be addressed as part of the parent ticket (#101016).

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[banderror]

Closed this one. Brandon's concerns regarding the index upgrade/rollover logic (#109276 (comment)) will be addressed as part of the parent ticket (#101016) which will be updated accordingly.