[RAC] [Rule Registry] Make data access via the RuleDataClient version-aware

[weltenwort]

ℹ️ This capability in the RuleDataClient is only required if the old indices are kept at their respective version. If the indices are all updated in-place on upgrade, the writer can just continue to assume an up-to-date index mapping.

📓 Summary

When updating alert documents in older indices (meaning indices whose mappings don't match the recent templates) the RuleDataClient must ensure that only appropriate fields are written. That means the documents will have to be "downgraded" to the mapping of their containing index.

🖼️ Background

The initial plan to ensure backwards-compatibility of the alerts-as-data indices assumes only additive mapping changes are permitted. This allows for old indices to remain unchanged and new mappings being adopted via alias rollovers (see #108941). As a consequence the code that writes to the alerts-as-data indices needs to update documents in older indices in a way that works with the respective old mappings.

✔️ Implementation plan / acceptance criteria

• Migration functions downgrade documents created with the recent mappings mind to a schema that can be written to older indices.
• Migration functions are composable so they can be applied in sequence to downgrade documents across multiple versions.
• The RuleDataService can be constructed with migration functions that match eventual changes to the technical mappings.
• Registrations performed via RuleDataService.initializeIndex() can contain migration functions that match eventual changes to the mappings in the registration-specific mappings.
• The RuleDataClient can be constructed with migration functions that consist of both the shared technical migrations as well as the registration-specific migrations.
• The RuleDataClient exposes and API that encapsulates alert document updates such that
  • written documents are "downgraded" according to their original versions such that they map their containing index' mapping
  • the number of search calls to fetch documents to-be-updated is minimal
  • the write operations are performed efficiently using one or two bulk calls

💡 Implementation hints

• The IndexInfo seems like a good way to pass the merged set of migration functions to the RuleDataClient.
• The new API on the RuleDataClient could look something like this:

  ruleDataClient.writeAlerts([
    {
      operation: 'update',
      alertUuid: '...',
      update: (old) => new,
    },
    {
      operation: 'add',
      alertUuid: '...',
      create: (defaults) => new,
    },
    // ...
  ])

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[elasticmachine]

Pinging @elastic/actionable-observability (Team: Actionable Observability)

[emma-raffenne]

This ticket is old and probably deprecated, feel free to reopen it if it is still relevant and needed.