[Logs UI] Use Kibana index patterns in the source configuration

[weltenwort]

⚠️ This is a meta issue.

Summary

Kibana index patterns will be the basis for analyst-created runtime fields and related features. The Logs UI wants to make use of that and should therefore base its index configuration on Kibana index patterns.

Task breakdown

KIP support

• [Logs UI] Support Kibana index pattern references in log source configuration #92650
• [Logs UI] Support Kibana index patterns in settings UI #92656
• [Logs UI] Validate and explain the Kibana index pattern settings #92695
• [Logs UI] Support querying runtime fields defined on Kibana index patterns #92881
• [Logs UI] [WIP] Prompt for a data view when log source not configured #98277
• Use KIP creation flyout

Working with runtime fields

These tasks are quite speculative and highly depend on the outcome of the ongoing design and UX research.

• Use runtime field editor flyout
• Add “refine log entry” workflow

Background

The indices read by the Logs UI are currently set as a plain Elasticsearch index ane pattern in the logAlias field of each source configuration saved object. While this is a flexible way to configure the set of indices, it makes integrating with the Kibana features that rely on Kibana index patterns very difficult. These include the aforementioned upcoming runtime fields, but also saved searches, visualizations and dashboards.

The historic arguments for implementing the index configuration that way have partly become void and the Kibana team is making an effort (#78376) to remove remaining hurdles:

Complexity of the user flow

History: Relying on a plain index name pattern means that we can offer a working out-of the box experience to the user. This was in contrast to the jarring "no index pattern found" message that greeted a user in discover when setting up a new Kibana instance or space. It required the user to leave the app context and create a Kibana index pattern in the management section.

Outlook: With the upcoming introduction of an index pattern creation flyout (#84725) the user can create new index patterns without leaving the app context.

Integrity of the configuration

History: Kibana index patterns used to store their field list in their saved object, which had to be refreshed explicitly by a user in the management section. That meant that the field list might be out of date or inappropriate for a user's permissions, which in turn could lead to unexpected breakage in the Logs UI. Furthermore, a deletion of the Kibana index pattern in the management section could leave the Logs UI source configuration in an inconsistent state because it refers to a non-existant Kibana index pattern.

Outlook: The Kibana index patterns don't store the full field list anymore. Only overrides are stored in a saved object, which removes the need for manual refreshes and reduces the dependence on the configuring user's permissions. To prevent breaking the integrity of reference graphs, saved objects can by now also declare dependencies on other saved objects.

Proposed solution

Instead of entering an index name pattern into an input text field, the user would be able to select one of the existing Kibana index patterns. If no suitable index pattern was found, the user could create it via the Kibana index pattern creation flyout without leaving the app.

Impact on the user flow

Instead of being dropped right into the log stream, the user would be presented with a prompt to set up the source configuration. Although this onboarding step could be seen as an obstacle, it has the advantage of making the possibility to change the underlying indices visible to the user.

Impact on the settings UI

Where the settings UI currently offers a text input field to enter an index name pattern, a list of available index patterns is shown. The listed elements can be checked for suitability and only selectable if they fulfill criteria like "has a @timestamp date field" and "has a message text field". Using the flyout created in (#84725) we could also offer the ability for the user to immediately create a new index pattern if desired.

Impact on alerting

The log threshold alert definitions currently store a reference to the source configuration, which the executors load to look up the indices and fields to use. This could be adapted to look up the index name pattern and timestamp field in the Kibana index pattern with a few localized changes.

Impact on the anomaly detection integration

The anomaly detection jobs currently capture the index name patterns and fields at job creation time. After jobs have been created, the UI warns if the source configuration and the job configurations diverge at a later time. This could be adapted to look up the index name pattern and timestamp field in the Kibana index pattern with a few localized changes.

Possible index pattern validation criteria

• @timestamp date field
• message text field

Impact on the source configuration saved object

Instead of storing a string in the logAlias field the saved object would store a property logIndices that contains an extensible descriptor:

interface LogIndexPatternReference {
  type: 'index-pattern';
  indexPattern: string;
}

interface LogIndexNameReference {
  type: 'index-name';
  indexName: string;
}

type LogIndexReference = LogIndexPatternReference | LogIndexNameReference;

The current logAlias field could be migrated to a LogIndexNameReference and continue to function as before even though the UI pushes the user towards using Kibana index patterns.

Backwards compatibility

The migration strategy described above combined with the choice of the legacy index name option in the UI could make this a backwards-compatible enhancement.

Outlook

• source switching in UI
• datastreams
• in-place index pattern editing
• index pattern cloning ("save as new")
• runtime fields

Related links

• Kibana index patterns roadmap: [Meta] Index Patterns Roadmap #78376
• Kibana index pattern creation flyout discussion: [discuss] Index pattern creation dialog #67711
• Kibana index pattern creation flyout implementation issue: [index patterns] Create index pattern flyout #84725

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)