[Internal]: document the recommended process for enabling logsDB on existing Integration-managed data streams

[tommyers-elastic]

Description

all the context is here

https://github.com/elastic/obs-integration-team/issues/551

@felixbarny is the best person to provide the technical details on the actual enablement proces and any information about impacts the setting might have (on ingest/storage perf etc).

Resources

https://www.elastic.co/docs/manage-data/data-store/data-streams/logs-data-stream

Which documentation set does this change impact?

Elastic On-Prem and Cloud (all)

Feature differences

n/a

What release is this request related to?

N/A

Serverless release

already released

Collaboration model

The engineering team

Point of contact.

Main contact: @felixbarny

Stakeholders: @daniela-elastic @tommyers-elastic

[felixbarny]

Happy to provide some general guidance.

Looks like we already have docs on how to use logs@custom in the following places:

• https://www.elastic.co/docs/solutions/observability/logs/logs-index-template-reference
• https://www.elastic.co/docs/reference/fleet/data-streams

We should extend it to add an example for how to enable LogsDB for all logs data streams or for a specific integration. Both documentation pages talk about something very similar, is there a way to consolidate the documentation? At a minimum, these should cross-link to one another, explaining in what context which doc page is relevant.

The Elasticsearch LogsDB documentation also has a section about upgrading to LogsDB. Can we link to the above docs or consolidate?

any information about impacts the setting might have (on ingest/storage perf etc).

The only place where we should document this is the existing LogsDB documentation: https://www.elastic.co/docs/manage-data/data-store/data-streams/logs-data-stream. We shouldn't fragment this to several places as it will make updating the information very difficult. We may update the docs if you think there's something missing.

[georgewallace]

This deals with logsdb which @marciw has been working on.

[felixbarny]

@georgewallace could you help prioritizing this issue? @lucabelluccini reported that this is causing confusion for users (specifically those using Elastic integrations). I think the bulk of the changes should be done in the platform LogsDB docs and then link out to integration specifics if needed.

Specifically, I think the Upgrade to LogsDB docs could be enhanced to mention tradeoffs for manually enabling LogsDB when it's not enabled by default. I think that includes mentioning that upgrading should be done with care if the cluster is operating at capacity already so that the additional ingest overhead of LogsDB adds a straw that breaks the camel's back. We should also mention how to use logs@custom to enable LogsDB. This is where we may also want to link to integration docs to enable LogsDB for specific integrations rather than all of them (which is what logs@custom would do).

[lucabelluccini]

I think we need to capture:

• What happens for Elastic Integrations, APM, Synthetics... with LogsDB
  • Serverless: enabled by default
  • Stateful: we enable logsdb on 9.x only if the data stream of the integration didn't exist yet - for existing data streams, we do not switch to logsdb
• How a user can switch to logsdb (or if we should discourage them doing so) a data stream of one integration (via the <dataset>@custom component template)
• What are the integrations in which we cannot/should not use logsdb.
• How a user can switch logsdb by default for all logs in pre 9.x (e.g. 8.17), how to make it behave like 9.x

I think we might need to involve both O11y & Security integrations leads or PMs for such inputs.

[georgewallace]

@marciw can you please take a look at this one?

[tommyers-elastic]

i added some draft content for a) checking the status of logsdb for a given integration and b) enabling logsdb for a given integration, or for all logs in the cluster

in both cases i found the method to be quite painful, and i think customers are going to have a hard time with it. i left some comments regarding this in the doc.

@felixbarny would you mind taking a look?

https://docs.google.com/document/d/1EsPIgMDRcX2y5_ineoHIaa_8v7IFNth9nMxZUisRBYU/edit?usp=sharing