(Bug) Modify Index Template to LogsDB

[stefnestor]

👋 howdy, team! ( cc: @asmith-elastic @anniegale9538 for Sev1 01849100 )

The existing docs for Creating LogsDB (v8, v9; believed ported via elastic/elasticsearch#118303) has users create an index template overriding previous which negates all Fleet-managed Integration index pipelines which breaks ingestion patterns across all logs-* 👻.

This adds in a warning header that for Fleet-managed integrations (which AFAIK is the only major exception) users should instead be adding the recommended setting into the component template per existing doc. TIA!

[eedugon]

I'm totally ok with the changes, although I believe we will still get problems caused by this doc (the provided example is a call for problems the way I see it), and my proposal would be to go one step further and apply a few more changes:

I think the example in create data stream using logs-* is a bad choice, as the users will lose our default component templates for ECS mappings and other stuff:

(and I want to believe that that's not on purpose, otherwise it should also be explained).

What I would suggest, for your consideration, is to explain in the create data stream section that it's just an example (the key is to show how index.mode needs to be set to logsdb), and use custom-logs-* in the pattern to avoid potential issues, as an example.

And then add a note with something like (feel free to improve the wording):

::::{note}
If you want to enable `logsdb` mode for data streams that are already covered by existing index templates, such as the default `logs-*-*` template or templates installed by Fleet Integrations, modify or extend the [component templates](LINK) associated with those index templates instead of creating a new one.

For example, the default `logs-*-*` index template is linked to a `logs@custom` component template that you can use for this purpose if you want all generic `logs-*-*` data streams to be created as `logsdb` data streams.
::::

So IMHO, no users should ever run the provided example in any case. But I might be missing something here.

As a an extra comment, integrations create index templates with priorities > 200. I completely agree with the mess that this doc can cause, but the documented example with priority: 101 shouldn't break integrations data ingestion as It has been reported.

[eedugon]

I'm totally ok with the change, but I've shared a message to propose a slightly bigger change.
We could do that in another PR though.

cc: @marciw , @mdbirnstiehl , feel free to share your thoughts here :)

[stefnestor]

Thanks @eedugon !

Agreed. Foremost care to stop gap that users using Fleet need to stop running this code and breaking their whole ingest. I like your call out that we can modify the index_patterns & have updated it to my-datastream-* in PR.

I would normally think this ballpark should go back to Elasticsearch+Fleet+Solutions Dev/Product for those teams to align/confirm how this should be setup+worded. In particular, this example reads to me like it was written to work for Elasticsearch but did not consider Fleet caveats. So does it+we also not know any other Fleet restrictions and/or break anything further down the pipeline for the Solutions (e.g. farther down the page if Elasticsearch Dev's host.name field type conflicts with the Integrations team it will break ingest+solutions again)? Part of me assumes not, but it falling on its face on the first implementation step makes me want confirmation just in case 🙂.

That's my opinion which brings me to my uncertainty: that discussion is for like 3 Dev teams but the PR now comes to docs-content but there aren't labels in this repo to tag those teams, so do I escalate this conversation to y'all to work with Dev/Product or how do we get folks involved in the conversation similar to how we used to?

[stefnestor]

Merged main to try to get rid of pipeline errors unrelated to my file's changes. ⏳

[lucabelluccini]

Thanks @stefnestor !

Yes that example at https://www.elastic.co/guide/en/elasticsearch/reference/current/logs-data-stream.html#how-to-use-logsds is very nasty.

I think this should be overlooked by the PM of Fleet + PMs of Integrations (and possibly the whole solution teams) and associated leads. So @nimarezainia , @jamiehynds, @daniela-elastic to begin with (for the integrations of o11y & security).

I've opened this elastic/integrations#12298 some time ago to have an official "guideline" to enable LogsDB on specific data streams.

[stefnestor]

Confirmed high-level discussion will continue in elastic/integrations#12298 .