Update and edit logsdb docs for logsdb / synthetic source GA #118303

marciw · 2024-12-09T23:26:36Z

This update focuses on licensing and core functionality for the GA release. All suggestions welcome; some will be moved to new issues (coming soon) for fast-follow or 9.0 improvements.

Updated previews

Logs data stream (entire topic)
Index settings (index mode)
Source fields (banner only)
Index management (screenshots)

Includes a general editing pass on changes made in #115205 (which didn't get a docs team review) 😿

Closes https://github.com/elastic/search-docs-team/issues/193

Note: I compared the docs to the in-progress blog post and didn't see any gaps, but I'll keep checking the blog post and will link to it from the docs when it's ready.

github-actions · 2024-12-09T23:26:49Z

Documentation preview:

✨ Changed pages

elasticsearchmachine · 2024-12-10T00:19:50Z

Pinging @elastic/es-docs (Team:Docs)

docs/reference/data-streams/logs.asciidoc

docs/reference/index-modules.asciidoc

docs/reference/data-streams/logs.asciidoc

leemthompo

Changes look good to me. How we call out subscription requirements is the biggest factual question for me.

Otherwise, my main qualm with this page is the structure/headings. Here's a potential heading structure that tries to regroup info, and might make the page easier to scan in the TOC:

Overview

Basic setup

Storage optimization
- Synthetic source
  - Fields without doc values
- Specialized codecs 

Sort settings
- Index sort defaults
- Sort orders and modes

Data validation defaults
- ignore_malformed  
- ignore_above
- ignore_dynamic_beyond_limit

Full settings reference

docs/reference/data-streams/logs.asciidoc

Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>

marciw · 2024-12-10T15:32:18Z

Changes look good to me. How we call out subscription requirements is the biggest factual question for me.

Otherwise, my main qualm with this page is the structure/headings. Here's a potential heading structure that tries to regroup info, and might make the page easier to scan in the TOC:

Thanks, agreed. Not sure I can tackle this now but will make it a "fast-follow" if not

…ogsdb

shainaraskas

some nits and questions for you! love the edits - it reads much clearer now

docs/reference/data-streams/logs.asciidoc

docs/reference/index-modules.asciidoc

docs/reference/indices/put-index-template.asciidoc

Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>

docs/reference/data-streams/logs.asciidoc

docs/reference/index-modules.asciidoc

Co-authored-by: Oleksandr Kolomiiets <olkolomiiets@gmail.com>

docs/reference/data-streams/logs.asciidoc

shainaraskas

logsdb changes lgtm!

are the tsdb "grandfathered" nuances covered anywhere? wondering how people who have old tsdb data streams should expect new ones to behave, or should account for differences between elasticsearch instances.

docs/reference/data-streams/logs.asciidoc

…#118303) * Update licensing; fix screenshots; edit generally * Small edit for clarity and style * Update docs/reference/index-modules.asciidoc Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> * Apply changes from review Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> * Address review comments * Match similar change from review * More changes from review * Apply suggestions from review Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> * Apply suggestions from review Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> * Update docs/reference/data-streams/logs.asciidoc Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> * Apply suggestions from review Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> * Apply suggestions from review * Change to general subscription note * Apply suggestions from review Co-authored-by: Oleksandr Kolomiiets <olkolomiiets@gmail.com> * Apply suggestions from review Co-authored-by: Oleksandr Kolomiiets <olkolomiiets@gmail.com> * Apply suggestions from review; additional edits * Apply suggestions from review; clarity tweaks * Restore previous paragraph structure and context --------- Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> Co-authored-by: Oleksandr Kolomiiets <olkolomiiets@gmail.com> (cherry picked from commit ae9bb90)

shainaraskas · 2024-12-11T20:16:04Z

💚 All backports created successfully

Status	Branch	Result
✅	8.17

Questions ?

Please refer to the Backport tool documentation

shainaraskas · 2024-12-11T20:17:50Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Questions ?

Please refer to the Backport tool documentation

…#118303) * Update licensing; fix screenshots; edit generally * Small edit for clarity and style * Update docs/reference/index-modules.asciidoc Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> * Apply changes from review Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> * Address review comments * Match similar change from review * More changes from review * Apply suggestions from review Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> * Apply suggestions from review Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> * Update docs/reference/data-streams/logs.asciidoc Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> * Apply suggestions from review Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> * Apply suggestions from review * Change to general subscription note * Apply suggestions from review Co-authored-by: Oleksandr Kolomiiets <olkolomiiets@gmail.com> * Apply suggestions from review Co-authored-by: Oleksandr Kolomiiets <olkolomiiets@gmail.com> * Apply suggestions from review; additional edits * Apply suggestions from review; clarity tweaks * Restore previous paragraph structure and context --------- Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> Co-authored-by: Oleksandr Kolomiiets <olkolomiiets@gmail.com> (cherry picked from commit ae9bb90)

#118504) * Update licensing; fix screenshots; edit generally * Small edit for clarity and style * Update docs/reference/index-modules.asciidoc Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> * Apply changes from review Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> * Address review comments * Match similar change from review * More changes from review * Apply suggestions from review Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> * Apply suggestions from review Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> * Update docs/reference/data-streams/logs.asciidoc Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> * Apply suggestions from review Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> * Apply suggestions from review * Change to general subscription note * Apply suggestions from review Co-authored-by: Oleksandr Kolomiiets <olkolomiiets@gmail.com> * Apply suggestions from review Co-authored-by: Oleksandr Kolomiiets <olkolomiiets@gmail.com> * Apply suggestions from review; additional edits * Apply suggestions from review; clarity tweaks * Restore previous paragraph structure and context --------- Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> Co-authored-by: Oleksandr Kolomiiets <olkolomiiets@gmail.com> (cherry picked from commit ae9bb90) Co-authored-by: Marci W <333176+marciw@users.noreply.github.com>

#118505) * Update licensing; fix screenshots; edit generally * Small edit for clarity and style * Update docs/reference/index-modules.asciidoc Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> * Apply changes from review Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> * Address review comments * Match similar change from review * More changes from review * Apply suggestions from review Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> * Apply suggestions from review Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> * Update docs/reference/data-streams/logs.asciidoc Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> * Apply suggestions from review Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> * Apply suggestions from review * Change to general subscription note * Apply suggestions from review Co-authored-by: Oleksandr Kolomiiets <olkolomiiets@gmail.com> * Apply suggestions from review Co-authored-by: Oleksandr Kolomiiets <olkolomiiets@gmail.com> * Apply suggestions from review; additional edits * Apply suggestions from review; clarity tweaks * Restore previous paragraph structure and context --------- Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> Co-authored-by: Oleksandr Kolomiiets <olkolomiiets@gmail.com> (cherry picked from commit ae9bb90) Co-authored-by: Marci W <333176+marciw@users.noreply.github.com>

tylerperk · 2024-12-11T22:41:15Z

Hi @marciw thanks for all the changes. I'll try to do a full review very soon. I did already notice one thing: In https://elasticsearch_bk_118303.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/index-modules.html#_static_index_settings it lists "tsds" as a valid index mode. AFAIK this is not valid and should be time_series. TSDS is the common abbreviation for that but is not an actual setting value for ES.

marciw · 2024-12-12T15:09:01Z

Hi @marciw thanks for all the changes. I'll try to do a full review very soon. I did already notice one thing: In https://elasticsearch_bk_118303.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/index-modules.html#_static_index_settings it lists "tsds" as a valid index mode. AFAIK this is not valid and should be time_series. TSDS is the common abbreviation for that but is not an actual setting value for ES.

oof, I think that's been there for a long time -- good catch. Fixed in #118579

…#118303) (elastic#118504) * Update licensing; fix screenshots; edit generally * Small edit for clarity and style * Update docs/reference/index-modules.asciidoc Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> * Apply changes from review Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> * Address review comments * Match similar change from review * More changes from review * Apply suggestions from review Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> * Apply suggestions from review Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> * Update docs/reference/data-streams/logs.asciidoc Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> * Apply suggestions from review Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> * Apply suggestions from review * Change to general subscription note * Apply suggestions from review Co-authored-by: Oleksandr Kolomiiets <olkolomiiets@gmail.com> * Apply suggestions from review Co-authored-by: Oleksandr Kolomiiets <olkolomiiets@gmail.com> * Apply suggestions from review; additional edits * Apply suggestions from review; clarity tweaks * Restore previous paragraph structure and context --------- Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> Co-authored-by: Oleksandr Kolomiiets <olkolomiiets@gmail.com> (cherry picked from commit ae9bb90) Co-authored-by: Marci W <333176+marciw@users.noreply.github.com>

@asmith-elastic

👋 howdy, team! ( cc: @asmith-elastic @anniegale9538 for Sev1 01849100 ) The existing docs for Creating LogsDB ([v8](https://www.elastic.co/guide/en/elasticsearch/reference/current/logs-data-stream.html#how-to-use-logsds), [v9](https://docs-v3-preview.elastic.dev/elastic/docs-content/tree/main/manage-data/data-store/data-streams/logs-data-stream); believed ported via elastic/elasticsearch#118303) has users create an index template overriding previous which negates all Fleet-managed Integration index pipelines which breaks ingestion patterns across all `logs-*` 👻. This adds in a warning header that for Fleet-managed integrations (which AFAIK is the only major exception) users should instead be adding the recommended setting [into the component template per existing doc](https://www.elastic.co/guide/en/observability/current/logs-index-template.html#custom-logs-template-edit). TIA! --------- Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>

Update licensing; fix screenshots; edit generally

5f5db01

marciw added >docs General docs changes Team:Docs Meta label for docs team v9.0.0 v8.17.0 v8.17.1 labels Dec 9, 2024

marciw added 2 commits December 9, 2024 18:42

Small edit for clarity and style

b908878

Merge branch 'main' into mw-logsdb

4d8736e

marciw marked this pull request as ready for review December 10, 2024 00:19

marciw requested review from leemthompo, salvatore-campagna, shainaraskas and tylerperk December 10, 2024 00:19

marciw mentioned this pull request Dec 10, 2024

Improve Logsdb docs including default values #115205

Merged

martijnvg reviewed Dec 10, 2024

View reviewed changes

docs/reference/data-streams/logs.asciidoc Outdated Show resolved Hide resolved

leemthompo reviewed Dec 10, 2024

View reviewed changes

docs/reference/index-modules.asciidoc Outdated Show resolved Hide resolved

docs/reference/data-streams/logs.asciidoc Outdated Show resolved Hide resolved

leemthompo reviewed Dec 10, 2024

View reviewed changes

docs/reference/data-streams/logs.asciidoc Outdated Show resolved Hide resolved

marciw and others added 2 commits December 10, 2024 10:29

Update docs/reference/index-modules.asciidoc

4a01131

Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>

Apply changes from review

ce7a558

Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>

marciw added 4 commits December 10, 2024 10:44

Address review comments

737ea70

Merge branch 'main' into mw-logsdb

b2a34e2

Match similar change from review

c5cbf91

Merge branch 'mw-logsdb' of github.com:marciw/elasticsearch into mw-l…

d0cf775

…ogsdb

shainaraskas reviewed Dec 10, 2024

View reviewed changes

marciw and others added 3 commits December 10, 2024 10:53

More changes from review

f01ce45

Apply suggestions from review

0ddf0e3

Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>

Apply suggestions from review

0765a05

Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>

Merge branch 'main' into mw-logsdb

773cb92

lkts reviewed Dec 10, 2024

View reviewed changes

marciw and others added 4 commits December 10, 2024 17:55

Apply suggestions from review

140c3f8

Co-authored-by: Oleksandr Kolomiiets <olkolomiiets@gmail.com>

Apply suggestions from review

7c3fb05

Co-authored-by: Oleksandr Kolomiiets <olkolomiiets@gmail.com>

Apply suggestions from review; additional edits

e9bfa0e

Merge branch 'main' into mw-logsdb

4ebd4fa

salvatore-campagna reviewed Dec 11, 2024

View reviewed changes

docs/reference/data-streams/logs.asciidoc Show resolved Hide resolved

shainaraskas approved these changes Dec 11, 2024

View reviewed changes

docs/reference/data-streams/logs.asciidoc Outdated Show resolved Hide resolved

docs/reference/data-streams/logs.asciidoc Outdated Show resolved Hide resolved

marciw removed the v9.0.0 label Dec 11, 2024

marciw added 2 commits December 11, 2024 11:42

Apply suggestions from review; clarity tweaks

c9419d1

Merge branch 'main' into mw-logsdb

61da7ab

shainaraskas approved these changes Dec 11, 2024

View reviewed changes

docs/reference/data-streams/logs.asciidoc Outdated Show resolved Hide resolved

marciw added 2 commits December 11, 2024 12:57

Restore previous paragraph structure and context

4620a37

Merge branch 'main' into mw-logsdb

fccda9c

marciw merged commit ae9bb90 into elastic:main Dec 11, 2024
5 checks passed

marciw added the auto-backport Automatically create backport pull requests when merged label Dec 11, 2024

shainaraskas mentioned this pull request Dec 11, 2024

[8.17] Update and edit logsdb docs for logsdb / synthetic source GA (#118303) #118504

Merged

shainaraskas mentioned this pull request Dec 11, 2024

[8.x] Update and edit logsdb docs for logsdb / synthetic source GA (#118303) #118505

Merged

marciw mentioned this pull request Dec 12, 2024

Fix invalid index mode #118579

Merged

stefnestor mentioned this pull request Mar 27, 2025

(Bug) Modify Index Template to LogsDB elastic/docs-content#962

Merged

Update and edit logsdb docs for logsdb / synthetic source GA #118303

Update and edit logsdb docs for logsdb / synthetic source GA #118303

Uh oh!

Conversation

marciw commented Dec 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 9, 2024

Uh oh!

elasticsearchmachine commented Dec 10, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

leemthompo left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

marciw commented Dec 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shainaraskas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shainaraskas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shainaraskas commented Dec 11, 2024

💚 All backports created successfully

Questions ?

Uh oh!

shainaraskas commented Dec 11, 2024

💚 All backports created successfully

Questions ?

Uh oh!

tylerperk commented Dec 11, 2024

Uh oh!

marciw commented Dec 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

marciw commented Dec 9, 2024 •

edited

Loading

leemthompo left a comment •

edited

Loading

marciw commented Dec 10, 2024 •

edited

Loading

marciw commented Dec 12, 2024 •

edited

Loading