Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move kibana reporting data stream settings into component template #107581

Conversation

dakrone
Copy link
Member

@dakrone dakrone commented Apr 17, 2024

Previously these were contained in the index template, however, Kibana needs to be able to make overrides to only the settings, so factoring these out would allow them to do this (in such a way that they can be overridden by the kibana-reporting@custom component template as well).

Relates to #97765

Previously these were contained in the index template, however, Kibana needs to be able to make
overrides to only the settings, so factoring these out would allow them to do this (in such a way
that they can be overridden by the `kibana-reporting@custom` component template as well).

Relates to elastic#97765
@dakrone dakrone requested a review from tsullivan April 17, 2024 17:48
@elasticsearchmachine elasticsearchmachine added the Team:Data Management Meta label for data/management team label Apr 17, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

Copy link
Member

@tsullivan tsullivan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@nielsbauman nielsbauman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Apologies for the wait @tsullivan

@dakrone
Copy link
Member Author

dakrone commented May 8, 2024

@elasticmachine update branch

@dakrone dakrone added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label May 8, 2024
@elasticsearchmachine elasticsearchmachine merged commit ef8b610 into elastic:main May 8, 2024
15 checks passed
@dakrone dakrone deleted the add-settings-component-template-for-kibana-reporting branch May 8, 2024 20:07
markjhoy pushed a commit to markjhoy/elasticsearch that referenced this pull request May 9, 2024
…lastic#107581)

Previously these were contained in the index template, however, Kibana
needs to be able to make overrides to only the settings, so factoring
these out would allow them to do this (in such a way that they can be
overridden by the `kibana-reporting@custom` component template as well).

Relates to elastic#97765
tsullivan added a commit to elastic/kibana that referenced this pull request May 21, 2024
## Summary

Closes #161608

* [X] Depends on elastic/elasticsearch#97765
* [x] Depends on elastic/elasticsearch#107581
* [x] Add create a new report job and check the details of the templated
data stream.
* [x] Run Discover tests in Flaky Test Runner:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5999

## Release Note

Reporting internal storage has been changed from using regular indices
to a data stream configuration for a more efficient sharding strategy.
This change is not expected to have any impact to users.

## Screenshots

### Upgrade test (manual process)
Using a report generated before this change, and a report generated
after "upgrading":

![image](https://github.com/elastic/kibana/assets/908371/f92193d8-70d6-4fa5-b1b7-8f6c1a0a5e9f)
Even though the two reports are in different types storage, they are
still managed by the same policy:

![image](https://github.com/elastic/kibana/assets/908371/9bd68d99-72ed-4cf0-bef9-55e644f039b7)
Looking at the details of the policy shows how the different types of
storage are used:

![image](https://github.com/elastic/kibana/assets/908371/6c0d1f80-97cb-4990-b2a8-45deab7528bc)

### Log lines

Initial startup in clean environment
```
[2024-05-13T13:22:49.138-07:00][INFO ][plugins.reporting.store] Creating ILM policy for reporting data stream: kibana-reporting
[2024-05-13T13:22:53.337-07:00][INFO ][plugins.reporting.store] Linking ILM policy to reporting data stream: .kibana-reporting, component template: kibana-reporting@custom
```

Kibana restart with ES running continuously
```
[2024-05-13T13:24:32.733-07:00][DEBUG][plugins.reporting.store] Found ILM policy kibana-reporting; skipping creation.
[2024-05-13T13:24:32.733-07:00][INFO ][plugins.reporting.store] Linking ILM policy to reporting data stream: .kibana-reporting, component template: kibana-reporting@custom
```

### Checklist

- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
~~See
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5302
(internal link)~~
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) :Data Management/Data streams Data streams and their lifecycles >non-issue Team:Data Management Meta label for data/management team v8.15.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants