Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fleet] fix index template from datastream name #166941

Merged
merged 4 commits into from Sep 22, 2023

Conversation

juliaElastic
Copy link
Contributor

@juliaElastic juliaElastic commented Sep 21, 2023

Summary

Resolve #164269

Some context why I picked this up now: #162772 (comment)

To verify:

  • Make sure 8.8+ apm package is installed
  • Create data stream PUT _data_stream/metrics-apm.app.default-default
  • Reinstall apm package from API or UI
  • Check kibana info logs, expect to not see simulate template error and rollover like below
[2023-09-21T15:54:36.559+02:00][INFO ][plugins.fleet] Mappings update for metrics-apm.app.default-default failed due to ResponseError: illegal_argument_exception
        Root causes:
                illegal_argument_exception: unable to simulate template [metrics-apm.app.default] that does not exist
[2023-09-21T15:54:36.559+02:00][INFO ][plugins.fleet] Triggering a rollover for metrics-apm.app.default-default

Checklist

@juliaElastic juliaElastic added release_note:fix backport:prev-minor Backport to the previous minor version (i.e. one version back from main) labels Sep 21, 2023
@juliaElastic juliaElastic self-assigned this Sep 21, 2023
@apmmachine
Copy link
Contributor

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
  • /oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@juliaElastic juliaElastic marked this pull request as ready for review September 21, 2023 14:57
@juliaElastic juliaElastic requested a review from a team as a code owner September 21, 2023 14:57
esClient: ElasticsearchClient,
dataStreamName: string
): Promise<string> {
const dataStream = await esClient.indices.getDataStream({
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just double checking, does this getDataStream call work even if the stream has no data?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good question, I'll check

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

checked and if the data stream doesn't exist, it is not hitting this code path

@botelastic botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Sep 21, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

Copy link
Contributor

@hop-dev hop-dev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't believe we didn't always do it this way, makes me scared there is some niche reason that we did :D

@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #50 / transform basic license transform - actions "after all" hook in "transform - actions"
  • [job] [logs] FTR Configs #50 / transform basic license transform - actions starting batch transform with pivot configuration start transform

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @juliaElastic

@juliaElastic juliaElastic merged commit 3ff82f2 into elastic:main Sep 22, 2023
25 checks passed
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Sep 22, 2023
## Summary

Resolve elastic#164269

Some context why I picked this up now:
elastic#162772 (comment)

To verify:
- Make sure 8.8+ apm package is installed
- Create data stream `PUT _data_stream/metrics-apm.app.default-default`
- Reinstall apm package from API or UI
- Check kibana info logs, expect to not see simulate template error and
rollover like below

```
[2023-09-21T15:54:36.559+02:00][INFO ][plugins.fleet] Mappings update for metrics-apm.app.default-default failed due to ResponseError: illegal_argument_exception
        Root causes:
                illegal_argument_exception: unable to simulate template [metrics-apm.app.default] that does not exist
[2023-09-21T15:54:36.559+02:00][INFO ][plugins.fleet] Triggering a rollover for metrics-apm.app.default-default
```

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

(cherry picked from commit 3ff82f2)
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
8.10

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Sep 26, 2023
…7008)

# Backport

This will backport the following commits from `main` to `8.10`:
- [[Fleet] fix index template from datastream name
(#166941)](#166941)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Julia
Bardi","email":"90178898+juliaElastic@users.noreply.github.com"},"sourceCommit":{"committedDate":"2023-09-22T08:20:05Z","message":"[Fleet]
fix index template from datastream name (#166941)\n\n##
Summary\r\n\r\nResolve
#164269 context why
I picked this up
now:\r\nhttps://github.com//issues/162772#issuecomment-1728031080\r\n\r\nTo
verify:\r\n- Make sure 8.8+ apm package is installed\r\n- Create data
stream `PUT _data_stream/metrics-apm.app.default-default`\r\n- Reinstall
apm package from API or UI\r\n- Check kibana info logs, expect to not
see simulate template error and\r\nrollover like
below\r\n\r\n```\r\n[2023-09-21T15:54:36.559+02:00][INFO
][plugins.fleet] Mappings update for metrics-apm.app.default-default
failed due to ResponseError: illegal_argument_exception\r\n Root
causes:\r\n illegal_argument_exception: unable to simulate template
[metrics-apm.app.default] that does not
exist\r\n[2023-09-21T15:54:36.559+02:00][INFO ][plugins.fleet]
Triggering a rollover for
metrics-apm.app.default-default\r\n```\r\n\r\n\r\n### Checklist\r\n\r\n-
[x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"3ff82f2c17e532fed5d5544ed9bbae4f6e7331af","branchLabelMapping":{"^v8.11.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Fleet","backport:prev-minor","v8.11.0"],"number":166941,"url":"#166941
fix index template from datastream name (#166941)\n\n##
Summary\r\n\r\nResolve
#164269 context why
I picked this up
now:\r\nhttps://github.com//issues/162772#issuecomment-1728031080\r\n\r\nTo
verify:\r\n- Make sure 8.8+ apm package is installed\r\n- Create data
stream `PUT _data_stream/metrics-apm.app.default-default`\r\n- Reinstall
apm package from API or UI\r\n- Check kibana info logs, expect to not
see simulate template error and\r\nrollover like
below\r\n\r\n```\r\n[2023-09-21T15:54:36.559+02:00][INFO
][plugins.fleet] Mappings update for metrics-apm.app.default-default
failed due to ResponseError: illegal_argument_exception\r\n Root
causes:\r\n illegal_argument_exception: unable to simulate template
[metrics-apm.app.default] that does not
exist\r\n[2023-09-21T15:54:36.559+02:00][INFO ][plugins.fleet]
Triggering a rollover for
metrics-apm.app.default-default\r\n```\r\n\r\n\r\n### Checklist\r\n\r\n-
[x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"3ff82f2c17e532fed5d5544ed9bbae4f6e7331af"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.11.0","labelRegex":"^v8.11.0$","isSourceBranch":true,"state":"MERGED","url":"#166941
fix index template from datastream name (#166941)\n\n##
Summary\r\n\r\nResolve
#164269 context why
I picked this up
now:\r\nhttps://github.com//issues/162772#issuecomment-1728031080\r\n\r\nTo
verify:\r\n- Make sure 8.8+ apm package is installed\r\n- Create data
stream `PUT _data_stream/metrics-apm.app.default-default`\r\n- Reinstall
apm package from API or UI\r\n- Check kibana info logs, expect to not
see simulate template error and\r\nrollover like
below\r\n\r\n```\r\n[2023-09-21T15:54:36.559+02:00][INFO
][plugins.fleet] Mappings update for metrics-apm.app.default-default
failed due to ResponseError: illegal_argument_exception\r\n Root
causes:\r\n illegal_argument_exception: unable to simulate template
[metrics-apm.app.default] that does not
exist\r\n[2023-09-21T15:54:36.559+02:00][INFO ][plugins.fleet]
Triggering a rollover for
metrics-apm.app.default-default\r\n```\r\n\r\n\r\n### Checklist\r\n\r\n-
[x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"3ff82f2c17e532fed5d5544ed9bbae4f6e7331af"}}]}]
BACKPORT-->

Co-authored-by: Julia Bardi <90178898+juliaElastic@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:prev-minor Backport to the previous minor version (i.e. one version back from main) release_note:fix Team:Fleet Team label for Observability Data Collection Fleet team v8.10.3 v8.11.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Fleet] Simulate template doesn't always use the right index template
6 participants