[Fleet] Fix conflicting dynamic template mappings for intermediate objects #175970
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When there are multiple dynamic templates with the same name, we only use the first appearance. Add an exception for intermediate objects, so we use the last appearance. This way we avoid that less specific dynamic templates created for intermediate objects match before more specific dynamic templates defined in packages.
jsoriano
added
the
Team:Fleet
|
Pinging @elastic/fleet (Team:Fleet) |
🤖 GitHub commentsExpand to view the GitHub comments
Just comment with:
|
jsoriano
added
backport:prev-minor
x-pack/plugins/fleet/server/services/epm/elasticsearch/template/template.ts
Show resolved
Hide resolved
💚 Build Succeeded
Metrics [docs]
History
To update your PR or re-run it, just comment with: cc @jsoriano |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Jan 31, 2024
…jects (elastic#175970) When there are multiple dynamic templates with the same name, we only use the first appearance. Add an exception for intermediate objects, so we use the last appearance. This way we avoid that less specific dynamic templates created for intermediate objects match before more specific dynamic templates defined in packages. This was a problem for example in the Prometheus package, where there are definitions for `prometheus.*.value` and `prometheus.*.histogram`. For both cases Fleet tries to create a dynamic template for the intermediate objects `prometheus.*`. If this dynamic template is inserted before the dynamic template for the histograms, it matches for them, causing mapping errors on ingestion, like the following one: ``` mapper [prometheus.queue_duration.histogram.values] cannot be changed from type [float] to [long] ``` With this change, when multiple definitions generate the same dynamic template for intermediate objects, only the last one ends up in the template. These intermediate objects were introduced in 8.12, after elastic#169981. (cherry picked from commit 7274f6e)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Jan 31, 2024
…iate objects (#175970) (#175991) # Backport This will backport the following commits from `main` to `8.12`: - [[Fleet] Fix conflicting dynamic template mappings for intermediate objects (#175970)](#175970) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Jaime Soriano Pastor","email":"jaime.soriano@elastic.co"},"sourceCommit":{"committedDate":"2024-01-31T14:51:39Z","message":"[Fleet] Fix conflicting dynamic template mappings for intermediate objects (#175970)\n\nWhen there are multiple dynamic templates with the same name, we only\r\nuse the first appearance.\r\nAdd an exception for intermediate objects, so we use the last\r\nappearance. This way we avoid that less specific dynamic templates\r\ncreated for intermediate objects match before more specific dynamic\r\ntemplates defined in packages.\r\n\r\nThis was a problem for example in the Prometheus package, where there\r\nare definitions for `prometheus.*.value` and `prometheus.*.histogram`.\r\nFor both cases Fleet tries to create a dynamic template for the\r\nintermediate objects `prometheus.*`. If this dynamic template is\r\ninserted before the dynamic template for the histograms, it matches for\r\nthem, causing mapping errors on ingestion, like the following one:\r\n```\r\nmapper [prometheus.queue_duration.histogram.values] cannot be changed from type [float] to [long]\r\n```\r\nWith this change, when multiple definitions generate the same dynamic\r\ntemplate for intermediate objects, only the last one ends up in the\r\ntemplate.\r\n\r\nThese intermediate objects were introduced in 8.12, after https://github.com/elastic/kibana/pull/169981.","sha":"7274f6e2e152928418a677fcb6b9c3550b8a6227","branchLabelMapping":{"^v8.13.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Fleet","backport:prev-minor","v8.12.1","v8.13.0"],"title":"[Fleet] Fix conflicting dynamic template mappings for intermediate objects","number":175970,"url":"https://github.com/elastic/kibana/pull/175970","mergeCommit":{"message":"[Fleet] Fix conflicting dynamic template mappings for intermediate objects (#175970)\n\nWhen there are multiple dynamic templates with the same name, we only\r\nuse the first appearance.\r\nAdd an exception for intermediate objects, so we use the last\r\nappearance. This way we avoid that less specific dynamic templates\r\ncreated for intermediate objects match before more specific dynamic\r\ntemplates defined in packages.\r\n\r\nThis was a problem for example in the Prometheus package, where there\r\nare definitions for `prometheus.*.value` and `prometheus.*.histogram`.\r\nFor both cases Fleet tries to create a dynamic template for the\r\nintermediate objects `prometheus.*`. If this dynamic template is\r\ninserted before the dynamic template for the histograms, it matches for\r\nthem, causing mapping errors on ingestion, like the following one:\r\n```\r\nmapper [prometheus.queue_duration.histogram.values] cannot be changed from type [float] to [long]\r\n```\r\nWith this change, when multiple definitions generate the same dynamic\r\ntemplate for intermediate objects, only the last one ends up in the\r\ntemplate.\r\n\r\nThese intermediate objects were introduced in 8.12, after https://github.com/elastic/kibana/pull/169981.","sha":"7274f6e2e152928418a677fcb6b9c3550b8a6227"}},"sourceBranch":"main","suggestedTargetBranches":["8.12"],"targetPullRequestStates":[{"branch":"8.12","label":"v8.12.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.13.0","branchLabelMappingKey":"^v8.13.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/175970","number":175970,"mergeCommit":{"message":"[Fleet] Fix conflicting dynamic template mappings for intermediate objects (#175970)\n\nWhen there are multiple dynamic templates with the same name, we only\r\nuse the first appearance.\r\nAdd an exception for intermediate objects, so we use the last\r\nappearance. This way we avoid that less specific dynamic templates\r\ncreated for intermediate objects match before more specific dynamic\r\ntemplates defined in packages.\r\n\r\nThis was a problem for example in the Prometheus package, where there\r\nare definitions for `prometheus.*.value` and `prometheus.*.histogram`.\r\nFor both cases Fleet tries to create a dynamic template for the\r\nintermediate objects `prometheus.*`. If this dynamic template is\r\ninserted before the dynamic template for the histograms, it matches for\r\nthem, causing mapping errors on ingestion, like the following one:\r\n```\r\nmapper [prometheus.queue_duration.histogram.values] cannot be changed from type [float] to [long]\r\n```\r\nWith this change, when multiple definitions generate the same dynamic\r\ntemplate for intermediate objects, only the last one ends up in the\r\ntemplate.\r\n\r\nThese intermediate objects were introduced in 8.12, after https://github.com/elastic/kibana/pull/169981.","sha":"7274f6e2e152928418a677fcb6b9c3550b8a6227"}}]}] BACKPORT--> Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
jloleysens
added a commit
to jloleysens/kibana
that referenced
this pull request
Feb 1, 2024
* main: use build hash in FTR tests [Security Solution] Fix moderate typo (elastic#175883) [Fleet] Fix conflicting dynamic template mappings for intermediate objects (elastic#175970) [Visualize] Prevent overwriting managed content (elastic#175274) [SLO] Add/edit form mark optional fields (elastic#175807) skip failing test suite (elastic#175984) [data views] Provide method of excluding data tiers when getting field list (elastic#167946) [Dataset quality] State management (elastic#174906) [Cloud Security] add is_internal config option for outputs (elastic#175546)
CoenWarmer
pushed a commit
to CoenWarmer/kibana
that referenced
this pull request
Feb 15, 2024
…jects (elastic#175970) When there are multiple dynamic templates with the same name, we only use the first appearance. Add an exception for intermediate objects, so we use the last appearance. This way we avoid that less specific dynamic templates created for intermediate objects match before more specific dynamic templates defined in packages. This was a problem for example in the Prometheus package, where there are definitions for `prometheus.*.value` and `prometheus.*.histogram`. For both cases Fleet tries to create a dynamic template for the intermediate objects `prometheus.*`. If this dynamic template is inserted before the dynamic template for the histograms, it matches for them, causing mapping errors on ingestion, like the following one: ``` mapper [prometheus.queue_duration.histogram.values] cannot be changed from type [float] to [long] ``` With this change, when multiple definitions generate the same dynamic template for intermediate objects, only the last one ends up in the template. These intermediate objects were introduced in 8.12, after elastic#169981.
fkanout
pushed a commit
to fkanout/kibana
that referenced
this pull request
Mar 4, 2024
…jects (elastic#175970) When there are multiple dynamic templates with the same name, we only use the first appearance. Add an exception for intermediate objects, so we use the last appearance. This way we avoid that less specific dynamic templates created for intermediate objects match before more specific dynamic templates defined in packages. This was a problem for example in the Prometheus package, where there are definitions for `prometheus.*.value` and `prometheus.*.histogram`. For both cases Fleet tries to create a dynamic template for the intermediate objects `prometheus.*`. If this dynamic template is inserted before the dynamic template for the histograms, it matches for them, causing mapping errors on ingestion, like the following one: ``` mapper [prometheus.queue_duration.histogram.values] cannot be changed from type [float] to [long] ``` With this change, when multiple definitions generate the same dynamic template for intermediate objects, only the last one ends up in the template. These intermediate objects were introduced in 8.12, after elastic#169981.
Merged
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
When there are multiple dynamic templates with the same name, we only use the first appearance.
Add an exception for intermediate objects, so we use the last appearance. This way we avoid that less specific dynamic templates created for intermediate objects match before more specific dynamic templates defined in packages.
This was a problem for example in the Prometheus package, where there are definitions for
prometheus.*.valueandprometheus.*.histogram. For both cases Fleet tries to create a dynamic template for the intermediate objectsprometheus.*. If this dynamic template is inserted before the dynamic template for the histograms, it matches for them, causing mapping errors on ingestion, like the following one:With this change, when multiple definitions generate the same dynamic template for intermediate objects, only the last one ends up in the template.
These intermediate objects were introduced in 8.12, after #169981.
Checklist