[ML] AIOps: Fixes missing field caps filters for log rate analysis. #181109
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
walterra
added
:ml
v8.14.0
v8.15.0
release_note:fix
Feature:ML/AIOps
walterra
requested review from
darnautov,
jgowdyelastic and
peteharverson
and removed request for
jgowdyelastic
|
Pinging @elastic/ml-ui (:ml) |
peteharverson
approved these changes
Apr 18, 2024
💚 Build Succeeded
Metrics [docs]
History
To update your PR or re-run it, just comment with: cc @walterra |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Apr 22, 2024
…astic#181109) ## Summary Part of elastic#172981. Field caps requests can be heavy calls in larger clusters. For all other queries for log rate analysis we were applying filters based on the time range selection. This was missing from the field caps call. The following parameters were added to improve the call: - `index_filter`: Adds a range filter to only get field caps from indices spanning the deviation time range. - `filters`: `-metadata` was added to not return fields like `_id` and esp. `_tier`. We previously had a manually check for `_tier` which is now unnecessary using this option. - `types`: Previously we fetched all field types and then filtered out the ones we don't support. This option allows us to pass in the supported fields right away and not return unsupported ones in the first place. ---- Here are examples that show how `index_filter` get applied correctly: Here the deviation selection spans only 1 month and that is reflected in the response from the field caps call: <img width="1026" alt="image" src="https://github.com/elastic/kibana/assets/230104/50a00e5a-2b59-4ae5-9d50-e2ed766f68f5"> ``` { indices: [ 'gallery-2021-11' ], fields: { ... } } ``` Now the deviation selection covers more months: <img width="1010" alt="image" src="https://github.com/elastic/kibana/assets/230104/0034a19e-b136-4261-9761-6b48fdf45989"> ``` { indices: [ 'gallery-2021-09', 'gallery-2021-10', 'gallery-2021-11', 'gallery-2021-12', 'gallery-2022-01' ], fields: { ... } } ``` ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) (cherry picked from commit 078dd22)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Apr 22, 2024
…is. (#181109) (#181302) # Backport This will backport the following commits from `main` to `8.14`: - [[ML] AIOps: Fix missing field caps filters for log rate analysis. (#181109)](#181109) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Walter Rafelsberger","email":"walter.rafelsberger@elastic.co"},"sourceCommit":{"committedDate":"2024-04-22T12:06:59Z","message":"[ML] AIOps: Fix missing field caps filters for log rate analysis. (#181109)\n\n## Summary\r\n\r\nPart of #172981.\r\n\r\nField caps requests can be heavy calls in larger clusters. For all other\r\nqueries for log rate analysis we were applying filters based on the time\r\nrange selection. This was missing from the field caps call. The\r\nfollowing parameters were added to improve the call:\r\n\r\n- `index_filter`: Adds a range filter to only get field caps from\r\nindices spanning the deviation time range.\r\n- `filters`: `-metadata` was added to not return fields like `_id` and\r\nesp. `_tier`. We previously had a manually check for `_tier` which is\r\nnow unnecessary using this option.\r\n- `types`: Previously we fetched all field types and then filtered out\r\nthe ones we don't support. This option allows us to pass in the\r\nsupported fields right away and not return unsupported ones in the first\r\nplace.\r\n\r\n----\r\n\r\nHere are examples that show how `index_filter` get applied correctly:\r\n\r\nHere the deviation selection spans only 1 month and that is reflected in\r\nthe response from the field caps call:\r\n\r\n<img width=\"1026\" alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/50a00e5a-2b59-4ae5-9d50-e2ed766f68f5\">\r\n\r\n```\r\n{\r\n indices: [ 'gallery-2021-11' ],\r\n fields: {\r\n ...\r\n }\r\n}\r\n```\r\n\r\nNow the deviation selection covers more months:\r\n\r\n<img width=\"1010\" alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/0034a19e-b136-4261-9761-6b48fdf45989\">\r\n\r\n```\r\n{\r\n indices: [\r\n 'gallery-2021-09',\r\n 'gallery-2021-10',\r\n 'gallery-2021-11',\r\n 'gallery-2021-12',\r\n 'gallery-2022-01'\r\n ],\r\n fields: {\r\n ...\r\n }\r\n}\r\n```\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n- [x] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"078dd22c47d7ecea4e18b64fb302205a799febd9","branchLabelMapping":{"^v8.15.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix",":ml","Feature:ML/AIOps","v8.14.0","v8.15.0"],"title":"[ML] AIOps: Fix missing field caps filters for log rate analysis.","number":181109,"url":"#181109 AIOps: Fix missing field caps filters for log rate analysis. (#181109)\n\n## Summary\r\n\r\nPart of #172981.\r\n\r\nField caps requests can be heavy calls in larger clusters. For all other\r\nqueries for log rate analysis we were applying filters based on the time\r\nrange selection. This was missing from the field caps call. The\r\nfollowing parameters were added to improve the call:\r\n\r\n- `index_filter`: Adds a range filter to only get field caps from\r\nindices spanning the deviation time range.\r\n- `filters`: `-metadata` was added to not return fields like `_id` and\r\nesp. `_tier`. We previously had a manually check for `_tier` which is\r\nnow unnecessary using this option.\r\n- `types`: Previously we fetched all field types and then filtered out\r\nthe ones we don't support. This option allows us to pass in the\r\nsupported fields right away and not return unsupported ones in the first\r\nplace.\r\n\r\n----\r\n\r\nHere are examples that show how `index_filter` get applied correctly:\r\n\r\nHere the deviation selection spans only 1 month and that is reflected in\r\nthe response from the field caps call:\r\n\r\n<img width=\"1026\" alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/50a00e5a-2b59-4ae5-9d50-e2ed766f68f5\">\r\n\r\n```\r\n{\r\n indices: [ 'gallery-2021-11' ],\r\n fields: {\r\n ...\r\n }\r\n}\r\n```\r\n\r\nNow the deviation selection covers more months:\r\n\r\n<img width=\"1010\" alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/0034a19e-b136-4261-9761-6b48fdf45989\">\r\n\r\n```\r\n{\r\n indices: [\r\n 'gallery-2021-09',\r\n 'gallery-2021-10',\r\n 'gallery-2021-11',\r\n 'gallery-2021-12',\r\n 'gallery-2022-01'\r\n ],\r\n fields: {\r\n ...\r\n }\r\n}\r\n```\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n- [x] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"078dd22c47d7ecea4e18b64fb302205a799febd9"}},"sourceBranch":"main","suggestedTargetBranches":["8.14"],"targetPullRequestStates":[{"branch":"8.14","label":"v8.14.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.15.0","branchLabelMappingKey":"^v8.15.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/181109","number":181109,"mergeCommit":{"message":"[ML] AIOps: Fix missing field caps filters for log rate analysis. (#181109)\n\n## Summary\r\n\r\nPart of #172981.\r\n\r\nField caps requests can be heavy calls in larger clusters. For all other\r\nqueries for log rate analysis we were applying filters based on the time\r\nrange selection. This was missing from the field caps call. The\r\nfollowing parameters were added to improve the call:\r\n\r\n- `index_filter`: Adds a range filter to only get field caps from\r\nindices spanning the deviation time range.\r\n- `filters`: `-metadata` was added to not return fields like `_id` and\r\nesp. `_tier`. We previously had a manually check for `_tier` which is\r\nnow unnecessary using this option.\r\n- `types`: Previously we fetched all field types and then filtered out\r\nthe ones we don't support. This option allows us to pass in the\r\nsupported fields right away and not return unsupported ones in the first\r\nplace.\r\n\r\n----\r\n\r\nHere are examples that show how `index_filter` get applied correctly:\r\n\r\nHere the deviation selection spans only 1 month and that is reflected in\r\nthe response from the field caps call:\r\n\r\n<img width=\"1026\" alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/50a00e5a-2b59-4ae5-9d50-e2ed766f68f5\">\r\n\r\n```\r\n{\r\n indices: [ 'gallery-2021-11' ],\r\n fields: {\r\n ...\r\n }\r\n}\r\n```\r\n\r\nNow the deviation selection covers more months:\r\n\r\n<img width=\"1010\" alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/0034a19e-b136-4261-9761-6b48fdf45989\">\r\n\r\n```\r\n{\r\n indices: [\r\n 'gallery-2021-09',\r\n 'gallery-2021-10',\r\n 'gallery-2021-11',\r\n 'gallery-2021-12',\r\n 'gallery-2022-01'\r\n ],\r\n fields: {\r\n ...\r\n }\r\n}\r\n```\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n- [x] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"078dd22c47d7ecea4e18b64fb302205a799febd9"}}]}] BACKPORT--> Co-authored-by: Walter Rafelsberger <walter.rafelsberger@elastic.co>
10 tasks
szabosteve
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Part of #172981.
Field caps requests can be heavy calls in larger clusters. For all other queries for log rate analysis we were applying filters based on the time range selection. This was missing from the field caps call. The following parameters were added to improve the call:
index_filter: Adds a range filter to only get field caps from indices spanning the deviation time range.filters:-metadatawas added to not return fields like_idand esp._tier. We previously had a manually check for_tierwhich is now unnecessary using this option.types: Previously we fetched all field types and then filtered out the ones we don't support. This option allows us to pass in the supported fields right away and not return unsupported ones in the first place.Here are examples that show how
index_filterget applied correctly:Here the deviation selection spans only 1 month and that is reflected in the response from the field caps call:
Now the deviation selection covers more months:
Checklist