[ML] AIOps Log Rate Analysis: 8.14 improvements. #172981
Assignees
Labels
Feature:ML/AIOps
ML AIOps features: Change Point Detection, Log Pattern Analysis, Log Rate Analysis
Meta
:ml
v8.14.0
Comments
peteharverson
added
Meta
:ml
Feature:ML/AIOps
|
Pinging @elastic/ml-ui (:ml) |
peteharverson
changed the title
peteharverson
changed the title
peteharverson
changed the title
This was referenced Apr 18, 2024
walterra
changed the title
walterra
added a commit
that referenced
this issue
Apr 18, 2024
…detected. (#180980) ## Summary Part of #172981. This fixes to not run log rate analysis twice when no spike/dip detected and a user needs to adapt the initial selection. When a user clicks in an area of the histogram chart that's not a highlighted change point, the click will just trigger the baseline/deviation time range selection, but it will not automatically run the analysis. Instead, an updated prompt is shown below the chart that explains that the baseline/deviation can be adjusted via dragging and the analysis can be run via the button below that description. Initial view after loading the page: <img width="1040" alt="image" src="https://github.com/elastic/kibana/assets/230104/90e8c390-af2a-45e2-8d11-cfd42285200b"> User clicked in an area that's not covered by the highlighted change point: <img width="1026" alt="image" src="https://github.com/elastic/kibana/assets/230104/050a07e0-c5e6-4639-a854-83fae10b125b"> ### Checklist Delete any items that are not applicable to this PR. - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Apr 18, 2024
…detected. (elastic#180980) ## Summary Part of elastic#172981. This fixes to not run log rate analysis twice when no spike/dip detected and a user needs to adapt the initial selection. When a user clicks in an area of the histogram chart that's not a highlighted change point, the click will just trigger the baseline/deviation time range selection, but it will not automatically run the analysis. Instead, an updated prompt is shown below the chart that explains that the baseline/deviation can be adjusted via dragging and the analysis can be run via the button below that description. Initial view after loading the page: <img width="1040" alt="image" src="https://github.com/elastic/kibana/assets/230104/90e8c390-af2a-45e2-8d11-cfd42285200b"> User clicked in an area that's not covered by the highlighted change point: <img width="1026" alt="image" src="https://github.com/elastic/kibana/assets/230104/050a07e0-c5e6-4639-a854-83fae10b125b"> ### Checklist Delete any items that are not applicable to this PR. - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) (cherry picked from commit 6fdcd8d)
kibanamachine
added a commit
that referenced
this issue
Apr 18, 2024
…ke/dip detected. (#180980) (#181128) # Backport This will backport the following commits from `main` to `8.14`: - [[ML] AIOps: Fix to not run log rate analysis twice when no spike/dip detected. (#180980)](#180980) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Walter Rafelsberger","email":"walter.rafelsberger@elastic.co"},"sourceCommit":{"committedDate":"2024-04-18T09:35:16Z","message":"[ML] AIOps: Fix to not run log rate analysis twice when no spike/dip detected. (#180980)\n\n## Summary\r\n\r\nPart of #172981.\r\n\r\nThis fixes to not run log rate analysis twice when no spike/dip detected\r\nand a user needs to adapt the initial selection. When a user clicks in\r\nan area of the histogram chart that's not a highlighted change point,\r\nthe click will just trigger the baseline/deviation time range selection,\r\nbut it will not automatically run the analysis. Instead, an updated\r\nprompt is shown below the chart that explains that the\r\nbaseline/deviation can be adjusted via dragging and the analysis can be\r\nrun via the button below that description.\r\n\r\nInitial view after loading the page:\r\n\r\n<img width=\"1040\" alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/90e8c390-af2a-45e2-8d11-cfd42285200b\">\r\n\r\nUser clicked in an area that's not covered by the highlighted change\r\npoint:\r\n\r\n<img width=\"1026\" alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/050a07e0-c5e6-4639-a854-83fae10b125b\">\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [x] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"6fdcd8d7b80515ce9361666f5264f8966d9b54e7","branchLabelMapping":{"^v8.15.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix",":ml","Feature:ML/AIOps","v8.14.0","v8.15.0"],"title":"[ML] AIOps: Fix to not run log rate analysis twice when no spike/dip detected.","number":180980,"url":"#180980 AIOps: Fix to not run log rate analysis twice when no spike/dip detected. (#180980)\n\n## Summary\r\n\r\nPart of #172981.\r\n\r\nThis fixes to not run log rate analysis twice when no spike/dip detected\r\nand a user needs to adapt the initial selection. When a user clicks in\r\nan area of the histogram chart that's not a highlighted change point,\r\nthe click will just trigger the baseline/deviation time range selection,\r\nbut it will not automatically run the analysis. Instead, an updated\r\nprompt is shown below the chart that explains that the\r\nbaseline/deviation can be adjusted via dragging and the analysis can be\r\nrun via the button below that description.\r\n\r\nInitial view after loading the page:\r\n\r\n<img width=\"1040\" alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/90e8c390-af2a-45e2-8d11-cfd42285200b\">\r\n\r\nUser clicked in an area that's not covered by the highlighted change\r\npoint:\r\n\r\n<img width=\"1026\" alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/050a07e0-c5e6-4639-a854-83fae10b125b\">\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [x] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"6fdcd8d7b80515ce9361666f5264f8966d9b54e7"}},"sourceBranch":"main","suggestedTargetBranches":["8.14"],"targetPullRequestStates":[{"branch":"8.14","label":"v8.14.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.15.0","branchLabelMappingKey":"^v8.15.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/180980","number":180980,"mergeCommit":{"message":"[ML] AIOps: Fix to not run log rate analysis twice when no spike/dip detected. (#180980)\n\n## Summary\r\n\r\nPart of #172981.\r\n\r\nThis fixes to not run log rate analysis twice when no spike/dip detected\r\nand a user needs to adapt the initial selection. When a user clicks in\r\nan area of the histogram chart that's not a highlighted change point,\r\nthe click will just trigger the baseline/deviation time range selection,\r\nbut it will not automatically run the analysis. Instead, an updated\r\nprompt is shown below the chart that explains that the\r\nbaseline/deviation can be adjusted via dragging and the analysis can be\r\nrun via the button below that description.\r\n\r\nInitial view after loading the page:\r\n\r\n<img width=\"1040\" alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/90e8c390-af2a-45e2-8d11-cfd42285200b\">\r\n\r\nUser clicked in an area that's not covered by the highlighted change\r\npoint:\r\n\r\n<img width=\"1026\" alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/050a07e0-c5e6-4639-a854-83fae10b125b\">\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [x] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"6fdcd8d7b80515ce9361666f5264f8966d9b54e7"}}]}] BACKPORT--> Co-authored-by: Walter Rafelsberger <walter.rafelsberger@elastic.co>
walterra
added a commit
that referenced
this issue
Apr 22, 2024
…81109) ## Summary Part of #172981. Field caps requests can be heavy calls in larger clusters. For all other queries for log rate analysis we were applying filters based on the time range selection. This was missing from the field caps call. The following parameters were added to improve the call: - `index_filter`: Adds a range filter to only get field caps from indices spanning the deviation time range. - `filters`: `-metadata` was added to not return fields like `_id` and esp. `_tier`. We previously had a manually check for `_tier` which is now unnecessary using this option. - `types`: Previously we fetched all field types and then filtered out the ones we don't support. This option allows us to pass in the supported fields right away and not return unsupported ones in the first place. ---- Here are examples that show how `index_filter` get applied correctly: Here the deviation selection spans only 1 month and that is reflected in the response from the field caps call: <img width="1026" alt="image" src="https://github.com/elastic/kibana/assets/230104/50a00e5a-2b59-4ae5-9d50-e2ed766f68f5"> ``` { indices: [ 'gallery-2021-11' ], fields: { ... } } ``` Now the deviation selection covers more months: <img width="1010" alt="image" src="https://github.com/elastic/kibana/assets/230104/0034a19e-b136-4261-9761-6b48fdf45989"> ``` { indices: [ 'gallery-2021-09', 'gallery-2021-10', 'gallery-2021-11', 'gallery-2021-12', 'gallery-2022-01' ], fields: { ... } } ``` ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Apr 22, 2024
…astic#181109) ## Summary Part of elastic#172981. Field caps requests can be heavy calls in larger clusters. For all other queries for log rate analysis we were applying filters based on the time range selection. This was missing from the field caps call. The following parameters were added to improve the call: - `index_filter`: Adds a range filter to only get field caps from indices spanning the deviation time range. - `filters`: `-metadata` was added to not return fields like `_id` and esp. `_tier`. We previously had a manually check for `_tier` which is now unnecessary using this option. - `types`: Previously we fetched all field types and then filtered out the ones we don't support. This option allows us to pass in the supported fields right away and not return unsupported ones in the first place. ---- Here are examples that show how `index_filter` get applied correctly: Here the deviation selection spans only 1 month and that is reflected in the response from the field caps call: <img width="1026" alt="image" src="https://github.com/elastic/kibana/assets/230104/50a00e5a-2b59-4ae5-9d50-e2ed766f68f5"> ``` { indices: [ 'gallery-2021-11' ], fields: { ... } } ``` Now the deviation selection covers more months: <img width="1010" alt="image" src="https://github.com/elastic/kibana/assets/230104/0034a19e-b136-4261-9761-6b48fdf45989"> ``` { indices: [ 'gallery-2021-09', 'gallery-2021-10', 'gallery-2021-11', 'gallery-2021-12', 'gallery-2022-01' ], fields: { ... } } ``` ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) (cherry picked from commit 078dd22)
|
Created follow up #181111, closing this one. |
kibanamachine
added a commit
that referenced
this issue
Apr 22, 2024
…is. (#181109) (#181302) # Backport This will backport the following commits from `main` to `8.14`: - [[ML] AIOps: Fix missing field caps filters for log rate analysis. (#181109)](#181109) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Walter Rafelsberger","email":"walter.rafelsberger@elastic.co"},"sourceCommit":{"committedDate":"2024-04-22T12:06:59Z","message":"[ML] AIOps: Fix missing field caps filters for log rate analysis. (#181109)\n\n## Summary\r\n\r\nPart of #172981.\r\n\r\nField caps requests can be heavy calls in larger clusters. For all other\r\nqueries for log rate analysis we were applying filters based on the time\r\nrange selection. This was missing from the field caps call. The\r\nfollowing parameters were added to improve the call:\r\n\r\n- `index_filter`: Adds a range filter to only get field caps from\r\nindices spanning the deviation time range.\r\n- `filters`: `-metadata` was added to not return fields like `_id` and\r\nesp. `_tier`. We previously had a manually check for `_tier` which is\r\nnow unnecessary using this option.\r\n- `types`: Previously we fetched all field types and then filtered out\r\nthe ones we don't support. This option allows us to pass in the\r\nsupported fields right away and not return unsupported ones in the first\r\nplace.\r\n\r\n----\r\n\r\nHere are examples that show how `index_filter` get applied correctly:\r\n\r\nHere the deviation selection spans only 1 month and that is reflected in\r\nthe response from the field caps call:\r\n\r\n<img width=\"1026\" alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/50a00e5a-2b59-4ae5-9d50-e2ed766f68f5\">\r\n\r\n```\r\n{\r\n indices: [ 'gallery-2021-11' ],\r\n fields: {\r\n ...\r\n }\r\n}\r\n```\r\n\r\nNow the deviation selection covers more months:\r\n\r\n<img width=\"1010\" alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/0034a19e-b136-4261-9761-6b48fdf45989\">\r\n\r\n```\r\n{\r\n indices: [\r\n 'gallery-2021-09',\r\n 'gallery-2021-10',\r\n 'gallery-2021-11',\r\n 'gallery-2021-12',\r\n 'gallery-2022-01'\r\n ],\r\n fields: {\r\n ...\r\n }\r\n}\r\n```\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n- [x] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"078dd22c47d7ecea4e18b64fb302205a799febd9","branchLabelMapping":{"^v8.15.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix",":ml","Feature:ML/AIOps","v8.14.0","v8.15.0"],"title":"[ML] AIOps: Fix missing field caps filters for log rate analysis.","number":181109,"url":"#181109 AIOps: Fix missing field caps filters for log rate analysis. (#181109)\n\n## Summary\r\n\r\nPart of #172981.\r\n\r\nField caps requests can be heavy calls in larger clusters. For all other\r\nqueries for log rate analysis we were applying filters based on the time\r\nrange selection. This was missing from the field caps call. The\r\nfollowing parameters were added to improve the call:\r\n\r\n- `index_filter`: Adds a range filter to only get field caps from\r\nindices spanning the deviation time range.\r\n- `filters`: `-metadata` was added to not return fields like `_id` and\r\nesp. `_tier`. We previously had a manually check for `_tier` which is\r\nnow unnecessary using this option.\r\n- `types`: Previously we fetched all field types and then filtered out\r\nthe ones we don't support. This option allows us to pass in the\r\nsupported fields right away and not return unsupported ones in the first\r\nplace.\r\n\r\n----\r\n\r\nHere are examples that show how `index_filter` get applied correctly:\r\n\r\nHere the deviation selection spans only 1 month and that is reflected in\r\nthe response from the field caps call:\r\n\r\n<img width=\"1026\" alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/50a00e5a-2b59-4ae5-9d50-e2ed766f68f5\">\r\n\r\n```\r\n{\r\n indices: [ 'gallery-2021-11' ],\r\n fields: {\r\n ...\r\n }\r\n}\r\n```\r\n\r\nNow the deviation selection covers more months:\r\n\r\n<img width=\"1010\" alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/0034a19e-b136-4261-9761-6b48fdf45989\">\r\n\r\n```\r\n{\r\n indices: [\r\n 'gallery-2021-09',\r\n 'gallery-2021-10',\r\n 'gallery-2021-11',\r\n 'gallery-2021-12',\r\n 'gallery-2022-01'\r\n ],\r\n fields: {\r\n ...\r\n }\r\n}\r\n```\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n- [x] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"078dd22c47d7ecea4e18b64fb302205a799febd9"}},"sourceBranch":"main","suggestedTargetBranches":["8.14"],"targetPullRequestStates":[{"branch":"8.14","label":"v8.14.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.15.0","branchLabelMappingKey":"^v8.15.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/181109","number":181109,"mergeCommit":{"message":"[ML] AIOps: Fix missing field caps filters for log rate analysis. (#181109)\n\n## Summary\r\n\r\nPart of #172981.\r\n\r\nField caps requests can be heavy calls in larger clusters. For all other\r\nqueries for log rate analysis we were applying filters based on the time\r\nrange selection. This was missing from the field caps call. The\r\nfollowing parameters were added to improve the call:\r\n\r\n- `index_filter`: Adds a range filter to only get field caps from\r\nindices spanning the deviation time range.\r\n- `filters`: `-metadata` was added to not return fields like `_id` and\r\nesp. `_tier`. We previously had a manually check for `_tier` which is\r\nnow unnecessary using this option.\r\n- `types`: Previously we fetched all field types and then filtered out\r\nthe ones we don't support. This option allows us to pass in the\r\nsupported fields right away and not return unsupported ones in the first\r\nplace.\r\n\r\n----\r\n\r\nHere are examples that show how `index_filter` get applied correctly:\r\n\r\nHere the deviation selection spans only 1 month and that is reflected in\r\nthe response from the field caps call:\r\n\r\n<img width=\"1026\" alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/50a00e5a-2b59-4ae5-9d50-e2ed766f68f5\">\r\n\r\n```\r\n{\r\n indices: [ 'gallery-2021-11' ],\r\n fields: {\r\n ...\r\n }\r\n}\r\n```\r\n\r\nNow the deviation selection covers more months:\r\n\r\n<img width=\"1010\" alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/0034a19e-b136-4261-9761-6b48fdf45989\">\r\n\r\n```\r\n{\r\n indices: [\r\n 'gallery-2021-09',\r\n 'gallery-2021-10',\r\n 'gallery-2021-11',\r\n 'gallery-2021-12',\r\n 'gallery-2022-01'\r\n ],\r\n fields: {\r\n ...\r\n }\r\n}\r\n```\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n- [x] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"078dd22c47d7ecea4e18b64fb302205a799febd9"}}]}] BACKPORT--> Co-authored-by: Walter Rafelsberger <walter.rafelsberger@elastic.co>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Follow up to #168459.
UI
Analysis
The text was updated successfully, but these errors were encountered: