Skip to content

Commit

Permalink
[8.14] [ML] AIOps: Fix missing field caps filters for log rate analys…
Browse files Browse the repository at this point in the history 
…is. (#181109) (#181302)

# Backport

This will backport the following commits from `main` to `8.14`:
- [[ML] AIOps: Fix missing field caps filters for log rate analysis.
(#181109)](#181109)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Walter
Rafelsberger","email":"walter.rafelsberger@elastic.co"},"sourceCommit":{"committedDate":"2024-04-22T12:06:59Z","message":"[ML]
AIOps: Fix missing field caps filters for log rate analysis.
(#181109)\n\n## Summary\r\n\r\nPart of #172981.\r\n\r\nField caps
requests can be heavy calls in larger clusters. For all other\r\nqueries
for log rate analysis we were applying filters based on the
time\r\nrange selection. This was missing from the field caps call.
The\r\nfollowing parameters were added to improve the call:\r\n\r\n-
`index_filter`: Adds a range filter to only get field caps
from\r\nindices spanning the deviation time range.\r\n- `filters`:
`-metadata` was added to not return fields like `_id` and\r\nesp.
`_tier`. We previously had a manually check for `_tier` which is\r\nnow
unnecessary using this option.\r\n- `types`: Previously we fetched all
field types and then filtered out\r\nthe ones we don't support. This
option allows us to pass in the\r\nsupported fields right away and not
return unsupported ones in the first\r\nplace.\r\n\r\n----\r\n\r\nHere
are examples that show how `index_filter` get applied
correctly:\r\n\r\nHere the deviation selection spans only 1 month and
that is reflected in\r\nthe response from the field caps
call:\r\n\r\n<img width=\"1026\"
alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/50a00e5a-2b59-4ae5-9d50-e2ed766f68f5\">\r\n\r\n```\r\n{\r\n
indices: [ 'gallery-2021-11' ],\r\n fields: {\r\n ...\r\n
}\r\n}\r\n```\r\n\r\nNow the deviation selection covers more
months:\r\n\r\n<img width=\"1010\"
alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/0034a19e-b136-4261-9761-6b48fdf45989\">\r\n\r\n```\r\n{\r\n
indices: [\r\n 'gallery-2021-09',\r\n 'gallery-2021-10',\r\n
'gallery-2021-11',\r\n 'gallery-2021-12',\r\n 'gallery-2022-01'\r\n
],\r\n fields: {\r\n ...\r\n }\r\n}\r\n```\r\n\r\n### Checklist\r\n\r\n-
[x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n- [x] This was checked for breaking
API changes and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"078dd22c47d7ecea4e18b64fb302205a799febd9","branchLabelMapping":{"^v8.15.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix",":ml","Feature:ML/AIOps","v8.14.0","v8.15.0"],"title":"[ML]
AIOps: Fix missing field caps filters for log rate
analysis.","number":181109,"url":"#181109
AIOps: Fix missing field caps filters for log rate analysis.
(#181109)\n\n## Summary\r\n\r\nPart of #172981.\r\n\r\nField caps
requests can be heavy calls in larger clusters. For all other\r\nqueries
for log rate analysis we were applying filters based on the
time\r\nrange selection. This was missing from the field caps call.
The\r\nfollowing parameters were added to improve the call:\r\n\r\n-
`index_filter`: Adds a range filter to only get field caps
from\r\nindices spanning the deviation time range.\r\n- `filters`:
`-metadata` was added to not return fields like `_id` and\r\nesp.
`_tier`. We previously had a manually check for `_tier` which is\r\nnow
unnecessary using this option.\r\n- `types`: Previously we fetched all
field types and then filtered out\r\nthe ones we don't support. This
option allows us to pass in the\r\nsupported fields right away and not
return unsupported ones in the first\r\nplace.\r\n\r\n----\r\n\r\nHere
are examples that show how `index_filter` get applied
correctly:\r\n\r\nHere the deviation selection spans only 1 month and
that is reflected in\r\nthe response from the field caps
call:\r\n\r\n<img width=\"1026\"
alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/50a00e5a-2b59-4ae5-9d50-e2ed766f68f5\">\r\n\r\n```\r\n{\r\n
indices: [ 'gallery-2021-11' ],\r\n fields: {\r\n ...\r\n
}\r\n}\r\n```\r\n\r\nNow the deviation selection covers more
months:\r\n\r\n<img width=\"1010\"
alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/0034a19e-b136-4261-9761-6b48fdf45989\">\r\n\r\n```\r\n{\r\n
indices: [\r\n 'gallery-2021-09',\r\n 'gallery-2021-10',\r\n
'gallery-2021-11',\r\n 'gallery-2021-12',\r\n 'gallery-2022-01'\r\n
],\r\n fields: {\r\n ...\r\n }\r\n}\r\n```\r\n\r\n### Checklist\r\n\r\n-
[x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n- [x] This was checked for breaking
API changes and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"078dd22c47d7ecea4e18b64fb302205a799febd9"}},"sourceBranch":"main","suggestedTargetBranches":["8.14"],"targetPullRequestStates":[{"branch":"8.14","label":"v8.14.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.15.0","branchLabelMappingKey":"^v8.15.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/181109","number":181109,"mergeCommit":{"message":"[ML]
AIOps: Fix missing field caps filters for log rate analysis.
(#181109)\n\n## Summary\r\n\r\nPart of #172981.\r\n\r\nField caps
requests can be heavy calls in larger clusters. For all other\r\nqueries
for log rate analysis we were applying filters based on the
time\r\nrange selection. This was missing from the field caps call.
The\r\nfollowing parameters were added to improve the call:\r\n\r\n-
`index_filter`: Adds a range filter to only get field caps
from\r\nindices spanning the deviation time range.\r\n- `filters`:
`-metadata` was added to not return fields like `_id` and\r\nesp.
`_tier`. We previously had a manually check for `_tier` which is\r\nnow
unnecessary using this option.\r\n- `types`: Previously we fetched all
field types and then filtered out\r\nthe ones we don't support. This
option allows us to pass in the\r\nsupported fields right away and not
return unsupported ones in the first\r\nplace.\r\n\r\n----\r\n\r\nHere
are examples that show how `index_filter` get applied
correctly:\r\n\r\nHere the deviation selection spans only 1 month and
that is reflected in\r\nthe response from the field caps
call:\r\n\r\n<img width=\"1026\"
alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/50a00e5a-2b59-4ae5-9d50-e2ed766f68f5\">\r\n\r\n```\r\n{\r\n
indices: [ 'gallery-2021-11' ],\r\n fields: {\r\n ...\r\n
}\r\n}\r\n```\r\n\r\nNow the deviation selection covers more
months:\r\n\r\n<img width=\"1010\"
alt=\"image\"\r\nsrc=\"https://github.com/elastic/kibana/assets/230104/0034a19e-b136-4261-9761-6b48fdf45989\">\r\n\r\n```\r\n{\r\n
indices: [\r\n 'gallery-2021-09',\r\n 'gallery-2021-10',\r\n
'gallery-2021-11',\r\n 'gallery-2021-12',\r\n 'gallery-2022-01'\r\n
],\r\n fields: {\r\n ...\r\n }\r\n}\r\n```\r\n\r\n### Checklist\r\n\r\n-
[x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n- [x] This was checked for breaking
API changes and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"078dd22c47d7ecea4e18b64fb302205a799febd9"}}]}]
BACKPORT-->

Co-authored-by: Walter Rafelsberger <walter.rafelsberger@elastic.co>
  • Loading branch information
@kibanamachine @walterra
kibanamachine and walterra committed Apr 22, 2024
1 parent c25443d commit ee8b102
Show file tree
Hide file tree
Showing 5 changed files with 562 additions and 261 deletions.
210 changes: 40 additions & 170 deletions x-pack/packages/ml/aiops_log_rate_analysis/queries/__mocks__/field_caps_ecommerce.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,228 +6,100 @@
*/

export const fieldCapsEcommerceMock = {
indices: ['ft_ecommerce'],
indices: ['kibana_sample_data_ecommerce'],
fields: {
'products.manufacturer': {
text: { type: 'text', metadata_field: false, searchable: true, aggregatable: false },
},
'products.discount_amount': {
half_float: {
type: 'half_float',
metadata_field: false,
searchable: true,
aggregatable: true,
},
},
'products.base_unit_price': {
half_float: {
type: 'half_float',
metadata_field: false,
searchable: true,
aggregatable: true,
},
},
type: {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
'products.discount_percentage': {
half_float: {
type: 'half_float',
metadata_field: false,
searchable: true,
aggregatable: true,
},
},
'products._id.keyword': {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
day_of_week_i: {
integer: { type: 'integer', metadata_field: false, searchable: true, aggregatable: true },
},
total_quantity: {
integer: { type: 'integer', metadata_field: false, searchable: true, aggregatable: true },
},
total_unique_products: {
integer: { type: 'integer', metadata_field: false, searchable: true, aggregatable: true },
},
taxless_total_price: {
half_float: {
type: 'half_float',
metadata_field: false,
searchable: true,
aggregatable: true,
},
},
'geoip.continent_name': {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
sku: {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
_version: {
_version: { type: '_version', metadata_field: true, searchable: false, aggregatable: true },
},
'customer_full_name.keyword': {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
'category.keyword': {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
'products.taxless_price': {
half_float: {
type: 'half_float',
metadata_field: false,
searchable: true,
aggregatable: true,
},
},
'products.quantity': {
integer: { type: 'integer', metadata_field: false, searchable: true, aggregatable: true },
},
'products.price': {
half_float: {
type: 'half_float',
metadata_field: false,
searchable: true,
aggregatable: true,
},
},
customer_first_name: {
text: { type: 'text', metadata_field: false, searchable: true, aggregatable: false },
},
customer_phone: {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
'geoip.region_name': {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
_tier: {
keyword: { type: 'keyword', metadata_field: true, searchable: true, aggregatable: true },
},
_seq_no: {
_seq_no: { type: '_seq_no', metadata_field: true, searchable: true, aggregatable: true },
},
customer_full_name: {
text: { type: 'text', metadata_field: false, searchable: true, aggregatable: false },
},
'geoip.country_iso_code': {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
_source: {
_source: { type: '_source', metadata_field: true, searchable: false, aggregatable: false },
},
_id: { _id: { type: '_id', metadata_field: true, searchable: true, aggregatable: false } },
order_id: {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
'products._id': {
text: { type: 'text', metadata_field: false, searchable: true, aggregatable: false },
},
'products.product_name.keyword': {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
_index: {
_index: { type: '_index', metadata_field: true, searchable: true, aggregatable: true },
},
'products.product_id': {
long: { type: 'long', metadata_field: false, searchable: true, aggregatable: true },
},
'products.category': {
text: { type: 'text', metadata_field: false, searchable: true, aggregatable: false },
},
'products.manufacturer.keyword': {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
type: {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
manufacturer: {
text: { type: 'text', metadata_field: false, searchable: true, aggregatable: false },
},
products: {
object: { type: 'object', metadata_field: false, searchable: false, aggregatable: false },
},
'products.unit_discount_amount': {
half_float: {
type: 'half_float',
metadata_field: false,
searchable: true,
aggregatable: true,
},
},
customer_last_name: {
text: { type: 'text', metadata_field: false, searchable: true, aggregatable: false },
},
'geoip.location': {
geo_point: { type: 'geo_point', metadata_field: false, searchable: true, aggregatable: true },
},
'products.tax_amount': {
half_float: {
type: 'half_float',
metadata_field: false,
searchable: true,
aggregatable: true,
},
'products._id.keyword': {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
'products.product_name': {
text: { type: 'text', metadata_field: false, searchable: true, aggregatable: false },
},
'products.min_price': {
half_float: {
type: 'half_float',
metadata_field: false,
searchable: true,
aggregatable: true,
},
},
'manufacturer.keyword': {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
'products.taxful_price': {
half_float: {
type: 'half_float',
metadata_field: false,
searchable: true,
aggregatable: true,
},
},
currency: {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
'products.base_price': {
half_float: {
type: 'half_float',
metadata_field: false,
searchable: true,
aggregatable: true,
},
'geoip.continent_name': {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
event: {
object: { type: 'object', metadata_field: false, searchable: false, aggregatable: false },
},
sku: {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
email: {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
'customer_full_name.keyword': {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
day_of_week: {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
'customer_last_name.keyword': {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
'products.sku': {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
'customer_last_name.keyword': {
'category.keyword': {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
geoip: {
object: { type: 'object', metadata_field: false, searchable: false, aggregatable: false },
},
customer_first_name: {
text: { type: 'text', metadata_field: false, searchable: true, aggregatable: false },
},
customer_phone: {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
'products.category.keyword': {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
'geoip.city_name': {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
order_date: {
date: { type: 'date', metadata_field: false, searchable: true, aggregatable: true },
'geoip.region_name': {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
'customer_first_name.keyword': {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
'products.created_on': {
date: { type: 'date', metadata_field: false, searchable: true, aggregatable: true },
customer_full_name: {
text: { type: 'text', metadata_field: false, searchable: true, aggregatable: false },
},
'geoip.country_iso_code': {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
category: {
text: { type: 'text', metadata_field: false, searchable: true, aggregatable: false },
Expand All @@ -238,16 +110,14 @@ export const fieldCapsEcommerceMock = {
user: {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
customer_gender: {
order_id: {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
'event.dataset': {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
taxful_total_price: {
half_float: {
type: 'half_float',
metadata_field: false,
searchable: true,
aggregatable: true,
},
customer_gender: {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
},
};
21 changes: 1 addition & 20 deletions x-pack/packages/ml/aiops_log_rate_analysis/queries/__mocks__/field_caps_large_arrays.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,27 +6,8 @@
*/

export const fieldCapsLargeArraysMock = {
indices: ['large_arrays'],
indices: ['large_array'],
fields: {
_tier: {
keyword: { type: 'keyword', metadata_field: true, searchable: true, aggregatable: true },
},
_seq_no: {
_seq_no: { type: '_seq_no', metadata_field: true, searchable: true, aggregatable: true },
},
'@timestamp': {
date: { type: 'date', metadata_field: false, searchable: true, aggregatable: true },
},
_index: {
_index: { type: '_index', metadata_field: true, searchable: true, aggregatable: true },
},
_source: {
_source: { type: '_source', metadata_field: true, searchable: false, aggregatable: false },
},
_id: { _id: { type: '_id', metadata_field: true, searchable: true, aggregatable: false } },
_version: {
_version: { type: '_version', metadata_field: true, searchable: false, aggregatable: true },
},
items: {
keyword: { type: 'keyword', metadata_field: false, searchable: true, aggregatable: true },
},
Expand Down
Loading

0 comments on commit ee8b102

Please sign in to comment.