Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Response Ops][Alerting] ES query rule error Executing Rule default:.es-query has resulted in Error: Cannot set properties of undefined (setting 'order') - TypeError: Cannot set properties of undefined (setting 'order') #182755

Closed
ymao1 opened this issue May 6, 2024 · 5 comments
Closed

[Response Ops][Alerting] ES query rule error Executing Rule default:.es-query has resulted in Error: Cannot set properties of undefined (setting 'order') - TypeError: Cannot set properties of undefined (setting 'order') #182755

ymao1 opened this issue May 6, 2024 · 5 comments
Assignees
@ymao1
Labels
Feature:Alerting Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams)

Comments

@ymao1
Copy link
Contributor

ymao1 commented May 6, 2024

Seen in the logs of the staging overview cluster. Looks like it's happening in the build_agg function

kibana/x-pack/plugins/triggers_actions_ui/common/data/lib/build_agg.ts

Line 117 in 0c0cf0a

aggParent.aggs.groupAgg.terms!.order = {

@ymao1 ymao1 added Feature:Alerting Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) labels May 6, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/response-ops (Team:ResponseOps)

@pmuellr
Copy link
Member

pmuellr commented May 7, 2024

Nice! I could have sworn I saw a message like in the logs recently, and had remembered while opening a different issue, so tried to find it again. But couldn't. I think I looked in qa and prod, but not staging 🤦🏻‍♂️

OHHHH .... I don't see it in serverless, only in ESS!

I checked prod, and it's happening with other fields as well. For searching, use message: "Cannot read properties of undefined" AND tags:".es-query" AND tags:"rule-run-failed"

@ymao1 ymao1 self-assigned this May 7, 2024
@pmuellr
Copy link
Member

pmuellr commented May 7, 2024

Some of the fields/locations I've found over the last 90 days:

  • type
  • gte
  • timeFieldName
  • index
  • actionId
  • getFlapping
  • scheduledAt
  • meta
  • fields
  • title

I wouldn't be surprised if some of these were caused by calls to ES that went bad somehow - which are likely not as critical as when it's not an ES call that goes bad. So we may be able to defer fixing some of these, if needed.

type:

Executing Rule default:.es-query:0e2d43dd-27d2-4e69-bbac-5a9a0ebd1c45 has resulted in Error: Cannot read properties of undefined (reading 'type') - TypeError: Cannot read properties of undefined (reading 'type')
    at /usr/share/kibana/node_modules/@kbn/es-query/src/filters/build_filters/range_filter.js:103:58
    at /usr/share/kibana/node_modules/lodash/lodash.js:13469:38
    at /usr/share/kibana/node_modules/lodash/lodash.js:4967:15
    at baseForOwn (/usr/share/kibana/node_modules/lodash/lodash.js:3032:24)
    at mapValues (/usr/share/kibana/node_modules/lodash/lodash.js:13468:7)
    at buildRangeFilter (/usr/share/kibana/node_modules/@kbn/es-query/src/filters/build_filters/range_filter.js:103:34)
    at updateSearchSource (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:68:50)
    at fetchSearchSourceQuery (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:43:7)

gte:

Executing Rule default:.es-query:55024590-a989-11ee-8c16-dbbdc60ea0be has resulted in Error: Cannot read properties of undefined (reading 'gte') - TypeError: Cannot read properties of undefined (reading 'gte')
    at updateSearchSource (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:70:130)
    at fetchSearchSourceQuery (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:41:7)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at executor (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/executor.js:84:8)

timeFieldName:

xecuting Rule default:.es-query:93138bc0-e76c-11ee-8563-83a10873f2c2 has resulted in Error: Cannot read properties of undefined (reading 'timeFieldName') - TypeError: Cannot read properties of undefined (reading 'timeFieldName')
    at updateSearchSource (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:59:51)
    at fetchSearchSourceQuery (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:40:7)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at executor (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/executor.js:87:8)
    at Object.executor (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/rule_type.js:180:14)

index:

Executing Rule default:.es-query:74b1a3e2-f7f0-4d72-8d2f-74b996eba09f has resulted in Error: Cannot read properties of undefined (reading 'index') - TypeError: Cannot read properties of undefined (reading 'index')
    at /usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:151:21
    at Array.map (<anonymous>)
    at updateFilterReferences (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:150:26)
    at generateLink (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:131:26)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at fetchSearchSourceQuery (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:46:16)
    at executor (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/executor.js:82:26)
    at Object.executor (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/rule_type.js:168:14)

actionId:

Executing Rule default:.es-query:fa369db0-143c-11ed-8815-654215629985 has resulted in Error: Cannot read properties of undefined (reading 'actionId') - TypeError: Cannot read properties of undefined (reading 'actionId')
    at /usr/share/kibana/x-pack/plugins/actions/server/create_execute_function.js:150:38
    at Array.map (<anonymous>)
    at ActionsClient.execute [as bulkExecutionEnqueuer] (/usr/share/kibana/x-pack/plugins/actions/server/create_execute_function.js:149:65)

getFlapping:

Executing Rule default:.es-query:f99cfd60-e85a-11ee-ad8d-2120ea18016e has resulted in Error: Cannot read properties of undefined (reading 'getFlapping') - TypeError: Cannot read properties of undefined (reading 'getFlapping')
    at getAlertsForNotification (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/lib/get_alerts_for_notification.js:26:30)
    at LegacyAlertsClient.processAndLogAlerts (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/alerts_client/legacy_alerts_client.js:114:54)
    at AlertsClient.processAndLogAlerts (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/alerts_client/alerts_client.js:183:29)
    at /usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:385:20

scheduledAt:

Executing Rule default:.es-query:2eed01d0-fca2-11ee-8f3a-2f515cdd4944 has resulted in Error: Cannot read properties of undefined (reading 'scheduledAt') - TypeError: Cannot read properties of undefined (reading 'scheduledAt')
    at savedObjectToConcreteTaskInstance (/usr/share/kibana/node_modules/@kbn/task-manager-plugin/server/task_store.js:423:50)
    at /usr/share/kibana/node_modules/@kbn/task-manager-plugin/server/task_store.js:116:49
    at Array.map (<anonymous>)
    at TaskStore.bulkSchedule (/usr/share/kibana/node_modules/@kbn/task-manager-plugin/server/task_store.js:116:39)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at TaskScheduling.bulkSchedule (/usr/share/kibana/node_modules/@kbn/task-manager-plugin/server/task_scheduling.js:94:12)
    at ActionsClient.execute [as bulkExecutionEnqueuer] (/usr/share/kibana/node_modules/@kbn/actions-plugin/server/create_execute_function.js:167:5)
    at ExecutionHandler.run (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/execution_handler.js:245:11)
    at /usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:376:37

meta:

Executing Rule default:.es-query:2eed01d0-fca2-11ee-8f3a-2f515cdd4944 has resulted in Error: Cannot read properties of undefined (reading 'meta') - TypeError: Cannot read properties of undefined (reading 'meta')
    at /usr/share/kibana/node_modules/@kbn/actions-plugin/server/authorization/get_authorization_mode_by_source.js:47:54
    at Array.reduce (<anonymous>)
    at getBulkAuthorizationModeBySource (/usr/share/kibana/node_modules/@kbn/actions-plugin/server/authorization/get_authorization_mode_by_source.js:43:47)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at ActionsClient.bulkEnqueueExecution (/usr/share/kibana/node_modules/@kbn/actions-plugin/server/actions_client.js:570:24)
    at ExecutionHandler.run (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/execution_handler.js:245:11)
    at /usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:376:37

fields:

Executing Rule default:.es-query:9fdd3cdc-cd87-4846-89bf-0bf76bfa46c5 has resulted in Error: Cannot read properties of undefined (reading 'fields') - TypeError: Cannot read properties of undefined (reading 'fields')
    at updateSearchSource (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:61:23)
    at fetchSearchSourceQuery (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:38:24)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at executor (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/executor.js:78:26)
    at Object.executor (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/rule_type.js:162:14)
    at /usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:318:26

title:

Executing Rule default:.es-query:ce83a050-dcb8-11ee-8cfd-776c9fe84c58 has resulted in Error: Cannot read properties of undefined (reading 'title') - TypeError: Cannot read properties of undefined (reading 'title')
    at getSearchParamsFromRequest (/usr/share/kibana/src/plugins/data/common/search/search_source/fetch/get_search_params.js:44:32)
    at SearchSource.fetchSearch$ (/usr/share/kibana/src/plugins/data/common/search/search_source/search_source.js:431:58)
    at /usr/share/kibana/src/plugins/data/common/search/search_source/search_source.js:298:19
    at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/switchMap.js:17:35
    at OperatorSubscriber._this._next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/OperatorSubscriber.js:33:21)

@ymao1
Copy link
Contributor Author

ymao1 commented May 7, 2024 •
edited

Error counts from the production logs

Bug with ES query multi-terms
Cannot set properties of undefined (setting 'order') - TypeError: Cannot set properties of undefined (setting 'order')
24,200 instances of this error across 2 clusters in the last week

Related to search source
Cannot read properties of undefined (reading 'type') - TypeError: Cannot read properties of undefined (reading 'type')
833,075 instances of this error spread out over hundreds of clusters in the last week

Cannot read properties of undefined (reading 'gte') - TypeError: Cannot read properties of undefined (reading 'gte')
10,355 instances of this error over 3 clusters in the last week

Cannot read properties of undefined (reading 'timeFieldName') - TypeError: Cannot read properties of undefined (reading 'timeFieldName')
8,006 instances of this error over 2 clusters in the last week

Cannot read properties of undefined (reading 'index') - TypeError: Cannot read properties of undefined (reading 'index')
5,993 instances of this error over 2 clusters in the last week

Cannot read properties of undefined (reading 'fields') - TypeError: Cannot read properties of undefined (reading 'fields')
0 instances of this error in the last week

Cannot read properties of undefined (reading 'title') - TypeError: Cannot read properties of undefined (reading 'title')
0 instances of this error in the last week

Accessing field from attributes returned by saved objects client - points to possible ES issues
Cannot read properties of undefined (reading 'actionId') - TypeError: Cannot read properties of undefined (reading 'actionId')
50 instances of this error over 3 clusters in the last week

Cannot read properties of undefined (reading 'scheduledAt') - TypeError: Cannot read properties of undefined (reading 'scheduledAt')
0 instances of this error in the last week

Cannot read properties of undefined (reading 'meta') - TypeError: Cannot read properties of undefined (reading 'meta')
0 instances of this error in the last week

Other
Cannot read properties of undefined (reading 'getFlapping') - TypeError: Cannot read properties of undefined (reading 'getFlapping')
0 instances of this error in the last week
Looking back over the last 90 days, we only see this error show up in version 8.11.1 so it seems it may have been fixed at some point

@ymao1 ymao1 mentioned this issue May 7, 2024
Merged
@ymao1
Copy link
Contributor Author

ymao1 commented May 9, 2024

ES query bug fixed with #182865
Search source timefield bug fixed with #182883

Created followup issues for the remaining search source errors #183068 and for the ES related errors #183072

Closing this issue in favor of those

@ymao1 ymao1 closed this as completed May 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ymao1 ymao1

Labels
Feature:Alerting Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams)
Projects
No open projects
AppEx: ResponseOps - Execution & Connectors
Status: Done
Milestone
No milestone
Development

No branches or pull requests
3 participants
@pmuellr @ymao1 @elasticmachine