[ML] APM Latency Correlations: Field/value candidates prioritization #107370

walterra · 2021-08-02T09:32:23Z

Summary

Part of #106381

Makes sure fields defined in FIELDS_TO_ADD_AS_CANDIDATE and prefixed with one of FIELD_PREFIX_TO_ADD_AS_CANDIDATE get queried first when retrieving the correlation and ks-test value.
Correctly consider the includeFrozen parameter.
The bulk of the PR is a refactor:
- Moves query_* files to queries directory
- Introduces asyncSearchServiceStateProvider to manage the state of the async search service in isolation so that we no longer mutate individual vars or plain objects.
- Introduces asyncSearchServiceLogProvider and extends the log to not only store messages but original error messages retrieved from ES too.
- Refactors some more functions in separate files and adds unit tests.
- Removes some deprecated code no longer needed.

Checklist

Unit or functional tests were updated or added to match the most common scenarios
This was checked for breaking API changes and was labeled appropriately

elasticmachine · 2021-08-02T14:56:56Z

Pinging @elastic/ml-ui (:ml)

elasticmachine · 2021-08-02T14:56:58Z

Pinging @elastic/apm-ui (Team:apm)

smith · 2021-08-03T18:27:29Z

.../plugins/apm/server/lib/search_strategies/correlations/queries/query_histograms_generator.ts

+    if (
+      params === undefined ||
+      item === undefined ||
+      state.getState().isCancelled


Since this is used a few times inside this for loop it might make sense to assign it to a variable outside the loop.

The process could have been cancelled by the user in parallel while the for loop is still running, that's why we always need to check for the latest state of isCancelled, if we assign it once outside the loop it could be out of date. I added a helper getter so we can use state.getIsCancelled() in af7afce to simplify the usage a bit.

x-pack/plugins/apm/server/lib/search_strategies/correlations/queries/query_field_value_pairs.ts

...s/apm/server/lib/search_strategies/correlations/queries/get_prioritized_field_value_pairs.ts

x-pack/plugins/apm/server/lib/search_strategies/correlations/queries/get_request_base.test.ts

qn895 · 2021-08-03T19:16:38Z

LGTM, just left a few comments 🎉 Tested the overall behavior and everything looks good, will look into testing the new includeFrozen option.

sorenlouv · 2021-08-04T11:42:43Z

x-pack/plugins/apm/server/lib/search_strategies/correlations/queries/query_field_value_pairs.ts

@@ -48,17 +48,17 @@ export const fetchTransactionDurationFieldValuePairs = async (
  esClient: ElasticsearchClient,
  params: SearchServiceFetchParams,
  fieldCandidates: Field[],
-  progress: AsyncSearchProviderProgress
+  state: AsyncSearchServiceState
 ): Promise<FieldValuePairs> => {
  const fieldValuePairs: FieldValuePairs = [];

  let fieldValuePairsProgress = 1;

  for (let i = 0; i < fieldCandidates.length; i++) {


Perhaps this would become easier to read if you create a utility function that makes it explicit that we want to iterate sequentially through the array (as opposed to doing it in parallel with arr.map):

async function runInSequence(items, fn) { const results = []; for (const item of items) { results.push(await fn(item)); } return results; }

And using it like:

const fieldValuePairs = runInSequence(fieldCandidates, async (fieldName) => { })

Good idea, refactored in dd7636c.

walterra · 2021-08-09T08:30:35Z

@elasticmachine merge upstream

walterra · 2021-08-09T09:28:16Z

@elasticmachine merge upstream

kibanamachine · 2021-08-09T11:48:24Z

💛 Build succeeded, but was flaky

continuous-integration/kibana-ci/pull-request
Commit: 0a11475
Storybooks Preview
Documentation Changes
Flaky suites:
- kibana-ciGroup7

Test Failures

Kibana Pipeline / general / Chrome UI Functional Tests.test/functional/apps/management/_index_pattern_create_delete·js.management creating and deleting default index validation can display errors

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 11 times on tracked branches: https://github.com/elastic/kibana/issues/107831

[00:00:00]       │
[00:00:00]         └-: management
[00:00:00]           └-> "before all" hook in "management"
[00:00:00]           └-> "before all" hook in "management"
[00:00:00]             │ info [test/functional/fixtures/es_archiver/logstash_functional] Unloading indices from "mappings.json"
[00:00:00]             │ info [test/functional/fixtures/es_archiver/logstash_functional] Unloading indices from "data.json.gz"
[00:00:01]             │ info [test/functional/fixtures/es_archiver/makelogs] Loading "mappings.json"
[00:00:01]             │ info [test/functional/fixtures/es_archiver/makelogs] Loading "data.json.gz"
[00:00:01]             │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [logstash-2015.09.17] creating index, cause [api], templates [], shards [1]/[0]
[00:00:01]             │ info [o.e.c.r.a.AllocationService] [node-01] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[logstash-2015.09.17][0]]])." previous.health="YELLOW" reason="shards started [[logstash-2015.09.17][0]]"
[00:00:01]             │ info [test/functional/fixtures/es_archiver/makelogs] Created index "logstash-2015.09.17"
[00:00:01]             │ debg [test/functional/fixtures/es_archiver/makelogs] "logstash-2015.09.17" settings {"index":{"analysis":{"analyzer":{"url":{"max_token_length":"1000","tokenizer":"uax_url_email","type":"standard"}}},"number_of_replicas":"0","number_of_shards":"1"}}
[00:00:01]             │ info [o.e.x.i.IndexLifecycleTransition] [node-01] moving index [.ds-ilm-history-5-2021.08.09-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [ilm-history-ilm-policy]
[00:00:01]             │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [logstash-2015.09.18] creating index, cause [api], templates [], shards [1]/[0]
[00:00:01]             │ info [o.e.c.r.a.AllocationService] [node-01] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[logstash-2015.09.18][0]]])." previous.health="YELLOW" reason="shards started [[logstash-2015.09.18][0]]"
[00:00:01]             │ info [test/functional/fixtures/es_archiver/makelogs] Created index "logstash-2015.09.18"
[00:00:01]             │ debg [test/functional/fixtures/es_archiver/makelogs] "logstash-2015.09.18" settings {"index":{"analysis":{"analyzer":{"url":{"max_token_length":"1000","tokenizer":"uax_url_email","type":"standard"}}},"number_of_replicas":"0","number_of_shards":"1"}}
[00:00:01]             │ info [o.e.x.i.IndexLifecycleTransition] [node-01] moving index [.ds-ilm-history-5-2021.08.09-000001] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [ilm-history-ilm-policy]
[00:00:01]             │ info [test/functional/fixtures/es_archiver/makelogs] Indexed 101 docs into "logstash-2015.09.17"
[00:00:01]             │ info [test/functional/fixtures/es_archiver/makelogs] Indexed 301 docs into "logstash-2015.09.18"
[00:00:02]           └-: 
[00:00:02]             └-> "before all" hook in ""
[00:00:25]             └-: creating and deleting default index
[00:00:25]               └-> "before all" hook in "creating and deleting default index"
[00:00:25]               └-> "before all" hook in "creating and deleting default index"
[00:00:25]                 │ debg replacing kibana config doc: {}
[00:00:26]                 │ debg navigating to settings url: http://localhost:61151/app/management
[00:00:26]                 │ debg navigate to: http://localhost:61151/app/management
[00:00:26]                 │ debg browser[INFO] http://localhost:61151/app/management?_t=1628504342187 281 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:26]                 │
[00:00:26]                 │ debg browser[INFO] http://localhost:61151/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:26]                 │ debg ... sleep(700) start
[00:00:26]                 │ debg ... sleep(700) end
[00:00:26]                 │ debg returned from get, calling refresh
[00:00:27]                 │ debg browser[INFO] http://localhost:61151/app/management?_t=1628504342187 281 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:27]                 │
[00:00:27]                 │ debg browser[INFO] http://localhost:61151/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:28]                 │ debg currentUrl = http://localhost:61151/app/management
[00:00:28]                 │          appUrl = http://localhost:61151/app/management
[00:00:28]                 │ debg TestSubjects.find(kibanaChrome)
[00:00:28]                 │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:00:28]                 │ debg ... sleep(501) start
[00:00:28]                 │ debg ... sleep(501) end
[00:00:28]                 │ debg in navigateTo url = http://localhost:61151/app/management
[00:00:28]                 │ debg clickKibanaIndexPatterns link
[00:00:28]                 │ debg TestSubjects.click(indexPatterns)
[00:00:28]                 │ debg Find.clickByCssSelector('[data-test-subj="indexPatterns"]') with timeout=10000
[00:00:28]                 │ debg Find.findByCssSelector('[data-test-subj="indexPatterns"]') with timeout=10000
[00:00:28]                 │ debg isGlobalLoadingIndicatorVisible
[00:00:28]                 │ debg TestSubjects.exists(globalLoadingIndicator)
[00:00:28]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:00:30]                 │ debg --- retry.tryForTime error: [data-test-subj="globalLoadingIndicator"] is not displayed
[00:00:30]                 │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:00:30]                 │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:00:33]               └-: validation
[00:00:33]                 └-> "before all" hook for "can display errors"
[00:00:33]                 └-> can display errors
[00:00:33]                   └-> "before each" hook: global before each for "can display errors"
[00:00:33]                   │ debg TestSubjects.exists(createIndexPatternButtonFlyout)
[00:00:33]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="createIndexPatternButtonFlyout"]') with timeout=2500
[00:00:36]                   │ debg --- retry.tryForTime error: [data-test-subj="createIndexPatternButtonFlyout"] is not displayed
[00:00:36]                   │ debg TestSubjects.exists(createIndexPatternButton)
[00:00:36]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="createIndexPatternButton"]') with timeout=2500
[00:00:36]                   │ debg TestSubjects.click(createIndexPatternButton)
[00:00:36]                   │ debg Find.clickByCssSelector('[data-test-subj="createIndexPatternButton"]') with timeout=10000
[00:00:36]                   │ debg Find.findByCssSelector('[data-test-subj="createIndexPatternButton"]') with timeout=10000
[00:00:36]                   │ debg TestSubjects.exists(createIndexPatternButtonFlyout)
[00:00:36]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="createIndexPatternButtonFlyout"]') with timeout=2500
[00:00:36]                   │ debg TestSubjects.click(createIndexPatternButtonFlyout)
[00:00:36]                   │ debg Find.clickByCssSelector('[data-test-subj="createIndexPatternButtonFlyout"]') with timeout=10000
[00:00:36]                   │ debg Find.findByCssSelector('[data-test-subj="createIndexPatternButtonFlyout"]') with timeout=10000
[00:00:36]                   │ debg setIndexPatternField(log*)
[00:00:36]                   │ debg TestSubjects.find(createIndexPatternNameInput)
[00:00:36]                   │ debg Find.findByCssSelector('[data-test-subj="createIndexPatternNameInput"]') with timeout=10000
[00:00:37]                   │ debg setIndexPatternField set to log*
[00:00:37]                   │ debg TestSubjects.find(saveIndexPatternButton)
[00:00:37]                   │ debg Find.findByCssSelector('[data-test-subj="saveIndexPatternButton"]') with timeout=10000
[00:00:37]                   │ debg Find.findByClassName('euiFormErrorText') with timeout=10000
[00:00:47]                   │ info Taking screenshot "/dev/shm/workspace/parallel/15/kibana/test/functional/screenshots/failure/management  creating and deleting default index validation can display errors.png"
[00:00:47]                   │ info Current URL is: http://localhost:61151/app/management/kibana/indexPatterns/patterns/3eb85680-f8fb-11eb-84cb-8de80e1ce81f#/?_a=(tab:indexedFields)
[00:00:47]                   │ info Saving page source to: /dev/shm/workspace/parallel/15/kibana/test/functional/failure_debug/html/management  creating and deleting default index validation can display errors.html
[00:00:47]                   └- ✖ fail: management  creating and deleting default index validation can display errors
[00:00:47]                   │      TimeoutError: Waiting for element to be located By(css selector, .euiFormErrorText)
[00:00:47]                   │ Wait timed out after 10031ms
[00:00:47]                   │       at /dev/shm/workspace/parallel/15/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
[00:00:47]                   │       at runMicrotasks (<anonymous>)
[00:00:47]                   │       at processTicksAndRejections (internal/process/task_queues.js:95:5)
[00:00:47]                   │ 
[00:00:47]                   │

Stack Trace

TimeoutError: Waiting for element to be located By(css selector, .euiFormErrorText)
Wait timed out after 10031ms
    at /dev/shm/workspace/parallel/15/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  remoteStacktrace: ''
}

Metrics [docs]

✅ unchanged

History

💔 Build #143585 failed b9a60c0
💚 Build #142982 succeeded c04c2e7
💚 Build #142942 succeeded dd7636c
💚 Build #142902 succeeded 0a75e15
💚 Build #142291 succeeded 78362e3

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @walterra

walterra · 2021-08-10T08:04:38Z

@sqren this is ready for another look

…lastic#107370) - Makes sure fields defined in `FIELDS_TO_ADD_AS_CANDIDATE` and prefixed with one of `FIELD_PREFIX_TO_ADD_AS_CANDIDATE` get queried first when retrieving the `correlation` and `ks-test` value. - Correctly consider the `includeFrozen` parameter. - The bulk of the PR is a refactor: - Moves `query_*` files to `queries` directory - Introduces `asyncSearchServiceStateProvider` to manage the state of the async search service in isolation so that we no longer mutate individual vars or plain objects. - Introduces `asyncSearchServiceLogProvider` and extends the log to not only store messages but original error messages retrieved from ES too. - Refactors some more functions in separate files and adds unit tests. - Removes some deprecated code no longer needed.

kibanamachine · 2021-08-11T08:49:22Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

…-png-pdf-report-type * 'master' of github.com:elastic/kibana: (101 commits) [ML] APM Latency Correlations: Field/value candidates prioritization (elastic#107370) [Reporting] Add lenience to a test on the order of asserted logs (elastic#108135) [Lens] fix do not submit invalid query in filtered metric (elastic#107542) skip flaky test (elastic#108043) fix newly introduced type error (elastic#107593) [Reporting] server side code clean up (elastic#106940) [build_ts_refs] improve caches, allow building a subset of projects (elastic#107981) [APM] Add new ftr_e2e to kibana CI and remove current e2e tests. (elastic#107593) add manage rules link to alerts dropdown (elastic#107950) [ML] Enable Index data visualizer document count chart to update time range query (elastic#106438) [Security Solutions][Detection Engine] Fixes "undefined" crash for author field by adding a migration for it (elastic#107230) [Actions UI] Fixed Jira Api token label. (elastic#107776) [Alerting UI] Fixed display permissions for edit/delete buttons when user has read only access. (elastic#107996) [Maps] fix code owners (elastic#108106) Update EMS landing page url (elastic#108102) Do not render page header for loading domains (elastic#108078) Update dependency @elastic/charts to v33.2.2 (elastic#107939) [APM] Display throughput as tps (instead of tpm) when bucket size < 60 seconds (elastic#107850) [Fleet] Fix all category count (elastic#108089) [Security Solution][Bug] - Disable alert table RBAC until fields sorted (elastic#108034) ... # Conflicts: # x-pack/plugins/reporting/server/export_types/common/generate_png.ts # x-pack/plugins/reporting/server/lib/screenshots/index.ts # x-pack/plugins/reporting/server/lib/screenshots/observable.test.ts # x-pack/plugins/reporting/server/lib/screenshots/observable.ts

…107370) (#108152) - Makes sure fields defined in `FIELDS_TO_ADD_AS_CANDIDATE` and prefixed with one of `FIELD_PREFIX_TO_ADD_AS_CANDIDATE` get queried first when retrieving the `correlation` and `ks-test` value. - Correctly consider the `includeFrozen` parameter. - The bulk of the PR is a refactor: - Moves `query_*` files to `queries` directory - Introduces `asyncSearchServiceStateProvider` to manage the state of the async search service in isolation so that we no longer mutate individual vars or plain objects. - Introduces `asyncSearchServiceLogProvider` and extends the log to not only store messages but original error messages retrieved from ES too. - Refactors some more functions in separate files and adds unit tests. - Removes some deprecated code no longer needed. Co-authored-by: Walter Rafelsberger <walter@elastic.co>

walterra added release_note:enhancement Team:APM All issues that need APM UI Team support :ml v8.0.0 apm:correlations v7.15.0 labels Aug 2, 2021

walterra self-assigned this Aug 2, 2021

walterra mentioned this pull request Aug 2, 2021

[APM] Latency correlations 7.15 meta issue #106381

Closed

24 tasks

walterra force-pushed the ml-apm-correlations-field-priorization branch from 201dd65 to 3964729 Compare August 2, 2021 10:14

walterra added 7 commits August 2, 2021 15:28

[ML] Cleanup, remove deprecated code.

5af838e

[ML] Moves queries to their own directory.

bc1c708

[ML] Introduce service state.

85501ca

[ML] Move generator to its own file.

fe7b5f8

[ML] Sort field value candidates by priorization.

6043797

[ML] Adds jest unit tests.

0fc2a5b

[ML] Consider includeFrozen parameter.

78362e3

walterra force-pushed the ml-apm-correlations-field-priorization branch from 6c0a11f to 78362e3 Compare August 2, 2021 13:29

walterra marked this pull request as ready for review August 2, 2021 14:56

walterra requested a review from a team as a code owner August 2, 2021 14:56

walterra requested a review from qn895 August 2, 2021 14:57

smith approved these changes Aug 3, 2021

View reviewed changes

qn895 reviewed Aug 3, 2021

View reviewed changes

x-pack/plugins/apm/server/lib/search_strategies/correlations/queries/query_field_value_pairs.ts Outdated Show resolved Hide resolved

qn895 reviewed Aug 3, 2021

View reviewed changes

...s/apm/server/lib/search_strategies/correlations/queries/get_prioritized_field_value_pairs.ts Outdated Show resolved Hide resolved

qn895 reviewed Aug 3, 2021

View reviewed changes

x-pack/plugins/apm/server/lib/search_strategies/correlations/queries/get_request_base.test.ts Show resolved Hide resolved

walterra added 3 commits August 4, 2021 12:37

[ML] Adds getter for isCancelled.

af7afce

[ML] Fix buckets check.

99b1976

[ML] Tweak getPrioritizedFieldValuePairs.

0a75e15

sorenlouv reviewed Aug 4, 2021

View reviewed changes

walterra added 3 commits August 4, 2021 14:57

[ML] Consider optional includeFrozen.

933a5d6

[ML] Refactor with helper function to emphasize sequential fetching.

dd7636c

Merge branch 'master' into ml-apm-correlations-field-priorization

c04c2e7

qn895 approved these changes Aug 5, 2021

View reviewed changes

Merge branch 'master' into ml-apm-correlations-field-priorization

b9a60c0

Merge branch 'master' into ml-apm-correlations-field-priorization

0a11475

walterra added the auto-backport Deprecated: Automatically backport this PR after it's merged label Aug 11, 2021

sorenlouv approved these changes Aug 11, 2021

View reviewed changes

walterra merged commit 86c17da into elastic:master Aug 11, 2021

kibanamachine mentioned this pull request Aug 11, 2021

[7.x] [ML] APM Latency Correlations: Field/value candidates prioritization (#107370) #108152

Merged

walterra mentioned this pull request Aug 24, 2021

[ML] Increase Test Coverage 7.15 #106113

Closed

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] APM Latency Correlations: Field/value candidates prioritization #107370

[ML] APM Latency Correlations: Field/value candidates prioritization #107370

walterra commented Aug 2, 2021 •

edited

elasticmachine commented Aug 2, 2021

elasticmachine commented Aug 2, 2021

smith Aug 3, 2021

walterra Aug 4, 2021

qn895 commented Aug 3, 2021

sorenlouv Aug 4, 2021 •

edited

walterra Aug 4, 2021

walterra commented Aug 9, 2021

walterra commented Aug 9, 2021

kibanamachine commented Aug 9, 2021

Standard Out

Stack Trace

walterra commented Aug 10, 2021

kibanamachine commented Aug 11, 2021

[ML] APM Latency Correlations: Field/value candidates prioritization #107370

[ML] APM Latency Correlations: Field/value candidates prioritization #107370

Conversation

walterra commented Aug 2, 2021 • edited

Summary

Checklist

elasticmachine commented Aug 2, 2021

elasticmachine commented Aug 2, 2021

smith Aug 3, 2021

Choose a reason for hiding this comment

walterra Aug 4, 2021

Choose a reason for hiding this comment

qn895 commented Aug 3, 2021

sorenlouv Aug 4, 2021 • edited

Choose a reason for hiding this comment

walterra Aug 4, 2021

Choose a reason for hiding this comment

walterra commented Aug 9, 2021

walterra commented Aug 9, 2021

kibanamachine commented Aug 9, 2021

💛 Build succeeded, but was flaky

Test Failures

Standard Out

Stack Trace

Metrics [docs]

History

walterra commented Aug 10, 2021

kibanamachine commented Aug 11, 2021

💚 Backport successful

walterra commented Aug 2, 2021 •

edited

sorenlouv Aug 4, 2021 •

edited