Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adds apm instrumentation to task manager's task runner #55356

Merged
merged 2 commits into from Feb 11, 2020

Conversation

@pmuellr
Copy link
Contributor

pmuellr commented Jan 20, 2020

Summary

This adds APM support to task manager by instrumenting task manager's task runner with apm transactions.

To use in Kibana with yarn start, create a file config/apm.dev.js with the following contents:

module.exports = {
  active: true,
  secretToken: '<token>',
  serverUrl: '<url>',
  // transactionSampleRate: 1.0,
  // metricsInterval: '5s',
};

You'll then need an APM server running. For running Kibana off master, I was able to run a production cloud 7.5.1 deployment, with apm enabled (the default), and use the token and url provided by the deployment. Start your local Kibana via yarn start, and you'll start seeing numbers in your APM instance.

Here's an example of what a pretty simple instrumentation (the first commit) can provide:

image

const trans = apm.startTransaction(
`taskManager run ${this.instance.taskType}`,
'taskManager',
this.instance.taskType,

This comment has been minimized.

Copy link
@pmuellr

pmuellr Feb 10, 2020

Author Contributor

The name and type (first two parms ^^^) of apm.startTransaction() seem useful, in terms of how the UI ends up making use of them. The subtype and action (last two parms) I'm less sure of, as I'm not sure they come up directly in the UI. Since these end up being other plugins tasks running, seems unlikely we can consistently get more specific per-task info. And alerting and actions already have alertType and actionType specific task types, which makes for nice grouping.

Thinking of just removing the last two.

This comment has been minimized.

Copy link
@pmuellr

pmuellr Feb 11, 2020

Author Contributor

resolved

return this.processResult(validatedResult);
} catch (err) {
this.logger.error(`Task ${this} failed: ${err}`);
// in error scenario, we can not get the RunResult
// re-use modifiedContext's state, which is correct as of beforeRun
if (trans) trans.end();

This comment has been minimized.

Copy link
@pmuellr

pmuellr Feb 10, 2020

Author Contributor

probably want to use trans.end('success') and one for failure as well

This comment has been minimized.

Copy link
@pmuellr

pmuellr Feb 11, 2020

Author Contributor

resolved

@@ -252,6 +253,7 @@ export class TaskStore {
)
);

const trans = apm.startTransaction(`taskManager markAvailableTasksAsClaimed`, 'taskManager');

This comment has been minimized.

Copy link
@pmuellr

pmuellr Feb 10, 2020

Author Contributor

I think for this call, we should look into using transaction.setLabel() to add the queue size. Hopefully it will show up somewhere in the UI, but ideally I'd like some sort of specialzed ui for this anyway, so while it may not be immediately useful, at least it's getting written somewhere.

This comment has been minimized.

Copy link
@pmuellr

pmuellr Feb 11, 2020

Author Contributor

label values must be strings, so not as useful as I thought

This comment has been minimized.

Copy link
@pmuellr

pmuellr Feb 11, 2020

Author Contributor

also, we'd need to actually get the queue size, but I wouldn't want the cost of getting that to be included here, so we'll have to figure out some other story for the queue size ...

Adds some apm transaction boundaries for parts of task manager, so that
they will show up in APM as new types of transactions.  Should provide
some visibility into the ES calls made by task manager for alerting and
actions, especially under stress testing scenarios.
@pmuellr pmuellr force-pushed the pmuellr:taskManager/apm-ize branch from bd09593 to 76451dd Feb 11, 2020
@elasticmachine

This comment has been minimized.

Copy link
Contributor

elasticmachine commented Feb 11, 2020

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

@pmuellr pmuellr marked this pull request as ready for review Feb 11, 2020
@pmuellr pmuellr requested a review from elastic/kibana-alerting-services as a code owner Feb 11, 2020
@dgieselaar dgieselaar requested a review from elastic/apm-ui Feb 11, 2020
@sqren
sqren approved these changes Feb 11, 2020
Copy link
Member

sqren left a comment

This is super exciting to see!

Copy link
Contributor

gmmorris left a comment

LGTM

@pmuellr

This comment has been minimized.

Copy link
Contributor Author

pmuellr commented Feb 11, 2020

@elasticmachine merge upstream

@kibanamachine

This comment has been minimized.

Copy link

kibanamachine commented Feb 11, 2020

💛 Build succeeded, but was flaky


Test Failures

Kibana Pipeline / kibana-xpack-agent / Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/advanced_settings/feature_controls/advanced_settings_spaces·ts.Advanced Settings spaces feature controls space with Advanced Settings disabled redirects to management home

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         └-: Advanced Settings
[00:00:00]           └-> "before all" hook
[00:03:03]           └-: spaces feature controls
[00:03:03]             └-> "before all" hook
[00:03:03]             └-> "before all" hook
[00:03:03]               │ info [logstash_functional] Loading "mappings.json"
[00:03:03]               │ info [logstash_functional] Loading "data.json.gz"
[00:03:04]               │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-oraclelinux-tests-xl-1581457584437083876] [logstash-2015.09.22] creating index, cause [api], templates [], shards [1]/[0], mappings [_doc]
[00:03:04]               │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-oraclelinux-tests-xl-1581457584437083876] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[logstash-2015.09.22][0]]]).
[00:03:04]               │ info [logstash_functional] Created index "logstash-2015.09.22"
[00:03:04]               │ debg [logstash_functional] "logstash-2015.09.22" settings {"index":{"analysis":{"analyzer":{"url":{"max_token_length":"1000","tokenizer":"uax_url_email","type":"standard"}}},"number_of_replicas":"0","number_of_shards":"1"}}
[00:03:04]               │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-oraclelinux-tests-xl-1581457584437083876] [logstash-2015.09.20] creating index, cause [api], templates [], shards [1]/[0], mappings [_doc]
[00:03:04]               │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-oraclelinux-tests-xl-1581457584437083876] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[logstash-2015.09.20][0]]]).
[00:03:04]               │ info [logstash_functional] Created index "logstash-2015.09.20"
[00:03:04]               │ debg [logstash_functional] "logstash-2015.09.20" settings {"index":{"analysis":{"analyzer":{"url":{"max_token_length":"1000","tokenizer":"uax_url_email","type":"standard"}}},"number_of_replicas":"0","number_of_shards":"1"}}
[00:03:04]               │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-oraclelinux-tests-xl-1581457584437083876] [logstash-2015.09.21] creating index, cause [api], templates [], shards [1]/[0], mappings [_doc]
[00:03:04]               │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-oraclelinux-tests-xl-1581457584437083876] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[logstash-2015.09.21][0]]]).
[00:03:04]               │ info [logstash_functional] Created index "logstash-2015.09.21"
[00:03:04]               │ debg [logstash_functional] "logstash-2015.09.21" settings {"index":{"analysis":{"analyzer":{"url":{"max_token_length":"1000","tokenizer":"uax_url_email","type":"standard"}}},"number_of_replicas":"0","number_of_shards":"1"}}
[00:03:13]               │ info progress: 7522
[00:03:20]               │ info [logstash_functional] Indexed 4633 docs into "logstash-2015.09.22"
[00:03:20]               │ info [logstash_functional] Indexed 4757 docs into "logstash-2015.09.20"
[00:03:20]               │ info [logstash_functional] Indexed 4614 docs into "logstash-2015.09.21"
[00:04:24]             └-: space with Advanced Settings disabled
[00:04:24]               └-> "before all" hook
[00:04:24]               └-> "before all" hook
[00:04:24]                 │ info [empty_kibana] Loading "mappings.json"
[00:04:24]                 │ info [empty_kibana] Loading "data.json.gz"
[00:04:24]                 │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-oraclelinux-tests-xl-1581457584437083876] [.kibana_1/Jlpc8ny3TfCk48ViE8n4Ww] deleting index
[00:04:24]                 │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-oraclelinux-tests-xl-1581457584437083876] [.kibana_2/JoBT2WN_Sui36PR3T_zCGQ] deleting index
[00:04:24]                 │ info [empty_kibana] Deleted existing index [".kibana_2",".kibana_1"]
[00:04:24]                 │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-oraclelinux-tests-xl-1581457584437083876] [.kibana] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[00:04:24]                 │ info [empty_kibana] Created index ".kibana"
[00:04:24]                 │ debg [empty_kibana] ".kibana" settings {"index":{"number_of_replicas":"1","number_of_shards":"1"}}
[00:04:24]                 │ info [empty_kibana] Indexed 2 docs into ".kibana"
[00:04:26]                 │ info Creating index .kibana_2.
[00:04:26]                 │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-oraclelinux-tests-xl-1581457584437083876] [.kibana_2] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[00:04:26]                 │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-oraclelinux-tests-xl-1581457584437083876] updating number_of_replicas to [0] for indices [.kibana_2]
[00:04:26]                 │ info Reindexing .kibana to .kibana_1
[00:04:26]                 │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-oraclelinux-tests-xl-1581457584437083876] [.kibana_1] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[00:04:26]                 │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-oraclelinux-tests-xl-1581457584437083876] updating number_of_replicas to [0] for indices [.kibana_1]
[00:04:26]                 │ info [o.e.t.LoggingTaskListener] [kibana-ci-immutable-oraclelinux-tests-xl-1581457584437083876] 5343 finished with response BulkByScrollResponse[took=31.9ms,timed_out=false,sliceId=null,updated=0,created=2,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]
[00:04:26]                 │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-oraclelinux-tests-xl-1581457584437083876] [.kibana/5E7N9jCqSFeA1qQxS-wLWw] deleting index
[00:04:26]                 │ info Migrating .kibana_1 saved objects to .kibana_2
[00:04:26]                 │ debg Migrating saved objects config:6.0.0-alpha1, space:default
[00:04:26]                 │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-oraclelinux-tests-xl-1581457584437083876] [.kibana_2/VrBXepTcQ1yfTuU7dQVQ-A] update_mapping [_doc]
[00:04:26]                 │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-oraclelinux-tests-xl-1581457584437083876] [.kibana_2/VrBXepTcQ1yfTuU7dQVQ-A] update_mapping [_doc]
[00:04:26]                 │ info Pointing alias .kibana to .kibana_2.
[00:04:26]                 │ info Finished in 631ms.
[00:04:26]                 │ debg applying update to kibana config: {"accessibility:disableAnimations":true,"dateFormat:tz":"UTC"}
[00:04:27]                 │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-oraclelinux-tests-xl-1581457584437083876] [.kibana_2/VrBXepTcQ1yfTuU7dQVQ-A] update_mapping [_doc]
[00:04:28]                 │ debg creating space
[00:04:29]                 │ debg created space
[00:04:29]               └-> redirects to management home
[00:04:29]                 └-> "before each" hook: global before each
[00:04:29]                 │ debg navigateToActualUrl http://localhost:6131/s/custom_space/app/kibana#management/kibana/settings
[00:04:30]                 │ debg browser[INFO] http://localhost:6131/s/custom_space/app/kibana?_t=1581459755373#management/kibana/settings 350 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:04:30]                 │
[00:04:30]                 │ debg browser[INFO] http://localhost:6131/s/custom_space/bundles/app/kibana/bootstrap.js 8:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:04:30]                 │ debg TestSubjects.exists(managementHome)
[00:04:30]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="managementHome"]') with timeout=10000
[00:04:33]                 │ debg --- retry.tryForTime error: [data-test-subj="managementHome"] is not displayed
[00:04:37]                 │ debg browser[INFO] http://localhost:6131/built_assets/dlls/vendors_3.bundle.dll.js 181:139970 "INFO: 2020-02-11T22:22:40Z
[00:04:37]                 │        Adding connection to http://localhost:6131/s/custom_space/elasticsearch
[00:04:37]                 │
[00:04:37]                 │      "
[00:04:37]                 │ debg browser[INFO] http://localhost:6131/s/custom_space/app/kibana#/management 350 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:04:37]                 │
[00:04:37]                 │ debg browser[INFO] http://localhost:6131/s/custom_space/bundles/app/kibana/bootstrap.js 8:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:04:37]                 │ debg --- retry.tryForTime failed again with the same message...
[00:04:40]                 │ debg --- retry.tryForTime failed again with the same message...
[00:04:41]                 │ info Taking screenshot "/dev/shm/workspace/kibana/x-pack/test/functional/screenshots/failure/Advanced Settings spaces feature controls space with Advanced Settings disabled redirects to management home.png"
[00:04:41]                 │ debg browser[INFO] http://localhost:6131/built_assets/dlls/vendors_3.bundle.dll.js 181:139970 "INFO: 2020-02-11T22:22:47Z
[00:04:41]                 │        Adding connection to http://localhost:6131/s/custom_space/elasticsearch
[00:04:41]                 │
[00:04:41]                 │      "
[00:04:44]                 │ info Current URL is: http://localhost:6131/s/custom_space/app/kibana#/management?_g=()
[00:04:44]                 │ info Saving page source to: /dev/shm/workspace/kibana/x-pack/test/functional/failure_debug/html/Advanced Settings spaces feature controls space with Advanced Settings disabled redirects to management home.html
[00:04:44]                 └- ✖ fail: "Advanced Settings spaces feature controls space with Advanced Settings disabled redirects to management home"
[00:04:44]                 │

Stack Trace

Error: expected testSubject(managementHome) to exist
    at TestSubjects.existOrFail (/dev/shm/workspace/kibana/test/functional/services/test_subjects.ts:60:15)

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@pmuellr pmuellr merged commit f304f88 into elastic:master Feb 11, 2020
56 checks passed
56 checks passed
API integration tests node scripts/functional_tests --config test/api_integration/config.js --bail --debug
Details
Browser tests yarn run grunt test:browser-ci
Details
Build kbn_tp_sample_panel_action yarn build
Details
CLA All commits in pull request signed
Details
Check core API changes node scripts/check_core_api_changes
Details
Check file casing node scripts/check_file_casing --quiet
Details
Check licenses node scripts/check_licenses --dev
Details
Check lockfile symlinks node scripts/check_lockfile_symlinks --quiet
Details
Example functional tests node scripts/functional_tests --config test/examples/config.js --bail --debug
Details
Firefox smoke test node scripts/functional_tests --bail --debug --kibana-install-dir /dev/shm/workspace/kibana/build/oss/kibana-8.0.0-SNAPSHOT-linux-x86_64 --include-tag smoke --config test/functional/config.firefox.js
Details
Functional tests / Group 1 yarn run grunt run:functionalTests_ciGroup1
Details
Functional tests / Group 10 yarn run grunt run:functionalTests_ciGroup10
Details
Functional tests / Group 11 yarn run grunt run:functionalTests_ciGroup11
Details
Functional tests / Group 12 yarn run grunt run:functionalTests_ciGroup12
Details
Functional tests / Group 2 yarn run grunt run:functionalTests_ciGroup2
Details
Functional tests / Group 3 yarn run grunt run:functionalTests_ciGroup3
Details
Functional tests / Group 4 yarn run grunt run:functionalTests_ciGroup4
Details
Functional tests / Group 5 yarn run grunt run:functionalTests_ciGroup5
Details
Functional tests / Group 6 yarn run grunt run:functionalTests_ciGroup6
Details
Functional tests / Group 7 yarn run grunt run:functionalTests_ciGroup7
Details
Functional tests / Group 8 yarn run grunt run:functionalTests_ciGroup8
Details
Functional tests / Group 9 yarn run grunt run:functionalTests_ciGroup9
Details
Internationalization check node scripts/i18n_check --ignore-missing
Details
Interpreter functional tests node scripts/functional_tests --config test/interpreter_functional/config.ts --bail --debug --kibana-install-dir /dev/shm/workspace/kibana/build/oss/kibana-8.0.0-SNAPSHOT-linux-x86_64-2
Details
Jest integration tests yarn run grunt test:jest_integration
Details
Jest tests yarn run grunt test:jest
Details
Kibana accessibility tests node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/kibana/build/oss/kibana-8.0.0-SNAPSHOT-linux-x86_64 --config test/accessibility/config.ts
Details
Mocha tests node scripts/mocha
Details
Plugin functional tests node scripts/functional_tests --config test/plugin_functional/config.js --bail --debug
Details
Project tests yarn run grunt test:projects
Details
Type check node scripts/type_check
Details
TypeScript - all files belong to a TypeScript project node scripts/check_ts_projects
Details
Verify NOTICE.txt node scripts/notice --validate
Details
Verify dependency versions yarn run grunt verifyDependencyVersions
Details
X-Pack Chrome Functional tests / Group 1 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-2 --include-tag ciGroup1
Details
X-Pack Chrome Functional tests / Group 10 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-11 --include-tag ciGroup10
Details
X-Pack Chrome Functional tests / Group 2 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-3 --include-tag ciGroup2
Details
X-Pack Chrome Functional tests / Group 3 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-4 --include-tag ciGroup3
Details
X-Pack Chrome Functional tests / Group 4 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-5 --include-tag ciGroup4
Details
X-Pack Chrome Functional tests / Group 5 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-6 --include-tag ciGroup5
Details
X-Pack Chrome Functional tests / Group 6 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-7 --include-tag ciGroup6
Details
X-Pack Chrome Functional tests / Group 7 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-8 --include-tag ciGroup7
Details
X-Pack Chrome Functional tests / Group 8 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-9 --include-tag ciGroup8
Details
X-Pack Chrome Functional tests / Group 9 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-10 --include-tag ciGroup9
Details
X-Pack Jest node scripts/jest --ci --verbose
Details
X-Pack Karma Tests yarn test:browser
Details
X-Pack SIEM cyclic dependency test node legacy/plugins/siem/scripts/check_circular_deps
Details
X-Pack accessibility tests node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana --config test/accessibility/config.ts
Details
X-Pack firefox smoke test node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-1 --include-tag smoke --config test/functional/config.firefox.js
Details
elasticsearch-ci/docs Build finished.
Details
eslint node scripts/eslint --no-cache
Details
kibana-ci Build finished.
Details
prbot:outdated
prbot:release note labels
prbot:release version labels
sasslint node scripts/sasslint
Details
pmuellr added a commit to pmuellr/kibana that referenced this pull request Feb 11, 2020
Adds some apm transaction boundaries for parts of task manager, so that
they will show up in APM as new types of transactions.  Should provide
some visibility into the ES calls made by task manager for alerting and
actions, especially under stress testing scenarios.
pmuellr added a commit that referenced this pull request Feb 12, 2020
Adds some apm transaction boundaries for parts of task manager, so that
they will show up in APM as new types of transactions.  Should provide
some visibility into the ES calls made by task manager for alerting and
actions, especially under stress testing scenarios.
@pmuellr pmuellr added the backported label Feb 12, 2020
gmmorris added a commit to gmmorris/kibana that referenced this pull request Feb 12, 2020
* master:
  [Canvas] Move sample data and feature registration to canvas np plugin (elastic#56564)
  instrument task manager with apm transactions (elastic#55356)
  displays Alert Instance state on Alert Details page (elastic#56842)
  Adding the Accessibility Statement to docs (elastic#57153)
  [Uptime] Remove redundant adapter function (elastic#56980)
  [SIEM][Detection Engine] Backend end-to-end tests
  [Uptime] Added tests for pages (elastic#56736)
  Updating to kind-of@6.0.3 (elastic#57367)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants
You can’t perform that action at this time.