Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[APM] Add log statements for flaky test #53775

Merged
merged 4 commits into from Dec 24, 2019
Merged

Conversation

@sqren
Copy link
Member

sqren commented Dec 23, 2019

I've been unable to reproduce a test failure that occasionally happens on CI. For details about the failure see #51764.

I have a hunch that a race condition is causing the test to fail. The test that is failing tries to verify that an existing agent config can be found when given the correct arguments. However, sometimes it is not found.
My hypothesis is the agent configuration is either not created correctly, or is created in parallel with the search request, causing the search request to be made before the config is created.

I was having problems seeing log.debug/log.info so resorted to console.log. Would prefer not using console.log though.

Important log lines indicating that the agent configuration was created:

[2019-12-23T22:54:27.872Z] Creating agent configuration
[2019-12-23T22:54:27.872Z]                │ proc [kibana] Could not get dynamic index pattern because indices "apm-*,apm-*,apm-*" don't exist
[2019-12-23T22:54:27.872Z]                │ proc [kibana]   log   [22:54:27.850] [info][apm][apm][plugins] Hitting: /api/apm/settings/agent-configuration/new with test-service/undefined
[2019-12-23T22:54:27.872Z]                │ proc [kibana]   log   [22:54:27.865] [info][apm][apm][plugins] Created agent configuration
[2019-12-23T22:54:27.872Z] Agent configuration created

Important log lines indicating that the agent configuration was found:

[2019-12-23T22:54:30.435Z] Requesting: /api/apm/settings/agent-configuration/search. Expecting: response
[2019-12-23T22:54:30.435Z]                │ proc [kibana]   log   [22:54:30.292] [info][apm][apm][plugins] Hitting: /api/apm/settings/agent-configuration/search for test-service/undefined
[2019-12-23T22:54:30.435Z]                │ proc [kibana]   log   [22:54:30.317] [info][apm][apm][plugins] Config was found for test-service/undefined
[2019-12-23T22:54:30.436Z] Responded: /api/apm/settings/agent-configuration/search
[2019-12-23T22:54:30.436Z]                └- ✓ pass  (1.3s) "apis APM apm feature controls APIs can be accessed by global_all user"
@sqren sqren requested a review from elastic/apm-ui as a code owner Dec 23, 2019
@kibanamachine

This comment has been minimized.

Copy link

kibanamachine commented Dec 23, 2019

💚 Build Succeeded

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@sqren sqren force-pushed the sqren:log-flaky-tests branch from df40484 to 0faa8bf Dec 23, 2019
@kibanamachine

This comment has been minimized.

Copy link

kibanamachine commented Dec 23, 2019

💚 Build Succeeded

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@kibanamachine

This comment has been minimized.

Copy link

kibanamachine commented Dec 23, 2019

💚 Build Succeeded

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@sqren sqren added the Team:apm label Dec 24, 2019
@elasticmachine

This comment has been minimized.

Copy link
Contributor

elasticmachine commented Dec 24, 2019

Pinging @elastic/apm-ui (Team:apm)

@dgieselaar

This comment has been minimized.

Copy link
Contributor

dgieselaar commented Dec 24, 2019

Would it help to log the output in the failing test of: whether the index exists, and a search without filters (e.g. output all documents)?

sqren added 4 commits Dec 23, 2019
@sqren sqren force-pushed the sqren:log-flaky-tests branch from a22b5d6 to 6b2c986 Dec 24, 2019
@kibanamachine

This comment has been minimized.

Copy link

kibanamachine commented Dec 24, 2019

💚 Build Succeeded

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@sqren

This comment has been minimized.

Copy link
Member Author

sqren commented Dec 24, 2019

Would it help to log the output in the failing test of: whether the index exists, and a search without filters (e.g. output all documents)?

Good idea. Adding in 6b2c986

@sqren sqren merged commit 53513f6 into elastic:master Dec 24, 2019
55 checks passed
55 checks passed
API integration tests node scripts/functional_tests --config test/api_integration/config.js --bail --debug
Details
Browser tests yarn run grunt test:browser-ci
Details
Build kbn_tp_sample_panel_action yarn build
Details
CLA All commits in pull request signed
Details
Check core API changes node scripts/check_core_api_changes
Details
Check file casing node scripts/check_file_casing --quiet
Details
Check licenses node scripts/check_licenses --dev
Details
Example functional tests node scripts/functional_tests --config test/examples/config.js --bail --debug
Details
Firefox smoke test node scripts/functional_tests --bail --debug --kibana-install-dir /dev/shm/workspace/kibana/build/oss/kibana-8.0.0-SNAPSHOT-linux-x86_64 --include-tag smoke --config test/functional/config.firefox.js
Details
Functional tests / Group 1 yarn run grunt run:functionalTests_ciGroup1
Details
Functional tests / Group 10 yarn run grunt run:functionalTests_ciGroup10
Details
Functional tests / Group 11 yarn run grunt run:functionalTests_ciGroup11
Details
Functional tests / Group 12 yarn run grunt run:functionalTests_ciGroup12
Details
Functional tests / Group 2 yarn run grunt run:functionalTests_ciGroup2
Details
Functional tests / Group 3 yarn run grunt run:functionalTests_ciGroup3
Details
Functional tests / Group 4 yarn run grunt run:functionalTests_ciGroup4
Details
Functional tests / Group 5 yarn run grunt run:functionalTests_ciGroup5
Details
Functional tests / Group 6 yarn run grunt run:functionalTests_ciGroup6
Details
Functional tests / Group 7 yarn run grunt run:functionalTests_ciGroup7
Details
Functional tests / Group 8 yarn run grunt run:functionalTests_ciGroup8
Details
Functional tests / Group 9 yarn run grunt run:functionalTests_ciGroup9
Details
Internationalization check node scripts/i18n_check --ignore-missing
Details
Interpreter functional tests node scripts/functional_tests --config test/interpreter_functional/config.ts --bail --debug --kibana-install-dir /dev/shm/workspace/kibana/build/oss/kibana-8.0.0-SNAPSHOT-linux-x86_64-2
Details
Jest integration tests yarn run grunt test:jest_integration
Details
Jest tests yarn run grunt test:jest
Details
Kibana accessibility tests node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/kibana/build/oss/kibana-8.0.0-SNAPSHOT-linux-x86_64 --config test/accessibility/config.ts
Details
Mocha tests node scripts/mocha
Details
Plugin functional tests node scripts/functional_tests --config test/plugin_functional/config.js --bail --debug
Details
Project tests yarn run grunt test:projects
Details
Type check node scripts/type_check
Details
TypeScript - all files belong to a TypeScript project node scripts/check_ts_projects
Details
Verify NOTICE.txt node scripts/notice --validate
Details
Verify dependency versions yarn run grunt verifyDependencyVersions
Details
X-Pack Chrome Functional tests / Group 1 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-2 --include-tag ciGroup1
Details
X-Pack Chrome Functional tests / Group 10 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-11 --include-tag ciGroup10
Details
X-Pack Chrome Functional tests / Group 2 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-3 --include-tag ciGroup2
Details
X-Pack Chrome Functional tests / Group 3 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-4 --include-tag ciGroup3
Details
X-Pack Chrome Functional tests / Group 4 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-5 --include-tag ciGroup4
Details
X-Pack Chrome Functional tests / Group 5 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-6 --include-tag ciGroup5
Details
X-Pack Chrome Functional tests / Group 6 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-7 --include-tag ciGroup6
Details
X-Pack Chrome Functional tests / Group 7 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-8 --include-tag ciGroup7
Details
X-Pack Chrome Functional tests / Group 8 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-9 --include-tag ciGroup8
Details
X-Pack Chrome Functional tests / Group 9 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-10 --include-tag ciGroup9
Details
X-Pack Jest node scripts/jest --ci --verbose
Details
X-Pack Karma Tests yarn test:browser
Details
X-Pack SIEM cyclic dependency test node legacy/plugins/siem/scripts/check_circular_deps
Details
X-Pack accessibility tests node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana --config test/accessibility/config.ts
Details
X-Pack firefox smoke test node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-1 --include-tag smoke --config test/functional/config.firefox.js
Details
elasticsearch-ci/docs Build finished.
Details
eslint node scripts/eslint --no-cache
Details
kibana-ci Build finished.
Details
prbot:outdated
prbot:release note labels
prbot:release version labels
sasslint node scripts/sasslint
Details
@sqren sqren deleted the sqren:log-flaky-tests branch Dec 24, 2019
gmmorris added a commit to gmmorris/kibana that referenced this pull request Dec 27, 2019
* master:
  Possibility to filter when testing scripted fields (elastic#35379) (elastic#44220)
  Update maps telemetry mappings to account for recent updates (elastic#53803)
  [Maps] Only show legend when layer is visible (elastic#53781)
  remove use of experimental fs.promises api (elastic#53346)
  [APM] Add log statements for flaky test (elastic#53775)
jloleysens added a commit to jloleysens/kibana that referenced this pull request Dec 30, 2019
…le-saved-objects

* 'master' of github.com:elastic/kibana: (250 commits)
  Allow chromeless applications to render via non-/app routes (elastic#51527)
  Add server rendering service to enable standalone route rendering (elastic#52161)
  Possibility to filter when testing scripted fields (elastic#35379) (elastic#44220)
  Update maps telemetry mappings to account for recent updates (elastic#53803)
  [Maps] Only show legend when layer is visible (elastic#53781)
  remove use of experimental fs.promises api (elastic#53346)
  [APM] Add log statements for flaky test (elastic#53775)
  [APM] Transaction page throws unhandled exception if transactions doesn't have  `http.request` (elastic#53760)
  Licensing plugin functional tests (elastic#53580)
  [Lens] Disable saving visualization until there are no changes to the document (elastic#52982)
  [Monitoring] Added safeguard for some EUI components (elastic#53318)
  [Vega] Shim new platform - cleanup vega_visualization dependencies (elastic#53605)
  Display changed field formats without requiring hard page refresh. (elastic#53746)
  Upgrade EUI to v17.3.1 (elastic#53655)
  [APM] Fix missing apm indicies (elastic#53541)
  Disable inspector for timelion (elastic#53747)
  Clean up search servie (elastic#53701)
  [Dashboard] Grid: removing double handler (elastic#53707)
  Remove SavedObjectRegistryProvider from codebase (elastic#53455)
  Move ui/courier into data shim plugin (elastic#52359)
  ...
jkelastic added a commit to jkelastic/kibana that referenced this pull request Jan 8, 2020
* [APM] Add log statements for flaky test

* Improve logging

* Improve logging

* Log full index on error
sqren added a commit to sqren/kibana that referenced this pull request Feb 6, 2020
* [APM] Add log statements for flaky test

* Improve logging

* Improve logging

* Log full index on error
sqren added a commit that referenced this pull request Feb 11, 2020
* [APM] Add log statements for flaky test

* Improve logging

* Improve logging

* Log full index on error

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.