docs: Add test file location -> test runner table #34986

rudolf · 2019-04-12T13:18:45Z

Summary

Following discussions around #34557 this adds documentation to CONTRIBUTING.md to try to address the following pain point:

On a bad day my workflow looks like this:

Make some changes

Try to test only the affected changes.

Give up and push to CI

Find something else to do while I wait for test results (~40mins)

Repeat

To improve this, I'd like to be able to:

Easily find out how to run a specific test file so that I can quickly run the tests most likely to be affected by my change (or even better, have a single test command that takes the file I'd like to test as argument).

If CI fails I want to know how to run the failing test cases locally so that I can have a short feedback loop when addressing these.

Please add suggestions for further test locations/runner invocations if there's anything I missed.

I wasn't able to figure out the following:

How to identify from a file location if a test might be a karma test or how to run that specific file.
How to run functional tests if the config file is a typescript file

The following table outlines possible test file locations and how to invoke them:

Test location	Runner command (working directory is kibana root)
`src/*/.test.js` `src/*/.test.ts`	`yarn test:jest -t regexp [test path]`
`src//__tests__//.js` `packages/kbn-datemath/test//.js` `packages/kbn-dev-utils/src//__tests__//.js` `tasks//__tests__//.js`	`node scripts/mocha --grep=regexp [test path]`
`/integration_tests//*.test.js`	`node scripts/jest_integration -t regexp [test path]`
`test/integration//config.js` `test/functional/*/config.js` `x-pack/test/integration/*/config.js` `x-pack/test/functional/config.js`	`node scripts/functional_tests --config test/[directory]/config.js --grep=regexp`
`x-pack/*/.test.js` `x-pack/*/.test.ts`	`cd x-pack && yarn test:jest -t regexp [test path]`
`x-pack/test/integration/*/config.ts`	??

Test runner arguments:

Where applicable, the optional arguments -t=regexp or --grep=regexp will only run tests or test suites whose descriptions matches the regular expression.
[test path] is the relative path to the test file.

Examples:

Run the entire elasticsearch_service test suite with yarn:
yarn test:jest src/core/server/elasticsearch/elasticsearch_service.test.ts
Run the jest test case whose description matches 'stops both admin and data clients':
yarn test:jest -t 'stops both admin and data clients' src/core/server/elasticsearch/elasticsearch_service.test.ts
Run the x-pack api integration test case whose description matches the given string:
node scripts/functional_tests --config x-pack/test/api_integration/config.js --grep='apis Monitoring Beats list with restarted beat instance should load multiple clusters'

elasticmachine · 2019-04-12T13:25:54Z

Pinging @elastic/kibana-operations

CONTRIBUTING.md

elasticmachine · 2019-04-12T13:56:44Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-04-12T14:42:00Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-04-12T15:22:45Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

cjcenizal · 2019-04-12T15:36:05Z

CONTRIBUTING.md

+| `**/integration_tests/**/*.test.js`                                                                                                                     | `node scripts/jest_integration -t regexp [test path]`                                   |
+| `test/*integration/**/config.js`<br>`test/*functional/**/config.js`<br>`x-pack/test/*integration/**/config.js`<br>`x-pack/test/*functional/config.js`   | `node scripts/functional_tests --config test/[directory]/config.js --grep=regexp`       |
+| `x-pack/**/*.test.js`<br>`x-pack/**/*.test.ts`                                                                                                          | `cd x-pack && yarn test:jest -t regexp [test path]`                                     |
+| `x-pack/test/*integration/**/config.ts`                                                                                                                 | ??                                                                                      |


I think the value for this row can be:

cd x-pack && node scripts/functional_tests_server.js node scripts/functional_test_runner.js --config x-pack/test/api_integration/config.js --grep "Some string from the test case name"

We probably want to keep the X-Pack specific docs here: https://github.com/elastic/kibana/blob/master/x-pack/README.md until we unify all of the tooling.

SGTM. It would be nice to cross-link between those two docs, in that case.

The problem I ran into here was how do I run functional tests if the config file is written as a typescript file as opposed to a javascript file?

cjcenizal · 2019-04-12T15:36:18Z

CONTRIBUTING.md

+
+The following table outlines possible test file locations and how to invoke them:
+
+| Test location                                                                                                                                           | Runner command (working directory is kibana root)                                       |


how about adding a column at the beginning of the table called “Type”? Values for each row could be “Jest”, “Mocha”, “Jest + API”, “Functional”, “Jest (X-Pack”), and “Integration (X-Pack)”

I think having the rows be by type would help simplify this quite a bit. They would then correspond to the locations they cover and the command to execute.

rudolf · 2019-04-16T12:48:16Z

@cjcenizal @tylersmalley I played with grouping tests by "type" and splitting x-pack tests out of this file (just pasting as a comment, didn't push any code yet). It felt weird to group integration and functional tests under the "functional" type, wouldn't it be two distinct types "functional" and "integration"?

Type	Test location	Runner command (working directory is kibana root)
Jest	`src/*/.test.js` `src/*/.test.ts`	`yarn test:jest -t regexp [test path]`
Mocha	`src//__tests__//.js` `packages/kbn-datemath/test//.js` `packages/kbn-dev-utils/src//__tests__//.js` `tasks//__tests__//.js`	`node scripts/mocha --grep=regexp [test path]`
Jest Integration	`/integration_tests//*.test.js`	`node scripts/jest_integration -t regexp [test path]`
Functional	`test/functional/*/config.js`	`node scripts/functional_tests --config test/[directory]/config.js --grep=regexp`
Integration	`test/integration/*/config.js`	`node scripts/functional_tests --config test/[directory]/config.js --grep=regexp`

Type	Test location	Runner command (working directory is kibana root)
X-Pack Jest	`x-pack/*/.test.js` `x-pack/*/.test.ts`	`cd x-pack && yarn test:jest -t regexp [test path]`
X-Pack Integration	`x-pack/test/integration/*/config.js`	`cd x-pack && node scripts/functional_tests --config test/[directory]/config.js --grep=regexp`
X-Pack Functional	`x-pack/test/*functional/config.js`	`cd x-pack && node scripts/functional_tests --config test/[directory]/config.js --grep=regexp`

But then I wondered if it would be simpler to group it under the test runner, in which case both functional and integration tests uses the "functional test runner":

Test runner	Test location	Runner command (working directory is kibana root)
Jest	`src/*/.test.js` `src/*/.test.ts`	`yarn test:jest -t regexp [test path]`
Jest (integration)	`/integration_tests//*.test.js`	`node scripts/jest_integration -t regexp [test path]`
Mocha	`src//__tests__//.js` `packages/kbn-datemath/test//.js` `packages/kbn-dev-utils/src//__tests__//.js` `tasks//__tests__//.js`	`node scripts/mocha --grep=regexp [test path]`
Functional	`test/integration//config.js` `test/functional/**/config.js`	`node scripts/functional_tests --config test/[directory]/config.js --grep=regexp`

Test runner	Test location	Runner command (working directory is kibana root)
Jest	`x-pack/*/.test.js` `x-pack/*/.test.ts`	`cd x-pack && yarn test:jest -t regexp [test path]`
Functional	`x-pack/test/integration//config.js` `x-pack/test/functional/config.js`	`cd x-pack && node scripts/functional_tests --config test/[directory]/config.js --grep=regexp`

elasticmachine · 2019-04-16T13:57:51Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-04-16T19:22:50Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

tylersmalley

Just a couple small changes - thanks for your work on this!

tylersmalley · 2019-04-17T14:56:47Z

CONTRIBUTING.md

+
+| Test runner        | Test location                                                                                                                                           | Runner command (working directory is kibana root)                                       |
+| -----------------  | ------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
+| Jest               | `src/**/*.test.js`<br>`src/**/*.test.ts`                                                                                                                | `yarn test:jest -t regexp [test path]`                                                  |


Can we update this to node scripts/jest

tylersmalley · 2019-04-17T14:59:36Z

CONTRIBUTING.md

+| Jest               | `src/**/*.test.js`<br>`src/**/*.test.ts`                                                                                                                | `yarn test:jest -t regexp [test path]`                                                  |
+| Jest (integration) | `**/integration_tests/**/*.test.js`                                                                                                                     | `node scripts/jest_integration -t regexp [test path]`                                   |
+| Mocha              | `src/**/__tests__/**/*.js`<br>`packages/kbn-datemath/test/**/*.js`<br>`packages/kbn-dev-utils/src/**/__tests__/**/*.js`<br>`tasks/**/__tests__/**/*.js` | `node scripts/mocha --grep=regexp [test path]`                                          |
+| Functional         | `test/*integration/**/config.js`<br>`test/*functional/**/config.js`                                                                                     | `node scripts/functional_tests --config test/[directory]/config.js --grep=regexp`       |


While this will run the tests, I am concerned folks will use this for debugging which takes a large amount of time to start Kibana/ES each time. We have details here regarding how to run for development: https://www.elastic.co/guide/en/kibana/current/development-functional-tests.html#_running_functional_tests

I would suggest we update this command to outline the two:

node scripts/functional_tests_server --config test/[directory]/config.js

node scripts/functional_test_runner --config test/[directory]/config.js --grep=regexp

tylersmalley · 2019-04-17T15:00:48Z

x-pack/README.md

+## Running specific tests
+| Test runner  | Test location                                                                       | Runner command (working directory is kibana/x-pack)                                     |
+| ------------ | ----------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
+| Jest         | `x-pack/**/*.test.js`<br>`x-pack/**/*.test.ts`                                      | `cd x-pack && yarn test:jest -t regexp [test path]`                                     |


Same feedback as for Kibana - use scripts/jest and outline the separate server/runner functional test scripts.

tylersmalley · 2019-04-17T15:02:12Z

CONTRIBUTING.md

+    `yarn test:jest src/core/server/elasticsearch/elasticsearch_service.test.ts`
+  - Run the jest test case whose description matches 'stops both admin and data clients':
+    `yarn test:jest -t 'stops both admin and data clients' src/core/server/elasticsearch/elasticsearch_service.test.ts`
+  - Run the x-pack api integration test case whose description matches the given string:


We can remove this X-Pack example since you included in the X-Pack docs

cjcenizal

💯 This is fantastic! Thanks so much for doing this.

elasticmachine · 2019-04-18T21:00:21Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

* docs: Add test file location -> test runner table * Use scripts/mocha instead of if yarn test:mocha and clarify arguments * docs: add some examples of running tests suites with grep patterns * Split out x-pack testing docs, add test runner column * Split out x-pack testing docs, add test runner column #2 * Use node scripts for jest and split functional runner/server

docs: Add test file location -> test runner table

3b9ad4c

rudolf requested a review from a team April 12, 2019 13:25

rudolf added Team:Operations Team label for Operations Team review v7.2.0 v8.0.0 labels Apr 12, 2019

mshustov reviewed Apr 12, 2019

View reviewed changes

CONTRIBUTING.md Outdated Show resolved Hide resolved

mshustov reviewed Apr 12, 2019

View reviewed changes

CONTRIBUTING.md Outdated Show resolved Hide resolved

Use scripts/mocha instead of if yarn test:mocha and clarify arguments

0b17e3e

docs: add some examples of running tests suites with grep patterns

5cdcdf4

jbudz approved these changes Apr 12, 2019

View reviewed changes

mshustov approved these changes Apr 12, 2019

View reviewed changes

cjcenizal reviewed Apr 12, 2019

View reviewed changes

rudolf mentioned this pull request Apr 15, 2019

Development Experience #34557

Closed

Split out x-pack testing docs, add test runner column

4b6074c

rudolf requested review from tylersmalley and cjcenizal and removed request for tylersmalley April 16, 2019 13:36

Split out x-pack testing docs, add test runner column #2

bf89d8d

tylersmalley suggested changes Apr 17, 2019

View reviewed changes

Use node scripts for jest and split functional runner/server

82c54af

rudolf requested a review from tylersmalley April 18, 2019 20:16

cjcenizal approved these changes Apr 18, 2019

View reviewed changes

tylersmalley approved these changes Apr 18, 2019

View reviewed changes

rudolf merged commit bb121ef into elastic:master Apr 19, 2019

rudolf mentioned this pull request Apr 19, 2019

[7.x] docs: Add test file location -> test runner table (#34986) #35345

Merged

rudolf deleted the test-runner-docs branch April 19, 2019 07:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: Add test file location -> test runner table #34986

docs: Add test file location -> test runner table #34986

rudolf commented Apr 12, 2019 •

edited by cjcenizal

elasticmachine commented Apr 12, 2019

elasticmachine commented Apr 12, 2019

elasticmachine commented Apr 12, 2019

elasticmachine commented Apr 12, 2019

cjcenizal Apr 12, 2019

tylersmalley Apr 15, 2019 •

edited

cjcenizal Apr 15, 2019

rudolf Apr 16, 2019

cjcenizal Apr 12, 2019

tylersmalley Apr 15, 2019

rudolf commented Apr 16, 2019

elasticmachine commented Apr 16, 2019

elasticmachine commented Apr 16, 2019

tylersmalley left a comment

tylersmalley Apr 17, 2019

tylersmalley Apr 17, 2019

tylersmalley Apr 17, 2019

tylersmalley Apr 17, 2019

cjcenizal left a comment

elasticmachine commented Apr 18, 2019


		The following table outlines possible test file locations and how to invoke them:

		\| Test location \| Runner command (working directory is kibana root) \|

docs: Add test file location -> test runner table #34986

docs: Add test file location -> test runner table #34986

Conversation

rudolf commented Apr 12, 2019 • edited by cjcenizal

Summary

elasticmachine commented Apr 12, 2019

elasticmachine commented Apr 12, 2019

💚 Build Succeeded

elasticmachine commented Apr 12, 2019

💚 Build Succeeded

elasticmachine commented Apr 12, 2019

💚 Build Succeeded

Choose a reason for hiding this comment

tylersmalley Apr 15, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rudolf commented Apr 16, 2019

elasticmachine commented Apr 16, 2019

💚 Build Succeeded

elasticmachine commented Apr 16, 2019

💚 Build Succeeded

tylersmalley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjcenizal left a comment

Choose a reason for hiding this comment

elasticmachine commented Apr 18, 2019

💚 Build Succeeded

rudolf commented Apr 12, 2019 •

edited by cjcenizal

tylersmalley Apr 15, 2019 •

edited