[functionalTestRunner] replace intern #10910

spalger · 2017-03-28T05:38:30Z

In an effort to make functional/selenium testing more accessible to plugins and external projects, this pr removes intern and replaces it with a custom solution.

Quick info

src/functional_test_runner: source for the bare bones functional test runner. Most functionality via services in config
tests/functional/config.js: Kibana's functional_test_runner config file, plugins will have one just like this
- config file defines
  - which test files to load
  - providers for the the Services and PageObjects
  - config values used by some services
services:
- default:
  - config: use config.get() to read values from the config file)
  - log: use log.debug() and log.error() just like console.log but with more organized output
  - lifecycle: designed for use in other services, calls handlers at certain lifecycle points
    - lifecycle.on(phaseName, handler): register a listener for a specific phase that will be called at that point. The handler can be async and return a promise if it needs to do async work.
    - phases: beforeLoadTests, beforeTests, beforeEachTest, cleanup, phaseStart, phaseEnd
- kibana (see tests/functional/services):
  - remote: leadfoot remote that provides access to automated browser (previously injected into each PageObject)
  - esArchiver: load/unload archives stored with the esArchiver ([esArchiver] combine elasticDump and ScenarioManager #10359)
  - es: elasticsearch client
  - find: helpers for remote findBy* methods that log and manage timeouts (previously PageObjects.common.findBy*)
  - retry: helpers for retrying logic (previously PageObjects.common.try*)
  - testSubjects: helpers for reading/interacting with test subjects (previously PageObjects.common.find/click/findAllTestSubect)
  - docTable: helpers for interacting with doc tables
  - pointSeriesVis: helpers for interacting with point series visualizations

Running the CLI

This is the second script to be added to the root scripts/ directory. Access it with node scripts/functional_test_runner and give it the --help flag to see what it can do.

Debugging Tests

I suggest installing the NIM Chrome Extension, turning on the "auto" setting, and then using the new node.js --inspect flag.

node --debug-brk --inspect scripts/functional_test_runner

Upstream prs:

rhoboat · 2017-04-04T00:22:32Z

crosslink to asciidocs from CONTRIBUTING.md
write functional-testing docs
make sure all the links in the doc are pointing to master
reinstate unit testing before merging
set default loglevel to debug or info, decide
- decided that in jenkins, the default will be debug and the cli tool will run in info

rhoboat · 2017-04-04T01:43:01Z

rhoboat · 2017-04-06T01:39:30Z

23 Tests Failing, all in Management app

rhoboat · 2017-04-06T14:54:09Z

jenkins, test it

rhoboat · 2017-04-06T19:43:59Z

4 Tests Failing, management app

rhoboat · 2017-04-06T21:12:14Z

1 Test failing, management https://kibana-ci.elastic.co/job/elastic+kibana+pull-request+multijob-selenium/3484/consoleFull

rhoboat · 2017-04-06T21:28:50Z

@stacey-gammon I tagged you because I incorporated your changes in here, and specifically for the dashboard tests. It's a big PR, so if you wanted to just look at those, I'd be grateful.

@LeeDr Would like your feedback on the way the tests look and how the functional-test-runner works. Since you're traveling, if you aren't able to take a close look just yet, it's no big deal, I'll tag others, too!

For everyone, I/we will be writing documentation that will help this go live. I can ping you when documentation is ready.

rhoboat · 2017-04-06T21:35:18Z

@spalger Would like to add you as reviewer. I'm not sure if I'm including files and directories that should no longer exist. Can you please check?

…op-level suite

kimjoar · 2017-04-11T22:03:55Z

🎉

epixa · 2017-04-11T22:12:03Z

Awesome! Great job on this @spalger and @archanid.

Let's backport it into 5.x as well.

@gammon

* [functional_test_runner] replace functional testing tools with custom/pluggable solution * [functional_test_runner] Convert unit tests to commonjs format * [functional_test_runner] Fix dashboard test in wrong mode * [functional_test_runner] Add dashboardLandingPage test subject * [functional_test_runner] Get Visualize page object * [functional_test_runner] Fix outdated references * [functional_test_runner] Fix more outdated refs * [functional_test_runner] Remove duplicate tests * [functional_test_runner] Improve test readability * [functional_test_runner] 😞 So many duplicate methods * [functional_test_runner] Move mgmt `before` outside toplevel describe * [functional_test_runner] Settings page obj missing methods * [functional_test_runner] Add improvements from @gammon * [functional_test_runner] Fix return statements in async funcs * [functional_test_runner] Move before() to correct scope * [functional_test_runner] Add after() hooks to remove index patterns * [functional_test_runner] Attempt to fix vertical bar chart tests * [functional_test_runner] Clean up * [functional_test_runner] Reinstate unit tests * [functional_test_runner] Set default loglevel back to info * [functional_test_runner] Replace `context`s with `describe`s * [functional_test_runner] Better error handling * [functional_test_runner] Add in new Tile Map tests * Incorporate changes from master * [functional_test_runner] validate that every test file has a single top-level suite * Update contributing doc with link to full doc * [docs] Spelling and grammar fixes * docs: writing and running functional tests * [docs] Move plugin doc to plugin area * [docs] Housekeeping. Doc in wrong place * [docs] Remove dup doc file * [grunt] Only run mocha_setup when running tests, not every grunt task

@gammon

* [functional_test_runner] replace functional testing tools with custom/pluggable solution * [functional_test_runner] Convert unit tests to commonjs format * [functional_test_runner] Fix dashboard test in wrong mode * [functional_test_runner] Add dashboardLandingPage test subject * [functional_test_runner] Get Visualize page object * [functional_test_runner] Fix outdated references * [functional_test_runner] Fix more outdated refs * [functional_test_runner] Remove duplicate tests * [functional_test_runner] Improve test readability * [functional_test_runner] 😞 So many duplicate methods * [functional_test_runner] Move mgmt `before` outside toplevel describe * [functional_test_runner] Settings page obj missing methods * [functional_test_runner] Add improvements from @gammon * [functional_test_runner] Fix return statements in async funcs * [functional_test_runner] Move before() to correct scope * [functional_test_runner] Add after() hooks to remove index patterns * [functional_test_runner] Attempt to fix vertical bar chart tests * [functional_test_runner] Clean up * [functional_test_runner] Reinstate unit tests * [functional_test_runner] Set default loglevel back to info * [functional_test_runner] Replace `context`s with `describe`s * [functional_test_runner] Better error handling * [functional_test_runner] Add in new Tile Map tests * Incorporate changes from master * [functional_test_runner] validate that every test file has a single top-level suite * Update contributing doc with link to full doc * [docs] Spelling and grammar fixes * docs: writing and running functional tests * [docs] Move plugin doc to plugin area * [docs] Housekeeping. Doc in wrong place * [docs] Remove dup doc file * [grunt] Only run mocha_setup when running tests, not every grunt task

@gammon

* [esArchiver] combine elasticDump and ScenarioManager (#10359) * As a part of bringing functional testing to plugins, esArchiver gives these plugins a way to capture and reload es indexes without needing to write a bunch of custom code. It works similarly to the elasticDump and ScenarioManager tools that it replaces. Differences: - Streaming implementation allows for much larger archives - CLI for creating and using archives - Configurable archive location - Stores the data in gzipped files (better for source control, searching, large archives) - Automatically identifies and upgrades Kibana config documents Methods: - `#load(name)`: import an archive - `#loadIfNeeded(name)`: import an archive, but skip the documents what belong to any existing index - `#unload(name)`: delete the indexes stored in an archive CLI operations: - `./bin/es_archiver save <name> [index patterns...]`: save the mapping and documents in one or more indexes that match the wild-card patterns into an the `<name>` archive - `./bin/es_archiver load <name>`: load the mapping and documents from the `<name>` archive * [functional_tests/common/nagivate] check for statusPage * [es_archiver] move bins into new scripts dir * [functional_tests/apps/context] use esArchiver * [esArchiver] general improvements after showing to a few folks - remove auto-upgrading config doc logic (until we have better access to kibana version info) - export unload command - remove preemptive checks in favor of reacting to errors - use type "doc" vs "hit" for doc records (consistency) - wrote a bunch of pending tests to think though and plan * [esArchiver] make log a stream that writes to itself * [esArchiver] fill in stats and archive format tests * [esArchiver] splitup action logic * [esArchiver/cli] fix cli --help output and comment * [esArchiver] remove type-based param coercion * [esArchiver/log] use strings for log levels * [esArchvier] remove unused var * [esArchiver/indexDocRecordsStream] add tests * [esArchive] fill in remaining tests * [esArchiver] fix dem tests * [eslint] remove unused vars * [esArchiver/loadIfNeeded] fix call to load() * [esArchiver] remove loadDumpData helpers * [esArchiver] update archives for 5.x * [functionalTestRunner] replace intern (#10910) * [functional_test_runner] replace functional testing tools with custom/pluggable solution * [functional_test_runner] Convert unit tests to commonjs format * [functional_test_runner] Fix dashboard test in wrong mode * [functional_test_runner] Add dashboardLandingPage test subject * [functional_test_runner] Get Visualize page object * [functional_test_runner] Fix outdated references * [functional_test_runner] Fix more outdated refs * [functional_test_runner] Remove duplicate tests * [functional_test_runner] Improve test readability * [functional_test_runner] 😞 So many duplicate methods * [functional_test_runner] Move mgmt `before` outside toplevel describe * [functional_test_runner] Settings page obj missing methods * [functional_test_runner] Add improvements from @gammon * [functional_test_runner] Fix return statements in async funcs * [functional_test_runner] Move before() to correct scope * [functional_test_runner] Add after() hooks to remove index patterns * [functional_test_runner] Attempt to fix vertical bar chart tests * [functional_test_runner] Clean up * [functional_test_runner] Reinstate unit tests * [functional_test_runner] Set default loglevel back to info * [functional_test_runner] Replace `context`s with `describe`s * [functional_test_runner] Better error handling * [functional_test_runner] Add in new Tile Map tests * Incorporate changes from master * [functional_test_runner] validate that every test file has a single top-level suite * Update contributing doc with link to full doc * [docs] Spelling and grammar fixes * docs: writing and running functional tests * [docs] Move plugin doc to plugin area * [docs] Housekeeping. Doc in wrong place * [docs] Remove dup doc file * [grunt] Only run mocha_setup when running tests, not every grunt task * [eslint] remove use of context()

This reverts commit 72f716b.

spalger assigned solostyle Mar 28, 2017

spalger mentioned this pull request Mar 28, 2017

WIP: [functional-tests] replace intern with custom test runner #9853

Closed

spalger added the Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc label Mar 28, 2017

spalger force-pushed the implement/functional-test-runner branch 3 times, most recently from b60c5a8 to d3be827 Compare March 29, 2017 17:32

rhoboat force-pushed the implement/functional-test-runner branch 2 times, most recently from ee68177 to 89800f1 Compare March 29, 2017 19:43

rhoboat assigned rhoboat and unassigned solostyle Mar 29, 2017

rhoboat force-pushed the implement/functional-test-runner branch 4 times, most recently from d90fbe4 to 5b8ea08 Compare April 3, 2017 21:35

rhoboat force-pushed the implement/functional-test-runner branch 2 times, most recently from c1415a6 to 25e4f9c Compare April 5, 2017 17:51

rhoboat mentioned this pull request Apr 5, 2017

Stabilize dashboard selenium tests #11004

Closed

rhoboat force-pushed the implement/functional-test-runner branch 2 times, most recently from 5bee23d to 73326bf Compare April 6, 2017 00:15

rhoboat force-pushed the implement/functional-test-runner branch from 4cf226f to 5fd1d3f Compare April 6, 2017 21:12

rhoboat requested review from LeeDr and stacey-gammon April 6, 2017 21:23

archanid and others added 16 commits April 11, 2017 16:08

[functional_test_runner] Attempt to fix vertical bar chart tests

cccfbfb

[functional_test_runner] Clean up

38dee88

[functional_test_runner] Reinstate unit tests

8d15b1a

[functional_test_runner] Set default loglevel back to info

ea0175b

[functional_test_runner] Replace contexts with describes

4a49e6e

[functional_test_runner] Better error handling

c126eb8

[functional_test_runner] Add in new Tile Map tests

4e63aef

Incorporate changes from master

c57f64c

docs: writing and running functional tests

ceb9a75

[functional_test_runner] validate that every test file has a single t…

0209314

…op-level suite

Update contributing doc with link to full doc

32cde4b

[docs] Spelling and grammar fixes

7b892ee

[docs] Move plugin doc to plugin area

c68dc2b

[docs] Housekeeping. Doc in wrong place

d21e85e

[docs] Remove dup doc file

f412485

[grunt] Only run mocha_setup when running tests, not every grunt task

d364014

rhoboat force-pushed the implement/functional-test-runner branch from 31e3438 to d364014 Compare April 11, 2017 21:14

rhoboat merged commit 9043476 into elastic:master Apr 11, 2017

epixa added test v5.4.0 labels Apr 11, 2017

spalger mentioned this pull request Apr 11, 2017

[5.x] [functionalTestRunner] replace intern (#10910) #11171

Merged

spalger added a commit that referenced this pull request Apr 12, 2017

Revert "[5.x] [functionalTestRunner] replace intern (#10910) (#11171)"

c3535a3

This reverts commit 72f716b.

kobelb mentioned this pull request Apr 12, 2017

Region map visualization #10937

Merged

25 tasks

jbudz mentioned this pull request Apr 17, 2017

Update chromedriver to 2.28 for Chrome 57 #11093

Closed

spalger deleted the implement/functional-test-runner branch August 18, 2020 17:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[functionalTestRunner] replace intern #10910

[functionalTestRunner] replace intern #10910

spalger commented Mar 28, 2017 •

edited by rhoboat

rhoboat commented Apr 4, 2017 •

edited

rhoboat commented Apr 4, 2017 •

edited

rhoboat commented Apr 6, 2017

rhoboat commented Apr 6, 2017

rhoboat commented Apr 6, 2017

rhoboat commented Apr 6, 2017 •

edited

rhoboat commented Apr 6, 2017

rhoboat commented Apr 6, 2017

kimjoar commented Apr 11, 2017

epixa commented Apr 11, 2017

[functionalTestRunner] replace intern #10910

[functionalTestRunner] replace intern #10910

Conversation

spalger commented Mar 28, 2017 • edited by rhoboat

rhoboat commented Apr 4, 2017 • edited

rhoboat commented Apr 4, 2017 • edited

rhoboat commented Apr 6, 2017

rhoboat commented Apr 6, 2017

rhoboat commented Apr 6, 2017

rhoboat commented Apr 6, 2017 • edited

rhoboat commented Apr 6, 2017

rhoboat commented Apr 6, 2017

kimjoar commented Apr 11, 2017

epixa commented Apr 11, 2017

spalger commented Mar 28, 2017 •

edited by rhoboat

rhoboat commented Apr 4, 2017 •

edited

rhoboat commented Apr 4, 2017 •

edited

rhoboat commented Apr 6, 2017 •

edited