New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable end-to-end test coverage #136

Merged
merged 4 commits into from Aug 14, 2017

Conversation

Projects
None yet
3 participants
@atodorov
Member

atodorov commented Aug 11, 2017

This is all the works necessary to support this. See the commit logs for more details but it boils down to:

  1. Instrument the application code (main.js, pages, etc) with babel-plugin-istanbul. Only do this if --with-coverage has been passed on the command line otherwise we mess up unit tests (more precisely coverage from them)

  2. In e2e tests make a slight modification how we chain and terminate Nightmare calls. This is necessary b/c we have to bring that coverage information from the browser scope (the page under test) to nodejs scope (the code doing the testing) and finally close the electron instance running the application.

WARNING: the above requires a bit of change in the way tests are written, in particular don't call .end() and done() but instead call the coverage handling code inside the last .then() method after all assertions have been executed. This is described in README but fyi <--- @jgiardino @henrywang @jkozol

NOTE I do use a soft of ugly and hackish eval() to execute my coverage helper. I've tried splitting this out into a proper module and instead calling a function from the module but for some reason it doesn't work. I've asked around several experienced nodejs developers and they couldn't figure it out. It works for now and is only a one line, which is easy to copy&paste. If necessary we can change it in the future, provided somebody figures out how. Honestly I've spent too much time on this just trying to figure out why it doesn't work when the code is split into a helper function instead of copying the same code into the last .then() method body.

  1. e2e coverage reports are stored under .nyc_output on the host and the nyc tool can be used to generate the reports.

  2. Finally I've switched from Coveralls to CodeCov in order to be able to unify the two coverage reports. This is the easiest way given that Travis executes everything in a separate environment and doesn't allow sharing of artifacts between them

You can see an example report against my fork at:
https://codecov.io/gh/atodorov/welder-web/commits

Enable babel-plugin-istanbul if --with-coverage is specified
this plugin instruments the application code and provides the
window.__coverage__ object inside the browser!

@atodorov atodorov requested review from henrywang and jgiardino Aug 11, 2017

@atodorov

This comment has been minimized.

Show comment
Hide comment
Member

atodorov commented Aug 11, 2017

@codecov-io

This comment has been minimized.

Show comment
Hide comment
@codecov-io

codecov-io Aug 11, 2017

Codecov Report

❗️ No coverage uploaded for pull request base (master@eea63c3). Click here to learn what that means.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff            @@
##             master     #136   +/-   ##
=========================================
  Coverage          ?   35.08%           
=========================================
  Files             ?       69           
  Lines             ?     2018           
  Branches          ?      279           
=========================================
  Hits              ?      708           
  Misses            ?     1105           
  Partials          ?      205

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eea63c3...11ba14b. Read the comment docs.

codecov-io commented Aug 11, 2017

Codecov Report

❗️ No coverage uploaded for pull request base (master@eea63c3). Click here to learn what that means.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff            @@
##             master     #136   +/-   ##
=========================================
  Coverage          ?   35.08%           
=========================================
  Files             ?       69           
  Lines             ?     2018           
  Branches          ?      279           
=========================================
  Hits              ?      708           
  Misses            ?     1105           
  Partials          ?      205

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eea63c3...11ba14b. Read the comment docs.

Show outdated Hide outdated .travis.yml
@henrywang

This comment has been minimized.

Show comment
Hide comment
@henrywang

henrywang Aug 14, 2017

Contributor

💯 👍

Contributor

henrywang commented Aug 14, 2017

💯 👍

atodorov added some commits Aug 9, 2017

Collect coverage from all end-to-end tests and save it under /tmp
after the application code has been instrumented we use
nightmare.evaluate() to fetch the data from browser scope back to
node scope and save it in a json file under /tmp. This file is
later used for reports.

Each nightmare sequence must not call .end() and done() because
this is done in coverage.js as the last step of the test!

NOTE: we're using eval() on the contents of utils/coverage.js
because all my attempts to successfully execute this piece of code
via a helper module failed! eval() in this scope is equivalent to
inserting the contents of utils/coverage.js into the test!
Build a Docker container with coverage for e2e testing
also mount .nyc_output/ under /tmp inside the running container.
This will ensure that coverage reports are saved directly onto the
host and we can pick them up from .travis.yml

Also add --verbose to test runner so we can see what tests have
been executed in the CI logs.
Report unit-test and end-to-end coverage results to CodeCov
we're switching from Coveralls because they don't support merging
of coverage results submitted via different build jobs. CodeCov
supports this plus it seems they have a better interface.

@atodorov atodorov merged commit 9c024c4 into weldr:master Aug 14, 2017

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@atodorov atodorov deleted the atodorov:e2e_coverage branch Aug 14, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment