Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable end-to-end test coverage #136

merged 4 commits into from Aug 14, 2017


Copy link

This is all the works necessary to support this. See the commit logs for more details but it boils down to:

  1. Instrument the application code (main.js, pages, etc) with babel-plugin-istanbul. Only do this if --with-coverage has been passed on the command line otherwise we mess up unit tests (more precisely coverage from them)

  2. In e2e tests make a slight modification how we chain and terminate Nightmare calls. This is necessary b/c we have to bring that coverage information from the browser scope (the page under test) to nodejs scope (the code doing the testing) and finally close the electron instance running the application.

WARNING: the above requires a bit of change in the way tests are written, in particular don't call .end() and done() but instead call the coverage handling code inside the last .then() method after all assertions have been executed. This is described in README but fyi <--- @jgiardino @henrywang @jkozol

NOTE I do use a soft of ugly and hackish eval() to execute my coverage helper. I've tried splitting this out into a proper module and instead calling a function from the module but for some reason it doesn't work. I've asked around several experienced nodejs developers and they couldn't figure it out. It works for now and is only a one line, which is easy to copy&paste. If necessary we can change it in the future, provided somebody figures out how. Honestly I've spent too much time on this just trying to figure out why it doesn't work when the code is split into a helper function instead of copying the same code into the last .then() method body.

  1. e2e coverage reports are stored under .nyc_output on the host and the nyc tool can be used to generate the reports.

  2. Finally I've switched from Coveralls to CodeCov in order to be able to unify the two coverage reports. This is the easiest way given that Travis executes everything in a separate environment and doesn't allow sharing of artifacts between them

You can see an example report against my fork at:

this plugin instruments the application code and provides the
window.__coverage__ object inside the browser!
Copy link
Contributor Author

Copy link

codecov-io commented Aug 11, 2017

Codecov Report

❗ No coverage uploaded for pull request base (master@eea63c3). Click here to learn what that means.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff            @@
##             master     #136   +/-   ##
  Coverage          ?   35.08%           
  Files             ?       69           
  Lines             ?     2018           
  Branches          ?      279           
  Hits              ?      708           
  Misses            ?     1105           
  Partials          ?      205

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eea63c3...11ba14b. Read the comment docs.

.travis.yml Outdated
@@ -10,9 +10,13 @@ cache:
- node_modules # NPM packages
- |
npm install codecov
npm install nyc --safe-dev
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

npm install nyc --save-dev or just npm install nyc. No difference here.

Copy link

💯 👍

1. Some cases may have identical coverage so the number of json files will be
equal or less to the number of test cases;
2. When constructing the Nightmare sequence makde sure you **DO NOT** call the
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make sure.

after the application code has been instrumented we use
nightmare.evaluate() to fetch the data from browser scope back to
node scope and save it in a json file under /tmp. This file is
later used for reports.

Each nightmare sequence must not call .end() and done() because
this is done in coverage.js as the last step of the test!

NOTE: we're using eval() on the contents of utils/coverage.js
because all my attempts to successfully execute this piece of code
via a helper module failed! eval() in this scope is equivalent to
inserting the contents of utils/coverage.js into the test!
also mount .nyc_output/ under /tmp inside the running container.
This will ensure that coverage reports are saved directly onto the
host and we can pick them up from .travis.yml

Also add --verbose to test runner so we can see what tests have
been executed in the CI logs.
we're switching from Coveralls because they don't support merging
of coverage results submitted via different build jobs. CodeCov
supports this plus it seems they have a better interface.
@atodorov atodorov merged commit 9c024c4 into osbuild:master Aug 14, 2017
@atodorov atodorov deleted the e2e_coverage branch August 14, 2017 08:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet

Successfully merging this pull request may close these issues.

None yet

3 participants