Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

code coverage with istanbul, mocha, and Selenium Webdriver running on Node.JS (WebDriverJS) #132

Open
attodorov opened this issue Dec 28, 2013 · 26 comments

Comments

@attodorov
Copy link

commented Dec 28, 2013

Hey,
I have the following scenario where I would like to collect code coverage and I would be glad to get some feedback about potential solutions using istanbul:

  • I have a web app - HTML, CSS * Javascript - that needs to be tested
  • it is mounted with ExpressJS and runs on some port, for instance 8080. There isn't any server-side logic, it's just js/css/html
  • the tests are written in mocha and use WebDriverJS to bring up the index.html, do whatever interactions the test needs to do (such as drag & drop , fill a form, etc.)

I can instrument my javascript resources with istanbul, then load the app using the instrumented scripts, but I am missing the link between the coverage collection and the reporting using my setup. when I look at the window object at runtime, I can see the code coverage metadata and its maps, but how do you think should I tackle collecting and reporting this data? Is there anything available or should I write my own lib to do that?

I cannot use istanbul's integration with mocha, because my app is actually loaded by WebDriver and lives in its own context there (including the window object and everything else). I actually have requirejs modules - my app is modularized, and I am testing as much as I can using pure unit tests with node's requirejs, as well as jsdom, but for a big part of the app that's not possible because it's highly interactive, so my closest bet is WebDriver.

On a separate note, I don't want to use tools like PhantomJS because it looks like they're quite unstable and I have no control on the browser. For example I get different pass rate when I run on Phantom vs real Chrome. I also can't run it on anything else than whatever phantom uses internally, I can't do FF, IEs, and so on.

I've also tried using other code coverage libs like Blanket, but it has the same issues with respect to my scenario. Istanbul has proven to be a lot easier to use and I use it all over in my other tests that don't rely on selenium's web driver.

Thanks
Angel

@swatiagarwal1284

This comment has been minimized.

Copy link

commented Jan 2, 2014

I also have the similar requirement and looking for the solution. Please update this post if you find any solution or workaround.

Thanks,
Swati

@gotwarlost

This comment has been minimized.

Copy link
Owner

commented Jan 2, 2014

Have you taken a look at istanbul-middleware for ideas on the general approach?

https://github.com/gotwarlost/istanbul-middleware/

It won't be directly usable since your app is not a node app but here is one set up that I can see working.

  1. Run your app on port, say 8000
  2. Run a basic express app with istanbul-middleware on port 8001
  3. Have a proxy on port 80 that forwards the /coverage path to the express app and all others to your main app
  4. At the end of your webdriver tests (for each page, before navigation) post the coverage object to /coverage
  5. After all your tests have run, download the coverage reports
@attodorov

This comment has been minimized.

Copy link
Author

commented Jan 6, 2014

Thanks, i got it working in the following way. (Btw, i am not sure why I need step 3. from your list).

  1. Create a simple node express app, this is the whole app:
var coverage = require('istanbul-middleware');
var express = require('express');
var app = express();

 app.use('/coverage', coverage.createHandler());

 app.listen(8888);
  1. Add the following logic to every test suite which uses selenium-webdriver:
after(function (done) {
    // post coverage info
    driver.switchTo().defaultContent();
    driver.executeScript("return window.__coverage__;").then(function (obj) {
        var str = JSON.stringify(obj);
        var options = {
            port: 8888,
            host: "localhost",
            path: "/coverage/client",
            method: "POST",
            headers: {
                "Content-Type": "application/json",
            }
        };
        var req = http.request(options, function (res) {
            console.log("\nFinished sending coverage data.");
            done();
        });
        req.write(str);
        req.end();
    });
})
test.after(function () {
    // at the end of the *whole* test run, download all coverage data and place it somewhere
    console.log("Closing browser.");
    driver.quit();
})

The only assumption that exists is that the js code loaded by the app under test needs to be manually instrumented with istanbul. In order to see report details, i am using the following command:

istanbul instrument src.original.js --output src.js --embed-source true

If i don't give it the "--embed-source true" option, I can't see detailed reports, just the summary. Makes sense, I guess.

Thanks,
Angel

@mahuntington

This comment has been minimized.

Copy link

commented Jun 20, 2014

Hi @attodorov and @gotwarlost I got most what you suggested working, but I'm unclear how to get the reports. In the example @attodorov gives just above, I console.log(res) within the http.request callback, and it prints out JSON. As far as I can tell this isn't the coverage report, though. Any suggestions?

Thanks,

Matt

@mekdev

This comment has been minimized.

Copy link

commented Sep 4, 2015

Guys this is great!

I have been looking all over and this is the only relevant info I have found for doing istanbul code coverage using selenium-webdriver to drive the UI.

I have some questions.

  1. @attodorov can you please briefly explain what you guys have to do for the step below ?
    // at the end of the whole test run, download all coverage data and place it somewhere
  2. Did you guys experience any skew in the report numbers, does the coverage report include the libraries that the app is using e.g. stuff in node_modules ? Did you have to so some source mapping to get this to work with accurate numbers ?
  3. @mahuntington were you guys able to get this to work after following the example ?

Thank you!

  • Mek
@mahuntington

This comment has been minimized.

Copy link

commented Sep 4, 2015

hey @mekdev! I was able to get it running. To be honest, it was a while ago, so I don't remember what I did specifically. Take a look at https://github.com/mahuntington/e2e-istanbul, though. See if that solves any of your issues.

@kiraLinden kiraLinden referenced this issue Jan 4, 2016
@dailystore

This comment has been minimized.

Copy link

commented May 14, 2017

That's great! I'm looking around and this thread seems to be relevant to my problem also. @mahuntington can you kindly elaborate how to get the instrumented code run please?

Thank you!

  • An
@mahuntington

This comment has been minimized.

Copy link

commented May 21, 2017

hey @dailystore, I've updated my repo (https://github.com/mahuntington/e2e-istanbul) with a README. Let me know if you have any other questions!

P.S. I've also updated the code a bit to work with the latest browser drivers

@dailystore

This comment has been minimized.

Copy link

commented May 23, 2017

@mahuntington Thank you! I'm able to get the coverage data against my own node server as well but the coverage report is not working. In the report detail, instead of showing lines of code with a nice format, it shows the instrumented code with no format so there is no way to know which line is covered or not. Are you aware of this issue?

@mahuntington

This comment has been minimized.

Copy link

commented May 23, 2017

Hmm, it worked for me. Can you put it up on github and paste a link?

@mahuntington

This comment has been minimized.

Copy link

commented May 23, 2017

Actually, @dailystore, I don't want to bug anyone else who's following this issue if there's a problem with my repo. Can you create an issue on my repo and we'll figure it out there?

@ORESoftware

This comment has been minimized.

Copy link

commented Jun 1, 2017

this was a GREAT thread...and I got it to work!

I wrote up an article on how to do this:

https://medium.com/@the1mills/front-end-javascript-test-coverage-with-istanbul-selenium-4b2be44e3e98

@attodorov I gave you credit for my findings :) thanks dogg

@kagarwal29

This comment has been minimized.

Copy link

commented Jun 14, 2017

Hello Guys,

I am taking help of this thread to get JS coverage report.

I am using code mentioned by @attodorov to setup server & posting coverage report from broser to server.But when I try to execute this line : 'driver.executeScript("return window.coverage;")', I get following error :

ReferenceError: window is not defined
    at /Users/kapaga/Desktop/Node/webdriver.js:23:19
    at ManagedPromise.invokeCallback_ (/Users/kapaga/Desktop/Node/node_modules/selenium-webdriver/lib/promise.js:1384:14)
    at TaskQueue.execute_ (/Users/kapaga/Desktop/Node/node_modules/selenium-webdriver/lib/promise.js:3092:14)
    at TaskQueue.executeNext_ (/Users/kapaga/Desktop/Node/node_modules/selenium-webdriver/lib/promise.js:3075:27)
    at asyncRun (/Users/kapaga/Desktop/Node/node_modules/selenium-webdriver/lib/promise.js:2935:27)
    at /Users/kapaga/Desktop/Node/node_modules/selenium-webdriver/lib/promise.js:676:7
    at process._tickCallback (internal/process/next_tick.js:109:7)

It seems Node doesn't support 'window' instead there is something called 'global' for Node.But when I try to use 'global', I get following error :

WebDriverError: unknown error: global is not defined
  (Session info: chrome=59.0.3071.86)
  (Driver info: chromedriver=2.28.455517 (2c6d2707d8ea850c862f04ac066724273981e88f),platform=Mac OS X 10.12.4 x86_64)
    at WebDriverError (/Users/kapaga/Desktop/Node/node_modules/selenium-webdriver/lib/error.js:27:5)
    at Object.checkLegacyResponse (/Users/kapaga/Desktop/Node/node_modules/selenium-webdriver/lib/error.js:517:15)
    at parseHttpResponse (/Users/kapaga/Desktop/Node/node_modules/selenium-webdriver/lib/http.js:509:13)
    at doSend.then.response (/Users/kapaga/Desktop/Node/node_modules/selenium-webdriver/lib/http.js:441:30)
    at process._tickCallback (internal/process/next_tick.js:109:7)
From: Task: WebDriver.executeScript()
    at thenableWebDriverProxy.schedule (/Users/kapaga/Desktop/Node/node_modules/selenium-webdriver/lib/webdriver.js:816:17)
    at thenableWebDriverProxy.executeScript (/Users/kapaga/Desktop/Node/node_modules/selenium-webdriver/lib/webdriver.js:887:16)
    at Object.<anonymous> (/Users/kapaga/Desktop/Node/webdriver.js:20:13)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.runMain (module.js:604:10)
    at run (bootstrap_node.js:393:7)

Can you please help me out here? I am new to Javascriot world ( & to be precise to the Coding world, as I am a QA) so it's highely likely that I might be missing out something very silly here.

Thanks

@mahuntington

This comment has been minimized.

Copy link

commented Jun 14, 2017

hey @kagarwal29. Node doesn't have a window, as far as I know, since it's purely server-side. Sounds like you're trying to run your tests in node, but in reality what you want to do is run browser tests. I have a walkthrough of something similar here

@kagarwal29

This comment has been minimized.

Copy link

commented Jun 19, 2017

Thanks @mahuntington for pointing me to you project. I imported the project & it works on my local. :) Your code is proving to be of great help to me.

@mahuntington

This comment has been minimized.

Copy link

commented Jun 19, 2017

great! @kagarwal29

@kagarwal29

This comment has been minimized.

Copy link

commented Jun 19, 2017

@mahuntington Can you please look into my query in following post ? Your response would be very helpful.

gotwarlost/istanbul-middleware#45 (comment)

@rsoman

This comment has been minimized.

Copy link

commented Jun 22, 2017

@attodorov , @mahuntington , @gotwarlost
Great thread indeed!
I got it working to the point of sending data successfully to node express server.
However, when trying to view or download coverage report, I am getting RangeError in JSON.stringify().
Error stack is as follows:

RangeError: Maximum call stack size exceeded
    at Object.stringify (native)
    at MemoryStore.Store.setObject (D:\HTML5 Unit Testing\Functional Coverage\node_modules\istanbul-middleware\node_modules\istanbul\lib\store\index.js:116:35)
    at D:\HTML5 Unit Testing\Functional Coverage\node_modules\istanbul-middleware\node_modules\istanbul\lib\collector.js:72:23
    at Array.forEach (native)
    at Object.Collector.add (D:\HTML5 Unit Testing\Functional Coverage\node_modules\istanbul-middleware\node_modules\istanbul\lib\collector.js:67:31)
    at Object.render (D:\HTML5 Unit Testing\Functional Coverage\node_modules\istanbul-middleware\lib\core.js:172:15)
    at D:\HTML5 Unit Testing\Functional Coverage\node_modules\istanbul-middleware\lib\handlers.js:47:14
    at Layer.handle [as handle_request] (D:\HTML5 Unit Testing\Functional Coverage\node_modules\express\lib\router\layer.js:95:5)
    at next (D:\HTML5 Unit Testing\Functional Coverage\node_modules\express\lib\router\route.js:137:13)
    at Route.dispatch (D:\HTML5 Unit Testing\Functional Coverage\node_modules\express\lib\router\route.js:112:3)

Ironically, when I am calling stringify on window.coverage object using executeScript() in selenium test class, it works without an issue.

I feel like I am missing something obvious and would appreciate some guidance.

Thanks in advance.

@kagarwal29

This comment has been minimized.

Copy link

commented Nov 25, 2017

@gotwarlost I have a quick query.How do I un-minify the JS present in the report? I want the JS present in the report to be in readable format.Currently it all comes in minified format.

I have tried passing debug:true,embedSource:true,noCompact:true to Instrumenter constructor but it is not doing anything.

@reeteshranjan

This comment has been minimized.

Copy link

commented Jan 31, 2018

@attodorov This is what I was looking for to do my cross-browser testing. I eventually built a tool of my own. If anyone wants to do cross-browser testing on platforms like browserstack, saucelabs and crossbrowsertesting and collect code coverage, you can use https://www.npmjs.com/package/cross-browser-tests-runner.

@ORESoftware

This comment has been minimized.

@silkentrance

This comment has been minimized.

Copy link

commented Mar 10, 2018

@gotwarlost since there is an article about this by @ORESoftware, why not close this for good?

@japborst

This comment has been minimized.

Copy link

commented Sep 15, 2018

Could we use the same approach to track coverage for electron apps?

@mahuntington

This comment has been minimized.

Copy link

commented Sep 15, 2018

@pavan447

This comment has been minimized.

Copy link

commented Dec 5, 2018

I am trying to find out code coverage for our backend application which is written on node js and my automation code written in Java using rest assured library. Both are in two different github repositories. Is there any way to find out code coverage for backend application using rest assured, please suggest.

@alex028502

This comment has been minimized.

Copy link

commented Mar 3, 2019

I made another example that might come in handy for somebody. It also merges the selenium code coverage with the unit test coverage.

https://github.com/alex028502/istanbulseleniumexample

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.