Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse code

readme

  • Loading branch information...
commit 71a87056b35aff0081a7b800e490c064f280abd6 1 parent af659c1
Yaron Naveh authored
71 README.md
Source Rendered
... ... @@ -1,2 +1,69 @@
1   -test-select
2   -===========
  1 +## test-creep
  2 +Node.js selective tests execution. *Seamlessly* integrates with [Mocha](http://visionmedia.github.com/mocha/) (more frameworks coming soon).
  3 +
  4 +
  5 +## What is selective test execution?
  6 +
  7 +It means running just the relevant subset of your tests instead of all of them. For example, if you have 200 tests, and 10 of them are related to some feature, then if you make a change to this feature you should run just the 10 tests and not the whole 200. test-select automatically chooses the relevant tests based on [istanbul](https://github.com/gotwarlost/istanbul) code coverage reports. All this is done for you behind the scenes and you can work normally with just Mocha.
  8 +
  9 +For more information visit [my blog](http://webservices20.blogspot.com/) or [my twitter](https://twitter.com/YaronNaveh).
  10 +
  11 +
  12 +## Usage
  13 +
  14 +1. You should use [Mocha](http://visionmedia.github.com/mocha/) in your project to run tests. You should use git as source control.
  15 +
  16 +2. You need to have Mocha installed locally and run it locally rather than globally:
  17 +
  18 + $> npm install mocha
  19 +
  20 + $> ./node_moduels/mocha/bin/mocha ./tests
  21 +
  22 +3. You need to install test-creep:
  23 +
  24 + $> npm install test-creep
  25 +
  26 +4. When you run mocha specify to run a special test before all other tests
  27 +
  28 + $> ./node_moduels/mocha/bin/mocha ./node_modules/test-creep/first.js ./tests
  29 +
  30 +first.js is bundled with test-select and monkey patchs mocha with required instrumentation.
  31 +
  32 +5. It is recommended to add ./.testdeps_.json to .gitignore (more on this file below
  33 +)
  34 +
  35 +## How does this work?
  36 +
  37 +The first time you execute the command all tests run. first.js monkey patches mocha with istanbul code coverage and tracks the coverage per test (rather than per the whole process). Then in your project root the test dependency file is created (./.testdeps_.json):
  38 +
  39 +
  40 +`````javascript
  41 + {
  42 + "should alert when dividing by zero": [
  43 + "tests/calc.js",
  44 + "lib/calc.js",
  45 + "lib/exceptions.js",
  46 +
  47 + ],
  48 +
  49 + "should multiply with negative numbers": [
  50 + "tests/negative.js",
  51 + "lib/calc.js",
  52 + ],
  53 + }
  54 +
  55 +`````
  56 +
  57 +Next time you run the test (assuming you add first.js to the command) test-creep runs 'git status' to see which files were added/deleted/modified since last commit. Then test-creep instructs mocha to only run tests that have dependency in changed files. In the example above, if you have uncommited changes only to lib/exceptions.js, then only the first test will be executed.
  58 +
  59 +At any moment you can run mocha without first.js in which case all tests and not just relevant ones will run.
  60 +
  61 +
  62 +## limitations
  63 +* Dependency between test and code is captured at file and not function granularity. So sometimes test-select can run more tests than actually requiered (there is no harm in that).
  64 +
  65 +* test-select cannot detect changes in global contexts. For example, if you have a one time global initialization of a dictionary, and some tests use this dictionary, then test-select will not mark these tests as dirty if there is a change in the initialization code.
  66 +
  67 +* If you have a test suite that runs super fast (< 2 seconds) then test-select will probably add more overhead than help. test-select sweet spot is in long running test suites, though whenever tests run for more than a couple of seconds it can save you time.
  68 +
  69 +For more information visit [my blog](http://webservices20.blogspot.com/) or [my twitter](https://twitter.com/YaronNaveh).
2  first.js
... ... @@ -1 +1 @@
1   -require("./selective")
  1 +require("./lib/selective")
1  lib/consts.js
... ... @@ -0,0 +1 @@
  1 +exports.depsFile = './.testdeps_.json'
179 lib/coverage.js
... ... @@ -0,0 +1,179 @@
  1 +var path = require('path'),
  2 + fs = require('fs'),
  3 + istanbul = require('istanbul'),
  4 + hook = istanbul.hook,
  5 + Instrumenter = istanbul.Instrumenter,
  6 + Collector = istanbul.Collector,
  7 + instrumenter = new Instrumenter(),
  8 + Report = istanbul.Report,
  9 + collector,
  10 + globalAdded = false,
  11 + fileMap = {};
  12 +
  13 +/**
  14 + * Facade for all coverage operations support node as well as browser cases
  15 + *
  16 + * Usage:
  17 + * ```
  18 + * //Node unit tests
  19 + * var coverage = require('/path/to/this/file');
  20 + * coverage.hookRequire(); // hooks require for instrumentation
  21 + * coverage.addInstrumentCandidate(file); // adds a file that needs to be instrumented; should be called before file is `require`d
  22 + *
  23 + * //Browser tests
  24 + * var coverage = require('/path/to/this/file');
  25 + * var instrumentedCode = coverage.instrumentFile(file); //alternatively, use `instrumentCode` if you have already loaded the code
  26 + * //collect coverage from the browser
  27 + * // this coverage will be stored as `window.__coverage__`
  28 + * // and...
  29 + * coverage.addCoverage(coverageObject); // rinse and repeat
  30 + * ```
  31 + *
  32 + * //in all cases, add an exit handler to the process
  33 + * process.once('exit', function () { coverage.writeReports(outputDir); }); //write coverage reports
  34 + */
  35 +
  36 +/**
  37 + * adds a file as a candidate for instrumentation when require is hooked
  38 + * @method addInstrumentCandidate
  39 + * @param file the file to add as an instrumentation candidate
  40 + */
  41 +function addInstrumentCandidate(file) {
  42 + file = path.resolve(file);
  43 + fileMap[file] = true;
  44 +}
  45 +/**
  46 + * hooks require to instrument all files that have been specified as instrumentation candidates
  47 + * @method hookRequire
  48 + * @param verbose true for debug messages
  49 + */
  50 +function hookRequire(verbose) {
  51 +
  52 + var matchFn = function (file) {
  53 + /*
  54 + var match = fileMap[file],
  55 + what = match ? 'Hooking' : 'NOT hooking';
  56 + if (verbose) { console.log(what + file); }
  57 + return match;
  58 + */
  59 + var res = file.indexOf("node_modules")==-1
  60 + var what = res ? 'Hooking ' : 'NOT hooking '
  61 + if (verbose) console.log(what + file)
  62 + return res
  63 +
  64 + }, transformFn = instrumenter.instrumentSync.bind(instrumenter);
  65 +
  66 + hook.hookRequire(matchFn, transformFn);
  67 +}
  68 +/**
  69 + * unhooks require hooks that have been installed
  70 + * @method unhookRequire
  71 + */
  72 +function unhookRequire() {
  73 + hook.unhookRequire();
  74 +}
  75 +
  76 +
  77 +function getCollector() {
  78 + return getCollectorInternal(false)
  79 +}
  80 +
  81 +
  82 +/**
  83 + * returns the coverage collector, creating one if necessary and automatically
  84 + * adding the contents of the global coverage object. You can use this method
  85 + * in an exit handler to get the accumulated coverage.
  86 +*/
  87 +function getCollectorInternal(createNew) {
  88 + if (!collector || createNew) {
  89 + collector = new Collector();
  90 + }
  91 +
  92 + if (globalAdded && !createNew) { return collector; }
  93 +
  94 + if (global['__coverage__']) {
  95 + collector.addInternal(global['__coverage__'], true);
  96 + globalAdded = true;
  97 + } else {
  98 + console.error('No global coverage found for the node process');
  99 + }
  100 + return collector;
  101 +}
  102 +/**
  103 + * adds coverage to the collector for browser test cases
  104 + * @param coverageObject the coverage object to add
  105 + */
  106 +function addCoverage(coverageObject) {
  107 + if (!collector) { collector = new Collector(); }
  108 + collector.add(coverageObject);
  109 +}
  110 +/**
  111 + * returns the merged coverage for the collector
  112 + */
  113 +function getFinalCoverage() {
  114 + return getCollector().getFinalCoverage();
  115 +}
  116 +
  117 +/**
  118 + * writes reports for an array of JSON files representing partial coverage information
  119 + * @method writeReportsFor
  120 + * @param fileList array of file names containing partial coverage objects
  121 + * @param dir the output directory for reports
  122 + */
  123 +function writeReportsFor(fileList, dir) {
  124 + var collector = new Collector();
  125 + fileList.forEach(function (file) {
  126 + var coverage = JSON.parse(fs.readFileSync(file, 'utf8'));
  127 + collector.addCoverage(coverage);
  128 + });
  129 + writeReportsInternal(dir, collector);
  130 +}
  131 +
  132 +/**
  133 + * writes reports for everything accumulated by the collector
  134 + * @method writeReports
  135 + * @param dir the output directory for reports
  136 + */
  137 +function writeReports(dir) {
  138 + writeReportsInternal(dir, getCollector());
  139 +}
  140 +
  141 +function writeReportsInternal(dir, collector) {
  142 + dir = dir || process.cwd();
  143 + var reports = [
  144 + Report.create('lcov', { dir: dir }),
  145 + Report.create('text'),
  146 + Report.create('text-summary')
  147 + ];
  148 + reports.forEach(function (report) { report.writeReport(collector, true); })
  149 +}
  150 +/**
  151 + * returns the instrumented version of the code specified
  152 + * @param {String} code the code to instrument
  153 + * @param {String} file the file from which the code was load
  154 + * @return {String} the instrumented version of the code in the file
  155 + */
  156 +function instrumentCode(code, filename) {
  157 + filename = path.resolve(filename);
  158 + return instrumenter.instrumentSync(code, filename);
  159 +}
  160 +/**
  161 + * returns the instrumented version of the code present in the specified file
  162 + * @param file the file to load
  163 + * @return {String} the instrumented version of the code in the file
  164 + */
  165 +function instrumentFile(file) {
  166 + filename = path.resolve(file);
  167 + return instrumentCode(fs.readFileSync(file, 'utf8'), file);
  168 +}
  169 +
  170 +module.exports = {
  171 + addInstrumentCandidate: addInstrumentCandidate,
  172 + hookRequire: hookRequire,
  173 + unhookRequire: unhookRequire,
  174 + instrumentCode: instrumentCode,
  175 + instrumentFile: instrumentFile,
  176 + addCoverage: addCoverage,
  177 + writeReports: writeReports,
  178 + getCollectorInternal: getCollectorInternal
  179 +};
206 lib/selective.js
... ... @@ -0,0 +1,206 @@
  1 +
  2 +var fs = require('fs')
  3 +var path = require('path')
  4 +var execSync = require('execSync');
  5 +var mocha = require('mocha')
  6 +var consts = require('./consts')
  7 +var coverage = require('./coverage');
  8 +
  9 +var selective = {
  10 + depsTree: {},
  11 + testsToRun: {},
  12 + verbose: false,
  13 +
  14 + init: function() {
  15 + selective.log('initializing selective execution...')
  16 + this.loadDepsTree()
  17 + this.loadChangedFiles()
  18 + },
  19 +
  20 + loadDepsTree: function() {
  21 + if (fs.existsSync(consts.depsFile)) {
  22 + var deps = fs.readFileSync(consts.depsFile)
  23 + this.depsTree = JSON.parse(deps)
  24 + selective.log('loading deps tree from disk:\n' + deps)
  25 + }
  26 + selective.log('done loading deps tree')
  27 + },
  28 +
  29 + saveDepsTree: function() {
  30 + var deps = JSON.stringify(this.depsTree, null, 4)
  31 + fs.writeFileSync(consts.depsFile, deps)
  32 + selective.log('saved deps tree to disk:\n' + deps)
  33 + },
  34 +
  35 + cleanCoverageCounts: function() {
  36 + selective.log('trying to clean coverage report...')
  37 + if (typeof __coverage__ == 'undefined') return
  38 +
  39 + for (var file in __coverage__) {
  40 + for (var line in __coverage__[file].s) {
  41 + __coverage__[file].s[line] = 0
  42 + }
  43 + }
  44 +
  45 + selective.log('coverage report clean')
  46 + },
  47 +
  48 + updateCoverageCounts: function(test) {
  49 + selective.log('updating coverage count for test '+test.title+'...')
  50 + var coverage = this.getCurrentCoverage()
  51 + selective.log('coverage for test:\n' + JSON.stringify(coverage, null, 4))
  52 + this.depsTree[test.title] = coverage
  53 + selective.log('total coverage:\n' + JSON.stringify(this.depsTree, null, 4))
  54 + },
  55 +
  56 + removeFromCoverage: function(test) {
  57 + selective.log('removing coverage count for test '+test.title+'...')
  58 + delete this.depsTree[test.title]
  59 + },
  60 +
  61 + getCurrentCoverage: function() {
  62 + if (typeof __coverage__ == 'undefined') return
  63 + //selective.log('current coverage:\n' + JSON.stringify(__coverage__, null, 4))
  64 + var res = []
  65 + for (var file in __coverage__) {
  66 + for (var line in __coverage__[file].s) {
  67 + if (__coverage__[file].s[line]>0) {
  68 + var relative = path.relative(process.cwd(), file)
  69 + res.push(relative)
  70 + break
  71 + }
  72 + }
  73 + }
  74 +
  75 + return res
  76 + },
  77 +
  78 + loadChangedFiles: function() {
  79 + selective.log('loading changed files...')
  80 + selective.log('process.env[gitstatus]: ' + process.env['gitstatus'])
  81 + var changedFiles = {}
  82 + //using env var is good for testing of the test-select library
  83 + var diff = process.env['gitstatus'] || execSync.stdout('git status');
  84 + selective.log('diff is:\n' + diff)
  85 + var rePattern = new RegExp(/(modified|deleted|added):\s*(.*)/g);
  86 + var match = rePattern.exec(diff)
  87 + while (match!=null) {
  88 + selective.log('changed file: ' + match[2])
  89 + changedFiles[match[2]] = true
  90 + match = rePattern.exec(diff)
  91 + }
  92 +
  93 + selective.log('deps tree:\n' + JSON.stringify(this.depsTree, null, 4))
  94 + selective.log('changed files:\n' + JSON.stringify(changedFiles, null, 4))
  95 +
  96 + for (var test in this.depsTree) {
  97 + this.testsToRun[test] = false
  98 + for (var file in this.depsTree[test])
  99 + {
  100 +
  101 + if (changedFiles[this.depsTree[test][file]]) {
  102 + this.testsToRun[test] = true
  103 + }
  104 + }
  105 + }
  106 +
  107 + selective.log('tests to run\n' + JSON.stringify(this.testsToRun, null, 4))
  108 +
  109 + },
  110 +
  111 + log: function(str) {
  112 + if (this.verbose) console.log(str + '\n')
  113 + }
  114 +
  115 +}
  116 +
  117 +
  118 +selective.log('starting selective execution')
  119 +selective.verbose = process.argv.indexOf('--verbose')!=-1
  120 +coverage.hookRequire(selective.verbose);
  121 +selective.init()
  122 +
  123 +
  124 +mocha.Runner.prototype.runTests = function(suite, fn) {
  125 + var self = this
  126 + , tests = suite.tests.slice()
  127 + , test;
  128 +
  129 + function next(err) {
  130 + // if we bail after first err
  131 + if (self.failures && suite._bail) return fn();
  132 +
  133 + // next test
  134 + test = tests.shift();
  135 +
  136 + // all done
  137 + if (!test) return fn();
  138 +
  139 + //**this is the line added for selective testing
  140 + if (selective.testsToRun[test.title]==false) {
  141 + selective.log('skipping test:\n' + test.title)
  142 + return next()
  143 + }
  144 +
  145 + // grep
  146 + var match = self._grep.test(test.fullTitle());
  147 + if (self._invert) match = !match;
  148 + if (!match) return next();
  149 +
  150 + // pending
  151 + if (test.pending) {
  152 + self.emit('pending', test);
  153 + self.emit('test end', test);
  154 + return next();
  155 + }
  156 +
  157 + // execute test and hook(s)
  158 + self.emit('test', self.test = test);
  159 + self.hookDown('beforeEach', function(){
  160 + self.currentRunnable = self.test;
  161 +
  162 + self.runTest(function(err){
  163 + test = self.test;
  164 +
  165 + if (err) {
  166 + self.fail(test, err);
  167 + self.emit('test end', test);
  168 + return self.hookUp('afterEach', next);
  169 + }
  170 +
  171 + test.state = 'passed';
  172 + self.emit('pass', test);
  173 + self.emit('test end', test);
  174 + self.hookUp('afterEach', next);
  175 + });
  176 +
  177 + });
  178 + }
  179 +
  180 + this.next = next;
  181 + next();
  182 +};
  183 +
  184 +var innerRunner = mocha.Runner
  185 +
  186 +mocha.Runner = function (suite) {
  187 + var runner = new innerRunner(suite)
  188 + runner.on('end', function() {
  189 + selective.saveDepsTree()
  190 + })
  191 + runner.on('test', function(test) {
  192 + selective.cleanCoverageCounts()
  193 + selective.log('start run test:\n' + test.title)
  194 + })
  195 + runner.on('pass', function(test) {
  196 + selective.updateCoverageCounts(test)
  197 + selective.log('end run test (pass):\n' + test.title)
  198 + })
  199 + runner.on('fail', function(test) {
  200 + selective.removeFromCoverage(test)
  201 + selective.log('end run test (fail):\n' + test.title)
  202 + })
  203 +
  204 + return runner
  205 +}
  206 +
7 package.json
... ... @@ -1,5 +1,5 @@
1 1 {
2   - "name": "test-select",
  2 + "name": "test-creep",
3 3 "version": "0.0.1",
4 4 "description": "Selective test execution",
5 5 "engines": { "node": ">=0.4.0" },
@@ -13,14 +13,13 @@
13 13 },
14 14 "repository" : {
15 15 "type":"git",
16   - "url":"https://github.com/yaronn/test-select.git" },
17   - "main": "./selective.js",
  16 + "url":"https://github.com/yaronn/test-select.git" },
18 17 "directories": { "lib": "./lib" },
19 18 "keywords": ["test", "testing", "selective test execution"],
20 19 "licenses": [{
21 20 "type" : "MIT License",
22 21 "url" : "http://www.opensource.org/licenses/mit-license.php" }],
23 22 "scripts": {
24   - "test": "./node_modules/mocha/bin/mocha tests.js ./test/tests.js"
  23 + "test": "./node_modules/mocha/bin/mocha first.js ./tests"
25 24 }
26 25 }
2  tests/test.js
... ... @@ -1,6 +1,6 @@
1 1 var execSync = require('execSync');
2 2 var fs = require('fs');
3   -var consts = require('../consts')
  3 +var consts = require('../lib/consts')
4 4 var assert = require('assert')
5 5
6 6 describe('selective test execution', function() {

0 comments on commit 71a8705

Please sign in to comment.
Something went wrong with that request. Please try again.