Running tests concurrently #215
Comments
|
JS is exclusively single-threaded - I'm not sure what other option you're proposing. |
|
I was under the impression that one test block wouldn't run before the last block with a plan finished, but that assumption might be bogus? If not, what I'm suggesting is calling all test blocks as soon as possible. |
|
@tjconcept currently/parallel/async tests are easy to get wrong and have weird edge cases in my experience and i'm glad tape doesn't do that.. although arguably there are cases for it, but other libs are built for that like https://github.com/vowsjs/vows |
|
Okay, fair. |
|
no i'm talking about the platform (not your code), now you have one more thing to manage, synchronizing your asynchronous tests to they run in the right order :( if you need to simulate a race condition or something you can still use t.plan and setTimeout in tape or if you use promises (in blue-tape) |
|
But that's my point; I would not want there to be any "wrong order". Tests should be independent thus being able to run concurrently without breaking each other. |
|
They should, of course! However, JS doesn't make it easy (or practically speaking, possible) to enforce that, and it's much harder to track down this class of bugs when test ordering is nondeterministic. |
|
Sure, I've tracked some down and it's absolutely horrible. |
|
When an error occurs, it's harder to isolate where when tests are running in parallel. Serial execution also make tests more deterministic, which is helpful when tracking down race conditions. |
Isn't that like saying: "write fewer tests, then your suite will pass more easily and you don't have to maintain as much"? What am I missing? |
|
I can see how the output might pose a challenge. Deterministic TAP output is a nice trait and mocha's approach will make it a bit less transparent. For those situations it would nice to have a way of forcing serial execution. |
|
At any rate, here are some options for parallel execution:
|
|
@substack - Would be nice to have a tape wrapper library that enables parallel execution of tests. |
|
You can use the prove command to execute tests parallel (http://perldoc.perl.org/prove.html). Although you don't get granular control over which tests get executed in parallel. I think it would be helpful to have a |
|
@wavded - What I've been considering is spawning a new process per test file. So the syntax for writing tests remains the same, the only thing that changes is how the runner runs the tests. I'm trying |
|
@Marak your comment reminded me of |
Why do you think the syntax has to change to be executed in parallel? |
|
I said the syntax would remain the same. Not change. |
Ah, sorry, I understood your comment as: "to keep the syntax the same, we need to spawn processes". |
|
@wavded - I'll probably be doing something similar. I'll put something together but I'm not sure if I will be able to open-source it. Hopefully someone releases a library that wraps tape for concurrent tests. |
|
I think running tests in parallel is something that should be done with processes. Someone can write a This means an entire file runs in very simple to debug run to completion and you can parallelize across multiple files. |
|
@Raynos - Are there any open-source solutions available for doing this? I'm working with a test suite with is going to require parrelization. I've already spec'd out a small tool for the job, but I don't think I will be able to open-source the project. Would much rather collaborate on an already existing project. |
|
@Marak Not that I know of. Maybe try contributing to |
|
hey all, I am writing a library called Suman that will compete with tape, ava etc and I need help finishing it. looking at this thread pains me because it's so obvious that test runners like tape and mocha could benefit greatly from parallelization. it runs tests in parallel or series according to the devs needs and runs tests suites in separate node.js processes. there's a whole host of other advantages as well. if you are willing to help out let me know. library is github.com/ORESoftware/suman, cheers |
|
@ORESoftware JS can't ever be parallel - it is single-threaded. the term you're looking for is "concurrent" which all of the frameworks you mentioned are. Separate processes works but precludes sharing objects across tests. You can do that just fine with tape by putting each "process"'s tests in a separate file, and running those in parallel with |
|
That's correct. Suman utilizes multiple cores by running each test in a
|
|
But you are wrong about this Justin - not all frameworks are concurrent.
|
How does that work? I have never seen dependency injection across threads, or is it more like sending some serialisable data?
I'm confused. Is it concurrent or multi-threaded or both?
Not at the same time, but interleaving, in AVA at least ;) |
|
Suman is just like AVA in terms of concurrency, they are basically the same in that respect. Each suite is designed to run in a separate process out of the box. For any given suite, you can have test cases run in "parallel" if you want using async I/O or async features of the event loop such as setTimeout. As a quick example. Here are test cases running in serial, like Mocha and Tape would always be like: https://gist.github.com/ORESoftware/bcab3bcaa36427ec7cce and here are test cases running in parallel: https://gist.github.com/ORESoftware/35305f45f6eea17de83f See the difference? :) this is with setTimeout, but the same behavior will happen with async I/O. the developer is in control, s/he can decide which test cases are appropriate for running in parallel or not. Parallel in this case is an OK word to use because it's more than likely that one test will start processing before another finishes, but if you prefer "interleaving", then pretend I said that instead. You might ask, well what about befores/afters/beforeEaches/afterEaches, can they be concurrent as well?...the answer is - befores/afters are always blocking, and beforeEaches and afterEaches are only blocking on a per test case basis, hope that makes sense. The simple way to prevent data sharing between tests is to attach data to each test, the same way you attach data to req in an Express server. Not that complicated. If you have any trouble running the above gists, please let me know, Suman is in beta. But Suman is going to give AVA a run for its money. I can talk about dependency injection on another thread, because that would be off topic. But Suman's biggest advantage over AVA would be that. |
|
I think tape should remain as it is. If concurrent tests fit my use case better I use Vows. |
|
Our test suite currently takes several minutes to run with coverage enabled. During this run, I can see 7 cores on my computer sitting idle. I'd love it if tape spawned 7 more node processes, divided tests equally among them, and consolidated the output. |
|
So it is possible to run your test concurrently (if by concurrently you mean running separate suites/files in a child process). I think the is what @keithkml was getting at? Here's my code for handling this: const spawn = require('child_process').spawn;
const numCPUs = require('os').cpus().length;
const Promise = require('bluebird');
const glob = Promise.promisify(require("glob"));
// The basic idea here is that we spawn a child process to run each test suite in
// our test folder up to the number of CPUs present in the system. We stream all
// the output back to the CLI and we handle everything as a promise to make the
// code easier to handle. This should speed up the tests significantly. NOTE there
// is a minor issue with the output being muddled up when passed through faucet. 😟
// This is causing two tests to fail only when passed through faucet (That I
// haven't defined). Use tap-summary instead. It handles output perfectly. 🕵💪
//
// Output:
//
// not ok 7 assert out of order
// not ok 8 unexpected additional plan
// ⨯ fail 2
//
// Other then this it all seems to be quite successful 🤓
function loadProcess(filename) {
return new Promise(function(resolve, reject) {
const process = spawn('node', [filename]);
const data = [];
process.stdout.on('data', function(d) {
data.push(d);
});
process.stderr.on('data', function(err) {
reject(err.toString());
});
process.on('exit', function() {
console.log(data.join(''));
resolve();
});
});
}
return glob(`${__dirname}/**/*.spec.js`)
.map((f) => loadProcess(f), {
concurrency: numCPUs
})This is the output from jdrew1303:~/workspace $ node ./test/runner.js | tap-summary
------------------------------------ Tests -------------------------------------
✔ Can naively parse a function. Dont expect anything too fancy. No ES6 😬 [pass: 1, fail: 0, duration: 168ms]
✔ Can create a new container [pass: 1, fail: 0, duration: 1ms]
✔ Can accept a simple binding [pass: 1, fail: 0, duration: 15ms]
✔ Parses a function declaration. [pass: 1, fail: 0, duration: 0ms]
✔ Parses a functional expression. [pass: 1, fail: 0, duration: 0ms]
✔ Parses a generator function. [pass: 1, fail: 0, duration: 0ms]
✔ Parses an arrow function [pass: 1, fail: 0, duration: 1ms]
✖ Parses complex default params [pass: 0, fail: 1, duration: 1ms]
✖ Parses object destructuring param definitions. [pass: 0, fail: 1, duration: 1ms]
✖ Parses object destructuring param definitions. [pass: 0, fail: 1, duration: 1ms]
✔ Parses a class constructor when passed through [pass: 1, fail: 0, duration: 4ms]
----------------------------------- Summary ------------------------------------
duration: 192ms
planned: 11
assertions: 11
pass: 8
fail: 3
------------------------------------ Fails -------------------------------------
# Parses complex default params
✖ should be equivalent
operator: deepEqual
expected: |-
{ params: [ 'it', 'parses', 'me' ], type: 'FunctionDeclaration' }
actual: |-
{ params: [ undefined, 'parses', 'me' ], type: 'FunctionDeclaration' }
at: Test.test (/home/ubuntu/workspace/test/paramParser.spec.js:65:7)
# Parses object destructuring param definitions.
✖ should be equivalent
operator: deepEqual
expected: |-
{ params: [ 'it', 'parses', 'me' ], type: 'FunctionDeclaration' }
actual: |-
{ params: [ undefined ], type: 'FunctionDeclaration' }
at: Test.test (/home/ubuntu/workspace/test/paramParser.spec.js:76:7)
✖ should be equivalent
operator: deepEqual
expected: |-
{ params: [ 'it', 'parses', 'me' ], type: 'FunctionDeclaration' }
actual: |-
{ params: [ undefined ], type: 'FunctionDeclaration' }
at: Test.test (/home/ubuntu/workspace/test/paramParser.spec.js:85:7)I'm not sure how this will handle in a CI environment. I haven't tried. Any help or ideas on this would be great. I wouldnt mind if this was a bit more robust. If these issues were resolved then I might wrap it with a cli lib and publish it. Hope this helps a bit 🙋🏻 |
|
@jdrew1303 Were you able to make it a CLI module? EDIT: I made |
|
Use AVA, Suman, or just put your tape tests in a bash file:
tape x &
tape y &
tape z &
wait
|
|
@ORESoftware i don't think that will exit nonzero if any of the subtasks do, tho. |
|
Good point dunno - maybe you can grab the exit codes somehow?
|
|
will report back upon answers received |
|
@ljharb et al, you can run things in parallel with a bash script like so: #!/usr/bin/env bash
arr=() # holds PIDs
commands=(
"echo 'hallelujah'; exit 0;"
"echo 'marzepan'; exit 3;"
)
clen=`expr "${#commands[@]}" - 1` # get length of commands - 1
for i in `seq 0 "$clen"`
do
# run the command via bash in subshell and push PID to array
(echo "${commands[$i]}" | bash) & arr+=($!)
done
len=`expr "${#arr[@]}" - 1` # get length of arr - 1
EXIT_CODE=0; # exit code of overall script
for i in `seq 0 "$len"`
do
pid="${arr[$i]}";
echo "PID => $pid"
wait ${pid} ; CODE="$?"
if [[ ${CODE} > 0 ]]; then
echo "at least one test failed";
EXIT_CODE=1;
fi
done
echo "EXIT_CODE => $EXIT_CODE"
exit "$EXIT_CODE"
throw your |
|
Actually this script is better, using #!/usr/bin/env bash
set -m # allow for job control
EXIT_CODE=0; # exit code of overall script
function foo() {
echo "CHLD exit code is $?"
if [[ $? > 0 ]]; then
echo "at least one test failed";
EXIT_CODE=1;
fi
}
trap 'foo' CHLD
DIRN=$(dirname "$0");
commands=(
"tape 'x'; exit 1;"
"tape 'y'; exit 0;"
"tape 'z'; exit 2;"
)
clen=`expr "${#commands[@]}" - 1` # get length of commands - 1
for i in `seq 0 "$clen"`; do
(echo "${commands[$i]}" | bash) & # run the command via bash in subshell
echo "$i ith command has been issued as a background job"
done
# wait for all to finish
wait;
echo "EXIT_CODE => $EXIT_CODE"
exit "$EXIT_CODE"
# end |
|
Just publish It's great for tests that have timeouts/network! |
|
@DiegoRBaquero I have tested here, but sadly the tests never exit when injecting a server. |
|
@iagomelanias You tested running tape's tests? tape's tests use tap too, which I don't support. |
|
For linux/mac users, moreutil's
|
|
@pbouzakis we'd have to verify that this tool captures exit codes from each subshell and then exits with 1 if any of the subshells exit with non-zero - my guess is that it does not. |
|
I hope it's okay to revive this inactive thread, but I created the aforementioned wrapper library for tape that runs tests concurrently (on the same thread). I hope this will come through for people looking here: |
|
With one of the latest Node.js versions it is now possible to use Worker Threads (the are like Web Workers in the Browser) directly in Node.js 😱 This should definitely help to implement parallel execution of tests in tape |
|
I've implemented a small concurrent runner that you can add inside single files. You replace class TestConcurrent {
constructor (tape) {
this._tape = tape
this._tests = []
this.testConcurrent = this.testConcurrent.bind(this)
this.runConcurrent = this.runConcurrent.bind(this)
}
testConcurrent (description, test) {
this._tests.push({description, test})
}
runConcurrent () {
this._tape('concurrent tests', async t => {
const plan = t.plan
let planCount = 0
t.plan = (incrementBy) => planCount += incrementBy
const promiseArray = []
for (const test of this._tests) {
const promise = new Promise(async resolve => {
t.comment(test.description)
await test.test(t)
resolve()
})
promiseArray.push(promise)
}
plan(planCount)
await Promise.all(promiseArray)
})
}
}
const test = require('tape')
const {testConcurrent, runConcurrent} = new TestConcurrent(test)
testConcurrent('test1', t => {
t.plan(1)
t.pass('done1')
})
testConcurrent('test2', async t => {
t.plan(1)
await new Promise(resolve => setTimeout(resolve, 4000))
t.pass('done2')
})
testConcurrent('test3', async t => {
t.plan(1)
await new Promise(resolve => setTimeout(resolve, 5000))
t.pass('done3')
})
test('regular test', t => {
t.plan(1)
t.pass('done')
})
runConcurrent() |
|
Using worker_threads isn't something that can be done transparently to tape test authors. It would be possible to run each file passed to Instead of any changes to tape here, I think that someone should invest the time to create a jest tape runner - that way, all of jest's already-solved parallelism can be applied to tape tests, pretty transparently for the test author, without adding complexity to tape. I would be more than happy to help maintain such a runner in a different repository, and publicize/advocate for it, if someone did the initial work to create it. In the meantime, I'm going to close this. |
Why did Tape choose to run tests serially? Is concurrent tests something we might see in the future?
The text was updated successfully, but these errors were encountered: