Running tests asynchronously

[ghost]

Is there a way for me to run the tests asynchronously.

I am testing my HTTP API for a database and each HTTP request takes 1-2 sec to finish and it would be really good if the tests are run asynchronously so each test doesn't have to wait for the test before to finish.

That would cut down the testing time to 1-2 sec instead of adding 1-2 sec for each test.

[tj]

wow that's really slow, I don't have a single suite so far over ~300ms, but no, and that's by-design, you give up quite a few things when you start running in parallel. ultra-slow tests like that should use an isolate or process per test

[ghost]

Why not let the user have the option to run them in asynch mode with eg "--asynch".

Not everyone wants to mock all database responses for every database in use.

It's better to inject data into the database and run the HTTP requests to manipulate them and assert the results :/

And when the database is changed in another version you will know it since some tests break. No false test results you get when you are mocking external systems.

Async nature is something that could help this kind of issue.

[tj]

that's what I did for expresso, it ends up being a huge mess. I dont mock any of mine and they're still fast

[tj]

plus all of the trashing of expresso making everything parallel actually made the tests nearly as slow

[ghost]

I'm on a mobile broadband so each HTTP connection takes 1-2 sec. The point is if I have a thorough test suite of all API methods and combinations it would probably be hundreds of tests. That would easily add up to minutes when testing.

What is the issue you run into with async testing on expresso?

[ghost]

You cannot cut your tests down to 300ms if you are testing your API's interaction with an external system with HTTP requests involved.

[tj]

sure, testing external serves is extremely slow, like knox with s3 etc, but that's not the focus of mocha, that's the focus of expresso

[tj]

in the future we could make files run parallel, but even that is not something I'm overly interested in

[ghost]

There was no before/beforeAll/after/afterAll in Expresso iirc. Perhaps that was what made it messy?

With these setup and teardown methods you have every test in isolation. You win a lot of time since the test after doesn't have to wait for the test before to finish. It isn't a chaining effect where the test after has to get some value from the test before, so why wait for it?

1. before()/beforeAll() setup the db and put some dummy text. after()/afterAll() tears down what is necessary.
2. Each test runs in isolation.

This is the one thing that was possible in Jasmine and not in Mocha.

This cuts down everything to 1-2 sec no matter how many connections you do. You could test an external system until it bleeds in just a matter of a sec from a slow broadband connecting using your laptop in an airplane! This is the soul of node.js. No one HTTP connection blocks everything.

Running files in parallell is not the way to go imo.

[tj]

because they're all parallel, there's no such thing as beforeEach or afterEach you would run into race conditions all over

[ghost]

If there wasn't a way to declare variables just for one test specifically. Using closures?

It's possible in Jasmine iirc.

[tj]

just function-local vars yes. I see nothing in jasmine-node for isolation

[ghost]

Then they were not in isolation in Jasmine. Lucky me that I didn't run into problems :)

If each test has access to it's own object with variables so it would access it like:

this.aVariableSetInBeforeEach, would that work?

http://yuilibrary.com/yui/docs/test/

Don't know if the tests are in isolation. I'll get back.

I don't think so, but wouldn't it work to just copy all variables set in a certain object in beforeEach() and call a test function in the context of the cloned object? Or just pass it as an argument to the test function. Each test function will have the same variables from beforeEach but each manipulation will just be for that specific test. Also the object from the first beforeEach() would be cloned to the nested multiple beforeEach() so they have access to whatever was specified, but each sibling beforeEach() can manipulate only the cloned one, thus in isolation.

Nothing I expect to be implemented. But does this work in theory? Just clone objects.

[ghost]

Someone else in the IRC was describing one of the issues with async testing.

"I have a test to handle the case for trying to connect too many connections to the database at once. if that test is executing while another test is trying to do some other query test...the other one might fail depending on when they execute"

I didn't take that one into account. I can understand that it's not just variable issues, it's a risk that the external system would change state and thus affecting other tests that would normally pass.

[nagyv]

Just to be sure that I understand this discussion correctly.

I was wondering whether a setup like

describe(
...
    describe(
    ...
    );
);
describe(
...
);

Could lead to race conditions if I change the system state inside each describe, but they are properly reverted using after/afterEach.

As far as I could understand the discussion above, no race conditions could emerge as nothing is called in parallel. Am I right? (I hope not, because then I'll have to find a different reason why my tests fail occasionally. :))

[tj]

@nagyv if you're executing other code in parallel and following that with a call to mocha's done() you may still have some races, it depends

[fakewaffle]

What about using/integrating https://github.com/caolan/async?

[kumar303]

There is a third party module (mocha.parallel) where you can opt-in a suite of tests by replacing describe() with parallel(). I think this should be pretty safe as long as you know your tests are not patching any shared objects. It can speed things up if the functions under test are IO bound.

[danielstjules]

Better async support is something that @boneskull is looking into for the mocha rewrite. mocha.parallel is a hack at best, and unfortunately, you'll almost certainly hit issues where it can't do what you want since you give up on most of the mocha API.