Unequal number of tests in each partition

[shripathee]

We are using the split option, however we find that the number of tests executed in each partition varies greatly. Does the splitting happen on the basis of number of tests or the number of the corresponding files? If it's the latter, it represents a problem for us since we have some files that contain a large number of tests and some which contain 2 or 3. We could still refactor our tests and create more number of files containing fewer test cases, but logically, shouldn't partitioning be done on the number of test cases?

[trentmwillis]

Partitioning is currently done by file, or, more accurately, by AMD module. Ideally, you are correct and partitioning would be done at the test level, but this is currently not possible as once a test() call is evaluated there is no way to remove that test from the queue. The current implementation thus relies on the TestLoader to not load test modules in order to create partitions.

In the future, I think this could be possible if the partitioning logic is pushed upstream into QUnit itself, but this is a long way off and likely not to be completed any time in the near future.

I think a note about this in the README might be beneficial.

[lolmaus]

We see Chrome error: browser disconnected when running tests. It is not related to a specific test, it always happens after a successful test. It and looks like a memory leak that piles up as more tests are run.

So far, we have no idea how to debug the memory leak. I'm looking into ways of working around it.

An obvious way would be to split the test into many partitions, but it doesn't help. Regardless of the amount of partitions (I tried --split=10, --split=100, --split=1000, --split=9999), the first partition runs 340-380 tests until it reaches a browser disconnect. Then the next partition runs exact same cases and crashes in the same place ± 1-2 tests.

Also, --parallel does not seem to parallelize. It runs partitions sequentially. If I remove --parallel, the tests will finish after the first browser disconnect.

Please advise.

[lolmaus]

I just ran ember exam --split=9999 --parallel --random, and the first partition ran 827 tests before it crashes. On our non-failing branch we have less than 2000 tests total.

It looks like splitting doesn't work at all, please help.

[lolmaus]

We're running Ember CLI 2.15.

[andrewzamojc-pd]

I had the same issue today. In testem.js I had parallel: '3' and I would see every partition running all the tests. I set parallel: '-1' and I think that's what got the split + parallel working for me. It may not exactly solve the "browser disconnects" but it seemed to help for me :S

[trentmwillis]

@lolmaus it sounds like you have several distinct issues. It would be tremendously helpful if you could (1) open new issues for each distinct issue you are experiencing (as they don't seem to be related to this issue), and (2) provide some form of a reproduction. Thanks!

[lolmaus]

@trentmwillis Here you go: #108.