The default "yarn test" command runs ALL tests 3 times

[begedin]

Problem

On my machine, when I run yarn test, or yarn test --silent, all tests execute 3 times. The desired behavior, since this actually executes ember exam --split=3 --weighted --parallel would be that 3 partitions would run, each running a fraction of the tests. In addition to that, the 3 partitions run consecutively, not in parallel.

This seems to be related to ember-cli/ember-exam#108 and, following the conversations there and adding parallel: -1 to testem.js seems to cause the partitions to execute in paralel, but each partition still runs all of the tests in the suite.

[joshsmith]

Technically we should change ember-try.json at this point, since I think I prefer the more explicit manual usage of ember exam directly in the .circleic/config.yml, for which parallelization was enabled. We could use the process.ENV in that file to dynamically pull from Circle's environment variables, but then that would be making ember-try's default case locked to Circle (unless we set fallback values or something instead).

A good question: is there even any indication that running the tests in parallel with ember-exam locally will actually speed up the run?

In a single container where you wanted to run two partitions, and run the third in a separate container, you would do ember exam --split=3 --partition=1 --partition=2 --parallel.

You could extend this to a non-containerized environment and run ember exam --split=3 --partition=1 --partition=2 --partition=3 --parallel, but I'm not sure I see any benefit versus running a single browser instance. It could certainly be benchmarked, though.

[begedin]

At this point, locally, due to the issue linked above, it's actually slower to enable parallelism (3x slower, to be exact). On CI, we disable parallelization anyway.

Assuming the issue is fixed, we ought to probably benchmark, but it would likely vary across user machines.

What we need to consider here is not just CI, though.

package.json is usually the entryway for a frontend project so people tend to refer to it to decide which commands to use for execution/testing.

With this being an ember app, most developers are used to ember test, ember server, etc. I'm used to ember exam and then to use that almost exclusively.

However, package.json says the command to run tests is yarn test, so it's safe to assume a bunch of people will use it and at this point, that command runs 3x as long on my machine as most other commands I'd use.

[joshsmith]

On CI, we disable parallelization anyway.

We use JOBS=2 to take advantage of two CPUs for things like broccoli, but then we have 4x parallelism with individual partitions in the test.

We can change from yarn test to just ember exam. Doesn't matter to me. Whatever will speed things back up for you, PR that.

[begedin]

It's not a matter of speeding up for me, since I use ember exam anyway.

It's a matter of there being a "default" command (yarn test) which runs 3x slower than it should, due to a bug that doesn't originate from us.

I'm thinking something like

• yarn test should just execute ember exam instead of ember exam --split=3 --weighted --parallel
• add some docs pointing the user on how they might try to speed it up by enabling parallelism, with a note that it may be buggy

[joshsmith]

Then we can just amend ember-try.json to use ember exam only. No need for docs IMO.