Tests emitting order #62

Closed
flore77 opened this Issue Jun 10, 2016 · 11 comments

Comments

Projects
None yet
4 participants
@flore77
Contributor

flore77 commented Jun 10, 2016

The Common Reporter Interface should also specify in which order the tests should be emitted.

The most natural would be the source order in which tests are emitted in the order they were written.

Mocha runs the tests in suite order, which means if it finds a suite that contains tests and other suites, it emits the start of the top level suite and then emits its tests, then the other suites, even if the test were the last in source order.

QUnit and Jasmine are executing the tests in the source order, but their reporters are not quite equal. Jasmine's reporter displays the tests always in the order they were emitted, but QUnit's reporter displays always the tests like Mocha.

The source order would be quite difficult to achieve for our adapters because we are not scanning the input file, we are only listening to events, and if the tests are executed in random order (QUnit and Jasmine have this capability), we will not know the source order. This is also available for Mocha, for example:

// Ex 1
describe('a', function() {
  describe('b', function() {
    ...
  });

  it('test', function() {});
});

// Ex 2
describe('a', function() {
  it('test', function() {});

  describe('b', function() {
    ...
  });
});

We cannot differentiate between the 2 above examples, to know when the test should be emitted in source order.

@flore77 flore77 referenced this issue Jun 11, 2016

Closed

Fix Jasmine Adapter #63

5 of 5 tasks complete
@flore77

This comment has been minimized.

Show comment
Hide comment
@flore77

flore77 Jul 25, 2016

Contributor

In my opinion the tests should be executed and reported in source order, because I find it the most natural way that also meet the expectation of the developer.

If an option is activated to run tests in another order (some test frameworks have this capability, i.e. to run tests in random order), only then they should be executed in that order and be reported in the order they have been executed to meet the expectation of the developer, give him feedback that the tests have been indeed ran in another one than the default one (source order).

To sum up tests should always be reported in the order they have been executed, which should default to source order execution.

What do you think about this ? @boneskull @searls @jzaefferer

This will be only a discussion for defining the standard for test frameworks that will implement it native, for our current adapters (like I explained in the first comment) we cannot know which the source order was, or at least I didn't identified a solution.

Contributor

flore77 commented Jul 25, 2016

In my opinion the tests should be executed and reported in source order, because I find it the most natural way that also meet the expectation of the developer.

If an option is activated to run tests in another order (some test frameworks have this capability, i.e. to run tests in random order), only then they should be executed in that order and be reported in the order they have been executed to meet the expectation of the developer, give him feedback that the tests have been indeed ran in another one than the default one (source order).

To sum up tests should always be reported in the order they have been executed, which should default to source order execution.

What do you think about this ? @boneskull @searls @jzaefferer

This will be only a discussion for defining the standard for test frameworks that will implement it native, for our current adapters (like I explained in the first comment) we cannot know which the source order was, or at least I didn't identified a solution.

@jzaefferer

This comment has been minimized.

Show comment
Hide comment
@jzaefferer

jzaefferer Jul 25, 2016

Contributor

To sum up tests should always be reported in the order they have been executed, which should default to source order execution.

That sounds good to me. That way the spec makes a recommendation, but doesn't force any particular implementation.

Contributor

jzaefferer commented Jul 25, 2016

To sum up tests should always be reported in the order they have been executed, which should default to source order execution.

That sounds good to me. That way the spec makes a recommendation, but doesn't force any particular implementation.

@searls

This comment has been minimized.

Show comment
Hide comment
@searls

searls Jul 25, 2016

I don't think it's wise to build a lot of assumptions on top of a default test ordering scheme, to be honest.

Some test frameworks (like RSpec and Minitest) have opted to run tests in random order by default, I believe in part because the same developers who most struggle with test pollution biting them are the ones least likely to discover and enable the option themselves. I've been thinking of implementing random-order tests for teenytest, and if I was able to get it to work effectively, I'd consider making it the default for the same reason (safety first).

Another reason to be wary of depending on execution order is that Node.js test runners like Ava run tests in separate processes and so execution order is unspecified by design, in the interest of maximizing speed (and with a side effect of warding against test pollution). This constraint would preclude its use.

searls commented Jul 25, 2016

I don't think it's wise to build a lot of assumptions on top of a default test ordering scheme, to be honest.

Some test frameworks (like RSpec and Minitest) have opted to run tests in random order by default, I believe in part because the same developers who most struggle with test pollution biting them are the ones least likely to discover and enable the option themselves. I've been thinking of implementing random-order tests for teenytest, and if I was able to get it to work effectively, I'd consider making it the default for the same reason (safety first).

Another reason to be wary of depending on execution order is that Node.js test runners like Ava run tests in separate processes and so execution order is unspecified by design, in the interest of maximizing speed (and with a side effect of warding against test pollution). This constraint would preclude its use.

@jzaefferer

This comment has been minimized.

Show comment
Hide comment
@jzaefferer

jzaefferer Jul 26, 2016

Contributor

Thanks for the input, those are useful references.

From that perspective, there's a difference between execution and output order. In QUnit, there's 3 types of execution order (source, rerun failed first, random), but the HTML reporter always outputs all tests in source order.

An equivalent console reporter would have to rewrite multiple lines to emulate that behaviour (not trivial, but certainly possible).

With that in mind, would it be reasonable to make a recommendation to:

  1. Don't assume any specific execution order
  2. Assume the list of suites and tests provided in runStart is in source order, and the output should honour that order.
Contributor

jzaefferer commented Jul 26, 2016

Thanks for the input, those are useful references.

From that perspective, there's a difference between execution and output order. In QUnit, there's 3 types of execution order (source, rerun failed first, random), but the HTML reporter always outputs all tests in source order.

An equivalent console reporter would have to rewrite multiple lines to emulate that behaviour (not trivial, but certainly possible).

With that in mind, would it be reasonable to make a recommendation to:

  1. Don't assume any specific execution order
  2. Assume the list of suites and tests provided in runStart is in source order, and the output should honour that order.
@boneskull

This comment has been minimized.

Show comment
Hide comment
@boneskull

boneskull Jul 26, 2016

tests should always be reported in the order they have been executed

This seems reasonable to me, as well as @jzaefferer's points above.

tests should always be reported in the order they have been executed

This seems reasonable to me, as well as @jzaefferer's points above.

@flore77

This comment has been minimized.

Show comment
Hide comment
@flore77

flore77 Jul 26, 2016

Contributor

there's a difference between execution and output order

This is a really great point.

I was saying in the above comment:

tests should always be reported in the order they have been executed

I was thinking to give the developer the feedback that his tests are really executed in another order if he has activated a certain option, but how @searls pointed out there are test frameworks which:

execution order is unspecified by design

So to maintain the feedback for the developers, test frameworks that have a default execution order, and have options to activate other types of execution, they can notify with a message in the end or beginning, something like: "Tests were executed in the order x".

With test frameworks which execution order is unpredictable, I think it would be annoying to get every time another report, so for output order I think definitely the order should be source order, because tests are describing the application flow, basically they are describing the application itself, personally when I write tests, I write them with some logic in mind and positioning them in a certain order, so when you look at them, you should be able to understand what is the application about.

I checked also Ava and despite to its concurrency (which is cool), it also reports tests in source order.

In the end here we are establishing a reporting standard, so I think it is good to not interfere with the execution order, but how @jzaefferer said we should:

  1. Don't assume any specific execution order
  2. Assume the list of suites and tests provided in runStart is in source order, and the output should honour that order.

I think it should be more than an assumption, it should be certainty that it is in source order.

Contributor

flore77 commented Jul 26, 2016

there's a difference between execution and output order

This is a really great point.

I was saying in the above comment:

tests should always be reported in the order they have been executed

I was thinking to give the developer the feedback that his tests are really executed in another order if he has activated a certain option, but how @searls pointed out there are test frameworks which:

execution order is unspecified by design

So to maintain the feedback for the developers, test frameworks that have a default execution order, and have options to activate other types of execution, they can notify with a message in the end or beginning, something like: "Tests were executed in the order x".

With test frameworks which execution order is unpredictable, I think it would be annoying to get every time another report, so for output order I think definitely the order should be source order, because tests are describing the application flow, basically they are describing the application itself, personally when I write tests, I write them with some logic in mind and positioning them in a certain order, so when you look at them, you should be able to understand what is the application about.

I checked also Ava and despite to its concurrency (which is cool), it also reports tests in source order.

In the end here we are establishing a reporting standard, so I think it is good to not interfere with the execution order, but how @jzaefferer said we should:

  1. Don't assume any specific execution order
  2. Assume the list of suites and tests provided in runStart is in source order, and the output should honour that order.

I think it should be more than an assumption, it should be certainty that it is in source order.

@jzaefferer

This comment has been minimized.

Show comment
Hide comment
@jzaefferer

jzaefferer Aug 1, 2016

Contributor

it should be certainty that it is in source order.

I'm still not sure if that helps. This doesn't seem crucial to make test frameworks and runners inter-operable.

That said, I'm okay with putting this in the spec as a requirement. We can include the examples discussed here to provide some background.

Contributor

jzaefferer commented Aug 1, 2016

it should be certainty that it is in source order.

I'm still not sure if that helps. This doesn't seem crucial to make test frameworks and runners inter-operable.

That said, I'm okay with putting this in the spec as a requirement. We can include the examples discussed here to provide some background.

@flore77 flore77 referenced this issue Aug 1, 2016

Merged

Update spec #84

@boneskull

This comment has been minimized.

Show comment
Hide comment
@boneskull

boneskull Aug 6, 2016

Not in this project anyway, but it's an idea that could use some thinking about. Limiting the experiment to JavaScript...

From my experience, how tests are run seems like a finite list (which grows slowly) of useful strategies. Maybe that's a CLI running tests in a node process; several node processes; maybe instead it launches a browser; maybe instead it hits a Selenium server; or even an Electron instance (recent development!).

Swap CLI with IDE. Or desktop interface. Or web interface (maybe CI servers fit into this bucket).

A user (or process) launches a process. An environment must be determined (or this is decided in advance), as well as the tests to run. Those tests may need preprocessing. The tests are delivered to the environment to be interpreted and executed by the framework. Results are delivered as specified by the user.

I'm not entirely convinced the framework has to be in the job of test execution itself, but that's how it currently stands.

If a team were to clearly define these edges, it could propose a specification for universal interoperability. Developers could mix and match parts (or use "presets") to tailor the testing experience to their specific use case.

Babel is extremely complex, but most projects can just get away with es2015-preset.

So, yeah, huge can of worms there. If anyone's gung-ho about bikeshedding this one, I'll join in :)

Not in this project anyway, but it's an idea that could use some thinking about. Limiting the experiment to JavaScript...

From my experience, how tests are run seems like a finite list (which grows slowly) of useful strategies. Maybe that's a CLI running tests in a node process; several node processes; maybe instead it launches a browser; maybe instead it hits a Selenium server; or even an Electron instance (recent development!).

Swap CLI with IDE. Or desktop interface. Or web interface (maybe CI servers fit into this bucket).

A user (or process) launches a process. An environment must be determined (or this is decided in advance), as well as the tests to run. Those tests may need preprocessing. The tests are delivered to the environment to be interpreted and executed by the framework. Results are delivered as specified by the user.

I'm not entirely convinced the framework has to be in the job of test execution itself, but that's how it currently stands.

If a team were to clearly define these edges, it could propose a specification for universal interoperability. Developers could mix and match parts (or use "presets") to tailor the testing experience to their specific use case.

Babel is extremely complex, but most projects can just get away with es2015-preset.

So, yeah, huge can of worms there. If anyone's gung-ho about bikeshedding this one, I'll join in :)

@boneskull

This comment has been minimized.

Show comment
Hide comment
@boneskull

boneskull Aug 6, 2016

(By "test execution" I mean "the test function" itself. A first pass at the idea should assume the test framework executes the tests itself, but challenging this assumption could be worth exploring later)

(By "test execution" I mean "the test function" itself. A first pass at the idea should assume the test framework executes the tests itself, but challenging this assumption could be worth exploring later)

@flore77

This comment has been minimized.

Show comment
Hide comment
@flore77

flore77 Sep 4, 2016

Contributor

Sounds interesting. There could be multiple smaller projects, like js-reporters, and all could be reunited under the same organization, something like, js-testing-standards, or something along this line. I am up for this 👍

Although we should move this discussion somewhere else, maybe in a new issue on this repo 😃

Contributor

flore77 commented Sep 4, 2016

Sounds interesting. There could be multiple smaller projects, like js-reporters, and all could be reunited under the same organization, something like, js-testing-standards, or something along this line. I am up for this 👍

Although we should move this discussion somewhere else, maybe in a new issue on this repo 😃

@flore77

This comment has been minimized.

Show comment
Hide comment
@flore77

flore77 Sep 4, 2016

Contributor

Fixed in #84.

Contributor

flore77 commented Sep 4, 2016

Fixed in #84.

@flore77 flore77 closed this Sep 4, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment