Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

RunWith support runs test suites #14

Closed
szeiger opened this Issue · 21 comments

8 participants

Stefan Zeiger diwa Jack Ganzha Ben McCann CHI-Sean Josh Suereth Peter Vlugter Rich Dougherty
Stefan Zeiger
Owner

The suite runner should be exempt from RunWith support to prevent tests in suites from running multiple times.

diwa

+1 for this.
For testing Play, I followed the helpful guide at http://livinginaplay.wordpress.com/2012/09/26/running-junit-test-suites-with-play-2-0/ and this is the last bit in making the experience seamless

Stefan Zeiger
Owner

@diwa: Using the suite runner with the new RunWith support makes test suites "work" in the first place, so your +1 is really arguing for breaking that Play support again ;-)

But rest assured that I have no intention of removing it without an option to enable it. For better control over what tests to run, please use 0.10-M1 which allows you to specify glob patterns against the individual test names. This pretty much obviates the need for the -tests option and sbt's own glob pattern support in the test-only command.

Jack Ganzha

Not sure if this issue is only related to @RunWith + suite runner, but I have tried to implement my own runner (to support some initializations that Play framework needs in order to run tests) and now all my tests are running twice. Here is the runner:

package test;

import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runner.notification.RunNotifier;
import org.junit.runners.model.InitializationError;

import play.test.Helpers;
import play.test.FakeApplication;

public class FakeApplicationRunner extends BlockJUnit4ClassRunner {

  public static FakeApplication app;

  public FakeApplicationRunner(Class<?> clazz) throws InitializationError {
    super(clazz);
  }

  public void run(final RunNotifier notifier) {
    app = Helpers.fakeApplication();
    Helpers.start(app);
    super.run(notifier);
    Helpers.stop(app);
  }
}

As far as I know, this is the recommend way to create a simple runner. Am I doing something wrong?

Jack Ganzha

I forgot to mention that that I'm trying both 0.10-M1and 0.9 versions of junit-interface.

Stefan Zeiger
Owner

@jackganzha How are you using that runner? You will get tests running twice if you have a runner that runs tests in a different class which is also discovered as a test class individually. With the current test interface for sbt, there is no way to avoid that (because all classes are discovered and run individually by sbt -- there is no concept of a test run that would allow you to examine all tests up-front and then decide what to run).

That makes me think that the feature for ignoring test suites should actually be more general. What we need is a configuration option with a list of Runner classes to ignore. Its default value would be the Suite runner.

Stefan Zeiger
Owner

Fixed in bb96bdd

Stefan Zeiger szeiger closed this
Jack Ganzha

@szeiger not sure about your question, but here is how I'm using the Runner:

The parent class for all my tests

@RunWith(FakeApplicationRunner.class)
public abstract class FixturesTest {
   ... // do some initialization using @BeforeClass/@AfterClass annotations
}

And the test itself:

public class UserTest extends test.FixturesTest {
   ... // plain old junit test using @Test annotation
}

If I correctly understand what you are saying, the child tests is detected twitter both by sbt and also because the runner. Is that correct?

Stefan Zeiger
Owner
Stefan Zeiger szeiger reopened this
Jack Ganzha

Hello @szeiger

Any news about this issue? Also, probably not related, but I tried to upgrade junit-interface to the most recent version in a Play project and suddenly the test-only command stops to work. I'm using (in sbt console):

 ~ test-only models.*

Is that stills right?

Stefan Zeiger
Owner

I've added a test case to junit-interface-test (szeiger/junit-interface-test@237b14a) but I cannot reproduce the problem. InheritedRunnerTest gets run only once, by the inherited FakeApplicationRunner.

Regarding tests not running any more in 0.10-M2: This version disables test suites again by default (which were enabled as a side-effect of adding RunWith support in 0.9). Check the --ignore-runners option.

Ben McCann

Here's a sample test that is running twice. It has @RunWith on the parent class, so that definitely sounds like the case you were wanting to reproduce:
playframework/playframework#1667
https://gist.github.com/asangamanage/6557758

CHI-Sean

This is a really old issue, but all of my test cases are being run twice due to this bug. Will it ever be fixed?

Ben McCann

@chi-shared is reporting this issue still with 0.11-RC1 which is what Play 2.3 uses: playframework/playframework#1667 (comment)

Ben McCann

@szeiger @jsuereth do you think there's a way we could add a test for this? maybe with scripted or something. i think that this gist would be a good test case for reproducing and making sure it's solved: https://gist.github.com/asangamanage/6557758

CHI-Sean
Josh Suereth
Owner
Jeff Wright j3ffw referenced this issue from a commit
Jeff Wright Add scripted test for #14 17efb20
Jeff Wright j3ffw referenced this issue from a commit in j3ffw/junit-interface
Jeff Wright Add scripted test for sbt#14 6835a4a
Peter Vlugter
Owner

I've been looking at this from the Play perspective. It certainly looks like sbt filters out test definitions with the same name:

https://github.com/sbt/sbt/blob/0.13.5/testing/src/main/scala/sbt/TestFramework.scala#L150

But Play sets fork in Test := true by default and it looks like the same 'distinct by name' filtering is not applied to forked tests. In this sample project:

https://github.com/retroryan/testPhantomJS

one of the tests runs twice. It doesn't if fork is disabled.

In sbt the 'distinct by name' could be moved to Tests.processOptions:

https://github.com/sbt/sbt/blob/0.13.5/main/actions/src/main/scala/sbt/Tests.scala#L114

which is also used by forked tests.

Ben McCann

The sample project that I got from one of the Play users is fixed by setting fork in Test := false so looks like you fix solved the problem for the case I was aware of! Here's the repro case https://github.com/benmccann/junit-interface-runwith

Jeff Wright j3ffw referenced this issue from a commit in j3ffw/junit-interface
Jeff Wright Add scripted test for sbt#14 2f5a46f
Ben McCann

We just need to merge #52 and then can mark this as fixed

Ben McCann

This can be closed now. It works with sbt 0.13.6-M2. See #58

Rich Dougherty

Reported fixed: #30 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.