Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

RunWith support runs test suites #14

szeiger opened this Issue Oct 8, 2012 · 21 comments


None yet
8 participants

szeiger commented Oct 8, 2012

The suite runner should be exempt from RunWith support to prevent tests in suites from running multiple times.

diwa-zz commented Oct 18, 2012

+1 for this.
For testing Play, I followed the helpful guide at http://livinginaplay.wordpress.com/2012/09/26/running-junit-test-suites-with-play-2-0/ and this is the last bit in making the experience seamless


szeiger commented Oct 18, 2012

@Diwa: Using the suite runner with the new RunWith support makes test suites "work" in the first place, so your +1 is really arguing for breaking that Play support again ;-)

But rest assured that I have no intention of removing it without an option to enable it. For better control over what tests to run, please use 0.10-M1 which allows you to specify glob patterns against the individual test names. This pretty much obviates the need for the -tests option and sbt's own glob pattern support in the test-only command.

Not sure if this issue is only related to @RunWith + suite runner, but I have tried to implement my own runner (to support some initializations that Play framework needs in order to run tests) and now all my tests are running twice. Here is the runner:

package test;

import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runner.notification.RunNotifier;
import org.junit.runners.model.InitializationError;

import play.test.Helpers;
import play.test.FakeApplication;

public class FakeApplicationRunner extends BlockJUnit4ClassRunner {

  public static FakeApplication app;

  public FakeApplicationRunner(Class<?> clazz) throws InitializationError {

  public void run(final RunNotifier notifier) {
    app = Helpers.fakeApplication();

As far as I know, this is the recommend way to create a simple runner. Am I doing something wrong?

I forgot to mention that that I'm trying both 0.10-M1and 0.9 versions of junit-interface.


szeiger commented Oct 22, 2012

@jackganzha How are you using that runner? You will get tests running twice if you have a runner that runs tests in a different class which is also discovered as a test class individually. With the current test interface for sbt, there is no way to avoid that (because all classes are discovered and run individually by sbt -- there is no concept of a test run that would allow you to examine all tests up-front and then decide what to run).

That makes me think that the feature for ignoring test suites should actually be more general. What we need is a configuration option with a list of Runner classes to ignore. Its default value would be the Suite runner.


szeiger commented Oct 22, 2012

Fixed in bb96bdd

@szeiger szeiger closed this Oct 22, 2012

@szeiger not sure about your question, but here is how I'm using the Runner:

The parent class for all my tests

public abstract class FixturesTest {
   ... // do some initialization using @BeforeClass/@AfterClass annotations

And the test itself:

public class UserTest extends test.FixturesTest {
   ... // plain old junit test using @Test annotation

If I correctly understand what you are saying, the child tests is detected twitter both by sbt and also because the runner. Is that correct?


szeiger commented Oct 23, 2012

Ah,the annotation is inherited. Maybe that's a different issue

Sent from my Android phone with K-9 Mail. Please excuse my brevity.

@szeiger szeiger reopened this Oct 23, 2012

Hello @szeiger

Any news about this issue? Also, probably not related, but I tried to upgrade junit-interface to the most recent version in a Play project and suddenly the test-only command stops to work. I'm using (in sbt console):

 ~ test-only models.*

Is that stills right?


szeiger commented Nov 23, 2012

I've added a test case to junit-interface-test (szeiger/junit-interface-test@237b14a) but I cannot reproduce the problem. InheritedRunnerTest gets run only once, by the inherited FakeApplicationRunner.

Regarding tests not running any more in 0.10-M2: This version disables test suites again by default (which were enabled as a side-effect of adding RunWith support in 0.9). Check the --ignore-runners option.


benmccann commented Sep 14, 2013

Here's a sample test that is running twice. It has @runwith on the parent class, so that definitely sounds like the case you were wanting to reproduce:

This is a really old issue, but all of my test cases are being run twice due to this bug. Will it ever be fixed?


benmccann commented Jun 19, 2014

@chi-shared is reporting this issue still with 0.11-RC1 which is what Play 2.3 uses: playframework/playframework#1667 (comment)


benmccann commented Jun 19, 2014

@szeiger @jsuereth do you think there's a way we could add a test for this? maybe with scripted or something. i think that this gist would be a good test case for reproducing and making sure it's solved: https://gist.github.com/asangamanage/6557758

Hi Ben,

I reported this on an open issue on the play side as well. I am using Play 2.3.



jsuereth commented Jun 21, 2014

Sure, a test would be highly welcome.

@j3ffw j3ffw pushed a commit to j3ffw/junit-interface that referenced this issue Jul 11, 2014

Jeff Wright Add scripted test for sbt#14 6835a4a

pvlugter commented Jul 12, 2014

I've been looking at this from the Play perspective. It certainly looks like sbt filters out test definitions with the same name:


But Play sets fork in Test := true by default and it looks like the same 'distinct by name' filtering is not applied to forked tests. In this sample project:


one of the tests runs twice. It doesn't if fork is disabled.

In sbt the 'distinct by name' could be moved to Tests.processOptions:


which is also used by forked tests.


benmccann commented Jul 17, 2014

The sample project that I got from one of the Play users is fixed by setting fork in Test := false so looks like you fix solved the problem for the case I was aware of! Here's the repro case https://github.com/benmccann/junit-interface-runwith

@j3ffw j3ffw pushed a commit to j3ffw/junit-interface that referenced this issue Jul 18, 2014

Jeff Wright Add scripted test for sbt#14 2f5a46f

benmccann commented Aug 2, 2014

We just need to merge #52 and then can mark this as fixed

@eed3si9n eed3si9n added a commit that referenced this issue Aug 16, 2014

@eed3si9n eed3si9n Merge pull request #52 from j3ffw/tests-run-once
Add scripted test for #14

benmccann commented Aug 16, 2014

This can be closed now. It works with sbt 0.13.6-M2. See #58

Reported fixed: #30 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment