Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SortMethodsWith allows the user to choose the order of execution of the methods within a test class #386

Merged
merged 3 commits into from Apr 9, 2012

Conversation

@matthewfarwell
Copy link
Contributor

commented Feb 19, 2012

The default order of execution of JUnit tests within a class
is deterministic but not predictable. Before 4.11, the
behaviour was to run the test methods in byte code order,
which pre-Java 7 was mostly predictable. Java 7 (and some
previous versions), does not guaranteee the order of execution,
which can change from run to run, so a deterministic sort was introduced.
As a rule, test method execution should be independent of
one another. However, there may be a number of dependent
tests either through error or by design. This class
allows the user to specify the order of execution of test methods.

There are four possibilities:

MethodSorters.DEFAULT: the default value, deterministic, but not predictable
MethodSorters.JVM: the order in which the tests are returned by the JVM, i.e. there is no sorting done
MethodSorters.NAME_ASC: sorted in order of method name, ascending
MethodSorters.NAME_DESC: sorter in order of method name, descending

This is an enhancement to the solution for #293 Sort test methods for predictability, in order that users can temporarily 'fix' their broken tests, from https://github.com/KentBeck/junit/pull/293#issuecomment-2095777

SortMethodsWith allows the user to choose the order of execution of t…
…he methods within a test class.

The default order of execution of JUnit tests within a class
is deterministic but not predictable. Before 4.11, the
behaviour was to run the test methods in byte code order,
which pre-Java 7 was mostly predictable. Java 7 (and some
previous versions), does not guaranteee the order of execution,
which can change from run to run, so a deterministic sort was introduced.
As a rule, test method execution should be independent of
one another. However, there may be a number of dependent
tests either through error or by design. This class
allows the user to specify the order of execution of test methods.

There are four possibilities:

MethodSorters.DEFAULT: the default value, deterministic, but not predictable
MethodSorters.JVM: the order in which the tests are returned by the JVM, i.e. there is no sorting done
MethodSorters.NAME_ASC: sorted in order of method name, ascending
MethodSorters.NAME_DESC: sorter in order of method name, descending
@jglick

This comment has been minimized.

Copy link
Contributor

commented Mar 2, 2012

Consider adding a mode BYTECODE which would match what users normally expect - order of methods as produced by javac - but without relying on the nondeterminism of Class.getDeclaredMethods. My b40493d (amended with 785a33b) shows that this is not particularly hard.

@dsaff

This comment has been minimized.

Copy link
Member

commented Mar 28, 2012

@matthewfarwell, are you working on responses to the requests here? Thanks.

@matthewfarwell

This comment has been minimized.

Copy link
Contributor Author

commented Mar 28, 2012

Yes, but give me a couple of days.

@matthewfarwell

This comment has been minimized.

Copy link
Contributor Author

commented Apr 4, 2012

@kcooney Thanks for the feedback Kevin (Cooney). I've pushed another commit with changes.

@matthewfarwell

This comment has been minimized.

Copy link
Contributor Author

commented Apr 4, 2012

@dsaff I've updated the code, could you please review what's been done.

There are two things for you to decide:

  1. Do you wish to create a standard interface for MethodSorter that anyone can implement, or leave it as I've written it? (see https://github.com/KentBeck/junit/pull/386#commitcomment-1011880)
  2. Do you want to add a BYTECODE as well, which would mean bytecode parsing? (see https://github.com/KentBeck/junit/pull/386#issuecomment-4286764)

If all is OK, I'll do the merge with master and resubmit the pull.

* Here is an example:
*
* <pre>
* &#064;SortMethodsWith(MethodSorters.NAME_ASC)

This comment has been minimized.

Copy link
@dsaff

dsaff Apr 6, 2012

Member

NAME_ASCENDING, right?

This comment has been minimized.

Copy link
@matthewfarwell

matthewfarwell Apr 6, 2012

Author Contributor

Yep, you're right. Changed.

void testOne() {}

@Test public void testNameAsc() {
assertEquals("[java.lang.Object alpha(int,double,java.lang.Thread), void beta(int[][]), void delta(), void epsilon(), int gamma(), void gamma(boolean)]", declaredMethods(DummySortWithNameAsc.class));

This comment has been minimized.

Copy link
@dsaff

dsaff Apr 6, 2012

Member

Can we write these tests in a way that doesn't end up with the lines so long?

This comment has been minimized.

Copy link
@matthewfarwell

matthewfarwell Apr 6, 2012

Author Contributor

Refactored to shorten lines, and reduce duplication at the same time.

@dsaff

This comment has been minimized.

Copy link
Member

commented Apr 6, 2012

Thanks for the work here, @matthewfarwell,

  1. We already have a Sorter interface, although we could use a better way to use it declaratively, which is a subject for a different pull request. Two general-purpose sorting methods could get very confusing. Instead, just to avoid confusion, maybe could we rename the annotation to @FixMethodOrder?

  2. It would take quite a deafening chorus of user feedback to convince me this project should take on the liabilities of the code complexity of the suggested BYTECODE.

A separate question: do we really think anyone will use NAME_DESCENDING?

@matthewfarwell

This comment has been minimized.

Copy link
Contributor Author

commented Apr 6, 2012

Annotation renamed to @FixMethodOrder.

Removed NAME_DESCENDING (you're right, noone will ever use this).

Not going to implement BYTECODE.

When people feel this is ready, I'll do a squash and merge with master.

dsaff added a commit that referenced this pull request Apr 9, 2012

Merge pull request #386 from matthewfarwell/sortwith
SortMethodsWith allows the user to choose the order of execution of the methods within a test class

@dsaff dsaff merged commit 560322d into junit-team:master Apr 9, 2012

@dsaff

This comment has been minimized.

Copy link
Member

commented Apr 9, 2012

Went ahead and merged. Thanks for the hard work!

@dsaff

This comment has been minimized.

Copy link
Member

commented on 7924760 May 18, 2012

Matthew,

Is it possible this is related to this bug report?

https://github.com/KentBeck/junit/issues/429

@schlm3

This comment has been minimized.

Copy link

commented Jun 15, 2012

@matthewfarwell Your last change to MethodSorters causes a wrong Javadoc-Entry for the JVM-Sorter:

/** Sorts the test methods by the method name, in reverse lexicographic order */
    JVM(null),
@matthewfarwell

This comment has been minimized.

Copy link
Contributor Author

commented Jun 16, 2012

@schim3 this has since been fixed elsewhere: It is now:

/** Leaves the test methods in the order returned by the JVM.
* Note that the order from the JVM may vary from run to run
*/

@KedneckInc

This comment has been minimized.

Copy link

commented Dec 1, 2013

I know I'm getting into this discussion way late, but...

I'd like to argue for having the option to run the tests in the order they're specified in the file. It's not that I'm writing tests that depend on that order, but that I put them in a specific order so that low level methods are tested and confirmed working before higher order methods that depend on the low order methods are tested. It makes it much easier for me to fix any issues that crop up, because I can depend on seeing the low order method tests at the beginning of the junit report, and not mixed up in everything else.

Now, I could do that by using the name ascending order, but that leads to artificial additions to test names that have to change every time I want to change the order.

test_001_foo()
test_002_bar()
test_003_baz()

Oops, it turns out 'bar' needs a method tested in 'baz'. Now I not only have to move the methods around, but I have to change their names too? Bleah.

@kcooney

This comment has been minimized.

Copy link
Member

commented Dec 1, 2013

Java doesn't provide a guaranteed way to get the order methods are declared in a class

@KedneckInc

This comment has been minimized.

Copy link

commented Dec 1, 2013

Thanks for your response. I guess I'm stuck with the artificial numbering
scheme.

On Sat, Nov 30, 2013 at 11:12 PM, Kevin Cooney notifications@github.comwrote:

Java doesn't provide a guaranteed way to get the order methods are
declared in a class


Reply to this email directly or view it on GitHubhttps://github.com//pull/386#issuecomment-29566988
.

@kcooney

This comment has been minimized.

Copy link
Member

commented Dec 1, 2013

@KedneckInc Have you considered using @RunWith(Enclosed.class)? You can put the "low order" tests in one nested class and the "high order" tests in another.

@bechte

This comment has been minimized.

Copy link
Contributor

commented Dec 1, 2013

@KedneckInc @kcooney and just not to forget the new HierarchicalContextRunner, which will allow you to create whole hierarchies of test. Normally, you'd test hierarchies with it, but it also fits perfectly for your use-case.

You find this runner on https://github.com/bechte/junit-hierarchicalcontextrunner/wiki and it can be easily added as a maven dependency to your project. Your test will look like this:

@RunWith(HierarchicalContextRunner.class)
public class FeatureTest {
  @Before
  public void setUp() {
    // like before
  }

  // Define a new "context" for higher level details
  public class HigherLevelDetails {
    //... test like normal...
  }

  // Define a new "context" for lower level details
  public class LowerLevelDetails {
    //... test like normal...
  }
}

Give it a try :-) and if you like let me know how it worked out for you.

All the best,

bechte

@KedneckInc

This comment has been minimized.

Copy link

commented Dec 30, 2013

@bechte Thanks for the suggestion! I'm still working out the best way to use it in my scenario, but it's definitely running the low level tests before the high level tests. At the least, that's the way the report is displayed in Eclipse. I've actually got 7 nested classes. I have run into an occasional problem, but I'm not sure if it's the variant of Eclipse I'm using (Spring Tool Suite), the way I'm using the hierarchy, or something else. It's not easily repeatable either, so no report on it at this time. One thing I need to try is getting the methods within the classes sorted alphabetically so that they happen in the same order every time. Just to keep me from getting lost when I run another test and the method results have rearranged ... again.

Can I use the

@FixMethodOrder(MethodSorters.NAME_ASCENDING)

attribute on each of the sub classes? Having it at the outermost class doesn't seem to affect the nested classes. At least, thats what I think I'm seeing.

Just to make things interesting, I've been using a Perl script to generate the java test code from a table that describes all the tests, their expected outputs, and the inputs to use for each test. That's working pretty good. With over a hundred tests, having the test code automatically generated seemed a good idea. At least that way I could be certain that all the tests are written identically.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.