Introduce mechanism for inter-test dependencies

[radai-rosenblatt]

would be nice to have some way of specifying inter-test dependencies.
this could be between different methods within the same class or between classes.

such a feature is very useful in large codebases with hundreds or thousands of tests. if something really basic breaks it could cause dozens of test classes to fail, and then tracking down the root cause depends on the developer choosing which test to debug - if you pick a complicated one it'll take a while.

with such a feature you could have all the higher-level tests depend on the basic tests so that such a failure will who up 1st on tests reports causing execution to skip the higher-level tests - saving on time and noise.

Related Issues

• Provide mechanism to order the sequence of tests #13
• Introduce first-class support for scenario tests #48

[mlevison]

Radai - why not simply tag some unit tests: core, primary or simple? Then use the command line to run those first. As a Scrum guy (odd person out here), I cringe at maintaining dependencies. I spend my days rooting them out of organizations not adding them back in. Cheers Mark

[radai-rosenblatt]

@mlevison - maintenance and usability nightmare, let me explain:

suppose i have a multi-module project. every module has "layered" test cases - starting at crud tests, then inter-class integration, then component integration etc. the number of "layers" is really up to the developers of every module, and it not fixed.

if i would like to get value out of these tags i'd need to encode them into my build system. at this point the tags are duplicated - they are defined on tests and in the build (which s manual effort to track and maintain in sync).

also - it would not carry over to people building from IDEs. if someone wants to run all tests in a module/project from his IDE he wont get the benefit of a fail-fast build order, he'll get whatever order the IDE wants to run them in.

also, this could work very nicely combined with multi-threaded runners.

[mlevison]

@Radai - key point I'm not on the JUnit team just an interested bystander. Feel free to throw rotten fruit my way. BTW the best rotten fruit is fermented into wine or beer. I understand your need. What we need to consider is how your need will be mis-used most of the time when its used outside of your team. Imagine someone who doesn't deeply understand the JUnit way: - Test A sets up a database state as a side effect of its work - Test B relies on test A and its state. - Test C relies on test A.... ...very rapidly we've made it easy for people to get away from the core point of unit tests (I like Mike Hill's definition: http://anarchycreek.com/2009/05/20/theyre-called-microtests/ ) Is it possible that the new extension point system would allow you handle this without adding anything to the core JUnit? You could achieve your goal and JUnit would stay true to its name? Alternatively, would the Scenario tests that are being described meet your needs? Cheers Mark

[radai-rosenblatt]

anything can be abused. I dont think something potentially being abused is reason not to include it.

dependencies are not really a scenario:

1. they are meant to describe completely different "layers" - so i can have a "crud" test layer, that tests the low level basic storage classes, then a "mid-tier" integration tests that tests my higher-level logic (and might spin up an in-mem DB and crud classes in @before), and on top of that higher-level integration tests that spin up an embedded DB and also an embedded http container and do rest calls.
   so although dependencies imply (crude) ordering they are not an "absolute" order - mid-tier integration would depend on "crud" and "mid-tier", but those 2 can run in parallel.

2. i feel scenario tests are meant for absolute ordering within a single test class - where youre actually building a scenario (user logs in, logged-in user adds item to cart, logged-in-user with cart checks out). it would be very abusive for me to use scenarios to get what i want. also - how do you say "run all tests with label "crud" as a scenario step?

[nipafx]

Potential for Abuse

anything can be abused.

Yes, anything can be abused but the likelihood of it happening and the damage it does depends a lot on what it is and how easy it is to misuse it. It is hence very necessary to look at the potential downsides of any feature.

I dont think something potentially being abused is reason not to include it.

Then you've not been following the development of Java very closely. Literally every feature that goes into the language and libraries is vetted closely by how much it could be abused. Good libraries and frameworks do the same.

So it does make sense for JUnit to consider the potential downsides of a feature and decline to implement it if they seem to weigh heavier than the benefits.

Back to Topic

Dependencies between individual tests across classes seem like a surefire way to get into a tangled mess but as I understand it that is not the proposal. Organizing tests in "layers", on the other hand, is an interesting thought.

It is indeed likely that it does not make sense to run certain tests unless other tests pass. Especially when JUnit is used to write unit as well as integration tests (or however you want to call them). I would also say it's likely that tags are used to express exactly that - it's even in the user guide and there was a matching question on SO.

So it looks like a possible solution would be a way to "layer" tests by tags and (important!) store that configuration in code so that it can get picked up by IDEs and build tools alike.

[radai-rosenblatt]

loose groups (tags) and the ability to specify high level layers of tests as you describe would suit my purpose very well. it would allow the low layers to run 1st and the higher layers to be skipped if the lower ones fail and yet be loose enough to allow concurrency in test execution.

so maybe @tag could be made into a meta-annotation, allowing for stuff like:

package com.acme.dbmodel;

@Tag("basic")
public @interface Basic {
}

package com.acme.restapi;

@Tag("basic")
public @interface Basic {
}

package com.acme.integration;

@Tag("integration")
@DependsOn({
   com.acme.dbmodel.Basic,
   com.acme.restapi.Basic
})
public @interface Integration {
}

[bokysan]

Let me put my kettle to the flame: dependencies do not make a lot of sense when writing unit tests. However, I believe they are essential for integration tests.

Let me explain:

• I agree unit tests should be completely atomic -- e.g. they shouldn't depend on any specific state of the component or previous tests.
• However, integration tests are often not atomic: when running a specific test you likely need to depend on the specific state of the system for the test to be successful.

Typical example would be testing a hypothetical "wallet application":

1. execute a "login" test first
2. execute a "deposit into wallet" test second
3. execute "pay something from the wallet" last

While it doesn't even make sense to run tests (2) and (3) if first one is not successful, the test (2) is a prerequisite for (3).

The only alternative I see would be to put the application into consistent state before every test, which would mean:

1. have a "login" test
2. have a "deposit into wallet" test: first by doing "login" (again), and "deposit into wallet" next
3. have a "pay something from the wallet": "login" (again), "deposit into wallet" (again), and "pay something from wallet" last

This does makes the tests atomic, but introduces several other issues, such as:

• the time it takes to test features is increased exponentially
• if login fails, all tests will fail, which makes it harder to debug and pinpoint the issue

Feel free to down vote this comment but please suggest a more optional way to do integration testing.

I do understand there's a potential to misuse this feature for unit tests, however I'm with @radai-rosenblatt on this one -- anything than can be abused will be abused. But this is solvable. E.g. this could be a feature that's either turned on explicitly (-Djunit5.im-sure-i-know-what-im-doing) and/or a separate JAR that needs to be included manually.

I'd be happy to hear your thoughts.

[mlevison]

Boky - I've got 5 minutes before the start of a class I'm running. Suggestion, use the Junit5 capability for different engines. Implement an integration engine. As separate engine it will be harder people to accidentally misuse. Sorry for the brevity. Cheers Mark

[AutomatedOwl]

Just my two cents - you discussed the implications of such feature on the best practice of unit tests. However, for a quite a long time, junit framework have been used for E2E tests as well, along with related frameworks such as Selenium and Appium, so I guess it should be taken into account while discussing the future of junit and the overall strategy.

[sbrannen]

@bokysan, what you've described sounds exactly like the proposed support for scenario tests (see issue #48).

Or did I misunderstand you?

[bokysan]

Yeah, @sbrannen, that sounds pretty similar.

However, I was following more TestNGs lines of reasoning (http://testng.org/doc/documentation-main.html#dependencies-with-annotations), where you specify conditions that need to be met (=specific tests run successfully) for the test to execute at all.

This lets the framework parallelize tests as it sees fit, as long as it satisfies the decision tree.

[sbrannen]

OK.

Please keep in mind, however, that #48 will address inter-dependencies between methods in the same test class, but it will not address inter-dependencies between methods in different test classes.

This lets the framework parallelize tests as it sees fit, as long as it satisfies the decision tree.

That should also be achievable after #48 has been implemented.

[Chealer]

I agree - it would be better to have a mechanism for specifying dependencies than ordering mechanisms.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. Thank you for your contribution.

[jbduncan]

This issue still sounds potentially useful as an extension to #48.

[igilbert]

#48 proposal does not state the "strong" inter-test dependency, which means when running a step3 test, its dependence tests would run as well.
Considering the scenoria of debugging the step3 test, an ordering mechanism would not run step1-2 tests at all. However, depend mechanism would run step1-2 tests. This is what is badlly needed in integration test.