-
-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Introduce mechanism for inter-test dependencies #607
Comments
Radai - why not simply tag some unit tests: core, primary or simple? Then use the command line to run those first.
As a Scrum guy (odd person out here), I cringe at maintaining dependencies. I spend my days rooting them out of organizations not adding them back in.
Cheers
Mark
|
@mlevison - maintenance and usability nightmare, let me explain: suppose i have a multi-module project. every module has "layered" test cases - starting at crud tests, then inter-class integration, then component integration etc. the number of "layers" is really up to the developers of every module, and it not fixed. if i would like to get value out of these tags i'd need to encode them into my build system. at this point the tags are duplicated - they are defined on tests and in the build (which s manual effort to track and maintain in sync). also - it would not carry over to people building from IDEs. if someone wants to run all tests in a module/project from his IDE he wont get the benefit of a fail-fast build order, he'll get whatever order the IDE wants to run them in. also, this could work very nicely combined with multi-threaded runners. |
@Radai - key point I'm not on the JUnit team just an interested bystander.
Feel free to throw rotten fruit my way. BTW the best rotten fruit is
fermented into wine or beer.
I understand your need. What we need to consider is how your need will be
mis-used most of the time when its used outside of your team.
Imagine someone who doesn't deeply understand the JUnit way:
- Test A sets up a database state as a side effect of its work
- Test B relies on test A and its state.
- Test C relies on test A....
...very rapidly we've made it easy for people to get away from the core
point of unit tests (I like Mike Hill's definition:
http://anarchycreek.com/2009/05/20/theyre-called-microtests/ )
Is it possible that the new extension point system would allow you handle
this without adding anything to the core JUnit? You could achieve your goal
and JUnit would stay true to its name?
Alternatively, would the Scenario tests that are being described meet your
needs?
Cheers
Mark
|
anything can be abused. I dont think something potentially being abused is reason not to include it. dependencies are not really a scenario:
|
Potential for Abuse
Yes, anything can be abused but the likelihood of it happening and the damage it does depends a lot on what it is and how easy it is to misuse it. It is hence very necessary to look at the potential downsides of any feature.
Then you've not been following the development of Java very closely. Literally every feature that goes into the language and libraries is vetted closely by how much it could be abused. Good libraries and frameworks do the same. So it does make sense for JUnit to consider the potential downsides of a feature and decline to implement it if they seem to weigh heavier than the benefits. Back to TopicDependencies between individual tests across classes seem like a surefire way to get into a tangled mess but as I understand it that is not the proposal. Organizing tests in "layers", on the other hand, is an interesting thought. It is indeed likely that it does not make sense to run certain tests unless other tests pass. Especially when JUnit is used to write unit as well as integration tests (or however you want to call them). I would also say it's likely that tags are used to express exactly that - it's even in the user guide and there was a matching question on SO. So it looks like a possible solution would be a way to "layer" tests by tags and (important!) store that configuration in code so that it can get picked up by IDEs and build tools alike. |
loose groups (tags) and the ability to specify high level layers of tests as you describe would suit my purpose very well. it would allow the low layers to run 1st and the higher layers to be skipped if the lower ones fail and yet be loose enough to allow concurrency in test execution. so maybe @tag could be made into a meta-annotation, allowing for stuff like: package com.acme.dbmodel;
@Tag("basic")
public @interface Basic {
}
package com.acme.restapi;
@Tag("basic")
public @interface Basic {
}
package com.acme.integration;
@Tag("integration")
@DependsOn({
com.acme.dbmodel.Basic,
com.acme.restapi.Basic
})
public @interface Integration {
} |
Let me put my kettle to the flame: dependencies do not make a lot of sense when writing unit tests. However, I believe they are essential for integration tests. Let me explain:
Typical example would be testing a hypothetical "wallet application":
While it doesn't even make sense to run tests (2) and (3) if first one is not successful, the test (2) is a prerequisite for (3). The only alternative I see would be to put the application into consistent state before every test, which would mean:
This does makes the tests atomic, but introduces several other issues, such as:
Feel free to down vote this comment but please suggest a more optional way to do integration testing. I do understand there's a potential to misuse this feature for unit tests, however I'm with @radai-rosenblatt on this one -- anything than can be abused will be abused. But this is solvable. E.g. this could be a feature that's either turned on explicitly ( I'd be happy to hear your thoughts. |
Boky - I've got 5 minutes before the start of a class I'm running.
Suggestion, use the Junit5 capability for different engines. Implement an
integration engine. As separate engine it will be harder people to
accidentally misuse.
Sorry for the brevity.
Cheers
Mark
|
Just my two cents - you discussed the implications of such feature on the best practice of unit tests. However, for a quite a long time, junit framework have been used for E2E tests as well, along with related frameworks such as Selenium and Appium, so I guess it should be taken into account while discussing the future of junit and the overall strategy. |
Yeah, @sbrannen, that sounds pretty similar. However, I was following more TestNGs lines of reasoning (http://testng.org/doc/documentation-main.html#dependencies-with-annotations), where you specify conditions that need to be met (=specific tests run successfully) for the test to execute at all. This lets the framework parallelize tests as it sees fit, as long as it satisfies the decision tree. |
OK. Please keep in mind, however, that #48 will address inter-dependencies between methods in the same test class, but it will not address inter-dependencies between methods in different test classes.
That should also be achievable after #48 has been implemented. |
I agree - it would be better to have a mechanism for specifying dependencies than ordering mechanisms. |
This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. Thank you for your contribution. |
This issue still sounds potentially useful as an extension to #48. |
#48 proposal does not state the "strong" inter-test dependency, which means when running a step3 test, its dependence tests would run as well. |
would be nice to have some way of specifying inter-test dependencies.
this could be between different methods within the same class or between classes.
such a feature is very useful in large codebases with hundreds or thousands of tests. if something really basic breaks it could cause dozens of test classes to fail, and then tracking down the root cause depends on the developer choosing which test to debug - if you pick a complicated one it'll take a while.
with such a feature you could have all the higher-level tests depend on the basic tests so that such a failure will who up 1st on tests reports causing execution to skip the higher-level tests - saving on time and noise.
Related Issues
The text was updated successfully, but these errors were encountered: