Add support for Teardown, a run-always Finalizer #57
Conversation
We're using btest to run system tests in a rather stateful environment and were looking for a hook / entry point to run after every test for cleanup and state checking. Finalizer almost met the bill, except for not run for failing tests. If running the executable given by Teardown fails, the whole test is considered failed. If Teardown succeeds, but the test failed, the test outcome is still failure.
awelzel
changed the title
Hmm - anything I should be doing about this? Seems @bbannier found that in #56 as well.. |
This was found and fixed by @ckreibich in #56, and we are waiting for a merge for that. |
| ``Teardown`` | ||
| An executable that will be executed after a test has run, | ||
| regardless of whether the test was successful or not. | ||
| It runs in the same directory as the test itself and receives | ||
| the name of the test and the number of failures as its parameters. | ||
| The return value indicates whether the test should indeed be | ||
| considered successful. By default, there's no teardown set. | ||
|
|
There was a problem hiding this comment.
Couple of questions about the semantics here:
- Does the exit value get to override only on success, or also on failure (in other words: should a failing teardown script turn a successful test into a failure?), Also, do you indeed need the change-of-success for your use case? Might be easier semantics to just ignore the exit code so that teardown gets to clean up whatever it needs, but otherwise doesn't impact test execution.
- What's the motivation for passing the number of failed tests so far? That seems ill-defined because one can't know when exactly the current test executes vs others, in particular with
-j.
There was a problem hiding this comment.
Does the exit value get to override only on success, or also on failure (in other words: should a failing teardown script turn a successful test into a failure?)
This is currently implemented. A failing Teardown causes an otherwise successful test to fail.
Also, do you indeed need the change-of-success for your use case? Might be easier semantics to just ignore the exit code so that teardown gets to clean up whatever it needs, but otherwise doesn't impact test execution.
Adding @deejgregor to chime in here. One use-case was to fail a test if it leaves the environment in an unexpected state. The Teardown name might not fit the bill for this.
@bbannier pointed out that there might be better approaches here. I'll mark the MR as RFC.
What's the motivation for passing the number of failed tests so far? That seems ill-defined because one can't know when exactly the current test executes vs others, in particular with -j.
The description isn't great. It's the number of cmd failures (failures variable) a given test had. No actual use-case currently, but could allow the teardown script to take conditional actions depending whether the test had succeeded or failed.
There was a problem hiding this comment.
Capturing the points from an offline discussion I had with @awelzel:
- I believe that if you want to validate that a test didn't leave broken external state you should explicitly test for that. Having this in the BTest test case would be most explicit, but it should also be possible to use a
Finalizerfor that. This wouldn't capture unexpected external state from failing tests, but if a test fails you already know that something is broken. - If you want to validate that each test starts with proper external state you could use an explicit test step or
Initializerto validate or reset the external state. Without knowing too much about the external state you are dealing with, resetting the state before a test seems superior as it would allow other tests to run after a prior test has broken the state.
An additional Teardown step introduces more places where e.g., a test's data or state could get mutated (in addition to the existing Finalizer), and I still don't see a clear cut use case for Teardown which would outweigh the complexity it would introduce. I already am slightly biased against Initializer/Finalizer since they seem to move test code (they can fail a test) out of the actual test case making it harder to follow what is happening; they also seem overly broad for something which can decide a test's status, but would be applied to all test cases. Teardown would add more of such functionality.
|
Fwiw, I just hit a scenario where I'd quite like an unconditional setup and teardown. I'm running btests on a |
|
Since it is unclear whether we actually want to solve the problem here let's close this for now. |
We're using btest to run system tests in a rather stateful environment
and were looking for a hook / entry point to run after every test for
cleanup and state checking.
Finalizer almost met the bill, except for not run for failing tests.
If running the executable given by Teardown fails, the whole test is
considered failed. If Teardown succeeds, but the test failed, the
test outcome is still failure.