AfterAll hook

[wjlroe]

Summary

Carrying on the work done in #858. This pull request aims to add a hook to replace using Kernel#at_exit for the end of a cucumber suite's cleanup tasks.

Motivation and Context

The problem with using Kernel#at_exit for test suite cleanup tasks is that it is very easy to accidentally clobber the exit status of the test suite. If you call any code that throws (and catches) exceptions, that will effectively overwrite the $ERROR_INFO value and turn an error status into a success status, which you likely don't want.

Due to the tricky way in which developers have to treat Kernel#at_exit, it would be preferable for cucumber to provide a hook for the same use cases except the exit status of the test suite would be preserved - anything that happens in the hook, you can't "break" the expected behaviour of cucumber.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

• I've added acceptance tests for my code
• I've added specs for my code
• I've completed the implementation for this feature
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• The name of the hook has been discussed and agreed upon.

[wjlroe]

My next action is to dive into the specs to drive these tests closer to the implementation, it's not going to be super quick because I'm working on this in fits and bursts here and there.

[mattwynne]

👍 @wjlroe

Feel free to ping me here on in gitter if you need some help finding your way around. There are some fairly gnarly bits of code you'll be getting into.

[mattwynne]

How are you getting on @wjlroe? Need any help?

[wjlroe]

@mattwynne I've just had a try at implementing this hook. It doesn't pass the tests, but you can see it is kinda working. I don't really understand what's going on though, so some pointers would be appreciated. The output from the AfterAll hooks doesn't get included in the output captured when the feature tests are run (but the puts code does execute), I don't know why that is. When I have a better understanding of all this, I'll add more specs, but right now this is exploratory code.

[brasmusson]

This should be moved up one line, so that the test_run_finished event means "all userland code that will be executed has been executed".

[brasmusson]

I have argued elsewhere that it should be possible by listening to the event stream exactly what userland code is executed (which would include AfterAll hooks). That means that we need new xxStarted and xxFinished events for these hooks that are not TestSteps (which Before, After and AfterStep hooks are), that are fired similar to the TestStepStarted and TestStepFinished events for TestSteps

When an AfterAll hook fails (an exception escapes it) I expect that the exit code in non-zero and the exceptions is captured by the formatters (by listening to the new xxFinished event).

[mattwynne]

Good point Bjorn. I wonder if we could abstract the name 'test step' so that it could cover steps in scenarios, before / after scenario hooks, and these new after all hooks?

[andrii-rymar]

@wjlroe do you have any plans to continue working on this?

[wjlroe]

I don't know what to do, the discussion above doesn't mean anything to me, and I'm not likely to get any time to work on this, so no.

[mattwynne]

Thanks for your efforts so far @wjlroe and sorry we lost you in the feedback discussion. I'm sure we can take it from here.

[gewargise]

Just wondering if there were any updates regarding an AfterAll hook?

[alexttransisland]

is it still valid? I was able to use

AfterConfiguration do |config|
  config.on_event :test_run_finished do |event|
    config.out_stream.puts "test run finished"
  end
end

with 3.0.0.pre.2 version.
Isn't it the same?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

[stale]

This issue has been automatically closed because of inactivity. You can support the Cucumber core team on opencollective.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.