Test failure cannot be detected

[nbigaouette]

I want to run my cucumber tests and fail the whole execution if a step fails.

Right now, there does not seem to be the case; Looking at Cucumber::run() the steps are simply executed in a while loop:
https://github.com/bbqsrc/cucumber-rust/blob/7140ba0d7314cd8863d7c290e8e31c2a/src/cucumber.rs#L145-L158

In my code base I isolate my call to Cucumber::run() in a separate crate (where all my steps/tests are defined) and re-use them in either a CLI or a webapp. This allows me to run some integration tests either locally or side-by-side with the system being tested (required due to auth issues).

Right now, I realize that I have no way of detecting if a test failed or not without looking at stdout/err, which makes running in service not as useful as it should be.

Searching the code base I found commits relating to exit code, for example 6bee788. This one uses std::process:exit(1) which is not what I need, but it seems the intent was to propagate the error to the caller.

I think the simplest would be to let Cucumber::run() return a Result<> instead of nothing? That way one can unwrap/assert/check/... the actual result of the test.

[bbqsrc]

I don't have time to look into this but I'd be happy to receive an experimental pull request that solves your problem then discuss the appropriate API changes to make it generally useful. 🙂

[bbqsrc]

So my understanding of your requirements based on your example PR is to be able to programmatically determine the outcome of a test run. Entirely reasonable.

Would it not make more sense for us to return an object similar to https://doc.rust-lang.org/std/process/struct.Output.html?

A standard "Output" object could contain total tests run, total errors, etc, with a basic success boolean for if anything failed, and then the complex data if you want to dig.

We could add a run_and_exit function that returns nothing, but exits the process with 1 if the success value is false. Does this sound good?

[nbigaouette]

Returning some kind of Output makes a lot of sense. I used a Result in the PR to show a success/failure dichotomy but it's tricky, for example how to treat skipped steps. Or timeout? I think it is hard to encode these multiple states in a Result.

I would be wary of using std::process::exit as it is quite "extreme". In that specific case returning a Result would be the thing to do I think.

So one function that returns an Output that allows the user to choose/judge how to handle the results and one that returns a Result where cucumber takes the decision itself about the success/failure. The former is useful, the latter I am not sure.

[bbqsrc]

To clarify, I proposed changing the return type of run to return this "Output" object, and add an additional convenience wrapper for the test runner that automatically returns an exit code on failure, which would be called run_and_exit.

[nbigaouette]

That's perfect! :D

[bbqsrc]

This is in main now. Will be in v0.8. :)