Exception thrown within Screenplay Task still allows test Process to finish with exit code 0

[nbarrett]

I'm currently in the process of moving a large 'legacy' codebase over from Cucumber/Geb/groovy to an all-java Serenity-powered and eventually screenplay solution. My focus at the moment is to do a 'MVP' and get the project team started with Serenity by:

• Replacing @RunWith(Cucumber.class) with @RunWith(CucumberWithSerenity.class)
• Instantiating the existing Geb Browser by passing it the serenity driver by means of Serenity.getWebdriverManager().getWebdriver()

This is working really well and all the reports are working fine with just those couple of changes!
However, part of the MVP is to see whether Screenplay also plays nicely with the existing codebase - e.g. can we have a mix of legacy steps AND new steps that use the Screenplay pattern, allowing a phased migration to the new patterns? There are around 600 Groovy geb page objects that will take some time to convert into Targets/Tasks/Questions.

That hybrid works too - except that I am having trouble getting Serenity to correctly catch exceptions that are thrown within Screenplay Tasks or Interactions.

The behaviour can be summarised like this:

• When a legacy cucumber step fails the test fails correctly and I get a non-zero exit code (as expected).
• When a Screenplay Task fails, the test failure is detected but the test runner fails with exit code 0 (not as expected).

For my test, I've mocked up a failure case as shown below:

    @Step("{0} creates an account")
    @Override
    public <T extends Actor> void performAs(T actor) {
        if (true) {
            throw new RuntimeException("forcing failure inside screenplay task");
        }
        actor.attemptsTo(
            Close.anyOpenClaimantSessions(),
            Complete.postcodeChecker(),
            Complete.numberOfChildrenQuestion(),
            Complete.andSubmitAccountDetails(),
            Complete.aboutYouDetails(),
            Complete.emailVerification()
        );

    }

Because we are using @RunWith(CucumberWithSerenity.class), screenplay Tasks are called from cucumber steps in the same way as in the todo demo project.
As you can see in the screenshot below, all the steps appear green, despite the console output on the right showing a step failure was detected:

The html test report renders perfectly by the way - it's just the exit code (and the rendering of all-green steps in the IDE runner) that is the problem.

I've been pondering over this one for a few days and am trying to recreate a sample project for you, but haven't been able to yet, so wondering whether there's anything obvious I might be doing wrong? I'm using Serenity 1.5.7 and I've hacked the todo demo project and that 'fails correctly'.

Thanks in advance @wakaleo!

[wakaleo]

Hmmm, what version are you on? I just tried this with 1.6.x and couldn't reproduce.

[nbarrett]

1.5.7 --^

[wakaleo]

That's odd. I can't reproduce with the code above, so maybe a sample project would help.

[nbarrett]

Sure okay - just wondered whether there was anything obvious that I was doing wrong regards configuration...I can't reproduce it either with todo-demo so it'll be a case of cutting down the configuration I have on my client project until I either find out what I've done wrong, or I can reproduce the problem with a GitHub project. I'll report back soon!

[nbarrett]

Creating a standalone example project that can reproduce this error is proving difficult given that my client use buck as their build tool within a monorepo. I've had to manually add the required dependencies into the existing module by manual cross referencing to the serenity pom files and then adding the required dependencies to fix compilation errors and then again at runtime to fix ClassNotFoundExceptions. I'll have a go though. The only additional discovery that I've since my last update is that the 'vanilla' cucumber json file that is produced - e.g. the one that is produced by the following qualifier:

    format = {"html:build/reports/cucumber-intellij",
              "json:build/reports/cucumber-intellij.json"},

does not contain the error message, where as the serenity json file does. I guess the test runner and the IDE cucumber plugin are only aware of the cucumber json file so it looks like a problem with the screenplay type exceptions not being passed to cucumber? Legacy exception ARE sent to the cucumber json. I could try and debug my project but knowing where to put the breakpoint is my issue!

[wakaleo]

If you can isolate the issue into a small test in your current project, put a breakpoint in the StepInterceptor class and see what it is doing.

[nbarrett]

Ah, that helps actually. The problem seems to be because the following methods are returning null:

which causes runTestStep() to return null as follows:

However, when I get a failure from a 'legacy' step I don't even land in these methods, so now trying to work out the entry point (if at all) to StepInterceptor...

[nbarrett]

Found it! It was all down to the fact that I had not called Serenity.throwExceptionsImmediately() at any point in my automation code. and the default value is:

private static boolean throwExceptionsImmediately = false;

I'm not sure why this is - but it seems like the codepath to that method is via SerenityConfigurer as follows:

    public static class SerenityConfigurer {
        public SerenityConfigurer throwExceptionsImmediately() {
            Serenity.throwExceptionsImmediately();
            return this;
        }
    }

I don't call SerenityConfigurer (or even know why I would need to). When I put Serenity.throwExceptionsImmediately(); just before I get my WebDriver instance, everything works fine! I'm not really sure what I should have done differently in my approach @wakaleo - any suggestions please?

[wakaleo]

If I remember correctly, this field is set to different values in Cucumber, JUnit and JBehave to account for their quirks in error handling. It should be a global thing, depending on the test framework, but I remember it being quite fiddly to get right.

[nbarrett]

Actually the runTestStep() behaviour and return values were the exactly the same between the todo demo app and my client project. What was different was the behaviour inside the Actor.perform() where the value of throwExceptionsImmediately affects whether to rethrow the exception.

[szaluk]

@nbarrett @wakaleo

I am seeing a similar issue in our code. We are using JUnit and in one of our tests we have an after method that does cleanup. That method calls some tasks. I noticed that if there is a failure in the task (performAs method) that error does not propagate up and fail the test. I had to call Serenity.throwExceptionsImmediately(); to get it to work.

I am not clear on what is happening though when I don't set throwExceptionsImmediately to true. Why does the failure propagate properly in a test method but not in a configuration (@after) method? Is setting throwExceptionsImmediately true the right thing to do in this situation? I wasn't able to find any documentation on this setting.

Thanks,
Steve