Override Serenity Step Exception Handling?

[bensashi]

I work on a project that has built up a large library of tests over the course of 4 or 5 years which are utilizing many Serenity step libraries and page objects. More recently, we have started to get requests to use our automated testing project to aid in large data staging efforts, as opposed to just for testing purposes.

To help illustrate, the system that we are using Serenity to test is basically case management software. For our tests, we have step functions that are used for creating a case and performing various functions within the context of a case. We are now being asked to do things like "create X number of cases" matching certain criteria.

I have been attempting to use write these staging scripts by essentially building loops to call some specific sequence of step functions over and over (until the desired number of records has been created). One thing that we have had to account for is that the likelihood of an exception occurring during execution of these scripts is higher than during execution of a single test, if for no other reason than they end up taking much more time to execute, and that increases the probability of an error occurring.

To account for this, I have tried (no pun intended) using try/catch blocks to recover from exceptions and move on. If the script is trying to create 50 records, and some unexpected UI issue prevents record 27 from being created successfully, the desired result is that the script breaks out of that loop iteration and starts creating record 28. For our needs, it is okay if a few exceptions occur, as long as the script can continue to run.

What I have found, however, is that I don't think this is feasible if the exception occurs within the context of a Serenity step function. It seems like Serenity will handle the exception itself and that step functions called in subsequent iterations of the loop are skipped. I understand from seeing other threads that this is probably expected behavior due to the reporting that Serenity does (having to run the remaining steps in dry-run mode).

What I'm wondering is: is there a solution that will allow us to utilize our existing step functions to perform UI interactions in the web application but leave the responsibility of handling any exceptions to our own code? We don't need the Serenity report for these data staging requests, so if there is some simple flag that will say "don't generate a report" and that would solve this problem, I think that would be a solution for us. Alternately, if there is some obvious solution that is better than trying to have large loops within a single JUnit test function, I'm totally open to that.

Thank you in advance.

[globalworming]

I am not sure your approach is a good one. my feeling is that using the UI to create a big batch of test data is a bad idea, especially when you say " unexpected UI issue". There shouldnt be many unexpected issues and the expected ones you should be able to address. There should be a very high reliability and if that's not given.. go fix it. Or use a more reliable interface. There probably is an API, the browser has to send requests somewhere :)

[bensashi]

I am not sure your approach is a good one. my feeling is that using the UI to create a big batch of test data is a bad idea, especially when you say " unexpected UI issue". There shouldnt be many unexpected issues and the expected ones you should be able to address. There should be a very high reliability and if that's not given.. go fix it. Or use a more reliable interface. There probably is an API, the browser has to send requests somewhere :)

Thanks. For what it's worth, in many of our tests (and these staging scripts) we are using an API to get the initial data created, since it is generally much faster to do it that way. In order to get these cases subsequently processed to whatever the desired state is, there are almost always some additional actions required that can only be performed via the UI.

In a perfect world, the third-party platform our application is built on would never experience a performance hiccup, and our code would be rock solid and able to handle all possible issues that might arise within the interface. Ultimately, we would like to be able to leverage code that is already written (some of which might throw an occasional exception) for a purpose other than just executing a test.

If we are calling a function that creates a record and it works 99 out of 100 times, this is generally acceptable to us for something that might be called a handful of times when used in a test. If that same function is used in a staging script and fails, I need a way to catch and recover from that. It may be that Serenity just isn't built in a way that can support our use case, but that's what I'm trying to find out.

[bensashi]

I've found some guidance in #652 that I believe might provide a path forward.

if (StepEventBus.getEventBus().aStepInTheCurrentTestHasFailed()) {
    LOGGER.warn("Resetting event bus due to previous failure");
    StepEventBus.getEventBus().clearStepFailures();
}

I'm going to try this and see if it will do the trick.

[bensashi]

Closing this, as the above snippet found in #652 appears to do exactly what I was attempting. The Serenity report still logs the step error, but once StepEventBus.getEventBus().clearStepFailures() is called, Serenity will attempt to execute any subsequent step function calls as normal.