Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reporting failures with mutliple examples result in nested steps #3393

Closed
Johan-Sap opened this issue Jan 31, 2024 · 7 comments
Closed

Reporting failures with mutliple examples result in nested steps #3393

Johan-Sap opened this issue Jan 31, 2024 · 7 comments

Comments

@Johan-Sap
Copy link
Contributor

What happened?

When I have a Scenario with multiple examples, and one of those (or all worst case) fail the report will show all steps of the next example as steps from the failed example.

It happens when the assert or failure is nested a bit deeper in the code (not stepdefinitions or stepslib but a step that is called from a method in a stepslib or deeper);
Below is a screenshot of the report which shows the second example to be nested in the first one;

image

Traced the issue back to this change
#3369

reverted this change locally and using that snapshot results in the correct behaviour.

What did you expect to happen?

All examples (regardless of passing or failing) will be shown as separate tests.

Below screenshot is of the report using my snapshot after reverting the code in change
https://github.com/Johan-Sap/serenity-core/commit/e2955c8aaf75798b1cb1f66c7b461b6a488f7f46

image

Serenity BDD version

As of 4.0.31

JDK version

20

Execution environment

MAC
Chrome Browser

How to reproduce the bug.

Repo:
https://github.com/Johan-Sap/eu_pvh_demo2

Run Command:
clean clearReports test

libs folder contains the snapshot of 4.0.47 in case you want to try that. Gradle should contain
testImplementation files('libs/*.jar', 'libs/serenity-core-4.0.47-SNAPSHOT.jar')

in the depedencies to test the snapshot.

How can we make it happen?

Work on this myself and propose a PR (with Serenity BDD team guidance)

@wakaleo
Copy link
Member

wakaleo commented Jan 31, 2024

This appears to be a persistent problem in the parallel execution logic. Thanks for your investigations. Unfortunately we can't undo that commit as it is where we implement support for parallel execution in Cucumber.

wakaleo added a commit that referenced this issue Feb 4, 2024
@Rangiferus
Copy link

I reported a similar effect some time ago in #3296.
I run Srenity in one thread, no parallelization.

@wakaleo
Copy link
Member

wakaleo commented Feb 6, 2024

Fixed in 4.1.0

@Johan-Sap
Copy link
Contributor Author

Perfect, tested it and seems to be working as expected indeed !

@Rangiferus
Copy link

Now, in 4.1.0 it's even worse. At least in all my cases with multiple examples when some of them fail on assertions specified in subtasks.
But not only. Even in regular scenarios (no examples) I'm getting a report with only one step visible and when it's expanded, then all steps are displayed mixed with log entries defined inside steps (that should be visible only when I expand their tasks) and with entries from @after hook. In some cases even the sequence of entries is totally wrong.

It's hard to reproduce and provide an example project to prove it due to complexity of my project.
cc: @wakaleo, @Johan-Sap

@wakaleo
Copy link
Member

wakaleo commented Feb 13, 2024

@Rangiferus Maybe dive into the code and see if you can see anything. I wont be able to look at a fix without a sample project to reproduce the issue though.

@wakaleo
Copy link
Member

wakaleo commented Feb 13, 2024

@Rangiferus It really sounds like there might be some project-specific issues though - we could look at your project directly, but it would have to be part of a commercial support package.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants