Serenity Reports including tests which are not executed

[nkhand]

Hi
Serenity Reports are showing tests which are not executed at all . I am executing just 1 test ( having 2 scenarios ) , but aggregate report is showing results for 5 tests . Serenity version used : 1.8.21

From console log :

(AfterStories)
Reports view generated with 2 stories (of which 0 pending) containing 2 scenarios (of which 0 pending)

[INFO] --- serenity-maven-plugin:1.8.21:aggregate (serenity-reports) @ artemis-implementation ---
[INFO] Generating test results for 5 tests
[INFO] Test results for 5 tests generated in 4583 ms

[INFO] --- serenity-maven-plugin:1.8.21:check (serenity-reports) @ artemis-implementation ---
[INFO] Checking Serenity test results
[INFO] ----------------------
[INFO] SERENITY TEST OUTCOMES
[INFO] ----------------------
[INFO] - Tests executed: 5
[INFO] - Tests passed: 2
[INFO] - Tests failed: 1
[INFO] - Tests with errors: 2
[INFO] - Tests pending: 0
[INFO] - Tests compromised: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

Please help to check quickly .

[nkhand]

I guess actual issue is for each execution session , reports are getting added over and not overwritten .Could you help how to make reports overwritten and not appended .

[wakaleo]

Aggregate and Check will find all the test results in the target directory, not only the ones you just ran. If you want an aggregate report for only the tests you just ran, run mvn clean first

[nkhand]

Thanks @wakaleo . My Maven goal in Eclipse is set to clean verify .

[wakaleo]

What are you seeing in the reports?

[scormaq]

Hi @wakaleo ,
I can confirm that this issue remains for Serenity 2.0.40 & Serenity-Cucumber4 1.0.6
I have next case:

1. Serenity-Cucumber4 framework, 5 feature files.
2. 2 feature files have tags like @app:rest (say, for REST API), 3 other don't (say, they are for UI)
3. I want to execute only these 2 tests for REST API, so I do next:

mvn clean verify -Dcucumber.options="--tags @app:rest"

4. In generated HTML report, in Functional Coverage Overview section, there are all 5 features. Features for UI are displayed as pending. This is inconsistent behavior, because there 0 pending/skipped/manual tests in summary review, on pie chart, on Test Results tab, in CSV results etc. Also, skipped features was not reported by serenity console logger.

Is there a way to display in functional coverage overview only tests which were actually executed?

Here is a screenshot for case above. I would like to somehow get rid of 'empty' features in Functional Coverage Overview

[wakaleo]

It is the nature of functional coverage to report not only features that were tested, but also features that were not tested. Pending can also mean that no tests were executed for a particular requirement.

Have you tried putting this in your Serenity properties?

serenity.exclude.unrelated.requirements.of.type=feature

[scormaq]

Hi @wakaleo , thanks for response.

Have you tried putting this in your Serenity properties?
serenity.exclude.unrelated.requirements.of.type=feature

I tried this, unfortunately didn't work.

It is the nature of functional coverage to report not only features that were tested, but also features that were not tested. Pending can also mean that no tests were executed for a particular requirement.

I agree with meaning of 'pending', but in my case tests were not pending, they were excluded, and therefore should not be reported at all.

Please correct me if I wrong, but pending tests are tests which tagged with @pending tag and executed. Such pending tests reported correctly.

Here, the report from my case, I tagged one of executed scenarios with @pending tag.

Pending can also mean that no tests were executed for a particular requirement.

This behavior is not very convenient when there many tests in project, but only small subset of them executed. In such case Functional Coverage section will have many tests, not actually related to coverage.

[scormaq]

Hi @wakaleo ,
so can it be considered as a valid defect that serenity.exclude.unrelated.requirements.of.type=feature property does not exclude features from coverage if they were not executed at all?

[barbaracabralamido]

Let me understand if the problem I had is the same the you're describing.

For me, I put the tag "@Manual" and "@manual-result:passed"
But the scenario is being executed, that is it right?

On the report is shown as pending (not passing):

AND the scenario is being executed and showing a BUILD Failure

The versions of the frameworks that I have are:

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<serenity.version>2.0.48</serenity.version>
		<serenity.maven.version>2.0.48</serenity.maven.version>
		<serenity.cucumber.version>1.9.37</serenity.cucumber.version>
		<slf4j-simple>1.6.1</slf4j-simple>
		<groovy-all>1.8.6</groovy-all>
		<junit>4.12</junit>
		<assertj-core>1.7.0</assertj-core>
		<commons-text>1.1</commons-text>
		<!-- Plugins -->
		<maven-enforcer-plugin>1.4</maven-enforcer-plugin>
		<maven-surefire-plugin>2.18</maven-surefire-plugin>
		<maven-failsafe-plugin>2.18</maven-failsafe-plugin>
		<maven-compiler-plugin>1.8</maven-compiler-plugin>
	</properties>

How can I have the manual scenario as "Passing" AND the build with a "Success" ?

[wakaleo]

You need to use versions 2.0.56 and 1.0.15

[scormaq]

@barbaracabralamido

For me, I put the tag "@Manual" and "@manual-result:passed"
But the scenario is being executed, that is it right?

The scenario is executed, because it is executed by Cucucmber, and Serenity just parses tags and reports proper result. If you want to have them passed, use tag @manual:passed (here's the article from John). Also If you want to avoid execution of tests, run them with --dry-run Cucumber option.

[ankush2704]

Hi @wakaleo as on today, I am still seeing the issue reported by @scormaq. I have below versions
<serenity.version>2.2.13</serenity.version> <serenity.cucumber.version>2.2.0</serenity.cucumber.version> <serenity.maven.version>2.2.13</serenity.maven.version>

I am seeing features in the report which are not the part of execution.
setting up serenity.exclude.unrelated.requirements.of.type property didn't help

[wakaleo]

@ankush2704 as a rule you will need to use filters to mask out any features you don't want appearing in the reports. Have you tried with the latest version?

[ankush2704]

No @wakaleo it is not possible to upgrade the version at the moment. We are on Cucumber 5 too.
What filters you are talking about? If I am having 100 features (100 feature files) and I am running 5 features then I need to mask 95 features so that they should not appear in report?
What about serenity.exclude.unrelated.requirements.of.type property then?