Serenity Report not showing all the scenarios outline mentioned in feature file though taking all screenshot for all scenario outline.

[rajaaryan4]

Feature file:-
@RunWith(CucumberWithSerenity.class)
@CucumberOptions(features = {"features"}, glue = {"stepdefinitons"},plugin={"pretty"})
public class serenity runner{

}

I have two scenario outline in feature file
@test1
Scenario Outline: Verification of Login as user 1
Given I login in to app as user
When I give credentials
Then I see the details page
Examples:
|x|y|
|abc|asx|

@test2
Scenario Outline: Verification of testing user
Give I login into app2 as user
When I give credentials
Then I see user name and details application
Examples:
|z|w|
|fdgdfg|gfdg|

Pom configuration:-
<serenity.version>2.0.23</serenity.version>
<serenity.maven.version>2.2.9</serenity.maven.version>
<serenity.cucumber.version>1.9.21</serenity.cucumber.version>
<maven.compiler.version>3.8.0</maven.compiler.version>


org.apache.maven.plugins
maven-surefire-plugin
2.22.1

false



maven-failsafe-plugin
2.22.1


%regex[.Runner.]
**/*TestSuite.java


<webdriver.base.url>${webdriver.base.url}</webdriver.base.url>

classes
${parallel.tests}
${parallel.tests}




integration-test
verify





net.serenity-bdd.maven.plugins
serenity-maven-plugin
${serenity.maven.version}

${tags}



serenity-reports
post-integration-test

aggregate

Goals:
clean verify -Dcucumber.options="--tags '@test1 or test2'"

Results:
Tests run: 2, failures: 0, Errors: 0, skipped: 0

but Serenity tests result shows as below

SERENITY TESTS: SUCCESS

|Test executed | 1
|Test passed | 1
|Tests with errors | 0
|Tests ignored/skipped | 0

The reports contains only @test2 Scenario outline but it does not contain @test1 scenario outline.
I checked the serenity report folder and it has sceenshot for @test1 Scenario outline too. I dont know why this is happening.

[wakaleo]

Try with recent and consistent versions, e.g. https://github.com/serenity-bdd/serenity-core/blob/master/README.md#what-is-the-latest-stable-version-i-should-use

[rajaaryan4]

I tried
<serenity.version>2.0.48</serenity.version>
<serenity.maven.version>2.0.48</serenity.maven.version>
<serenity.cucumber.version>1.9.37</serenity.cucumber.version>

I am still getting Generating test results for 1 tests only. Only test2 scenario outline is getting reflected in serenity report. I dont know why this is happening. It is due to scenario outline? But when I using only 'Scenario' instead of 'Scenario Outline' without any examples , the report is getting generated with right no of scenarios.

[wakaleo]

Scenario outlines should not be an issue. Please format your pom.xml content correctly so we can see what it contains.

[rajaaryan4]

Here is my POM configuration:-

<properties>
	<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<serenity.version>2.1.5</serenity.version>
		<serenity.maven.version>2.1.5</serenity.maven.version>
		<maven.compiler.version>3.8.1</maven.compiler.version>
		<serenity.cucumber.version>1.9.50</serenity.cucumber.version>
		<webdriver.base.url>chrome</webdriver.base.url>
		<tags></tags>
</properties>
<packaging>jar</packaging>
<repositories>
    <repository>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
        <id>central</id>
        <name>bintray</name>
        <url>http://jcenter.bintray.com</url>
    </repository>	
</repositories>
<pluginRepositories>
    <pluginRepository>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
        <id>central</id>
        <name>bintray-plugins</name>
        <url>http://jcenter.bintray.com</url>
    </pluginRepository>
</pluginRepositories>

<dependencies>
    <dependency>
        <groupId>ch.qos.logback</groupId>
        <artifactId>logback-classic</artifactId>
        <version>1.0.13</version>
    </dependency>

<dependency>
    <groupId>net.serenity-bdd</groupId>
    <artifactId>serenity-core</artifactId>
    <version>${serenity.version}</version>
    <scope>test</scope>
</dependency>

<dependency>
    <groupId>net.serenity-bdd</groupId>
    <artifactId>serenity-junit</artifactId>
    <version>${serenity.version}</version>
    <scope>test</scope>
</dependency>

<dependency>
    <groupId>net.serenity-bdd</groupId>
    <artifactId>serenity-screenplay</artifactId>
    <version>${serenity.version}</version>
    <scope>test</scope>
</dependency>

<dependency>
    <groupId>net.serenity-bdd</groupId>
    <artifactId>serenity-screenplay-webdriver</artifactId>
    <version>${serenity.version}</version>
    <scope>test</scope>
</dependency>

<dependency>
    <groupId>net.serenity-bdd</groupId>
    <artifactId>serenity-cucumber</artifactId>
    <version>${serenity.cucumber.version}</version>
    <scope>test</scope>
</dependency>

<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-simple</artifactId>
    <version>1.7.7</version>
    <scope>test</scope>
</dependency>

<dependency>
    <groupId>junit</groupId>
    <artifactId>junit</artifactId>
    <version>4.12</version>
    <scope>test</scope>
</dependency>

<dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-email</artifactId>
    <version>1.5</version>
</dependency>

<dependency>
    <groupId>org.apache.poi</groupId>
    <artifactId>poi</artifactId>
    <version>4.1.0</version>
</dependency>

    <dependency>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>exec-maven-plugin</artifactId>
        <version>3.0.0</version>
    </dependency>
</dependencies>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.22.1</version>
            <configuration>
                <skip>false</skip>
            </configuration>
        </plugin>
        <plugin>
	    <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-failsafe-plugin</artifactId>
            <version>2.22.1</version>
            <configuration>
                <includes>
                    <include>%regex[.*Runner.*]</include>
                    <include>**/Test*.java</include>
                </includes>
                <systemPropertyVariables>
	            <argLine>-Xmx512m</argLine>
                    <webdriver.base.url>${webdriver.base.url}</webdriver.base.url>
                </systemPropertyVariables>
            </configuration>
            <executions>
                <execution>
                    <goals>
                        <goal>integration-test</goal>
                        <goal>verify</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
	    <version>${maven-compiler-version}</version>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
        </plugin>
        <plugin>
            <groupId>net.serenity-bdd.maven.plugins</groupId>
            <artifactId>serenity-maven-plugin</artifactId>
            <version>${serenity.maven.version}</version>
            <configuration>
              <tags>${tags}</tags>
            </configuration>
            <executions>
                <execution>
                    <id>serenity-reports</id>
                    <phase>post-integration-test</phase>
                    <goals>
                        <goal>aggregate</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>

[cliviu]

hey @rajaaryan4 , can you try with the latest serenity-cucumber5 ?

[rajaaryan4]

Hi @cliviu , I tried with the latest serenity-cucumber5 and it gave the same result. Though all the scenarios outline were getting executed but only 1 result was showing up in the serenity report instead of two scenario outline.

Results:
Tests run: 2, failures: 0, Errors: 0, skipped: 0

but Serenity tests result shows as below
SERENITY TESTS: SUCCESS
|Test executed | 1
|Test passed | 1
|Tests with errors | 0
|Tests ignored/skipped | 0

[cliviu]

this is strange, it should work. Can you please post a small example to help us reproduce the problem ? Also the exact command that you are using for running the tests. Thanks !

[rajaaryan4]

Here is my Feature file

Feature:Story#:XXXXXXX
As a layman
I need to login into app
So that I verify a yyyy field

@Test_1
Scenario Outline: Verification of yyyy field as a 123 user
Given I log into gmail as an user
When I click login button
Then <field1> field should be visible on <section1>

Examples:
|field1|section1|
|yyyy|fgdfgdg|
|yyyya|fgdfgdg|
|yyyyb|fgdfgdg|
|yyyyv|fgdfgdg|
|yyyy|gggggg|
|yyyy|gggggg|


@Test_2
Scenario Outline: Verification of yyyy field as a 345 user
Give I log into gmail as a <user1> user
When I click on login button 
Then <field 2> should  not be visible <section2>

Examples:
|user1|field2|section2|
|vvvv@gmail.com|yyyy|uuuuu|
|nnnn@gmail.com|yyyy|uuuuu|

Here is my runner file:

@RunWith(CucumberWithSerenity.class)
@CucumberOptions(features = {"features"}, glue = {"stepdefinitons"},plugin={"pretty"})
public class serenityrunner{

}

Goals mentioned at runtime:- 
clean verify -Dcucumber.options="--tags '@Test_1 or @Test_2'" -Dusername="user" -Dpassword="password"


Result:-
It is executing both the scenarios mentioned in Scenario Outline but in the serenity report only Test 2 results are getting displayed. 


Do let me know if you need more information.

[cliviu]

the best thing will be to make a github project and share it with us. Exactly the think that you are running. So I will be sure that we are on the same page.

[rajaaryan4]

Hi It will be a while as the application I am trying to automate is controlled by client. I will try to create a sample project and share with you(this weekend).

[rajaaryan4]

Please find attached the sample project. I have tried to keep it close to the project I have created for client.
Scenario_Outline_Sample.zip

[wakaleo]

Try putting your feature files in a requirements directory structure under src/test/resources/features, not in src/test/java.

[rajaaryan4]

I tried doing that as you suggested. I moved the feature files in the requirement directory under src/test/resources But still the report got generated with only one test (Scenario Outline) instead of both the scenarios Outline. Though all the scenarios outline got executed.

[wakaleo]

Start with one of the starter projects and add your feature files, following the Cucumber/Serenity conventions.

[rajaaryan4]

I will start with a fresh project and will let you know.

[rajaaryan4]

I started fresh and took another project ,did everything following Cucumber/Serenity conventions but still only 1 scenario showing up in report. In my original project I am reading the value from an excel sheet. Is that the issue?

[cliviu]

hi @rajaaryan4. I suggest that you start with such a clean base https://github.com/serenity-bdd/serenity-cucumber5-starter . Just add a feature with two scenario outlines and convince yourself that the filtering works. On this clean base you can start to add your functionality little by little and recognize the part that is causing problems.

[cliviu]

@rajaaryan4 , please try this project : https://github.com/cliviu/serenity-cucumber5-starter.git and run the following command
mvn clean verify -Dcucumber.options="--tags '@test3 or @test5'"

[rajaaryan4]

@cliviu Thanks a lot. Will try and let you know.

[rajaaryan4]

@cliviu The issue is resolved. I created the project as suggested by you. I am able to see all the scenarios in the serenity report.