Issue #48 - Tagwise runners

[temyers]

No description provided.

[sugatmankar]

@temyers I agree with generic approach of Cucumber, if no tags provide then all tags runner should be generated. I have restricted this approach because of following reason.

Suppose a Scenario or Scenario Outline is tagged with multiple tags

In this eg 5 runner will get generate if we follow blank tag approach.

If project has bunch of features with bunch of Scenarios and Scenario Outlines and each tagged with multiple tags and if we don't provide tags then duplicate runners will be generated and this is meaning less to keep running as well as re-running also. So i think it will be good if we restrict users from adding blank tags. This branch has compulsion of tags, user must pass tags either in pom file or override it while running from command line.

[sugatmankar]

@temyers Bingo ..! while thinking about support for generic approach i got solution, i was very near to skip those duplication, if i use Gherkin to parse all tags and select only one from multiple, i would be able to do what is recommended, Yes i did it, Thanks Gherkin it is very easy to control any requirement with the help of it.

Now if no tags are specified, this branch will be able to generate all runner per tag present in project.
Only one compulsion is there Scenario or Scenario Outline must be tagged with at least one tag.

Now this branch has almost everything what we were expecting.

[sugatmankar]

@stevejensen22 please try now by getting latest update , let me know if you face any issue.

[stevejensen22]

My new runners look like this:

@CucumberOptions(strict = true,
          features = {"src/test/resources/skeleton/features/mbo/MBO Place Personal Order.feature"},
       tags = {"@automated"},
       plugin = {"json:target/1.json", "pretty"},
    monochrome = false,
    glue = { "skeleton.stepdefs" })
public class ParallelRunner01 extends AbstractTestNGCucumberTests {
}

@CucumberOptions(strict = true,
          features = {"src/test/resources/skeleton/features/mbo/MBO Place Personal Order.feature"},
       tags = {"@automated"},
       plugin = {"json:target/2.json", "pretty"},
    monochrome = false,
    glue = { "skeleton.stepdefs" })
public class ParallelRunner02 extends AbstractTestNGCucumberTests {
}

These runners will still result in duplicated scenarios. I thought the fix would append a line number similar to what you do with Scenario Outline's.

[sugatmankar]

@stev could you provide feature file. I think you have tagged Scenarios with same @automated tag. Tags must be unique per Scenario per Scenario Outlines.

[stevejensen22]

So I need to generate a unique tag for every scenario? All my scenarios share an @automated tag and then differ by application or priority. ie: @checkout @high.

I was thinking your change would find all the scenarios that fit my tags config in my pom.xml and then somehow generate a unique Runner class for each scenario.

ie <tags>"~@ignore", "@automated", "@high"</tags> would give me a ParallelRunner*.java for every Scenario that had an @automated and an @high tag, that was unique similarly to what is done for scenario outlines (features = {"src/test/resources/skeleton/features/mbo/MBO Place Personal Order.feature:23"}).

I thought the runner would look something like this:

@CucumberOptions(strict = true,
          features = {"src/test/resources/skeleton/features/mbo/MBO Place Personal Order.feature:16"},
       tags = {"~@ignore", "@automated", "@high"},
       plugin = {"json:target/1.json", "pretty"},
    monochrome = false,
    glue = { "skeleton.stepdefs" })
public class ParallelRunner01 extends AbstractTestNGCucumberTests {
}

[sugatmankar]

Yes put first tag unique like @12345 @automated.

[stevejensen22]

Here's a snippet from my feature file:

@mbo3 @high
Feature: Place Personal Orders in the Member Back Office
    I will be in My Orders Place Personal Order
    Testing Addresses and Credit Cards

    Background:
        Given I am a valid user
        And I have started a browser
        And I have logged
        And I have navigated to "place a personal order"
        And I select a random product on the page
        And I navigate to my shopping cart
        And I check out from the shopping cart page

    @automated
    Scenario: Place Personal Order, New Address and Existing Credit Card
        Given I have a credit card saved to my account
        And I have added a new shipping address manually
#            | information    |
#            | Full Name      |
#            | Address Line 1 |
#            | Zip Code       |
#            | City           |
#            | State          |
#            | Phone          |
        And I select a credit card and continue
        And I place my order
        Then I am taken to the order summary page

    @automated
    Scenario: Place Personal Order, New Address and New Credit Card
        Given I have added a new shipping address manually
        And I add a credit card manually
#            | information        |
#            | Name on Card       |
#            | Credit Card Number |
#            | Expiration Date    |
#            | Billing Address    |
#            | Billing City       |
#            | Billing State      |
#            | Billing Zip Code   |
#            | Billing Country    |
#            | CVV2 Security Code |
        And I place my order
        Then I am taken to the order summary page

[stevejensen22]

Yes put first tag unique like @12345 @automated.

So how do I use the <tags> config option if all my tags are unique?

[sugatmankar]

I tried hard for single tag on each Scenario but duplication raise . To achieve tag wise tag must unique. Outline won't allow as u mentioned.
features = {"src/test/resources/skeleton/features/mbo/MBO Place Personal Order.feature:16"},
tags = {"~@ignore", "@automated", "@high"}

If you provide line no followed by :line no , tags option can't be there.

[sugatmankar]

<tags> @tag1, @tag2 , @tag3 ,..... Comma separated tags <tags>

Or you are already using -Denv.args="@tag1,@tag2,@tag3, like this "

[stevejensen22]

So I need to remove the <tags> config from my pom.xml?

[sugatmankar]

@stevejensen22 Update code again.
No you don't need to remove config. Keep it as
<tags>"@automated","~@ignore", "${env.tag}"</tags>
And pass all tags that you want to run in command line.
like mvn clean verify -Denv.tag="@tag1,@tag2,@tag3,@all tags comma separated that you want to run"

I have removed compulsion of unique tag. Now you can put "@automated" tag to every scenario you want.

Update pom for one config element "filterScenarioAndOutlineByLines", now it will allow to generate both scenario and scenario outline by line no.

eg. suppose you have two scenarios at line no 16 and 25 then runner will look like

@CucumberOptions(strict = true,
          features = {"src/test/resources/skeleton/features/Place Personal Order, New Address and New Credit Card:16"},
       plugin = {"json:target/1.json", "pretty"},
    monochrome = false,
    glue = { "skeleton.stepdefs" })
public class ParallelRunner01 extends AbstractTestNGCucumberTests {
}

@CucumberOptions(strict = true,
          features = {"src/test/resources/skeleton/features/Place Personal Order, New Address and New Credit Card:25"},
       plugin = {"json:target/2.json", "pretty"},
    monochrome = false,
    glue = { "skeleton.stepdefs" })
public class ParallelRunner02 extends AbstractTestNGCucumberTests {
}

Let me know if you find any issue.

[stevejensen22]

Just tried this today and it is almost PERFECT!!!

The only thing I've seen is that <tags>"@automated","~@ignore", "${env.tag}"</tags> appears to be treating those values as OR's instead of AND's.

So that config generates a runner class for any feature that has @automated OR @${env.tag} thus generating way more classes than is needed

I think that just needs to be tweaked.
For cucumber:
"@automated", "@high" should be AND
while
"@automated, @high" should be OR.

Thank you so much!! This is awesome!

Edit: It also appears that <filterScenarioAndOutlineByLines> does not support feature level tags

ie:

@high_priority
Feature: Japan MBO My Team Enroll

If I pass "@high_priority" into the <tags> config, no runner classes are generated.

[stevejensen22]

Also found that using Examples specific tags also is not supported. ie:

    Scenario Outline: Do something with <otg nfr>
        Given I do something with <otg nfr>

        @automated
        Examples:
            | otg nfr |
            | otg     |

        @Jira @broken
        Examples:
            | otg nfr |
            | nfr     |

[sugatmankar]

Welcome @stevejensen22 ... I need to test more about AND OR cases , Scenario Outline with multiple examples i didn't added it yet. Wait for that else you are welcome to add those changes to my repository by forking it.
:)

[SasidharParnandi]

@sugatmankar : Following your response to my comment at #30
I got tagwiseOutlinewiseIssueRerun branch and tried to check re-run functionality.
It doesn't work.
When I set retryCount in config to any non zero value, I can see errors. The exception is in the runner class. When I looked at the runners, I found that the wrong arguments are passed to ReportBuilder when building the consolidated report.

I use the same ReportBuilder locally and can confirm the exception is valid. Not sure if this is "cucumber-reporting" dependency version issue. I checked on 2.2(which I use) as well as 2.8 latest.

Exception :

Parallel01IT.java:[251,43] constructor ReportBuilder in class net.masterthought.cucumber.ReportBuilder cannot be applied to given types;
[ERROR] required: java.util.List<java.lang.String>,net.masterthought.cucumber.Configuration
[ERROR] found: java.util.List<java.lang.String>,java.io.File,java.lang.String,java.lang.String,java.lang.String,boolean,boolean,boolean,boolean,boolean,java.lang.String,boolean
[ERROR] reason: actual and formal argument lists differ in length

It works if I change retryCount to 0, but obviously no ReRun functionality.

[sugatmankar]

@SasidharParnandi please use java version above 1.7 also check for compiler setting in your IDE. If you are using IntelliJ idea ignore error in pom file. Retry count you can specify only 0 to 5.

[SasidharParnandi]

I use Intellij idea and ignored POM errors. I use Java 1.8.
As I said above, I get error when I set retryCount to any non zero i.e. 1 in my case. I can clearly see that the runner class created with rerun functionality has exception in it.

Anyway, for now I have figured out an easy way to handle reruns. In my jenkins job, I have added a post build action to concatenate all the *.rerun files into a single rerun.txt after the maven test. I already have my ReRunFailed runner which I use today for running the failed tests.

[temyers]

Re-implemented in #68

[stevejensen22]

@temyers: Thank you so much for putting these changes into an official release (#68)! I'm so excited to get your implementation working with my configuration. I had previously used @sugatmankar's implementation by compiling the code locally and had everything working except OR logic with tags and SCENARIO based parallelization.

However, with the latest changes made in 2.0.0...

I can't use <parallelScheme>SCENARIO</parallelScheme> with <tags>"@tag"</tags> and
<filterFeaturesByTags>true</filterFeaturesByTags>.

If I use <parallelScheme>SCENARIO</parallelScheme> without the other two ☝️, it appears to be working, but running EVERY scenario simply isn't an option for me.

Here's an example error:

run_cukes(Mboviewmyteamenroll43IT)  Time elapsed: 0.02 sec  <<< FAILURE!
java.lang.IllegalArgumentException: Inconsistent filters: [@staging, 93]. Only one type [line,name,tag] can be used at once.

This may be a limitation related to the gherkin parser. Is there any way we can ignore this requirement (Only one type [line,name,tag] can be used at once.)? Thanks again! I'll be happy to provide any other information you'd like!

Here's my config:

<plugin>
    <groupId>com.github.temyers</groupId>
    <artifactId>cucumber-jvm-parallel-plugin</artifactId>
    <version>2.0.0</version>
    <executions>
        <execution>
            <id>generateRunners</id>
            <phase>generate-test-sources</phase>
            <goals>
                <goal>generateRunners</goal>
            </goals>
            <configuration>
                <glue>zija.stepdefs</glue>
                <!--<outputDirectory>src/test/java/zija/parallel</outputDirectory>-->
                <featuresDirectory>src/test/resources/zija</featuresDirectory>
                <cucumberOutputDir>target</cucumberOutputDir>
                <format>json</format>
                <strict>false</strict>
                <namingScheme>feature-title</namingScheme>
                <tags>"@staging"</tags>
                <filterFeaturesByTags>true</filterFeaturesByTags>
                <useTestNG>true</useTestNG>
                <!--<namingPattern>ParallelRunner{c}</namingPattern>-->
                <parallelScheme>SCENARIO</parallelScheme>
                <!--<filterScenarioAndOutlineByLines>true</filterScenarioAndOutlineByLines>-->
            </configuration>
        </execution>
    </executions>
</plugin>

[stevejensen22]

I think the problem is actually with the runner class that's created. The class needs to be generated based on the tag but executed based on the line number. So the runner class should not contain the tag. The line number alone is satisfactory after your logic that ONLY creates runner classes based on the tag with the line number.

Ex:
Current

import cucumber.api.CucumberOptions;
import cucumber.api.testng.AbstractTestNGCucumberTests;

@CucumberOptions(strict = true,
features = {"src/test/resources/zija/features/japan/Japan MBO My Team Enroll.feature:85"},
plugin = {"json:target/1.json", "pretty"},
monochrome = false,
tags = {"@staging"},
glue = { "zija.stepdefs" })
public class Japanmbomyteamenroll01IT extends AbstractTestNGCucumberTests {
}

Expected

import cucumber.api.CucumberOptions;
import cucumber.api.testng.AbstractTestNGCucumberTests;

@CucumberOptions(strict = true,
features = {"src/test/resources/zija/features/japan/Japan MBO My Team Enroll.feature:85"},
plugin = {"json:target/1.json", "pretty"},
monochrome = false,
glue = { "zija.stepdefs" })
public class Japanmbomyteamenroll01IT extends AbstractTestNGCucumberTests {
}

[temyers]

@stevejensen22
Damn, I thought I'd tested that scenario. I'll look into it. Alternatively, if you can add a PR with an IT test case to demonstrate that'd be really helpful cheers!

[sugatmankar]

@stevejensen22 @temyers
Please check in closed branch i have already implemented The class needs to be generated based on the tag but executed based on the line number. I have skipped tag getting added when there is line number, Its in template file with one bool variable.