Unable to run specific examples from scenarios by file name and line number

[WarioNeila86]

Hi @jan-molak, there seems to be an issue when running specific examples or scenarios by file name & line number in the latest version of protractor-cucumber-framework.

This is our current setup:

    "cucumber": "^5.0.3",
    "protractor": "^7.0.0",
    "protractor-cucumber-framework": "^6.1.1",

After upgrading protractor-cucumber-framework to 8.2.11 we're seeing the following issue:

Steps to reproduce:

• Having a feature with 2 or more examples, e.g.:

1. Feature: Dummy feature
2.  Scenario Outline: Dummy feature
3.    Given I load some website
4.    When I do action number <param>
5.    Then something happens
6. 
7.    Examples:
8.      | param |
9.      | 1     |
10.     | 2     |

• In your config from protractor.conf.js file, specify specs to run as specs: ['features/dummy.feature:10']
• Execute npx protractor protractor.conf.js

Expected result:

• Only example from line 10 is executed

Actual result:

• All examples are executed

After some investigation, this seems to be related to serenity, as we were able to reproduce the issue with the protractor-cucumber-framework-demo from serenity-js project.

Could you review this, please? This is a complete blocker for us.

[jan-molak]

Hi @WarioNeila86 and thanks for reporting this issue. It seems to be caused by a bug in Cucumber, which I've reported at cucumber/cucumber-js#1900

[WarioNeila86]

Hi again @jan-molak,

After reviewing this issue again, I have some doubts this is related to that issue reported in Cucumber, let me explain our findings:

With this setup, specs are always split by line and we can execute specific examples from scenario outline.

    "cucumber": "^5.0.3",
    "protractor": "^7.0.0",
    "protractor-cucumber-framework": "^6.1.1"

However, when we upgrade protractor-cucumber-framework, we see the issue mentioned above with all examples being executed in all instances.

    "cucumber": "^5.0.3",
    "protractor": "^7.0.0",
    "protractor-cucumber-framework": "^8.2.11"

It doesn't matter if we use relative or absolute paths for specs in protractor.config.js for any of the above we always get same results with same versions.

That's why I'm thinking issue might be related to serenity-js changes in protractor-cucumber-framework.

Thanks for your help.

[jan-molak]

Hey @WarioNeila86!

So the older versions of protractor-cucumber-framework, like 6.1.1 released 3 years ago, didn't use Serenity/JS, which has been introduced in version 8 of this module and replaced the original implementation.

Versions prior to 8 used to pass any paths to feature files they were given, verbatim to Cucumber. So for example if you specified C:\test\features\my.feature, C:\test\**\my.feature, /usr/jan/my.feature, ./my.feature, or my.feature that's exactly how they'd be passed on, and protractor-cucumber-framework would leave handling of any invalid paths to Cucumber.

As an experiment, given "protractor-cucumber-framework": "^6.1.1", try to specify the path to your feature file using an absolute value and line number. For example:

specs: [
  `${ path.resolve(__dirname, '/features/dummy.feature') }:10`,
]

I'd expect the above configuration to fail and run all the scenarios from dummy.feature, despite you providing the line number cucumber/cucumber-js#1900

Would you be able to confirm that, please?

---

Now, I believe that the right way to solve this problem would be to do it at the source and make sure that Cucumber supports both relative and absolute paths. Based on the conversation at cucumber/cucumber-js#1900 it seems like Cucumber 8 will get the fix, but for the older (and now deprecated) versions Serenity/JS would need to work around this.

I'll check if I could make the paths relative at the point where Serenity/JS hands them over to Cucumber.

[jan-molak]

@WarioNeila86 OK, I think I can work around this. Stay tuned.

[jan-molak]

@WarioNeila86 - can you please update to protractor-cucumber-framework version 8.3.0 and let me know the results on your end? Thanks.

[WarioNeila86]

Hi @jan-molak, I could finally test this and 8.3.0 is working as expected with both absolute and relative paths, thanks for the quick fix 👍

However, we're now seeing some failures in after hook coming from serenity-js:

[chrome #01-1] [13:58:03] INFO: ✖ After Hook
[chrome #01-1] [13:58:41] I/runnerCli - an AfterAll hook errored, process exiting: ../node_modules/@serenity-js/cucumber/src/listeners/legacy/cucumber-4.ts:23: 1 async operation has failed to complete within a 5s cue timeout:
[chrome #01-1] 10s 20ms - [ProtractorReporter] Invoking ProtractorRunner.afterEach...

[13:58:41] I/testLogger - 

[13:58:41] E/launcher - Runner process exited unexpectedly with error code: 1

I'm always getting 3 or 4 examples failing because of this.

I guess this might be unrelated to the original issue reported here, but if you can throw some light or suggest something to check/test here... I'd really appreciate it. I can create a different issue for this if needed.

Thanks again!

[jan-molak]

You're welcome, thanks for confirming.

However, we're now seeing some failures in after hook coming from serenity-js

Could you please try to fork https://github.com/serenity-js/protractor-cucumber-framework-demo/ and try to reproduce the issue?