Mixing absolute and relative paths breaks native and custom formatters

[jan-molak]

Hi team!

Consider:

• a Node.js project using Cucumber.js at /home/jan/project
• a features directory with an example.feature file inside it
• my current working directory is /home/jan/project
• a Linux or mac OS machine (I haven't checked on Windows)

When I run the latest Cucumber 7.3.1 as follows:

cucumber-js features/example.feature /home/jan/project/features/example.feature

the following issues occur:

Issue 1 - feature files get executed twice

Cucumber doesn't understand that the relative features/example.feature is the same as the absolute /home/jan/project/features/example.feature and runs the feature twice.

Issue 2 - scenario location undefined

Location of the duplicate scenarios returned by parseTestCaseAttempt and formatterHelpers.PickleParser.getPickleLocation(testCaseAttempt) is undefined.

This breaks native formatters, such as snippets:

TypeError: Cannot read property 'line' of undefined
    at Object.parseTestCaseAttempt (/home/jan/project/node_modules/@cucumber/cucumber/lib/formatter/helpers/test_case_attempt_parser.js:91:88)
    at Object.formatTestCaseAttempt (/home/jan/project/node_modules/@cucumber/cucumber/lib/formatter/helpers/test_case_attempt_formatter.js:77:47)
    at Object.formatIssue (/home/jan/project/node_modules/@cucumber/cucumber/lib/formatter/helpers/issue_helpers.js:26:68)
    at /home/jan/project/node_modules/@cucumber/cucumber/lib/formatter/summary_formatter.js:56:32
    at Array.forEach (<anonymous>)
    at ProgressFormatter.logIssues (/home/jan/project/node_modules/@cucumber/cucumber/lib/formatter/summary_formatter.js:55:16)
    at ProgressFormatter.logSummary (/home/jan/project/node_modules/@cucumber/cucumber/lib/formatter/summary_formatter.js:42:18)
    at EventEmitter.<anonymous> (/home/jan/project/node_modules/@cucumber/cucumber/lib/formatter/summary_formatter.js:22:22)
    at EventEmitter.emit (events.js:327:22)
    at Runtime.start (/home/jan/project/node_modules/@cucumber/cucumber/lib/runtime/index.js:118:31)
    at async Cli.run (/home/jan/project/node_modules/@cucumber/cucumber/lib/cli/index.js:205:23)
    at async Object.run [as default] (/home/jan/project/node_modules/@cucumber/cucumber/lib/cli/run.js:25:18)

as well as custom formatters, such as Serenity/JS - serenity-js/serenity-js#975

Issue 3 - Incorrect messages emitted

testCaseStarted and related messages are emitted twice:

{"meta":{"protocolVersion":"16.0.1","implementation":{"name":"cucumber-js","version":"7.3.1"},"cpu":{"name":"x64"},"os":{"name":"darwin","version":"20.6.0"},"runtime":{"name":"node.js","version":"14.16.0"}}}
{"source":{"data":"Feature: An example feature\n\n  Scenario: First\n\n    Given a step that passes\n","uri":"features/example.feature","mediaType":"text/x.cucumber.gherkin+plain"}}
{"gherkinDocument":{"feature":{"tags":[],"location":{"line":1,"column":1},"language":"en","keyword":"Feature","name":"An example feature","description":"","children":[{"scenario":{"id":"9bce6771-fdcf-46ff-9794-c4fc89352f23","tags":[],"location":{"line":3,"column":3},"keyword":"Scenario","name":"First","description":"","steps":[{"id":"aacb2b99-c0e1-4e06-a24f-1334a981769c","location":{"line":5,"column":5},"keyword":"Given ","text":"a step that passes"}],"examples":[]}}]},"comments":[],"uri":"features/example.feature"}}
{"pickle":{"id":"6813e6ed-ddbe-4c01-897e-2a0c95fb1a14","uri":"features/example.feature","astNodeIds":["9bce6771-fdcf-46ff-9794-c4fc89352f23"],"tags":[],"name":"First","language":"en","steps":[{"id":"77e6d2a4-e5b7-49a9-89a2-645d1bf780c0","text":"a step that passes","astNodeIds":["aacb2b99-c0e1-4e06-a24f-1334a981769c"]}]}}
{"source":{"data":"Feature: An example feature\n\n  Scenario: First\n\n    Given a step that passes\n","uri":"features/example.feature","mediaType":"text/x.cucumber.gherkin+plain"}}
{"gherkinDocument":{"feature":{"tags":[],"location":{"line":1,"column":1},"language":"en","keyword":"Feature","name":"An example feature","description":"","children":[{"scenario":{"id":"c04a6aaf-ff3f-4606-938a-7e7661f40a1f","tags":[],"location":{"line":3,"column":3},"keyword":"Scenario","name":"First","description":"","steps":[{"id":"1663e51f-847a-4105-988e-1b25aa4097e8","location":{"line":5,"column":5},"keyword":"Given ","text":"a step that passes"}],"examples":[]}}]},"comments":[],"uri":"features/example.feature"}}
{"pickle":{"id":"1d34b7d3-935c-4c86-99fb-8a5e9abb60cb","uri":"features/example.feature","astNodeIds":["c04a6aaf-ff3f-4606-938a-7e7661f40a1f"],"tags":[],"name":"First","language":"en","steps":[{"id":"56e6eed4-3786-46fa-bd27-ffb77a763d27","text":"a step that passes","astNodeIds":["1663e51f-847a-4105-988e-1b25aa4097e8"]}]}}
{"testRunStarted":{"timestamp":{"seconds":1630374491,"nanos":498000000}}}
{"testCase":{"pickleId":"6813e6ed-ddbe-4c01-897e-2a0c95fb1a14","id":"fad8f086-04ab-45e1-a8f0-92439c05b79d","testSteps":[{"id":"d367e991-c27d-4975-9d00-f0992699f8d5","pickleStepId":"77e6d2a4-e5b7-49a9-89a2-645d1bf780c0","stepDefinitionIds":[],"stepMatchArgumentsLists":[]}]}}
{"testCase":{"pickleId":"1d34b7d3-935c-4c86-99fb-8a5e9abb60cb","id":"2e4befdd-9e94-4769-b8b7-c7ba1cf7b3e1","testSteps":[{"id":"e4a01c2b-d27e-4a98-b654-1ab6817c829d","pickleStepId":"56e6eed4-3786-46fa-bd27-ffb77a763d27","stepDefinitionIds":[],"stepMatchArgumentsLists":[]}]}}
{"testCaseStarted":{"attempt":0,"testCaseId":"fad8f086-04ab-45e1-a8f0-92439c05b79d","id":"dfa85e2f-cfb1-40ee-951b-37f22799c9da","timestamp":{"seconds":1630374491,"nanos":500000000}}}
{"testStepStarted":{"testCaseStartedId":"dfa85e2f-cfb1-40ee-951b-37f22799c9da","testStepId":"d367e991-c27d-4975-9d00-f0992699f8d5","timestamp":{"seconds":1630374491,"nanos":500000000}}}
{"testStepFinished":{"testCaseStartedId":"dfa85e2f-cfb1-40ee-951b-37f22799c9da","testStepId":"d367e991-c27d-4975-9d00-f0992699f8d5","testStepResult":{"status":"UNDEFINED","duration":{"seconds":0,"nanos":0},"willBeRetried":false},"timestamp":{"seconds":1630374491,"nanos":500000000}}}
{"testCaseFinished":{"testCaseStartedId":"dfa85e2f-cfb1-40ee-951b-37f22799c9da","timestamp":{"seconds":1630374491,"nanos":500000000}}}
{"testCaseStarted":{"attempt":0,"testCaseId":"2e4befdd-9e94-4769-b8b7-c7ba1cf7b3e1","id":"4dad91f9-2022-4942-8ce1-dd48314982d9","timestamp":{"seconds":1630374491,"nanos":501000000}}}
{"testStepStarted":{"testCaseStartedId":"4dad91f9-2022-4942-8ce1-dd48314982d9","testStepId":"e4a01c2b-d27e-4a98-b654-1ab6817c829d","timestamp":{"seconds":1630374491,"nanos":501000000}}}
{"testStepFinished":{"testCaseStartedId":"4dad91f9-2022-4942-8ce1-dd48314982d9","testStepId":"e4a01c2b-d27e-4a98-b654-1ab6817c829d","testStepResult":{"status":"UNDEFINED","duration":{"seconds":0,"nanos":0},"willBeRetried":false},"timestamp":{"seconds":1630374491,"nanos":501000000}}}
{"testCaseFinished":{"testCaseStartedId":"4dad91f9-2022-4942-8ce1-dd48314982d9","timestamp":{"seconds":1630374491,"nanos":501000000}}}
{"testRunFinished":{"timestamp":{"seconds":1630374491,"nanos":501000000},"success":false}}

Impact

The above issues are particularly problematic when Cucumber rerun feature is used (where the @rerun.txt file contains relative paths to feature files), together with absolute paths to other feature files, for example:

cucumber-js @rerun /home/jan/project/features/example.feature

where @rerun.txt:

features/example.feature:3

Also, when external frameworks such as Serenity/JS (maybe Stryker too? @nicojs?) allow users to specify paths to feature files as globs and then resolve those to absolute paths.

To Reproduce

See https://github.com/jan-molak/cucumber-rerun-bug

Note that in package.json I'm using realpath to get the absolute path dynamically. This will probably not work on Windows, but you could replace that with the actual absolute path when debugging.

[nicojs]

Good find! Stryker passes glob expressions directly to cucumber-js a.t.m., so no issues as long as relative paths are used 🤷‍♂️

[davidjgoss]

@jan-molak FYI I think the changes from @nicojs here #1708 may alleviate issue 1 to some extent. Regardless we have work to do around absolute paths, I'll aim to fix before the next RC.

[davidjgoss]

Released in 8.4.0 https://github.com/cucumber/cucumber-js/releases/tag/v8.4.0