Skip to content

Commit

Permalink
cli: use kebab-case for --retry-tag-filter arg (#1293)
Browse files Browse the repository at this point in the history
* update error message

* switch to kebab-case

* issue warning when using camel case for --retry-tag-filter; make warnings testable

* update docs to reference correct case

* whoops remove arg we dont need

* remove arg we dont need

* create a console from stdout and warn using that

* log warnings to stderr not stdout

* pass both stdout and stderr to console

* rework again, do linting in argv parser

* simplify

* update CHANGELOG.md
  • Loading branch information
davidjgoss committed Mar 17, 2020
1 parent 552115f commit 0bbb4b9
Show file tree
Hide file tree
Showing 6 changed files with 55 additions and 17 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ Please see [CONTRIBUTING.md](https://github.com/cucumber/cucumber/blob/master/CO
#### Bug fixes

* don't execute BeforeAll and AfterAll hooks when in dry-run
* support correct case for `--retry-tag-filter` CLI argument

### [6.0.5](https://github.com/cucumber/cucumber-js/compare/v6.0.4...v6.0.5) (2019-11-13)

Expand Down
2 changes: 1 addition & 1 deletion docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ By default, cucumber-js runs the entire suite and reports all the failures. This
## Retry failing tests

Use `--retry <int>` to rerun tests that have been failing. This can be very helpful for flaky tests.
To only retry failing tests in a subset of test use `--retryTagFilter <EXPRESSION>` (use the same as in Use [Tags](#tags))
To only retry failing tests in a subset of test use `--retry-tag-filter <EXPRESSION>` (use the same as in Use [Tags](#tags))

## Transpilation

Expand Down
45 changes: 33 additions & 12 deletions features/retry.feature
Original file line number Diff line number Diff line change
@@ -1,17 +1,38 @@
Feature: Retry flaky tests

Using the `--retry` flag will retry failing tests for the specified number of times
Additionally using the `--retryTagFilter` flag will re-run only tests matching the tag expression
Additionally using the `--retry-tag-filter` flag will re-run only tests matching the tag expression

@spawn
Scenario: running Cucumber JS with --retryTagFilter but no positive --retry will fail
When I run cucumber-js with `--retryTagFilter @flaky`
Scenario: running Cucumber JS with --retry-tag-filter but no positive --retry will fail
When I run cucumber-js with `--retry-tag-filter @flaky`
Then the error output contains the text:
"""
Error: a positive --retry count must be specified when setting --retryTagFilter
Error: a positive --retry count must be specified when setting --retry-tag-filter
"""
And it fails

@spawn
Scenario: running Cucumber JS with --retryTagFilter in camel case will result in a warning
Given a file named "features/a.feature" with:
"""
Feature:
Scenario:
Given a step
"""
Given a file named "features/step_definitions/cucumber_steps.js" with:
"""
const {Given} = require('cucumber')
Given(/^a step$/, function() {})
"""
When I run cucumber-js with `--retry 1 --retryTagFilter @flaky`
Then the error output contains the text:
"""
the argument --retryTagFilter is deprecated and will be removed in a future release; please use --retry-tag-filter
"""
But it passes

Scenario: running Cucumber JS with negative --retry will fail
When I run cucumber-js with `--retry -1`
Then the error output contains the text:
Expand Down Expand Up @@ -322,7 +343,7 @@ Feature: Retry flaky tests
"""
And it fails

Scenario: retrying a flaky test matching --retryTagFilter will eventually make it pass
Scenario: retrying a flaky test matching --retry-tag-filter will eventually make it pass
Given a file named "features/a.feature" with:
"""
Feature:
Expand All @@ -344,12 +365,12 @@ Feature: Retry flaky tests
throw 'fail'
})
"""
When I run cucumber-js with `--retry 1 --retryTagFilter '@flaky'`
When I run cucumber-js with `--retry 1 --retry-tag-filter '@flaky'`
Then scenario "Flaky" attempt 0 step "Given a flaky step" has status "failed"
Then scenario "Flaky" attempt 1 step "Given a flaky step" has status "passed"
And it passes

Scenario: a flaky test not matching --retryTagFilter won't re-run and just fail
Scenario: a flaky test not matching --retry-tag-filter won't re-run and just fail
Given a file named "features/a.feature" with:
"""
Feature:
Expand All @@ -371,11 +392,11 @@ Feature: Retry flaky tests
throw 'fail'
})
"""
When I run cucumber-js with `--retry 1 --retryTagFilter '@not_flaky'`
When I run cucumber-js with `--retry 1 --retry-tag-filter '@not_flaky'`
Then scenario "Flaky" step "Given a flaky step" has status "failed"
And it fails

Scenario: retrying a flaky test matching --retryTagFilter will eventually make it pass but not-matching will not be retried (AND operator between tags)
Scenario: retrying a flaky test matching --retry-tag-filter will eventually make it pass but not-matching will not be retried (AND operator between tags)
Given a file named "features/a.feature" with:
"""
Feature:
Expand Down Expand Up @@ -410,13 +431,13 @@ Feature: Retry flaky tests
throw 'fail'
})
"""
When I run cucumber-js with `--retry 1 --retryTagFilter '@flaky and @anOtherTag'`
When I run cucumber-js with `--retry 1 --retry-tag-filter '@flaky and @anOtherTag'`
Then scenario "Flaky" attempt 0 step "Given a flaky step" has status "failed"
Then scenario "Flaky" attempt 1 step "Given a flaky step" has status "passed"
And scenario "Also Flaky" step "Given an other flaky step" has status "failed"
And it fails

Scenario: retrying a flaky test matching --retryTagFilter will eventually make it pass but not-matching will not be retried (OR operator between tags)
Scenario: retrying a flaky test matching --retry-tag-filter will eventually make it pass but not-matching will not be retried (OR operator between tags)
Given a file named "features/a.feature" with:
"""
Feature:
Expand Down Expand Up @@ -464,7 +485,7 @@ Feature: Retry flaky tests
throw 'fail'
})
"""
When I run cucumber-js with `--retry 1 --retryTagFilter '@anOtherTag or @oneMoreTag'`
When I run cucumber-js with `--retry 1 --retry-tag-filter '@anOtherTag or @oneMoreTag'`
Then scenario "Flaky" attempt 0 step "Given a flaky step" has status "failed"
And scenario "Flaky" attempt 1 step "Given a flaky step" has status "passed"
And scenario "Also Flaky" attempt 0 step "Given an other flaky step" has status "failed"
Expand Down
12 changes: 10 additions & 2 deletions src/cli/argv_parser.ts
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ const ArgvParser = {
validateRetryOptions(options: IParsedArgvOptions): void {
if (options.retryTagFilter !== '' && options.retry === 0) {
throw new Error(
'a positive --retry count must be specified when setting --retryTagFilter'
'a positive --retry count must be specified when setting --retry-tag-filter'
)
}
},
Expand Down Expand Up @@ -187,7 +187,7 @@ const ArgvParser = {
0
)
.option(
'--retryTagFilter <EXPRESSION>',
'--retryTagFilter, --retry-tag-filter <EXPRESSION>',
`only retries the features or scenarios with tags matching the expression (repeatable).
This option requires '--retry' to be specified.`,
ArgvParser.mergeTags,
Expand Down Expand Up @@ -223,6 +223,14 @@ const ArgvParser = {
args: program.args,
}
},

lint(fullArgv: string[]): void {
if (fullArgv.includes('--retryTagFilter')) {
console.warn(
'the argument --retryTagFilter is deprecated and will be removed in a future release; please use --retry-tag-filter'
)
}
},
}

export default ArgvParser
1 change: 1 addition & 0 deletions src/cli/configuration_builder.ts
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ export default class ConfigurationBuilder {
constructor({ argv, cwd }: INewConfigurationBuilderOptions) {
this.cwd = cwd

ArgvParser.lint(argv)
const parsedArgv = ArgvParser.parse(argv)
this.args = parsedArgv.args
this.options = parsedArgv.options
Expand Down
11 changes: 9 additions & 2 deletions src/cli/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import Gherkin from 'gherkin'
import { ISupportCodeLibrary } from '../support_code_library_builder/types'
import { IParsedArgvFormatOptions } from './argv_parser'
import { WriteStream } from 'fs'

const { incrementing, uuid } = IdGenerator

export interface ICliRunResult {
Expand Down Expand Up @@ -64,8 +65,14 @@ export default class Cli {
}

async getConfiguration(): Promise<IConfiguration> {
const fullArgv = await getExpandedArgv({ argv: this.argv, cwd: this.cwd })
return ConfigurationBuilder.build({ argv: fullArgv, cwd: this.cwd })
const fullArgv = await getExpandedArgv({
argv: this.argv,
cwd: this.cwd,
})
return ConfigurationBuilder.build({
argv: fullArgv,
cwd: this.cwd,
})
}

async initializeFormatters({
Expand Down

0 comments on commit 0bbb4b9

Please sign in to comment.