Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cli: use kebab-case for --retry-tag-filter arg #1293

Merged
merged 14 commits into from
Mar 17, 2020
2 changes: 1 addition & 1 deletion docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ By default, cucumber-js runs the entire suite and reports all the failures. This
## Retry failing tests

Use `--retry <int>` to rerun tests that have been failing. This can be very helpful for flaky tests.
To only retry failing tests in a subset of test use `--retryTagFilter <EXPRESSION>` (use the same as in Use [Tags](#tags))
To only retry failing tests in a subset of test use `--retry-tag-filter <EXPRESSION>` (use the same as in Use [Tags](#tags))

## Transpilation

Expand Down
44 changes: 32 additions & 12 deletions features/retry.feature
Original file line number Diff line number Diff line change
@@ -1,17 +1,37 @@
Feature: Retry flaky tests

Using the `--retry` flag will retry failing tests for the specified number of times
Additionally using the `--retryTagFilter` flag will re-run only tests matching the tag expression
Additionally using the `--retry-tag-filter` flag will re-run only tests matching the tag expression

@spawn
Scenario: running Cucumber JS with --retryTagFilter but no positive --retry will fail
When I run cucumber-js with `--retryTagFilter @flaky`
Scenario: running Cucumber JS with --retry-tag-filter but no positive --retry will fail
When I run cucumber-js with `--retry-tag-filter @flaky`
Then the error output contains the text:
"""
Error: a positive --retry count must be specified when setting --retryTagFilter
Error: a positive --retry count must be specified when setting --retry-tag-filter
"""
And it fails

Scenario: running Cucumber JS with --retryTagFilter in camel case will result in a warning
Given a file named "features/a.feature" with:
"""
Feature:
Scenario:
Given a step
"""
Given a file named "features/step_definitions/cucumber_steps.js" with:
"""
const {Given} = require('cucumber')

Given(/^a step$/, function() {})
"""
When I run cucumber-js with `--retry 1 --retryTagFilter @flaky`
Then it issues the warning:
"""
the argument --retryTagFilter is deprecated and will be removed in a future release; please use --retry-tag-filter
"""
But it passes

Scenario: running Cucumber JS with negative --retry will fail
When I run cucumber-js with `--retry -1`
Then the error output contains the text:
Expand Down Expand Up @@ -322,7 +342,7 @@ Feature: Retry flaky tests
"""
And it fails

Scenario: retrying a flaky test matching --retryTagFilter will eventually make it pass
Scenario: retrying a flaky test matching --retry-tag-filter will eventually make it pass
Given a file named "features/a.feature" with:
"""
Feature:
Expand All @@ -344,12 +364,12 @@ Feature: Retry flaky tests
throw 'fail'
})
"""
When I run cucumber-js with `--retry 1 --retryTagFilter '@flaky'`
When I run cucumber-js with `--retry 1 --retry-tag-filter '@flaky'`
Then scenario "Flaky" attempt 0 step "Given a flaky step" has status "failed"
Then scenario "Flaky" attempt 1 step "Given a flaky step" has status "passed"
And it passes

Scenario: a flaky test not matching --retryTagFilter won't re-run and just fail
Scenario: a flaky test not matching --retry-tag-filter won't re-run and just fail
Given a file named "features/a.feature" with:
"""
Feature:
Expand All @@ -371,11 +391,11 @@ Feature: Retry flaky tests
throw 'fail'
})
"""
When I run cucumber-js with `--retry 1 --retryTagFilter '@not_flaky'`
When I run cucumber-js with `--retry 1 --retry-tag-filter '@not_flaky'`
Then scenario "Flaky" step "Given a flaky step" has status "failed"
And it fails

Scenario: retrying a flaky test matching --retryTagFilter will eventually make it pass but not-matching will not be retried (AND operator between tags)
Scenario: retrying a flaky test matching --retry-tag-filter will eventually make it pass but not-matching will not be retried (AND operator between tags)
Given a file named "features/a.feature" with:
"""
Feature:
Expand Down Expand Up @@ -410,13 +430,13 @@ Feature: Retry flaky tests
throw 'fail'
})
"""
When I run cucumber-js with `--retry 1 --retryTagFilter '@flaky and @anOtherTag'`
When I run cucumber-js with `--retry 1 --retry-tag-filter '@flaky and @anOtherTag'`
Then scenario "Flaky" attempt 0 step "Given a flaky step" has status "failed"
Then scenario "Flaky" attempt 1 step "Given a flaky step" has status "passed"
And scenario "Also Flaky" step "Given an other flaky step" has status "failed"
And it fails

Scenario: retrying a flaky test matching --retryTagFilter will eventually make it pass but not-matching will not be retried (OR operator between tags)
Scenario: retrying a flaky test matching --retry-tag-filter will eventually make it pass but not-matching will not be retried (OR operator between tags)
Given a file named "features/a.feature" with:
"""
Feature:
Expand Down Expand Up @@ -464,7 +484,7 @@ Feature: Retry flaky tests
throw 'fail'
})
"""
When I run cucumber-js with `--retry 1 --retryTagFilter '@anOtherTag or @oneMoreTag'`
When I run cucumber-js with `--retry 1 --retry-tag-filter '@anOtherTag or @oneMoreTag'`
Then scenario "Flaky" attempt 0 step "Given a flaky step" has status "failed"
And scenario "Flaky" attempt 1 step "Given a flaky step" has status "passed"
And scenario "Also Flaky" attempt 0 step "Given an other flaky step" has status "failed"
Expand Down
5 changes: 5 additions & 0 deletions features/step_definitions/cli_steps.ts
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,11 @@ Then(/^it fails$/, function(this: World) {
this.verifiedLastRunError = true
})

Then(/^it issues the warning:$/, function(this: World, text: string) {
const warnings: string[] = this.lastRun.warnings
expect(warnings).to.include(text)
})

Then(/^it outputs the text:$/, function(this: World, text) {
const actualOutput = normalizeText(this.lastRun.output)
const expectedOutput = normalizeText(text)
Expand Down
7 changes: 7 additions & 0 deletions features/support/world.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ interface ILastRun {
errorOutput: string
envelopes: messages.IEnvelope[]
output: string
warnings: string[]
}

interface IRunResult {
Expand Down Expand Up @@ -49,6 +50,10 @@ export class World {
return arg
})
const cwd = this.tmpDir
const warnings: string[] = []
const warn = (message: string): void => {
warnings.push(message)
}

let result: IRunResult

Expand All @@ -64,6 +69,7 @@ export class World {
argv: args,
cwd,
stdout,
warn,
})
let error: any, stderr: string
try {
Expand Down Expand Up @@ -105,6 +111,7 @@ export class World {
errorOutput: result.stderr,
envelopes,
output: colors.strip(result.stdout),
warnings,
}
this.verifiedLastRunError = false
expect(this.lastRun.output).to.not.include('Unhandled rejection')
Expand Down
4 changes: 2 additions & 2 deletions src/cli/argv_parser.ts
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ const ArgvParser = {
validateRetryOptions(options: IParsedArgvOptions): void {
if (options.retryTagFilter !== '' && options.retry === 0) {
throw new Error(
'a positive --retry count must be specified when setting --retryTagFilter'
'a positive --retry count must be specified when setting --retry-tag-filter'
)
}
},
Expand Down Expand Up @@ -188,7 +188,7 @@ const ArgvParser = {
0
)
.option(
'--retryTagFilter <EXPRESSION>',
'--retryTagFilter, --retry-tag-filter <EXPRESSION>',
`only retries the features or scenarios with tags matching the expression (repeatable).
This option requires '--retry' to be specified.`,
ArgvParser.mergeTags,
Expand Down
25 changes: 21 additions & 4 deletions src/cli/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -42,16 +42,33 @@ export default class Cli {
private readonly argv: string[]
private readonly cwd: string
private readonly stdout: IFormatterStream
private readonly warn: (message: string) => void

constructor({ argv, cwd, stdout }) {
constructor({ argv, cwd, stdout, warn = console.warn }) {
davidjgoss marked this conversation as resolved.
Show resolved Hide resolved
this.argv = argv
this.cwd = cwd
this.stdout = stdout
this.warn = warn
}

async getConfiguration(): Promise<IConfiguration> {
const fullArgv = await getExpandedArgv({ argv: this.argv, cwd: this.cwd })
return ConfigurationBuilder.build({ argv: fullArgv, cwd: this.cwd })
const fullArgv = await getExpandedArgv({
argv: this.argv,
cwd: this.cwd,
})
this.lintArgv(fullArgv)
return ConfigurationBuilder.build({
argv: fullArgv,
cwd: this.cwd,
})
}

private lintArgv(fullArgv: string[]): void {
if (fullArgv.includes('--retryTagFilter')) {
this.warn(
'the argument --retryTagFilter is deprecated and will be removed in a future release; please use --retry-tag-filter'
)
}
davidjgoss marked this conversation as resolved.
Show resolved Hide resolved
}

async initializeFormatters({
Expand Down Expand Up @@ -83,7 +100,7 @@ export default class Cli {
}
if (type === 'progress-bar' && !(stream as TtyWriteStream).isTTY) {
const outputToName = outputTo === '' ? 'stdout' : outputTo
console.warn(
this.warn(
`Cannot use 'progress-bar' formatter for output to '${outputToName}' as not a TTY. Switching to 'progress' formatter.`
)
type = 'progress'
Expand Down