Proposal: remove strict and always fail on undefined/pending steps

[charlierudolph]

I have personally always use --strict and don't see why its needed. What is the reasoning behind undefined/pending steps exiting with status zero?

[aslakhellesoy]

Could you dig in the google group archives and/or github to find the original discussion? Probably 2008-2009 I'd guess

[aslakhellesoy]

@sebrose please wheigh in

[ajbw]

We don't use --strict, because we need a way of turning tests yellow without failing the entire build, so we are overloading pending and turning --strict off.

To explain our use case: we are checking routing to downstream HTTP services and don't 100% care if the service responds with 4xx or 5xx, because we don't control the downstream services, but we still would like to know (e.g., so that if people come to us and say "why isn't routing to service X working", we can say, "because service X is returning Y"). Therefore, we have step definitions that say "response code should not be" that return pending if they find a match (rather than raising an error).

We obviously would prefer that there was a clean, supported way to indicate a warning (as opposed to a fatal error), but as that doesn't seem to be supported in cucumber at this time, we use pending, and therefore require the ability to turn off --strict so that our builds don't fail.

[charlierudolph]

Coming back to this. I think this should be removed because of the poor practice it lets you walk into. When i first came onto cucumber-js, the builds looked like this. One of the first things I wanted to do when I joined was get the test suite to be entirely green (and using --strict).

I don't think the ability to have "passing" tests while they are incomplete is a practice we want to have as the default. I would be okay with making strict the default and allowing users some way to go --not-strict but I think that needs to be exception instead of the default behavior.

I don't think incomplete features should ever be committed to the main branch. They can be left on a feature branch or use tags to avoid running them. I don't see the use in running them as if an earlier step breaks, a developer might fix and not realize the feature is incomplete until later.

[paoloambrosio]

At the moment Cucumber-CPP runs the old Cucumber-TCK, but without implementing all features. This means that we have to use the default non-strict mode or we'd have to add tags to Cucumber-TCK.

Despite this need till we move away from it, I agree that it promotes bad practice so my 👍 to @charlierudolph's proposal of --not-strict.

[brasmusson]

Cucumber-JVM already have the ability to both turn on and turn off the boolean options, so to be consistent with the naming there, --no-strict rather then --not-strict should be used if reverting the default behaviour to fail on undefined or pending steps.

[charlierudolph]

The next version of cucumber-js is going to have strict as the default with a CLI option for --no-strict. Thus I'm going to close this. Please reopen if anyone wants to discuss further

[mattwynne]

@charlierudolph how about another ticket on this repo that documents what you did, with checkboxes for all the other implementations so we can keep track of which ones are working 'to spec'?

[charlierudolph]

Does that belong here @mattwynne? Wouldn't it be more useful to create issues on each repo instead?

[aslakhellesoy]

Let's keep all consolidation tickets here. Less work, easier to keep track of. We can use a (new) consolidation tag to group them.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.