Diagnostics support for undefined steps in scenario outlines

[kieran-ryan]

🤔 What's the problem you're trying to solve?

The Cucumber Language Service provides diagnostics support for whether steps in a scenario are 'undefined'. This support does not extend to scenario outlines. This is a result of the challenges associated with defining 'undefined steps' for scenario outlines in the context of parameters that reference examples table values (captured below).

Scenario with diagnostic for undefined step:

Scenario outline without diagnostic for undefined step:

✨ What's your proposed solution?

Substitute examples table values into steps, before passing the steps to the undefined step validation function - this ensures no change is required to that logic. Additionally to ensure clear behaviour, to provide diagnostics for:

• table value outliers that do not match the step definition
• steps with parameters that do not have an associated header in the examples table

⛏ Have you considered any alternatives or workarounds?

The language service is embedded into tools used by editors such as Monaco and Visual Studio Code and thus there does not appear to be a workaround available for end users.

📚 Any additional context?

Challenges

Parameters may not be considered a match of a step definition

If we were to remove the scenario outline check, we would incorrectly observe that our defined step ("I have {int} cukes in my belly"), would be marked as 'undefined'. This is a result of the raw <qty> parameter not matching the regular expression for the type int itself, rather than for the values of <qty> in the examples table itself.

Ignoring steps with parameters may lead to unclear behaviour

We could modify the check that prevents checking the test for undefined steps if it is a scenario outline, to instead prevent checking test steps that have parameters only...

step: function (step, diagnostics) {
  const parameters = step.text.match(/<.*?>/g);
  if (parameters) {
    return diagnostics;
  }

  ...

  return diagnostics;
}

....This would ensure that steps without parameters are correctly checked whether they are undefined. However this may create unclear behaviour: defined steps and steps without parameters will both appear without diagnostics; as if they are both defined - which is not the case. Though one might argue that with the present solution, it could be misinterpreted that all steps in a scenario outline are defined, as they will not appear with any diagnostics.

Substituting values may lead to unclear behaviour until the examples table is populated

Alternatively, for steps with parameters, we could iterate through the parameter's example table values, substituting them into the test, and running it against the 'undefined' step validation function - which would not require modification.

If the step was considered defined with any data, then we can overall consider the step defined and do not need to assign it a diagnostic. Though we would need to mark any 'undefined' values within the examples table itself.

This would only work once the examples table has content and has a matching parameter. This again presents unclear behaviour where a step with parameters could be misinterpreted as defined until its associated examples table value is in place. To rectify this, a diagnostic could be applied to the parameter that it 'requires an associated header in an examples table' until it is authored. This would ensure it is appropriately annotated and prompt the user to populate the table, at which point it would be checked whether it's undefined.