Implement possibility to overwrite the step status for cucumber messages/json

[artem-sokorskyi]

Current behavior

Basically, currently there's no way (at least known to me) how to overwrite the result status of the step on the scope of the cucumber message/jsonlogs.

Let's imagine there are some huge scenarios with a lot of steps in itself. Some of the assertion steps just check whether element contains expected text and these steps are in the middle of the scenario.

The problem is when the middle step fails, then next steps are skipped and runner gets to the next scenario.

Here's an idea to catch all errors from custom assertion methods (there's no way for doing that with cypress native commands like "should" or "and") like I mentioned about checking the text value of element, and push it to the array storing these errors. Following that, the runner will continue execution and my steps will be marked as passed.

On the end of scenario I would like to map all these errors from array with their corresponding steps to mark those as failed. Thus, in the html report my scenario will have several failed steps instead of just one with the rest to be skipped.

But if error happens in cypress command or any other uncatchable error, it will follow the usual behaviour when the scenario is immediately terminated and step is marked as failed. So not always it will be that runner executes all listed steps of the scenario (only those that I intend to catch).

Desired behavior

A way to mark steps where failures were caught to the "failed" status on the cucumber messages/json logs level thus my report will accurately represent the scenario execution result

Test code to reproduce

Here's just an example of how I would catch it inside my custom assertion method. Pay attention to the try/catch block in the bottom

`

  cy.get(elementPath).should((textContentList) => {
    const { isSensitive, sensitiveKey } = secureOptions;
    textContentList.forEach((currentText, index) => {
      const targetIndex = focusIndex || index;
      const lowerCaseTrimmedExpectedText = expectedText.toLowerCase().trim();
      const testResult = {
        contain: currentText.includes(lowerCaseTrimmedExpectedText),
        have: lowerCaseTrimmedExpectedText === currentText,
      }[equalityString];

      const printedCurrentValue = currentText;
      const printedExpectedValue = lowerCaseTrimmedExpectedText;

      const finalTestResult = isNegative ? !testResult : testResult;
      const assertionCoditionString = `${isNegative ? 'not ' : ''}${equalityString} ${assertion}`;

      const commandMsg = `**Actual ${assertion}:** ${printedCurrentValue}\n**Expected ${assertion}:** ${printedExpectedValue}\n**Test result:** ${finalTestResult}`;
      const errorMsg = `The "${assertionCoditionString}" condition is failed${
        textContentList.length === 1 && focusIndex == null
          ? ''
          : ` for the item #${targetIndex + 1} (count starts from 1)`
      }.\nThe ${printedCurrentValue} current value does not satisfy the ${printedExpectedValue}`;
      try {
        if (!finalTestResult) throw new Error(errorMsg);
        this.catchableErrors = this.catchableErrors.filter(
          (catchabelError) => !catchabelError.stepId === window.testState.pickleStep.id,
        );
      } catch (err) {
        const { argument, text: stepName, id: stepId } = window.testState.pickleStep;
        this.catchableErrors.push({ stepName, stepId, argument, errorMsg });
      }
    });
  });

`

Versions

• Cypress version: 13.3.1
• Preprocessor version: 18.0.6
• Node version: 18.17.1

Checklist

• [ x ] I've read the FAQ.
• [ x ] I've read instructions for logging issues.
• [ x ] I'm not using cypress-cucumber-preprocessor@4.3.1 (package name has changed and it is no longer the most recent version, see #689).

[badeball]

Native support for soft errors has been proposed before. I agree with Cucumber's maintainers, that the use cases for this are questionable and I remain hesistant in adding complexity to support it. Furthermore, I don't want to see any further drift between the user experience in cucumber-js and the processor. Thus, I am closing this issue.

[badeball]

As an alternative, you can store soft errors yourself and post-process the messages, as shown below.

Feature: soft errors
  Scenario: soft errors
    Given a step with soft error
    And a normal step

const { BeforeStep, AfterStep, Given } = require("@badeball/cypress-cucumber-preprocessor");

BeforeStep(function () {
  delete this.softError;
});

AfterStep(function ({ testStepId }) {
  if (this.softError) {
    cy.task("store-soft-error", {
      testStepId,
      message: this.softError.message
    });
  }
});

Given("a step with soft error", function () {
  this.softError = new Error("some error");
});

Given("a normal step", () => {});

const fs = require("fs/promises");
const { defineConfig } = require("cypress");
const createBundler = require("@bahmutov/cypress-esbuild-preprocessor");
const {
  addCucumberPreprocessorPlugin,
  afterRunHandler,
} = require("@badeball/cypress-cucumber-preprocessor");
const {
  createEsbuildPlugin,
} = require("@badeball/cypress-cucumber-preprocessor/esbuild");

async function setupNodeEvents(on, config) {
  // This is required for the preprocessor to be able to generate JSON reports after each run, and more,
  await addCucumberPreprocessorPlugin(on, config, {
    omitAfterRunHandler: true,
  });

  on(
    "file:preprocessor",
    createBundler({
      plugins: [createEsbuildPlugin(config)],
    })
  );

  const softErrors = [];

  on("task", {
    "store-soft-error": (softError) => {
      softErrors.push(softError);
      return true;
    },
  });

  on("after:run", async () => {
    await afterRunHandler(config);

    if (!config.isTextTerminal) {
      return true;
    }

    const messagesPath = "cucumber-messages.ndjson";

    const messages = (await fs.readFile(messagesPath))
      .toString()
      .trim()
      .split("\n")
      .map(JSON.parse);

    for (const message of messages) {
      if (message.testStepFinished) {
        const matchingSoftError = softErrors.find(
          (softError) =>
            softError.testStepId === message.testStepFinished.testStepId
        );

        if (matchingSoftError) {
          const { testStepResult } = message.testStepFinished;
  
          testStepResult.status = "FAILED";
          testStepResult.message = matchingSoftError.message;
        }
      }
    }

    await fs.writeFile(messagesPath, messages.map(JSON.stringify).join("\n"));

    return true;
  });

  // Make sure to return the config object as it might have been modified by the plugin.
  return config;
}

module.exports = defineConfig({
  e2e: {
    specPattern: "**/*.feature",
    setupNodeEvents,
  },
});

[artem-sokorskyi]

@badeball thank you a lot for providing this code sample!!! Especially the way how to store globally softErrors as you showed is wonderful

[artem-sokorskyi]

@badeball it works as a way to modify the messages.ndjson!

I'm using multiple cucumber html reporter which relies on the log.json and this file is generated before the "after:run" event in cypress. So no matter how I modify the messages.ndjson, it still has no effect on the log.json.

Any thoughts how to bypass that behaviour and be able to regenerate the log.json in the after:run event?)

[artem-sokorskyi]

@badeball I figured it out, thanks again. Without your code sample, I wouldn't pull it off

The code should be placed above await afterRunHandler(config); as this handler generates the log.json file