Option to retry beforeAll(before) hook

[bluprince13]

What would you like?

Copied across from kuceb/cypress-plugin-retries#50

If there was an option to retry the beforeAll(before) hook when it fails that would be great.

Why is this needed?

Currently if the before hook in any test suite fails, the whole E2E fails and we have to manually trigger the E2E again. This is a pain.

Other

No response

[Michel73]

In the before hook, we want to set up our test data in the backend. If something went wrong in the test case, we need to restore the test data before we start again.
So we would really appreciate if this becomes a feature.

[SenneVProceedix]

I have a similar issue. I'm looking into writing something to retry the before hook myself but it would be extremely helpfull if this could be added in future versions!

[Michel73]

@SenneVProceedix I wrote a little workaround. See my answer here: https://stackoverflow.com/questions/71285827/cypress-e2e-before-hook-not-working-on-retries/71377694#71377694

[SenneVProceedix]

Hi @Michel73 Thank you for the information. Do you have any information in regards to (safely) retrying the before hook?

[MDG-JHowley]

This is kinda horrible and hacky but technically you could get the mocha runner from a beforeEach hook and if it's a retry call any before hook functions again...

I use the cypress cucumber preprocessor plugin, so some of this is a bit specific but...

In support/index.js or similar

beforeEach(function () {
  /**
   * Support retries with mocha/Cucumber before hooks
   *
   * retries are per test(scenario) not per suite(feature).
   * In some cases the initial test state may be the cause of the issue or
   * may have been broken by the test (I guess this should be exceptional as otherwise our test isolation could be a problem...)
   *
   * Either way this code allows us to effectively call the before hook again - thereby resetting inital test state
   *
   * See test-retries.feature for examples
   */
  if (cy.state('test').currentRetry() > 0) {
    /**
     * 1. Get the mocha runner
     */
    const runnable = Cypress.mocha.getRunner().currentRunnable;

    /**
     * 2. get the parent suite
     *
     * the current runnable context is this beforeEach hook, we need the suite to find any before hooks
     */

    const parentSuite = runnable.parent.suites[0];

    /**
     * 3. get hooks which are part of this test and not from elsewhere e.g. plugins
     */

    const hooks = Cypress._.filter(parentSuite._beforeAll, (o) => {
      return Cypress._.startsWith(
        o.invocationDetails.originalFile,
        `cypress/definitions`
      );
    });

    /**
     * 4. run the original hooked function!
     */
    hooks.forEach((hook) => {
      cy.log(`Re-running before hook: ${hook.hookId} - ${hook.hookName}`).then(
        () => {
          hook.fn();
        }
      );
    });
  }
});

[Michel73]

@SenneVProceedix Sorry but I haven't any further information about this.

[wilsonpage]

/**
 * A `before()` alternative that gets run when a failing test is retried.
 *
 * By default cypress `before()` isn't run when a test below it fails
 * and is retried. Because we use `before()` as a place to setup state
 * before running assertions inside `it()` this means we can't make use
 * of cypress retry functionality to make our suites more reliable.
 *
 * https://github.com/cypress-io/cypress/issues/19458
 * https://stackoverflow.com/questions/71285827/cypress-e2e-before-hook-not-working-on-retries
 */
export const retryableBefore = (fn) => {
  let shouldRun = true;

  // we use beforeEach as cypress will run this on retry attempt
  // we just abort early if we detected that it's already run
  beforeEach(() => {
    if (!shouldRun) return;
    shouldRun = false;
    fn();
  });

  // When a test fails we flip the `shouldRun` flag back to true
  // so when cypress retries and runs the `beforeEach()` before
  // the test that failed, we'll run the `fn()` logic once more.
  Cypress.on('test:after:run', (result) => {
    if (result.state === 'failed') {
      if (result.currentRetry < result.retries) {
        shouldRun = true;
      }
    }
  });
};

Use in place of before():

describe('my suite', () => {
  retryableBefore(() => {
    // reset database and seed with test data …

    cy.visit('/some/page');
  });

  it('my test 2', () => {
    …
  });

  it('test 2', () => {
    …
  });
  
  describe('my suite', () => {
    retryableBefore(() => {
      // do something in ui
    });

    it('my test 3', () => {
      …
    });

    it('test 4', () => {
      …
    });
  });
});

If any of the tests fail and you have retries config set, Cypress will re-run the retryableBefore() block as expected.

[MDG-JHowley]

That's a lot cleaner than my mess

[InvisibleExo]

@wilsonpage what about cases where the function placed in the before all hook fails, but causes the the condition variable to update boolean value?

I'm doing something similar with my test. It works if another beforeEach/afterEach or the test itself fails, but from local tests against the before itself might cause user test errors. Work around that would be to place the condition varible in a resolve or then block to promise to be executed only after your hook fully executed and didn't retry after.

[stokrattt]

/**
 * A `before()` alternative that gets run when a failing test is retried.
 *
 * By default cypress `before()` isn't run when a test below it fails
 * and is retried. Because we use `before()` as a place to setup state
 * before running assertions inside `it()` this means we can't make use
 * of cypress retry functionality to make our suites more reliable.
 *
 * https://github.com/cypress-io/cypress/issues/19458
 * https://stackoverflow.com/questions/71285827/cypress-e2e-before-hook-not-working-on-retries
 */
export const retryableBefore = (fn) => {
  let shouldRun = true;

  // we use beforeEach as cypress will run this on retry attempt
  // we just abort early if we detected that it's already run
  beforeEach(() => {
    if (!shouldRun) return;
    shouldRun = false;
    fn();
  });

  // When a test fails we flip the `shouldRun` flag back to true
  // so when cypress retries and runs the `beforeEach()` before
  // the test that failed, we'll run the `fn()` logic once more.
  Cypress.on('test:after:run', (result) => {
    if (result.state === 'failed') {
      if (result.currentRetry < result.retries) {
        shouldRun = true;
      }
    }
  });
};

Use in place of before():

describe('my suite', () => {
  retryableBefore(() => {
    // reset database and seed with test data …

    cy.visit('/some/page');
  });

  it('my test 2', () => {
    …
  });

  it('test 2', () => {
    …
  });
  
  describe('my suite', () => {
    retryableBefore(() => {
      // do something in ui
    });

    it('my test 3', () => {
      …
    });

    it('test 4', () => {
      …
    });
  });
});

If any of the tests fail and you have retries config set, Cypress will re-run the retryableBefore() block as expected

Hi, in which file write this function retryableBefore?
Thanks for answer

[wilsonpage]

@stokrattt it doesn't need to be in any special file, it's just a simple JS function, you can place it anywhere and import or require() it where needed.

[stokrattt]

@wilsonpage thanks, but I have currentRetry is not defined in the runnable.currentRetry. Please help

[emilyrohrbough]

related to: #17321

[JohnnyDevNull]

/**
 * A `before()` alternative that gets run when a failing test is retried.
 *
 * By default cypress `before()` isn't run when a test below it fails
 * and is retried. Because we use `before()` as a place to setup state
 * before running assertions inside `it()` this means we can't make use
 * of cypress retry functionality to make our suites more reliable.
 *
 * https://github.com/cypress-io/cypress/issues/19458
 * https://stackoverflow.com/questions/71285827/cypress-e2e-before-hook-not-working-on-retries
 */
export const retryableBefore = (fn) => {
  let shouldRun = true;

  // we use beforeEach as cypress will run this on retry attempt
  // we just abort early if we detected that it's already run
  beforeEach(() => {
    if (!shouldRun) return;
    shouldRun = false;
    fn();
  });

  // When a test fails we flip the `shouldRun` flag back to true
  // so when cypress retries and runs the `beforeEach()` before
  // the test that failed, we'll run the `fn()` logic once more.
  Cypress.on('test:after:run', (result) => {
    if (result.state === 'failed') {
      if (result.currentRetry < result.retries) {
        shouldRun = true;
      }
    }
  });
};

Use in place of before():

describe('my suite', () => {
  retryableBefore(() => {
    // reset database and seed with test data …

    cy.visit('/some/page');
  });

  it('my test 2', () => {
    …
  });

  it('test 2', () => {
    …
  });
  
  describe('my suite', () => {
    retryableBefore(() => {
      // do something in ui
    });

    it('my test 3', () => {
      …
    });

    it('test 4', () => {
      …
    });
  });
});

If any of the tests fail and you have retries config set, Cypress will re-run the retryableBefore() block as expected.

We ran also into the issue and thanks to your solution it works now.
But I don't understand why the before() hook not runs by default on a retry, that does not make any sense, but it is as it is...

[stokrattt]

/**
 * A `before()` alternative that gets run when a failing test is retried.
 *
 * By default cypress `before()` isn't run when a test below it fails
 * and is retried. Because we use `before()` as a place to setup state
 * before running assertions inside `it()` this means we can't make use
 * of cypress retry functionality to make our suites more reliable.
 *
 * https://github.com/cypress-io/cypress/issues/19458
 * https://stackoverflow.com/questions/71285827/cypress-e2e-before-hook-not-working-on-retries
 */
export const retryableBefore = (fn) => {
  let shouldRun = true;

  // we use beforeEach as cypress will run this on retry attempt
  // we just abort early if we detected that it's already run
  beforeEach(() => {
    if (!shouldRun) return;
    shouldRun = false;
    fn();
  });

  // When a test fails we flip the `shouldRun` flag back to true
  // so when cypress retries and runs the `beforeEach()` before
  // the test that failed, we'll run the `fn()` logic once more.
  Cypress.on('test:after:run', (result) => {
    if (result.state === 'failed') {
      if (result.currentRetry < result.retries) {
        shouldRun = true;
      }
    }
  });
};

Use in place of before():

describe('my suite', () => {
  retryableBefore(() => {
    // reset database and seed with test data …

    cy.visit('/some/page');
  });

  it('my test 2', () => {
    …
  });

  it('test 2', () => {
    …
  });
  
  describe('my suite', () => {
    retryableBefore(() => {
      // do something in ui
    });

    it('my test 3', () => {
      …
    });

    it('test 4', () => {
      …
    });
  });
});

If any of the tests fail and you have retries config set, Cypress will re-run the retryableBefore() block as expected.

We ran also into the issue and thanks to your solution it works now. But I don't understand why the before() hook not runs by default on a retry, that does not make any sense, but it is as it is...

i have this problem too

[Loren-Johnson]

Due to various API and database limitations, we often need to use the UI to configure test state. This is expensive enough that we don't want to do it before every test, so we have been using before blocks to run the setup code once and then let several tests assert various aspects of that state. However, we've discovered that since there are no retries in the before blocks, if we get even a little flake in the setup code it can completely shut down whole suites of tests.

So, we're left with a choice:

1. Leave things flaky and unreliable
2. Run the setup code in a beforeEach block - which costs a lot of (unnecessary) time
3. Roll our own before block with retries

None of those are very good options, but number 3 is clearly the best of the bunch and is what we will employ until something better comes along.

[rodolfostahelin]

I'm impressed that there are 2 issues about the need of this feature, they are at least 2 and a half years old, with comments per year, and yet we don't have a solution from Cypress itself.

The issues:

• The one we are in, created on Dec 22, 2021.
• Provide a way to re-run/retry an entire spec upon failure #17321, created on Jul 14, 2021.