Add global before and after hooks (including parallel run via Dashboard)

[denis-domanskii]

What would you like?

Add global "before" and "after" hooks, which will be executed before all specs and after all specs respectively, including case with a parallel run: each hook should be called once per whole specs run, not for each cypress instance.

Why is this needed?

I want to use a third-party service inside cypress tests. This service must be inited with init() command before build start and deinited after last test with deinit() to finalise build, generate report and send status to a GitHub. And now there is no place in Cypress where I can inject this init() and deinit() service functions, especially for a parallel run case.

I can't run it before and after each spec (using your default global hooks), because this service need to know when the last test is finished to send final status to the GitHub. I also can't use some kind of process finish hooks, because I can have 2 or 3 or 10 parallel Cypress processes on a different boxes in my CI.

Probably, the best solution for this issue is to add beforeSuite() and afterSuite() cypress hooks, which will be called before and after all specs run. As I know it was already implemented for a non-parallel run in a Cypress v2 #818 but was broken in a v3. Also I see that "run:end" event was added to a v4 (#2331), but looks it was removed later. Why?

As for parallel run - it can be a little tricky but still possible: Cypress Dashboard managing specs order, so when Dashboard sends first spec to an instance - you can just add "isFirst:true" flag to this request. And this flag will fire beforeSuite() hook only on one Cypress instance. Same for last spec - Dashboard is also know that current spec is a last in a specs queue, so just send "isLast:true" to fire afterSuite() hook for a specific instance.

Current non-friendly approach is already caused a lot of pain to the third-party plugins authors (Happo folks for example). I also looked into Applitools and Percy plugins - poor guys manage suite state on its own service side! It's totally not cool when we have a Dashboard to manage state, but still need to additionally manage it in other places. Cypress has a great architecture, so you must know that it always better to manage the state in a one place. More user confusion examples: a recent issue, first SO question, second, third, fourth, fifth... and unanswered discussion here.

Probably I missed something and solution without "you can write your own test runner" is already exist, but I can't find it :( If I can help somehow (e.g. contribute some code) - I am ready. But it looks that full support is impossible without small Dashboard adjustments.

[jennifer-shehane]

Duplicate of #6665 I know this issue looks abandoned but we're actually planning to pick up the work on this in the next few sprints.

A way you could do this today is using our Module API https://on.cypress.io/module-api

[denis-domanskii]

A way you could do this today is using our Module API https://on.cypress.io/module-api

Thank you @jennifer-shehane, I will very wait for this feature. Could you please explain a little - how Module API can help me with my issue (including parallel run)?

[jennifer-shehane]

You can now listen to before:run, after:run, before:spec and after:spec events in the plugins file when setting the experimentalRunEvents configuration option to true. See the appropriate events doc for more details: https://on.cypress.io/writing-a-plugin#List-of-events

For the Module API, you could run any piece of code you want before calling cypress.run, and then after receiving the results from that run, run whatever code you want outside of it.

[denis-domanskii]

Hi @jennifer-shehane, thank you for answer. I propose to reopen my issue as a parallel flow wasn't covered by #6665. Current before:run and after:run events are fired for each parallel instance, but I'm asking for event which will be fired once for all instances. So probably an additional before:suite and after:suite events should be added.

[patrickshaughnessy]

Hi @denis-domanskii and @jennifer-shehane . I would also like to reopen this issue (or perhaps create a new one) just for the parallel run part.

The after:run additions in v6 are great! But we still have a need to do some additional cleanup after all tests have completed in a parallel run.

I understand that this is much more complicated than the after:all hook. Parallel runs on the dashboard are using different machines and some global process internal to the dashboard that listens for the addition of machines / tests. So there's not really a way to put a hook like after:suite that the process running the Cypress tests can access. I think it would have to be implemented like a webhook callback or similar fired from the Cypress dashboard when the suite is completed. Not sure if there are other ideas, but that's what I would imagine given the current architecture.

[rdhelms]

@denis-domanskii Did you ever find a solution for parallel runs? Or should this be reopened? We have the same use case.

[denis-domanskii]

Hi @patrickshaughnessy, @rdhelms, I still have no solution for it. I just ask the Cypress team in an official gitter chat (https://gitter.im/cypress-io/cypress?at=604b9391a3a2f04f1fefc466) and will create a separate issue if my message will be ignored.

@jennifer-shehane, as you see, the issue is still actual.

[patrickshaughnessy]

@denis-domanskii @rdhelms I think the only way you'll be able to achieve this currently is to have some process manager handling the Cypress parallel runs. E.g. We ended up running the parallel jobs via Jenkins. Jenkins kicks off the Cypress parallel runs on a number of virtual machines, and we can add whatever steps needed to the Jenkins pipeline after all processes have completed on all machines (since the Jenkins process is still running).

In parallel runs, the Cypress dashboard is always listening for new tests to be added from other machines. It can only detect the end of the parallel run via timeout (i.e. after 2 minutes without receiving a signal of a new test getting added to the queue). So it's not really technically possible to have some kind of after:suite hook, because the process would have to have already exited for the parallel run to be considered complete.

[rdhelms]

I wonder if there's a way to take advantage of the way mocha does its global setup and teardown fixtures? In the docs they specifically say that they're meant to "Work identically parallel mode, watch mode, and serial mode"
https://mochajs.org/#global-fixtures

---

These are as opposed to their root beforeAll and afterAll hooks, which run for each file when running in parallel
https://mochajs.org/#available-root-hooks

[jennifer-shehane]

Having a hook to run after all specfiles are done running in parallel will be more complex to implement. It will require communication with the Dashboard API. The Dashboard and Test Runner process do not truly know what the last spec file is that will run - the Dashboard sits open listening for new specs to arrive and if it doesn't get communication of more specs running after a specific runCompletionDelay, then it closes out the run as finished.

So, the Test Runner processes would have already exited at this point when the run is complete with the current architecture. We'd have to expose some webhook from the Dashboard to listen to or some option in the Test Runner to wait/poll for completion before exiting. We wouldn't want to expose this second option to everyone because it would slow down the runs for everyone with the extra wait.