Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

global setup / teardown hooks #4308

Closed
boneskull opened this issue Jun 1, 2020 · 12 comments
Closed

global setup / teardown hooks #4308

boneskull opened this issue Jun 1, 2020 · 12 comments

Comments

@boneskull
Copy link
Member

@boneskull boneskull commented Jun 1, 2020

With the addition of root hook plugins and parallel mode, an important use case may have been omitted.

In serial mode, a root before all (before()) hook can perform async setup (e.g., starting a test server) and a root after all (after()) can perform async teardown (closing the server). This is not perfect, given Mocha could crash and the server would not get cleaned up, but I think it's behavior people expect.

In parallel mode, this behavior not available. Root hooks "before all" and "after all" run once per file. If you want to run something once and only once, you're kind of stuck.

One could abuse mochaHooks by defining a Promise-returning function, and doing setup there...

exports.mochaHooks = async () => {
  // do something once only
});

... but there's no associated "teardown".

The only real workaround that I'm aware of is via script, e.g., {"scripts": {"test": "startServer && mocha; stopServer"}}. Top-level await suffers from the same problem as the above example--no associated teardown.

An idea for this would be to implement two new "hooks" in mochaHooks, e.g., begin and end.

  • These would be aggregated before Mocha begins running tests, similar to the behavior of the other hooks
  • The hooks are guaranteed (within reason) to run once and only once.
  • These hooks can share a context with each other, but nothing else. You cannot access the current test context from begin, for instance. You could define this.foo in begin and access it in end, I suppose.
  • Standard Node.js-style callbacks and Promise-returning functions allowed
  • They will always run in the main process

Alternatives:

  • Instead of using mochaHooks, use a different property.
  • Instead of having a single property, use mochaBegin and mochaEnd properties.
  • Name it something else, but be careful not to conflate it with existing hooks or interfaces (setup is going to be very confusing for those using the tdd interface, for example).

cc @nicojs, who may have opinions about how this should work.

@boneskull
Copy link
Member Author

@boneskull boneskull commented Jun 1, 2020

FWIW I think it may be confusing to use mochaHooks for this, since there isn't a 1:1 mapping between e.g., begin and any other hook.

@boneskull boneskull added this to To do in Maintainer TODO Jun 1, 2020
@boneskull
Copy link
Member Author

@boneskull boneskull commented Jun 9, 2020

@mochajs/core Any ideas? @jan-molak?

It would be helpful to have someone with this actual use case. Might not figure this out until somebody has a problem with how it works and complains about it. 😄

@boneskull boneskull added parallel and removed needs-feedback labels Jun 9, 2020
@rickcarrier
Copy link

@rickcarrier rickcarrier commented Jun 15, 2020

I see a need for a begin() and end() hooks that run once for all describes in parallel mode. I'm currently doing some setup multiple times in the beforeAll() because there isn't a better option that allows me to set up once

@boneskull
Copy link
Member Author

@boneskull boneskull commented Jun 15, 2020

yes, I think this is needed, but unsure what the API should look like. open to ideas.

won’t have time to implement anything for a couple weeks at least

@craigtaub
Copy link
Member

@craigtaub craigtaub commented Jun 26, 2020

Been looking into how Jest does things. Setups and teardowns provided by Jest:

  • Once-global - creates state only available for this context, not tests. You can add basic properties to the process.env global here, but not resources (limitation of nodejs, type-casts any props to a string).
  • Single-use per-file - creates and shares state inside the current suite context
  • Shared per-file - creates and shares state inside the current suite context

There is an example for getting it to work for Puppeteer (here) which uses 2 of the above.

Essentially nothing I found providing a global setup and teardown for setting and sharing state between processes.
They have many GH issues for this. Most of the discussion around the json serialisation of resources problem.

There might be some libs out there which could to help us, but needs more investigation. I'll see if I can find anything on AVA.
Hope this helps.

@boneskull
Copy link
Member Author

@boneskull boneskull commented Jun 26, 2020

Thanks. I think a mochaGlobalSetup and mochaGlobalTeardown would be appropriate then. I'll see if we can share the context object between them.

@boneskull
Copy link
Member Author

@boneskull boneskull commented Jun 26, 2020

I think--but am not sure--that we should not re-run these when watching files.

@jgehrcke
Copy link
Contributor

@jgehrcke jgehrcke commented Jul 7, 2020

I think a mochaGlobalSetup and mochaGlobalTeardown would be appropriate then

👍 This has little mental overhead and aligns with existing concepts. I was looking for something like this just today.

When we look at https://mochajs.org/#run-cycle-overview that visualization I think immediately makes the point that we need a hook at a hierarchy level closer to the root of things (I looked at this picture today and wondered: why would there be no "global hook"?, then found this issue here.)

The https://mochajs.org/#root-hook-plugins section was a bit repelling as of the complexity, and caveats.

I spent many years writing tests in the Python ecosystem using pytest. I must say that I miss the concept of scoped "fixtures" in the JavaScript testing ecosystem. I do not like the word "fixture" too much, but let's think of it as a dependency. An individual test can require a dependency. But a test suite (test module) can, too. But also the entire test session can require/request a dependency. What we think of as "global" here would be global within a "test session" in pytest. So, in pytest, if you define a "session-scoped fixture" then the session would "consume it" (run its code), once (because there is only one session, the current session, think "test run", comprised of many test modules), and before running the first test in the first module. For testing complex systems we've always made heavy use of this concept; with quite a bit of business logic being executed as part of global test suite (well, session) setup and teardown.

@boneskull
Copy link
Member Author

@boneskull boneskull commented Jul 7, 2020

The https://mochajs.org/#root-hook-plugins section was a bit repelling as of the complexity, and caveats.

Yes, if we add something like this we will reduce the complexity there (at least in terms of documentation). If you want to run a hook ("fixture") once, use mochaGlobalSetup; if you want to run it before all tests in a suite in a given file, use a before hook; if you want to run it before each test, use a beforeEach hook. There may not be any practical difference between mochaGlobalSetup and a before in Mocha's default serial mode, but if they are used as prescribed, the user can expect consistent behavior in serial and parallel mode.

(hope that made sense)

@boneskull boneskull moved this from To do to In progress in Maintainer TODO Jul 7, 2020
@boneskull boneskull self-assigned this Jul 7, 2020
@sarbbottam
Copy link

@sarbbottam sarbbottam commented Aug 31, 2020

Looking forward to a resolution to a global setup that runs only once for all the test files.

boneskull added a commit that referenced this issue Sep 2, 2020
boneskull added a commit that referenced this issue Sep 2, 2020
@boneskull boneskull closed this in 12db9db Sep 8, 2020
Maintainer TODO automation moved this from In progress to Done Sep 8, 2020
@boneskull
Copy link
Member Author

@boneskull boneskull commented Sep 8, 2020

This will be released in v8.2.0. I do not know yet when v8.2.0 will be released.

@rickcarrier
Copy link

@rickcarrier rickcarrier commented Sep 25, 2020

@boneskull Any estimate when v8.2.0 will be released? I'm trying to prioritize some work based on when this fix will be available. Thanks!

boneskull added a commit that referenced this issue Oct 12, 2020
- add "fixture flowchart wizard" to docs
- added source `.sketch` file to repo
- enabled skip of supporter image download in 11ty via env var `MOCHA_DOCS_SKIP_IMAGE_DOWNLOAD` for faster iteration
- made the links underline on hover
- Ref: #4308 

Signed-off-by: Christopher Hiller <boneskull@boneskull.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.