New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Please consider adding native support for ES modules #4842

Open
kirlat opened this Issue Nov 5, 2017 · 45 comments

Comments

Projects
None yet
@kirlat
Copy link

kirlat commented Nov 5, 2017

Do you want to request a feature or report a bug?
I want to request a feature.
What is the current behavior?
Right now Jest does not support test suites with import statement. They result in the following error:

SyntaxError: Unexpected token import
      
      at ScriptTransformer._transformAndBuildScript (node_modules/jest-runtime/build/script_transformer.js:305:17)
          at Generator.next (<anonymous>)
          at new Promise (<anonymous>)

What is the expected behavior?
Would be great if Jest supported ES modules natively.

Please provide your exact Jest configuration and mention your Jest, node, yarn/npm version and operating system.
Jest: 21.2.1
node: 8.9.0
npm: 5.5.1

Before, native support of ES modules was not possible since node.js did not support them. Starting from a few versions ago, node.js added support of ES modules with a flag (https://nodejs.org/api/esm.html). It would be absolutely great if Jest would match this with adding support of ES modules too, probably with a flag or even without it.

Node.js requires ES modules to have an .mjs extension. In order to support ES modules Jest needs to add a recognition of those extensions. Jest will also need to pass an --experimental-modules flag to node.js until node will implement support of modules without a flag. I'm not sure if any other required changes would be needed within Jest in order to support this. I can only hope it will not be terribly hard.

Ideally, it would be cool if Jest would recognize modules even in files without .mjs extensions since code that targets browsers do not use them, but I don't know if it is ever possible. Node.js provides loader hooks for that (https://nodejs.org/api/esm.html) but this still doesn't solve the issue with a reliable determination of what type of module the file is.

I believe ES modules are a great feature vastly superior to all existing JS module solutions. Having it implemented in node.js opens a door for Jest to have it too. This would allow developers stick to using the first truly standardized JS module format not only throughout development, but trough testing as well.

@SimenB

This comment has been minimized.

Copy link
Collaborator

SimenB commented Nov 6, 2017

Jest has its own require implementation (https://github.com/facebook/jest/blob/master/packages/jest-runtime/src/index.js), so it would be way more involved than just supporting the syntax and default to looking at .mjs. I'm also against activating experimental flags.

It might be possible to automatically transpile import/export, but implementing the semantics is a huge undertaking, and probably blocked for a year because of support for node 6.

@cpojer

This comment has been minimized.

Copy link
Contributor

cpojer commented Nov 6, 2017

I agree with SimenB. We need a number of hooks from the node team to be able to make this work together with the vm module. However, I think we should support this in the meantime, but not by mirroring the full native implementation but rather by using babel and compiling it to require inside of babel-jest. I think for testing purposes this will work fine and we do not have to provide the same guarantees the node runtime needs to provide anyway.

@SimenB

This comment has been minimized.

Copy link
Collaborator

SimenB commented Nov 6, 2017

So just adding babel-plugin-transform-es2015-modules-commonjs & babel-plugin-dynamic-import-node to babel-jest?

@cpojer

This comment has been minimized.

Copy link
Contributor

cpojer commented Nov 6, 2017

Yeah, that's what I was thinking.

@TrySound

This comment has been minimized.

Copy link

TrySound commented Nov 6, 2017

Guys, what about integrating https://github.com/standard-things/esm ? It's fast and maintain a lot of edge cases.

@cpojer

This comment has been minimized.

Copy link
Contributor

cpojer commented Nov 6, 2017

@TrySound what would that look like concretely? Can you make a prototype?

@SimenB

This comment has been minimized.

Copy link
Collaborator

SimenB commented Nov 6, 2017

We still have our own require-implementation (needed for mocks), so I don't think that'd help much.

And we need to work both with Node's rules and the browser's rules.

I'd be very happy to be corrected and have it work perfectly for us :D

@SimenB

This comment has been minimized.

Copy link
Collaborator

SimenB commented Nov 14, 2017

@std/esm apparently should just work with jest: https://twitter.com/jdalton/status/930257653292400640

Could anyone give it a whirl and come back with a PR for the docs? 🙂

@TrySound

This comment has been minimized.

Copy link

TrySound commented Nov 14, 2017

I guess users would like to have support everywhere, but I found it works only for dependencies of test files.

// test.js
require = require('@std/esm')(module, { esm: 'js', cjs: true });
const utils = require('./utils');
// utils.js
export { default as update } from './update';

It's better, but not ideal.

@gaearon

This comment has been minimized.

Copy link
Member

gaearon commented Nov 27, 2017

So just adding babel-plugin-transform-es2015-modules-commonjs & babel-plugin-dynamic-import-node to babel-jest?

I don't think this is a great solution, as it doesn't perform any "missing export" checks that are so valuable with ES modules. For example, in React repo I started running build more often just because Rollup finds these mistakes but Jest with es2015-modules-commonjs does not.

@std/esm apparently should just work with jest

It would be pretty great to invest some time into their interop. Pretty sure this hacky solution will break eventually: https://stackoverflow.com/questions/46433678/specify-code-to-run-before-any-jest-setup-happens. But if it's just a matter of exposing something on Jest side, would be cool to see it supported.

@awidjaja

This comment has been minimized.

Copy link

awidjaja commented Dec 9, 2017

@SimenB, in my mind the immediate step would not be too complex. What is urgent is to allow people to work with .mjs module, even if babel is helping behind the test scene. Otherwise people might have to find different testing solution if they want to use .mjs.

  1. Fix the testMatch property to allow .mjs file to be found
    (currently it doesn't work even the regex seems correct already, maybe there's a hard code somewhere rejecting .mjs extension)
  2. Pass --experimental-modules flag when running node.js so that it will run natively
  3. Jest should treat .mjs the exact same way as .js today (e.g. in jest require) -- the import statements are already allowed inside .js today, so only need to allow .mjs extension.

The final solution might be complex and takes time, but it is inevitable isn't it?

@jcguarinpenaranda

This comment has been minimized.

Copy link

jcguarinpenaranda commented Feb 2, 2018

Hello, has someone been able to fix this error?

@skkap

This comment has been minimized.

Copy link

skkap commented Feb 7, 2018

We are using .mjs with "node --experimental-modules" option. Any workaround?

@jdalton

This comment has been minimized.

Copy link
Contributor

jdalton commented Feb 7, 2018

We are using .mjs with "node --experimental-modules" option. Any workaround?

That's experimental and not fully fleshed out. There is lots of churn still with basic things, like how to import a builtin module, still up in the air. Projects like AVA have started allowing the use of @std/esm as their loader pipeline if used (bypassing Babel). Maybe jest could take a similar approach.

@SimenB

This comment has been minimized.

Copy link
Collaborator

SimenB commented Feb 7, 2018

Supporting @std/esm is something we want to do, help to implement it is more than welcome!

@jdalton

This comment has been minimized.

Copy link
Contributor

jdalton commented Feb 8, 2018

@SimenB Can you chat sometime on hangouts?

@jdalton

This comment has been minimized.

Copy link
Contributor

jdalton commented Mar 23, 2018

Hi @SimenB 👋

An esm user has contributed esm + Jest demo and I figured it might be a good starting point for creating a more official cow path.

Update:

The esm + Jest demo has been updated with basic module name mapping support.

@SimenB

This comment has been minimized.

Copy link
Collaborator

SimenB commented Mar 29, 2018

That's pretty cool! Thanks for sharing.

We'll have to figure out where we want the integration to be. How would it handle CSS (or other non-js assets) files? Should it just be a transform? What about the built-in babel transform? How should Jest behave when it comes to the incoming loaders, if it does affect anything?

@kenotron

This comment has been minimized.

Copy link

kenotron commented Mar 29, 2018

It seems like there may be a benefit for a community contrib esm-enabled alternate jest runner (or an unofficial / experimental flag) so we can make progress on something like that. Would there be interest in that from jest team?

@SimenB

This comment has been minimized.

Copy link
Collaborator

SimenB commented Mar 29, 2018

require is not implemented in a runner, it's in the runtime itself. Any contributions towards making it pluggable is very much welcome (ref #848).

I'm sure if you can get the example code @jdalton linked to to work without issues (or close to it), it should be simple enough to load in the esm loader behind a flag in jest itself. One thing I see as an issue is that it wants the real module global, not the fake one we create. Not sure if it means modules can leak between tests? I don't know what esm does with it under the hood. It also doesn't handle mocks, so mocks with import would still break

@jdalton

This comment has been minimized.

Copy link
Contributor

jdalton commented Mar 29, 2018

May just need some tweaking of how Jest taps into CJS. For example when mocking the module object, instead of using a plain object, it could use require("module") and then wrap/overwrite module.require to intercept requests and juggle as needed.

Update:

I'm now working on enabling Jest out of the box with esm (with hopefully nothing Jest has to do differently). I'll continue to experiment over the weekend but early signs look good.

@JasonCust

This comment has been minimized.

Copy link

JasonCust commented Apr 30, 2018

@jdalton Any update with the esm compatibility?

@jdalton

This comment has been minimized.

Copy link
Contributor

jdalton commented Apr 30, 2018

Hi @JasonCust, wow my comment has gotten some attention!

I have made progress in identifying the work needed to enable Jest support in esm. In my experiments I got Jest loading and evaluating tests written in ESM. The work required on the esm loader side is to make the way we handle vm.Script use more generic. At the moment we hook into that primarily for REPL use which assumes a single module. We have to make that a bit more generic for Jest support to shake out. It doesn't look like Jest will have to change anything. The refactor of our vm.Script support is still on my TODO and will still be tackled before I release things like experimental WASM support. At the moment I've been squashing bugs that have popped up around improved APM and mocking support.

@yanickrochon

This comment has been minimized.

Copy link

yanickrochon commented Aug 21, 2018

Been trying to make this feature work for the past two hours. There are so many partial solutions, half-documented answers, all different from each other... It used to be easy to test node.js packages.

@Volune

This comment has been minimized.

Copy link

Volune commented Nov 9, 2018

So just adding babel-plugin-transform-es2015-modules-commonjs & babel-plugin-dynamic-import-node to babel-jest?

What happened to that idea? Any issue in implementing it?

@SurenAt93

This comment has been minimized.

Copy link

SurenAt93 commented Dec 16, 2018

Hi @jdalton. I was wondering if you could keep us up to date about status of this issue. Sorry if I bother you, but I am waiting for this for a while, and it would be better if we'll receive an update at least for last/next six months. Thank you :)

@jdalton

This comment has been minimized.

Copy link
Contributor

jdalton commented Dec 16, 2018

Hi @SurenAt93!

Thank you for your patience. I'm hoping to have a release out by the end of the month with Jest support. With that you'll specify esm as a Jest transform.

@kenotron

This comment has been minimized.

Copy link

kenotron commented Dec 16, 2018

Cool. How is that transform different than Babel?

@jdalton

This comment has been minimized.

Copy link
Contributor

jdalton commented Dec 16, 2018

@kenotron

Using Jest's transform option is a way for me to side-load the esm loader. So while it is technically transforming source too, it's also wiring up the esm loader.

If the question is more what's the difference between Babel and esm. Babel is a collection of packages that transforms source, one of the targets could be ESM syntax. The esm loader is a zero dependency loader that simulates the runtime environment. So esm supports things like dynamic import(), passes relevant test262 specs (temporal dead zones, live bindings, upfront-errors, etc.), and supports loading a mix of CJS/ESM/WASM to taste by configuration.

@SurenAt93

This comment has been minimized.

Copy link

SurenAt93 commented Dec 19, 2018

@jdalton Thank you for you work and support!

@lukasholzer

This comment has been minimized.

Copy link

lukasholzer commented Dec 19, 2018

@igasparetto

This comment has been minimized.

Copy link

igasparetto commented Jan 5, 2019

Been trying to make this feature work for the past two hours. There are so many partial solutions, half-documented answers, all different from each other... It used to be easy to test node.js packages.

I agree.

I created a simple Vue project, which also manifests the problem.
https://github.com/igasparetto/vue-jest-test
I wasn't able to get it to work.

I followed the instructions from the following pages:

My machine (not sure it should matter):

  • Windows 10 Pro 64-bit
  • Node v8.9.4
  • Vue: 3.2.3
  • npm: 6.5.0
@igasparetto

This comment has been minimized.

Copy link

igasparetto commented Jan 7, 2019

@kenotron

Using Jest's transform option is a way for me to side-load the esm loader. So while it is technically transforming source too, it's also wiring up the esm loader.

If the question is more what's the difference between Babel and esm. Babel is a collection of packages that transforms source, one of the targets could be ESM syntax. The esm loader is a zero dependency loader that simulates the runtime environment. So esm supports things like dynamic import(), passes relevant test262 specs (temporal dead zones, live bindings, upfront-errors, etc.), and supports loading a mix of CJS/ESM/WASM to taste by configuration.

@kenotron could you provide us an update please?

@kenotron

This comment has been minimized.

Copy link

kenotron commented Jan 7, 2019

@igasparetto, it's @jdalton's work that should enable this. I hadn't tried that solution just yet.

@SimenB

This comment has been minimized.

Copy link
Collaborator

SimenB commented Jan 7, 2019

Latest status there I think is: https://twitter.com/jdalton/status/1080627279124934661

@jdalton

This comment has been minimized.

Copy link
Contributor

jdalton commented Jan 7, 2019

Yep! Sorry it's taking longer than I wanted. Locally I now have all relevant test262 tests passing. In the course of getting those up the mocking related scenario tests slipped so I have to pick them back up before I can release. It isn't insurmountable, it just takes a bit of time. I'm presenting at Covalence Conf on Jan 16th and hope to have a release by then. In other news, npm tink early adopted esm for its ESM syntax support 🤝.

michaelwittwer added a commit to shiftcode/dynamo-easy that referenced this issue Jan 8, 2019

@bfred-it bfred-it referenced this issue Jan 14, 2019

Merged

Add tests #1

@igasparetto

This comment has been minimized.

Copy link

igasparetto commented Jan 17, 2019

Jan 16th and hope to have a release by then

@jdalton I hope the presentation went well.

Have you got an update on this release please?

@manniL

This comment has been minimized.

Copy link

manniL commented Jan 17, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment