Targeting more than JSDOM

[nhunzaker]

Regarding facebook/react#5703, I wanted to open up conversation about the possibility of supporting JavaScript environments beyond JSDOM.

What does this look like?

Does it involve leaning on tools such as Karma, browser-run, or zuul?

Could it be as simple as providing a way to package a fully-encapsulated test as an environment-agnostic payload?

In any case. I thought it was worth starting discussion here.

[cpojer]

Thanks for opening this issue. I'll triage some older issues that are related to this task and close them in favor of this fresh discussion. Browser testing is part of my plan for Jest, however it isn't something we are planning to use at FB so it doesn't have a high priority. This is something where I'd be excited to support someone from the community – this is certainly a challenging task but it would make Jest even more compelling to people :)

browser-run looks really awesome. If you look at the jsdom environment: https://github.com/facebook/jest/blob/b01ab18e0c703a715302a6959e3fca9e6b0260cf/src/environments/JSDOMEnvironment.js#L36 Jest already has a way for environments to run code in any way they want.

This would be easy if it wasn't for the fact that this callback would be async but Jest relies on synchronous calls to require to load the next module. Most of this is done in https://github.com/facebook/jest/blob/master/src/HasteModuleLoader/HasteModuleLoader.js (soon to be renamed to Runtime or similar). What's interesting however is that almost nothing in the module loader is entirely node specific and it could be packaged up and run in a browser.

At that point, here is what we'll need to do:

• Abstract HasteModuleLoader a bit more into a server and a client component. This might just be about making the seams clear and to isolate the parts that should be shipped to the browser: all the runtime (jest object), require implementation and mock configuration should be available on the browser.
• ship jest-jasmine and jest-util to the browser. Might require a tiny bit of refactoring.
• Transform and package all JavaScript and ship it to the client. JS should be sent as a Map<path, codeString> – it has to be sent as string and should not immediately be run because Jest resets the module registry for every test. This is important for test isolation so that tests don't depend on each other. Since Jest doesn't know which files to run until they are needed, the big change here would be to bundle everything upfront. This wouldn't work at FB scale, but we don't intend to ever use a browser runner, so I don't mind this not being 100 % efficient.
• Add a "mock" path implementation so that module resolution can work based on the Map mentioned before.
• Every individual test should be run in its own browser environment, which should probably be an iframe. Again, this is to preserve Jest's sandboxing guarantees.

[nhunzaker]

As a community member, is there a component here that would be best for me to work towards? I'm still familiarizing myself with Jest internals, but it seems like the bulk of the work has to be done in HasteModuleLoader.

One of the attractive parts of Karma is how easy it is to integrate browserify and webpack. It is not difficult to test code bundled in the same way between test and dev. This is attractive to me. While not a direct concern to this issue, I am curious if Jest could integrate with these tools.

This is a difficult problem. Looking forward to digging in.

[cpojer]

I think we'll first need to think pretty hard on how to implement this in a way that it's gonna work well before we start writing any code. Once we have a good idea of what we are gonna build here, I can think more about the staging and smaller chunks of work.

I think one of the big questions is how to create the bundle. webpack/browserify will be insufficient for a few reasons:

• We don't know which actual modules we are going to use and run until they are required at runtime. I experimented with doing this statically (in Haste2 #599) but mocking rules are runtime dependent and the static resolution was just way too slow (like, 2 order of magnitudes slower and not worth it). For example, when you do require('Foo') in Jest, it might require Foo.js or __mocks__/Foo.js and we only know which one at runtime (and it might be different for different tests, too). If you can think of a way to ship both to the browser, that would be great. We actually have an exhaustive list of all mock files and all JS files inside of a project in Jest, however that only works for Facebook's @providesModule modules and not for modules that are real CommonJS or node_modules. We need to figure out a way to get all of the relevant JS files into the browser.
• Once we have a way of shipping the relevant code, it'll be a matter of separating some Jest code so it can be run in the browser. We need to ship our own require implementation (that we can extract from Runtime) with all the mocking rules applied and we need to be able to run JS from strings (using eval) instead of just dumping a bundle into a script pack.

I'm all for integrating with existing tools, but it seems that we need the ability to let Jest do it's runtime work of sandboxing and shipping its own require implementation.

Another option would be to make the runtime parts of Jest work in the browser and consider them as entirely separate from the cli runner. One example is the modularization we are doing already: jest-mock (https://github.com/facebook/jest/tree/master/packages/jest-mock) can already be used in any environment. The bundling would then be up to the developer (just get the JS to the client, and put it somewhere that Jest can execute it). Instead of jest-cli, it would be jest-browser and it would be a separate framework that shares the same core and could potentially run the same tests but with vastly different configurations and setup. I'm not sure if that's a good idea.

[jquense]

been thinking about this a bit. Mocking is one of the big features of Jest over other options., this in particular makes a browser runner hard since the execution environment really needs to understand modules. to that end I think using webpack, under the hood, makes a lot of sense.

webpack can more than handle the resolution and mocking concerns, with the right set of plugins (which can consume jest-resolve). the suggestion tho, is to use it as an implementation detail/node environment polyfill.

i need to spend more time consolidating my thoughts (just typing this up quickly on my phone) but I think the big bits that would be needed in jest is pluggable environments (jsdom, browser) and an environment agnostic runtime/runner. I suppose similar to karma s launchers but maybe less generic

[nhunzaker]

@jquense I don't want to run the risk of repeating you too much but I wanted to share some thoughts too.

I think the big bits that would be needed in jest is pluggable environments (jsdom, browser) and an environment agnostic runtime/runner

I wonder what the minimum required surface area would be for browser execution. What happens when --env (jsdom|node) gets called? What code path activates differently?

Could mocking be handled within node, leaning on existing work, with most code execution happening through passing a bundle of javascript to a browser? Could this be as simple as preparing a bundled javascript file, injecting that into a browser context, and reporting back results?

I think this is how Karma's test harness works. I gravitate towards keeping this as simple as dumping JS on the page and tracking console.log/window.onerror. Though I'm sure it's more complicated than that.

[jquense]

karma requires a preprocessor for any sort of modular code, such as karma-webpack.

Could mocking be handled within node, leaning on existing work, with most code execution happening through passing a bundle of javascript to a browser?

this is what im suggesting, but the above would require building a file concatenator based on module/depedency graphs with custom resolution logic. essentially webpack :P i was thinking we could save a ton of hard work by leaveraging webpack instead of builing something. the rest is figuring out api boundries between the test runner, environment and test providers

[jquense]

I think that there is a big difference between the feature set of something like Jest vs karma. karma is a simply a pluggable runner/framework for running a set of tests in many different environments. Jest on the other hand makes a lot of assumptions about the environment. Jest has wider scope than karma; its more like karma + jasmine/mocha + jsdom-launcher + karma-webpack + rewire/inject-loader. Because of this I don't think that a super flexible launcher api like Karma's is well suited to Jest.

To that end I think that what you really need for jest-browser to work is the ability to create a node like environment in the browser, to which I say that webpack is the most robust and flexible tool for doing this already out there. It's plugin system can more than handle the concerns made by @cpojer above, as well as all the jest config options. Undoubtable it'd be slower, but I think that could be isolated to the browser-runner bits so that fb wouldn't mind ;P

As soon as a get a bit of time i want to see where the sore points are in the existing jest infra, and where assumptions are being made about the runtime. I think that will put us in a good place to talk about how the API's would need to change in order to support this sort of use case. I feel pretty optimistic thou, the team as already done an amazing job as modularizing and extracting jest into lots of its constituent bits :)

[cpojer]

I appreciate this discussion got started again. To me this has always been exciting but also risky for the project. I'll support this as much as I can and have some thoughts as well.

My ideal solution to this would be somewhere in jest-runtime and jest-environment-*. The problem is that right now; Jest requires and compiles files "just-in-time" when they are required which means they need to be synchronous. Because of the mocking feature, the entire module system and file system needs to be virtualized in the browser, as well as the entire transform pipeline with babel.

However, there are two more ways of solving this, both equally exciting:

• We split up Jest into many packages and a bunch of them can be used in the browser if they are bundled up, like jest-matchers.
• We could actually turn require into a generator and use a websocket connection to communicate with "jest sever". The problem is that right now require's are synchronous but we'd need to make them async. I'm not sure how this would work (generator or async/await) but with a babel transform and changing the require implementation in Jest, given that we are in full control, could work:

// How it works right now, jest wraps every file in a function and passes its require implementation:
(function(require, ...) {
  const apple = require('apple');
});

// How it could work, using a babel transform:
(function*(require, ...) {
  const apple = yield require('apple');
});

This would run on the client; when we call into require we'd request the transformed module over a websocket connection.

[thymikee]

@cpojer I'm actually not into the topic but in case you missed it, async import() recently got to Stage-3: https://github.com/tc39/proposal-dynamic-import
and I believe it's used in Webpack2 now: webpack/webpack#3413

[cpojer]

@thymikee yeah good point but that also requires to write a transform from require to async import, it's very similar I guess.

[jquense]

i'm gonna guess that requesting files over the wire (even locally) is going to be too slow. My thought was that just shipping all the mocks for required packages would be simpler and probably faster overall? Maybe i'm missing something here but most jest mocks are statically knowable, right? detecting calls to jest.mock or require.requireMock isn't too bad. For ambiguous cases you can just ship both.

Because of the mocking feature, the entire module system and file system needs to be virtualized in the browser, as well as the entire transform pipeline with babe

This speaks to my earlier point, its a tough one. Writing a module env for the browser is going to be super fraught and complicated. The thought was to leavage work already done on that front via webpack, for this bit. Especially since almost every nook and crany of webpack is pluggable.

[thattomperson]

Sorry if you guys have thought of this already but if the require('./foo') needs to be done at runtime for mocking reasons, why not just write a "jest-loader" which could rewrite the require statement to something more like

let foo
if (jest.mocking('./foo')) {
    foo = require('./__mocks__/foo');
} else {
    foo = require('./foo');
}

and let webpack just include them both?

edit
NVM
I miss read the previous comment

[cpojer]

@thattomperson jest-runtime does this branching already.

[nhunzaker]

Playing around with this a bit more. I've been experimenting with running Jest inside of an Electron browser instance:

https://gist.github.com/nhunzaker/5fc6f3ea153319e3140275914773d5bc

Electron is pretty neat because I can just require Jest inside of the browser. Trouble is that I don't have encapsulation. Test globals leak into each other.

Ideally, I think it would be neat if each test suite opened an isolated browser instance.

I'm not sure if there's really any benefit here. Eventually I think it would be neat if you could mount a React app to the browser for testing, stepping through breakpoints and watching behavior play out in a real browser.

No real questions here, but maybe it gives someone else some inspiration.

[thymikee]

Wow, this is super cool you're working on this!

I think you should spawn new browser in ElectronEnvironment#constructor, just like we do with spawning jsdom env.
Keep us posted

[cpojer]

oh hell yeah @nhunzaker. I'm extremely excited about this. Do you have a guide to set this up?

If @thymikee's suggestion doesn't work, could you use iframes in runScript?

[nhunzaker]

@cpojer I've put up the code here:

https://github.com/nhunzaker/jest-electron-environment

I think I could use the contextIsolation option from BrowserWindow to spin up an isolated browser and arbitrarily execute some code. That would be a similar approach to spitting JS into an iframe.

What I'm struggling with is how to get the actual code for the test:
https://github.com/nhunzaker/jest-electron-environment/blob/master/src/jest-environment-electron.js#L27

☝️ I'm not too familiar with Jasmine script contexts (if I have that right). Is there a way to get the string content for the test?

[chrishyle]

While running Jest in an electron shell is an interesting project, I think it only solves part of the issue for real world developers. While a CLI runner or an electron environment is great for our developer workflow, we have also found many issues over the years by running our unit tests in real browsers (especially older browsers) that we didn't find in those CLI environments or Chrome alone. I think we still need a way to run Jest in real browsers, even if the electron environment is successful (and I hope it is!).

[nhunzaker]

I completely agree. As someone less familiar with Jest internals, I found Electron to be an attractive target. Electron just works, and opens up a lot of possibilities, such as recording test sessions and stepping through browser behavior using a debugger. Still, I don't want to express neglect for real browsers with that enthusiasm.

Unfortunately, I also haven't been able to make the time for those problems. It would be cool to establish a more concrete roadmap for how we can get to testing in real browsers. I still have a lot to learn.

[suchipi]

My two cents: I got Jest running in NW.js, which is pretty similar to Electron, but it doesn't have the same node/browser context separation, so it's pretty easy to get the test code. Proof of concept here: https://github.com/suchipi/jest-node-nw-example

It's using the same browser globals every run right now, but creating new iframes for each test shouldn't be too hard

[intellix]

ChromeHeadless is now available, with support on it's way to karma-chrome-launcher: karma-runner/karma-chrome-launcher#111
Would love to know how it performs in comparison to jsdom

[joelgriffith]

I've been cooking up something out in the woods in Washington, and it's this library: https://github.com/joelgriffith/navalia. This tool has the capacity to write tests in a high-level browser API, and can parallelize them in a single browser context (think of tabs). Once I harden the API a bit I'm going to write some content on how to achieve this wizardry, and folks can hopefully begin to have a better experience with E2E-style tests.

Here's a short post on how it can instrument chrome for coverage (for those of you who are curios about the API): https://codeburst.io/capturing-unused-application-code-2b7594a9fe06

[MattKunze]

One thing I haven't seen mentioned in this discussion (probably missed it from previous issues) is that one of one of the big wins I've seen before with browser based tests is the ability to run them on various platforms (IE/Edge, iOS on Safari, etc) to validate edge cases in those environments. It certainly doesn't cover everything you can do with automation tools, but it's been helpful to me in the past.

Getting it working on Electron or NW.js would be great from a debugging perspective, but actually being able to run it on the target platform would be even better

[nhunzaker]

i don't really know much about how electron apps work so i'm just trying random things right now and see what makes the build green

I'm kind of in the same boat, it's my first Electron project :), but I'd be happy to help out where I can.

[neojski]

https://github.com/facebook-atom/jest-electron-runner worked really well for me.

[kimamula]

Running Jest test code on browsers using Karma is just as easy as configuring Karma to use with Jasmine as usual and adding the following lines above where the tests are loaded.

window.jest = require('jest-mock');
window.expect = require('expect');

// if you are using snapshot testing, which cannot be run on browsers
expect.extend({
  toMatchSnapshot: () => ({ pass: true }),
  // ...
});

See https://github.com/kimamula/jest-karma-angular-demo for demo.

What I want to do is to run tests usually on JSDOM (as it is faster) but sometimes on browsers to make sure my code is compatible with target browsers, which is achieved by the above setup.

[suchipi]

@kimamula you can use fs-remote to run snapshot testing in browser(s); see here for an example.

One thing to note with your setup is that module mocking won't work.

[kimamula]

@suchipi Thanks for your feedback.
I will investigate if I can make module mocking work.

[guoyunhe]

There is a karma-jest plugin. However, it doesn't have a stable release yet.

[github-actions]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 30 days.

[eps1lon]

comment

[unional]

Shout out to https://github.com/kayahr/jest-electron-runner to continue the archived jest-electron-runner

[github-actions]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 30 days.

[emilio-martinez]

Activity