New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Targeting more than JSDOM #848

Open
nhunzaker opened this Issue Mar 30, 2016 · 44 comments

Comments

Projects
None yet
@nhunzaker
Contributor

nhunzaker commented Mar 30, 2016

Regarding facebook/react#5703, I wanted to open up conversation about the possibility of supporting JavaScript environments beyond JSDOM.

What does this look like?

Does it involve leaning on tools such as Karma, browser-run, or zuul?

Could it be as simple as providing a way to package a fully-encapsulated test as an environment-agnostic payload?

In any case. I thought it was worth starting discussion here.

@cpojer

This comment has been minimized.

Show comment
Hide comment
@cpojer

cpojer Mar 30, 2016

Contributor

Thanks for opening this issue. I'll triage some older issues that are related to this task and close them in favor of this fresh discussion. Browser testing is part of my plan for Jest, however it isn't something we are planning to use at FB so it doesn't have a high priority. This is something where I'd be excited to support someone from the community – this is certainly a challenging task but it would make Jest even more compelling to people :)

browser-run looks really awesome. If you look at the jsdom environment:

return this.global.eval(sourceText + '\n//# sourceURL=' + filename);
Jest already has a way for environments to run code in any way they want.

This would be easy if it wasn't for the fact that this callback would be async but Jest relies on synchronous calls to require to load the next module. Most of this is done in https://github.com/facebook/jest/blob/master/src/HasteModuleLoader/HasteModuleLoader.js (soon to be renamed to Runtime or similar). What's interesting however is that almost nothing in the module loader is entirely node specific and it could be packaged up and run in a browser.

At that point, here is what we'll need to do:

  • Abstract HasteModuleLoader a bit more into a server and a client component. This might just be about making the seams clear and to isolate the parts that should be shipped to the browser: all the runtime (jest object), require implementation and mock configuration should be available on the browser.
  • ship jest-jasmine and jest-util to the browser. Might require a tiny bit of refactoring.
  • Transform and package all JavaScript and ship it to the client. JS should be sent as a Map<path, codeString> – it has to be sent as string and should not immediately be run because Jest resets the module registry for every test. This is important for test isolation so that tests don't depend on each other. Since Jest doesn't know which files to run until they are needed, the big change here would be to bundle everything upfront. This wouldn't work at FB scale, but we don't intend to ever use a browser runner, so I don't mind this not being 100 % efficient.
  • Add a "mock" path implementation so that module resolution can work based on the Map mentioned before.
  • Every individual test should be run in its own browser environment, which should probably be an iframe. Again, this is to preserve Jest's sandboxing guarantees.
Contributor

cpojer commented Mar 30, 2016

Thanks for opening this issue. I'll triage some older issues that are related to this task and close them in favor of this fresh discussion. Browser testing is part of my plan for Jest, however it isn't something we are planning to use at FB so it doesn't have a high priority. This is something where I'd be excited to support someone from the community – this is certainly a challenging task but it would make Jest even more compelling to people :)

browser-run looks really awesome. If you look at the jsdom environment:

return this.global.eval(sourceText + '\n//# sourceURL=' + filename);
Jest already has a way for environments to run code in any way they want.

This would be easy if it wasn't for the fact that this callback would be async but Jest relies on synchronous calls to require to load the next module. Most of this is done in https://github.com/facebook/jest/blob/master/src/HasteModuleLoader/HasteModuleLoader.js (soon to be renamed to Runtime or similar). What's interesting however is that almost nothing in the module loader is entirely node specific and it could be packaged up and run in a browser.

At that point, here is what we'll need to do:

  • Abstract HasteModuleLoader a bit more into a server and a client component. This might just be about making the seams clear and to isolate the parts that should be shipped to the browser: all the runtime (jest object), require implementation and mock configuration should be available on the browser.
  • ship jest-jasmine and jest-util to the browser. Might require a tiny bit of refactoring.
  • Transform and package all JavaScript and ship it to the client. JS should be sent as a Map<path, codeString> – it has to be sent as string and should not immediately be run because Jest resets the module registry for every test. This is important for test isolation so that tests don't depend on each other. Since Jest doesn't know which files to run until they are needed, the big change here would be to bundle everything upfront. This wouldn't work at FB scale, but we don't intend to ever use a browser runner, so I don't mind this not being 100 % efficient.
  • Add a "mock" path implementation so that module resolution can work based on the Map mentioned before.
  • Every individual test should be run in its own browser environment, which should probably be an iframe. Again, this is to preserve Jest's sandboxing guarantees.
@nhunzaker

This comment has been minimized.

Show comment
Hide comment
@nhunzaker

nhunzaker Mar 30, 2016

Contributor

As a community member, is there a component here that would be best for me to work towards? I'm still familiarizing myself with Jest internals, but it seems like the bulk of the work has to be done in HasteModuleLoader.

One of the attractive parts of Karma is how easy it is to integrate browserify and webpack. It is not difficult to test code bundled in the same way between test and dev. This is attractive to me. While not a direct concern to this issue, I am curious if Jest could integrate with these tools.

This is a difficult problem. Looking forward to digging in.

Contributor

nhunzaker commented Mar 30, 2016

As a community member, is there a component here that would be best for me to work towards? I'm still familiarizing myself with Jest internals, but it seems like the bulk of the work has to be done in HasteModuleLoader.

One of the attractive parts of Karma is how easy it is to integrate browserify and webpack. It is not difficult to test code bundled in the same way between test and dev. This is attractive to me. While not a direct concern to this issue, I am curious if Jest could integrate with these tools.

This is a difficult problem. Looking forward to digging in.

@cpojer

This comment has been minimized.

Show comment
Hide comment
@cpojer

cpojer Mar 31, 2016

Contributor

I think we'll first need to think pretty hard on how to implement this in a way that it's gonna work well before we start writing any code. Once we have a good idea of what we are gonna build here, I can think more about the staging and smaller chunks of work.

I think one of the big questions is how to create the bundle. webpack/browserify will be insufficient for a few reasons:

  • We don't know which actual modules we are going to use and run until they are required at runtime. I experimented with doing this statically (in #599) but mocking rules are runtime dependent and the static resolution was just way too slow (like, 2 order of magnitudes slower and not worth it). For example, when you do require('Foo') in Jest, it might require Foo.js or __mocks__/Foo.js and we only know which one at runtime (and it might be different for different tests, too). If you can think of a way to ship both to the browser, that would be great. We actually have an exhaustive list of all mock files and all JS files inside of a project in Jest, however that only works for Facebook's @providesModule modules and not for modules that are real CommonJS or node_modules. We need to figure out a way to get all of the relevant JS files into the browser.
  • Once we have a way of shipping the relevant code, it'll be a matter of separating some Jest code so it can be run in the browser. We need to ship our own require implementation (that we can extract from Runtime) with all the mocking rules applied and we need to be able to run JS from strings (using eval) instead of just dumping a bundle into a script pack.

I'm all for integrating with existing tools, but it seems that we need the ability to let Jest do it's runtime work of sandboxing and shipping its own require implementation.

Another option would be to make the runtime parts of Jest work in the browser and consider them as entirely separate from the cli runner. One example is the modularization we are doing already: jest-mock (https://github.com/facebook/jest/tree/master/packages/jest-mock) can already be used in any environment. The bundling would then be up to the developer (just get the JS to the client, and put it somewhere that Jest can execute it). Instead of jest-cli, it would be jest-browser and it would be a separate framework that shares the same core and could potentially run the same tests but with vastly different configurations and setup. I'm not sure if that's a good idea.

Contributor

cpojer commented Mar 31, 2016

I think we'll first need to think pretty hard on how to implement this in a way that it's gonna work well before we start writing any code. Once we have a good idea of what we are gonna build here, I can think more about the staging and smaller chunks of work.

I think one of the big questions is how to create the bundle. webpack/browserify will be insufficient for a few reasons:

  • We don't know which actual modules we are going to use and run until they are required at runtime. I experimented with doing this statically (in #599) but mocking rules are runtime dependent and the static resolution was just way too slow (like, 2 order of magnitudes slower and not worth it). For example, when you do require('Foo') in Jest, it might require Foo.js or __mocks__/Foo.js and we only know which one at runtime (and it might be different for different tests, too). If you can think of a way to ship both to the browser, that would be great. We actually have an exhaustive list of all mock files and all JS files inside of a project in Jest, however that only works for Facebook's @providesModule modules and not for modules that are real CommonJS or node_modules. We need to figure out a way to get all of the relevant JS files into the browser.
  • Once we have a way of shipping the relevant code, it'll be a matter of separating some Jest code so it can be run in the browser. We need to ship our own require implementation (that we can extract from Runtime) with all the mocking rules applied and we need to be able to run JS from strings (using eval) instead of just dumping a bundle into a script pack.

I'm all for integrating with existing tools, but it seems that we need the ability to let Jest do it's runtime work of sandboxing and shipping its own require implementation.

Another option would be to make the runtime parts of Jest work in the browser and consider them as entirely separate from the cli runner. One example is the modularization we are doing already: jest-mock (https://github.com/facebook/jest/tree/master/packages/jest-mock) can already be used in any environment. The bundling would then be up to the developer (just get the JS to the client, and put it somewhere that Jest can execute it). Instead of jest-cli, it would be jest-browser and it would be a separate framework that shares the same core and could potentially run the same tests but with vastly different configurations and setup. I'm not sure if that's a good idea.

@jquense

This comment has been minimized.

Show comment
Hide comment
@jquense

jquense Dec 4, 2016

Contributor

been thinking about this a bit. Mocking is one of the big features of Jest over other options., this in particular makes a browser runner hard since the execution environment really needs to understand modules. to that end I think using webpack, under the hood, makes a lot of sense.

webpack can more than handle the resolution and mocking concerns, with the right set of plugins (which can consume jest-resolve). the suggestion tho, is to use it as an implementation detail/node environment polyfill.

i need to spend more time consolidating my thoughts (just typing this up quickly on my phone) but I think the big bits that would be needed in jest is pluggable environments (jsdom, browser) and an environment agnostic runtime/runner. I suppose similar to karma s launchers but maybe less generic

Contributor

jquense commented Dec 4, 2016

been thinking about this a bit. Mocking is one of the big features of Jest over other options., this in particular makes a browser runner hard since the execution environment really needs to understand modules. to that end I think using webpack, under the hood, makes a lot of sense.

webpack can more than handle the resolution and mocking concerns, with the right set of plugins (which can consume jest-resolve). the suggestion tho, is to use it as an implementation detail/node environment polyfill.

i need to spend more time consolidating my thoughts (just typing this up quickly on my phone) but I think the big bits that would be needed in jest is pluggable environments (jsdom, browser) and an environment agnostic runtime/runner. I suppose similar to karma s launchers but maybe less generic

@nhunzaker

This comment has been minimized.

Show comment
Hide comment
@nhunzaker

nhunzaker Dec 6, 2016

Contributor

@jquense I don't want to run the risk of repeating you too much but I wanted to share some thoughts too.

I think the big bits that would be needed in jest is pluggable environments (jsdom, browser) and an environment agnostic runtime/runner

I wonder what the minimum required surface area would be for browser execution. What happens when --env (jsdom|node) gets called? What code path activates differently?

Could mocking be handled within node, leaning on existing work, with most code execution happening through passing a bundle of javascript to a browser? Could this be as simple as preparing a bundled javascript file, injecting that into a browser context, and reporting back results?

I think this is how Karma's test harness works. I gravitate towards keeping this as simple as dumping JS on the page and tracking console.log/window.onerror. Though I'm sure it's more complicated than that.

Contributor

nhunzaker commented Dec 6, 2016

@jquense I don't want to run the risk of repeating you too much but I wanted to share some thoughts too.

I think the big bits that would be needed in jest is pluggable environments (jsdom, browser) and an environment agnostic runtime/runner

I wonder what the minimum required surface area would be for browser execution. What happens when --env (jsdom|node) gets called? What code path activates differently?

Could mocking be handled within node, leaning on existing work, with most code execution happening through passing a bundle of javascript to a browser? Could this be as simple as preparing a bundled javascript file, injecting that into a browser context, and reporting back results?

I think this is how Karma's test harness works. I gravitate towards keeping this as simple as dumping JS on the page and tracking console.log/window.onerror. Though I'm sure it's more complicated than that.

@jquense

This comment has been minimized.

Show comment
Hide comment
@jquense

jquense Dec 7, 2016

Contributor

karma requires a preprocessor for any sort of modular code, such as karma-webpack.

Could mocking be handled within node, leaning on existing work, with most code execution happening through passing a bundle of javascript to a browser?

this is what im suggesting, but the above would require building a file concatenator based on module/depedency graphs with custom resolution logic. essentially webpack :P i was thinking we could save a ton of hard work by leaveraging webpack instead of builing something. the rest is figuring out api boundries between the test runner, environment and test providers

Contributor

jquense commented Dec 7, 2016

karma requires a preprocessor for any sort of modular code, such as karma-webpack.

Could mocking be handled within node, leaning on existing work, with most code execution happening through passing a bundle of javascript to a browser?

this is what im suggesting, but the above would require building a file concatenator based on module/depedency graphs with custom resolution logic. essentially webpack :P i was thinking we could save a ton of hard work by leaveraging webpack instead of builing something. the rest is figuring out api boundries between the test runner, environment and test providers

@jquense

This comment has been minimized.

Show comment
Hide comment
@jquense

jquense Dec 7, 2016

Contributor

I think that there is a big difference between the feature set of something like Jest vs karma. karma is a simply a pluggable runner/framework for running a set of tests in many different environments. Jest on the other hand makes a lot of assumptions about the environment. Jest has wider scope than karma; its more like karma + jasmine/mocha + jsdom-launcher + karma-webpack + rewire/inject-loader. Because of this I don't think that a super flexible launcher api like Karma's is well suited to Jest.

To that end I think that what you really need for jest-browser to work is the ability to create a node like environment in the browser, to which I say that webpack is the most robust and flexible tool for doing this already out there. It's plugin system can more than handle the concerns made by @cpojer above, as well as all the jest config options. Undoubtable it'd be slower, but I think that could be isolated to the browser-runner bits so that fb wouldn't mind ;P

As soon as a get a bit of time i want to see where the sore points are in the existing jest infra, and where assumptions are being made about the runtime. I think that will put us in a good place to talk about how the API's would need to change in order to support this sort of use case. I feel pretty optimistic thou, the team as already done an amazing job as modularizing and extracting jest into lots of its constituent bits :)

Contributor

jquense commented Dec 7, 2016

I think that there is a big difference between the feature set of something like Jest vs karma. karma is a simply a pluggable runner/framework for running a set of tests in many different environments. Jest on the other hand makes a lot of assumptions about the environment. Jest has wider scope than karma; its more like karma + jasmine/mocha + jsdom-launcher + karma-webpack + rewire/inject-loader. Because of this I don't think that a super flexible launcher api like Karma's is well suited to Jest.

To that end I think that what you really need for jest-browser to work is the ability to create a node like environment in the browser, to which I say that webpack is the most robust and flexible tool for doing this already out there. It's plugin system can more than handle the concerns made by @cpojer above, as well as all the jest config options. Undoubtable it'd be slower, but I think that could be isolated to the browser-runner bits so that fb wouldn't mind ;P

As soon as a get a bit of time i want to see where the sore points are in the existing jest infra, and where assumptions are being made about the runtime. I think that will put us in a good place to talk about how the API's would need to change in order to support this sort of use case. I feel pretty optimistic thou, the team as already done an amazing job as modularizing and extracting jest into lots of its constituent bits :)

@cpojer

This comment has been minimized.

Show comment
Hide comment
@cpojer

cpojer Dec 8, 2016

Contributor

I appreciate this discussion got started again. To me this has always been exciting but also risky for the project. I'll support this as much as I can and have some thoughts as well.

My ideal solution to this would be somewhere in jest-runtime and jest-environment-*. The problem is that right now; Jest requires and compiles files "just-in-time" when they are required which means they need to be synchronous. Because of the mocking feature, the entire module system and file system needs to be virtualized in the browser, as well as the entire transform pipeline with babel.

However, there are two more ways of solving this, both equally exciting:

  • We split up Jest into many packages and a bunch of them can be used in the browser if they are bundled up, like jest-matchers.
  • We could actually turn require into a generator and use a websocket connection to communicate with "jest sever". The problem is that right now require's are synchronous but we'd need to make them async. I'm not sure how this would work (generator or async/await) but with a babel transform and changing the require implementation in Jest, given that we are in full control, could work:
// How it works right now, jest wraps every file in a function and passes its require implementation:
(function(require, ...) {
  const apple = require('apple');
});

// How it could work, using a babel transform:
(function*(require, ...) {
  const apple = yield require('apple');
});

This would run on the client; when we call into require we'd request the transformed module over a websocket connection.

Contributor

cpojer commented Dec 8, 2016

I appreciate this discussion got started again. To me this has always been exciting but also risky for the project. I'll support this as much as I can and have some thoughts as well.

My ideal solution to this would be somewhere in jest-runtime and jest-environment-*. The problem is that right now; Jest requires and compiles files "just-in-time" when they are required which means they need to be synchronous. Because of the mocking feature, the entire module system and file system needs to be virtualized in the browser, as well as the entire transform pipeline with babel.

However, there are two more ways of solving this, both equally exciting:

  • We split up Jest into many packages and a bunch of them can be used in the browser if they are bundled up, like jest-matchers.
  • We could actually turn require into a generator and use a websocket connection to communicate with "jest sever". The problem is that right now require's are synchronous but we'd need to make them async. I'm not sure how this would work (generator or async/await) but with a babel transform and changing the require implementation in Jest, given that we are in full control, could work:
// How it works right now, jest wraps every file in a function and passes its require implementation:
(function(require, ...) {
  const apple = require('apple');
});

// How it could work, using a babel transform:
(function*(require, ...) {
  const apple = yield require('apple');
});

This would run on the client; when we call into require we'd request the transformed module over a websocket connection.

@thymikee

This comment has been minimized.

Show comment
Hide comment
@thymikee

thymikee Dec 8, 2016

Collaborator

@cpojer I'm actually not into the topic but in case you missed it, async import() recently got to Stage-3: https://github.com/tc39/proposal-dynamic-import
and I believe it's used in Webpack2 now: webpack/webpack#3413

Collaborator

thymikee commented Dec 8, 2016

@cpojer I'm actually not into the topic but in case you missed it, async import() recently got to Stage-3: https://github.com/tc39/proposal-dynamic-import
and I believe it's used in Webpack2 now: webpack/webpack#3413

@cpojer

This comment has been minimized.

Show comment
Hide comment
@cpojer

cpojer Dec 8, 2016

Contributor

@thymikee yeah good point but that also requires to write a transform from require to async import, it's very similar I guess.

Contributor

cpojer commented Dec 8, 2016

@thymikee yeah good point but that also requires to write a transform from require to async import, it's very similar I guess.

@jquense

This comment has been minimized.

Show comment
Hide comment
@jquense

jquense Dec 8, 2016

Contributor

i'm gonna guess that requesting files over the wire (even locally) is going to be too slow. My thought was that just shipping all the mocks for required packages would be simpler and probably faster overall? Maybe i'm missing something here but most jest mocks are statically knowable, right? detecting calls to jest.mock or require.requireMock isn't too bad. For ambiguous cases you can just ship both.

Because of the mocking feature, the entire module system and file system needs to be virtualized in the browser, as well as the entire transform pipeline with babe

This speaks to my earlier point, its a tough one. Writing a module env for the browser is going to be super fraught and complicated. The thought was to leavage work already done on that front via webpack, for this bit. Especially since almost every nook and crany of webpack is pluggable.

Contributor

jquense commented Dec 8, 2016

i'm gonna guess that requesting files over the wire (even locally) is going to be too slow. My thought was that just shipping all the mocks for required packages would be simpler and probably faster overall? Maybe i'm missing something here but most jest mocks are statically knowable, right? detecting calls to jest.mock or require.requireMock isn't too bad. For ambiguous cases you can just ship both.

Because of the mocking feature, the entire module system and file system needs to be virtualized in the browser, as well as the entire transform pipeline with babe

This speaks to my earlier point, its a tough one. Writing a module env for the browser is going to be super fraught and complicated. The thought was to leavage work already done on that front via webpack, for this bit. Especially since almost every nook and crany of webpack is pluggable.

@ThatTomPerson

This comment has been minimized.

Show comment
Hide comment
@ThatTomPerson

ThatTomPerson Dec 15, 2016

Sorry if you guys have thought of this already but if the require('./foo') needs to be done at runtime for mocking reasons, why not just write a "jest-loader" which could rewrite the require statement to something more like

let foo
if (jest.mocking('./foo')) {
    foo = require('./__mocks__/foo');
} else {
    foo = require('./foo');
}

and let webpack just include them both?

edit
NVM
I miss read the previous comment

ThatTomPerson commented Dec 15, 2016

Sorry if you guys have thought of this already but if the require('./foo') needs to be done at runtime for mocking reasons, why not just write a "jest-loader" which could rewrite the require statement to something more like

let foo
if (jest.mocking('./foo')) {
    foo = require('./__mocks__/foo');
} else {
    foo = require('./foo');
}

and let webpack just include them both?

edit
NVM
I miss read the previous comment

@cpojer

This comment has been minimized.

Show comment
Hide comment
@cpojer

cpojer Dec 15, 2016

Contributor

@ThatTomPerson jest-runtime does this branching already.

Contributor

cpojer commented Dec 15, 2016

@ThatTomPerson jest-runtime does this branching already.

@nhunzaker

This comment has been minimized.

Show comment
Hide comment
@nhunzaker

nhunzaker Jan 27, 2017

Contributor

Playing around with this a bit more. I've been experimenting with running Jest inside of an Electron browser instance:

https://gist.github.com/nhunzaker/5fc6f3ea153319e3140275914773d5bc

Electron is pretty neat because I can just require Jest inside of the browser. Trouble is that I don't have encapsulation. Test globals leak into each other.

Ideally, I think it would be neat if each test suite opened an isolated browser instance.

I'm not sure if there's really any benefit here. Eventually I think it would be neat if you could mount a React app to the browser for testing, stepping through breakpoints and watching behavior play out in a real browser.

No real questions here, but maybe it gives someone else some inspiration.

Contributor

nhunzaker commented Jan 27, 2017

Playing around with this a bit more. I've been experimenting with running Jest inside of an Electron browser instance:

https://gist.github.com/nhunzaker/5fc6f3ea153319e3140275914773d5bc

Electron is pretty neat because I can just require Jest inside of the browser. Trouble is that I don't have encapsulation. Test globals leak into each other.

Ideally, I think it would be neat if each test suite opened an isolated browser instance.

I'm not sure if there's really any benefit here. Eventually I think it would be neat if you could mount a React app to the browser for testing, stepping through breakpoints and watching behavior play out in a real browser.

No real questions here, but maybe it gives someone else some inspiration.

@thymikee

This comment has been minimized.

Show comment
Hide comment
@thymikee

thymikee Jan 27, 2017

Collaborator

Wow, this is super cool you're working on this!

I think you should spawn new browser in ElectronEnvironment#constructor, just like we do with spawning jsdom env.
Keep us posted

Collaborator

thymikee commented Jan 27, 2017

Wow, this is super cool you're working on this!

I think you should spawn new browser in ElectronEnvironment#constructor, just like we do with spawning jsdom env.
Keep us posted

@cpojer

This comment has been minimized.

Show comment
Hide comment
@cpojer

cpojer Jan 27, 2017

Contributor

oh hell yeah @nhunzaker. I'm extremely excited about this. Do you have a guide to set this up?

If @thymikee's suggestion doesn't work, could you use iframes in runScript?

Contributor

cpojer commented Jan 27, 2017

oh hell yeah @nhunzaker. I'm extremely excited about this. Do you have a guide to set this up?

If @thymikee's suggestion doesn't work, could you use iframes in runScript?

@nhunzaker

This comment has been minimized.

Show comment
Hide comment
@nhunzaker

nhunzaker Jan 27, 2017

Contributor

@cpojer I've put up the code here:

https://github.com/nhunzaker/jest-electron-environment

I think I could use the contextIsolation option from BrowserWindow to spin up an isolated browser and arbitrarily execute some code. That would be a similar approach to spitting JS into an iframe.

What I'm struggling with is how to get the actual code for the test:
https://github.com/nhunzaker/jest-electron-environment/blob/master/src/jest-environment-electron.js#L27

☝️ I'm not too familiar with Jasmine script contexts (if I have that right). Is there a way to get the string content for the test?

Contributor

nhunzaker commented Jan 27, 2017

@cpojer I've put up the code here:

https://github.com/nhunzaker/jest-electron-environment

I think I could use the contextIsolation option from BrowserWindow to spin up an isolated browser and arbitrarily execute some code. That would be a similar approach to spitting JS into an iframe.

What I'm struggling with is how to get the actual code for the test:
https://github.com/nhunzaker/jest-electron-environment/blob/master/src/jest-environment-electron.js#L27

☝️ I'm not too familiar with Jasmine script contexts (if I have that right). Is there a way to get the string content for the test?

@chrishyle

This comment has been minimized.

Show comment
Hide comment
@chrishyle

chrishyle Feb 7, 2017

While running Jest in an electron shell is an interesting project, I think it only solves part of the issue for real world developers. While a CLI runner or an electron environment is great for our developer workflow, we have also found many issues over the years by running our unit tests in real browsers (especially older browsers) that we didn't find in those CLI environments or Chrome alone. I think we still need a way to run Jest in real browsers, even if the electron environment is successful (and I hope it is!).

While running Jest in an electron shell is an interesting project, I think it only solves part of the issue for real world developers. While a CLI runner or an electron environment is great for our developer workflow, we have also found many issues over the years by running our unit tests in real browsers (especially older browsers) that we didn't find in those CLI environments or Chrome alone. I think we still need a way to run Jest in real browsers, even if the electron environment is successful (and I hope it is!).

@nhunzaker

This comment has been minimized.

Show comment
Hide comment
@nhunzaker

nhunzaker Feb 8, 2017

Contributor

I completely agree. As someone less familiar with Jest internals, I found Electron to be an attractive target. Electron just works, and opens up a lot of possibilities, such as recording test sessions and stepping through browser behavior using a debugger. Still, I don't want to express neglect for real browsers with that enthusiasm.

Unfortunately, I also haven't been able to make the time for those problems. It would be cool to establish a more concrete roadmap for how we can get to testing in real browsers. I still have a lot to learn.

Contributor

nhunzaker commented Feb 8, 2017

I completely agree. As someone less familiar with Jest internals, I found Electron to be an attractive target. Electron just works, and opens up a lot of possibilities, such as recording test sessions and stepping through browser behavior using a debugger. Still, I don't want to express neglect for real browsers with that enthusiasm.

Unfortunately, I also haven't been able to make the time for those problems. It would be cool to establish a more concrete roadmap for how we can get to testing in real browsers. I still have a lot to learn.

@suchipi

This comment has been minimized.

Show comment
Hide comment
@suchipi

suchipi Feb 8, 2017

Contributor

My two cents: I got Jest running in NW.js, which is pretty similar to Electron, but it doesn't have the same node/browser context separation, so it's pretty easy to get the test code. Proof of concept here: https://github.com/suchipi/jest-node-nw-example

It's using the same browser globals every run right now, but creating new iframes for each test shouldn't be too hard

Contributor

suchipi commented Feb 8, 2017

My two cents: I got Jest running in NW.js, which is pretty similar to Electron, but it doesn't have the same node/browser context separation, so it's pretty easy to get the test code. Proof of concept here: https://github.com/suchipi/jest-node-nw-example

It's using the same browser globals every run right now, but creating new iframes for each test shouldn't be too hard

@intellix

This comment has been minimized.

Show comment
Hide comment
@intellix

intellix Apr 13, 2017

ChromeHeadless is now available, with support on it's way to karma-chrome-launcher: karma-runner/karma-chrome-launcher#111
Would love to know how it performs in comparison to jsdom

ChromeHeadless is now available, with support on it's way to karma-chrome-launcher: karma-runner/karma-chrome-launcher#111
Would love to know how it performs in comparison to jsdom

@joelgriffith

This comment has been minimized.

Show comment
Hide comment
@joelgriffith

joelgriffith Jul 10, 2017

I've been cooking up something out in the woods in Washington, and it's this library: https://github.com/joelgriffith/navalia. This tool has the capacity to write tests in a high-level browser API, and can parallelize them in a single browser context (think of tabs). Once I harden the API a bit I'm going to write some content on how to achieve this wizardry, and folks can hopefully begin to have a better experience with E2E-style tests.

Here's a short post on how it can instrument chrome for coverage (for those of you who are curios about the API): https://codeburst.io/capturing-unused-application-code-2b7594a9fe06

I've been cooking up something out in the woods in Washington, and it's this library: https://github.com/joelgriffith/navalia. This tool has the capacity to write tests in a high-level browser API, and can parallelize them in a single browser context (think of tabs). Once I harden the API a bit I'm going to write some content on how to achieve this wizardry, and folks can hopefully begin to have a better experience with E2E-style tests.

Here's a short post on how it can instrument chrome for coverage (for those of you who are curios about the API): https://codeburst.io/capturing-unused-application-code-2b7594a9fe06

@MattKunze

This comment has been minimized.

Show comment
Hide comment
@MattKunze

MattKunze Jul 19, 2017

One thing I haven't seen mentioned in this discussion (probably missed it from previous issues) is that one of one of the big wins I've seen before with browser based tests is the ability to run them on various platforms (IE/Edge, iOS on Safari, etc) to validate edge cases in those environments. It certainly doesn't cover everything you can do with automation tools, but it's been helpful to me in the past.

Getting it working on Electron or NW.js would be great from a debugging perspective, but actually being able to run it on the target platform would be even better

One thing I haven't seen mentioned in this discussion (probably missed it from previous issues) is that one of one of the big wins I've seen before with browser based tests is the ability to run them on various platforms (IE/Edge, iOS on Safari, etc) to validate edge cases in those environments. It certainly doesn't cover everything you can do with automation tools, but it's been helpful to me in the past.

Getting it working on Electron or NW.js would be great from a debugging perspective, but actually being able to run it on the target platform would be even better

@cpojer

This comment has been minimized.

Show comment
Hide comment
@cpojer

cpojer Aug 25, 2017

Contributor

Hey everyone! I thought I'd give an update on this thread, as it is one of our oldest issues. We actually made quite a lot of progress on this for Jest 21, although it's not as straightforward as "running Jest in the browser" yet. Currently tagged as beta, jest@test can run a number of modules inside of the browser, with your existing test runner there.

  • jest-matchers is renamed to expect and can be installed and used in the browser.
  • jest-mock can be used in the browser for mock functions, together with expect

This gives you a way to use Jest's awesome assertion library in any other test framework.

Further, Jest is slowly migrating away from Jasmine, and with jest-circus we are building a new test runner that will also work in the browser. This is not quite ready yet, but you could experiment with it if you want to.

There are more missing pieces that are being filled in. Recently I showed a demo of Python tests being run using Jest, see pyjest. This is also not polished yet, but in the future you'll be able to use Jest's runner to drive any other test framework on any platform you like, as long as you integrate it yourself :)

Yet another big step is to "virtualize" the require/mock system and extract it from jest-runtime. This will allow to use most pieces of Jest inside a test context rather than controlling everything from the outside. Once this is done, most pieces of Jest can be run in various places.

While Jest in the future will run just as it does today, once all the pieces from above are completed, we can run almost all of Jest in almost all places. This means that Jest's runner can be used to run current Jest tests, tests in Electron, tests in any browser. Together with some other plans, we may also be able to run Jest's runner within various places.

Note that there is nobody actively working towards this right now, so if you'd like to help out, please reach out to me.

Contributor

cpojer commented Aug 25, 2017

Hey everyone! I thought I'd give an update on this thread, as it is one of our oldest issues. We actually made quite a lot of progress on this for Jest 21, although it's not as straightforward as "running Jest in the browser" yet. Currently tagged as beta, jest@test can run a number of modules inside of the browser, with your existing test runner there.

  • jest-matchers is renamed to expect and can be installed and used in the browser.
  • jest-mock can be used in the browser for mock functions, together with expect

This gives you a way to use Jest's awesome assertion library in any other test framework.

Further, Jest is slowly migrating away from Jasmine, and with jest-circus we are building a new test runner that will also work in the browser. This is not quite ready yet, but you could experiment with it if you want to.

There are more missing pieces that are being filled in. Recently I showed a demo of Python tests being run using Jest, see pyjest. This is also not polished yet, but in the future you'll be able to use Jest's runner to drive any other test framework on any platform you like, as long as you integrate it yourself :)

Yet another big step is to "virtualize" the require/mock system and extract it from jest-runtime. This will allow to use most pieces of Jest inside a test context rather than controlling everything from the outside. Once this is done, most pieces of Jest can be run in various places.

While Jest in the future will run just as it does today, once all the pieces from above are completed, we can run almost all of Jest in almost all places. This means that Jest's runner can be used to run current Jest tests, tests in Electron, tests in any browser. Together with some other plans, we may also be able to run Jest's runner within various places.

Note that there is nobody actively working towards this right now, so if you'd like to help out, please reach out to me.

@suchipi

This comment has been minimized.

Show comment
Hide comment
@suchipi

suchipi Aug 25, 2017

Contributor

@cpojer I'm interested in contributing. I don't think I'm familiar enough with jest-runtime to pull it apart, but could I contribute to jest-circus somehow? I'm more familiar with that problem space.

Contributor

suchipi commented Aug 25, 2017

@cpojer I'm interested in contributing. I don't think I'm familiar enough with jest-runtime to pull it apart, but could I contribute to jest-circus somehow? I'm more familiar with that problem space.

@aaronabramov

This comment has been minimized.

Show comment
Hide comment
@aaronabramov

aaronabramov Aug 25, 2017

Member

@suchipi any help on jest-circus side would be appreciated!
some of the things that we need to do next is:

  • make it fully compatible with Jasmine (currently it has about 90% of the features)
  • refactor and modularize the integration between jest-circus and test_runner
  • write a lot of tests

let me know if you want to work on any of that!

Member

aaronabramov commented Aug 25, 2017

@suchipi any help on jest-circus side would be appreciated!
some of the things that we need to do next is:

  • make it fully compatible with Jasmine (currently it has about 90% of the features)
  • refactor and modularize the integration between jest-circus and test_runner
  • write a lot of tests

let me know if you want to work on any of that!

@suchipi

This comment has been minimized.

Show comment
Hide comment
@suchipi

suchipi Aug 25, 2017

Contributor

@aaronabramov Jasmine compatibility should be pretty straightforward. Is there a tracking issue outlining which features are missing that need to be added?

Contributor

suchipi commented Aug 25, 2017

@aaronabramov Jasmine compatibility should be pretty straightforward. Is there a tracking issue outlining which features are missing that need to be added?

@suchipi

This comment has been minimized.

Show comment
Hide comment
@suchipi

suchipi Aug 25, 2017

Contributor

I can do spelunking and figure it out if needed, but I don't want to add anything that you don't actually need; Jasmine as a whole has some overlap with expect, jest-mock, etc

Contributor

suchipi commented Aug 25, 2017

I can do spelunking and figure it out if needed, but I don't want to add anything that you don't actually need; Jasmine as a whole has some overlap with expect, jest-mock, etc

@lsjroberts lsjroberts referenced this issue Oct 18, 2017

Open

Add tests #25

@rdeltour

This comment has been minimized.

Show comment
Hide comment
@rdeltour

rdeltour Oct 28, 2017

In case it might help others, I've been experimenting with using Jest’s recent async environment API to run tests within headless Chrome with Puppeteer:

I'm pretty happy with the results!

I can't fully run everything in the browser since Puppeteer lacks an API to run vm.Scritp instances, so I have to get back to the Node context at some point... but it's very promising nonetheless! Kudos to the Jest team 👍

In case it might help others, I've been experimenting with using Jest’s recent async environment API to run tests within headless Chrome with Puppeteer:

I'm pretty happy with the results!

I can't fully run everything in the browser since Puppeteer lacks an API to run vm.Scritp instances, so I have to get back to the Node context at some point... but it's very promising nonetheless! Kudos to the Jest team 👍

@geovanisouza92

This comment has been minimized.

Show comment
Hide comment

@rdeltour Maybe you can get it working using https://github.com/patriksimek/vm2

@rdy

This comment has been minimized.

Show comment
Hide comment
@rdy

rdy Nov 23, 2017

I'm a maintainer of gulp-jasmine-browser which attempts to automate headless browser controls using the techniques described in this thread except using jasmine. We just finished our support for puppeteer and also have support for phanotm and slimer. I'm thinking of building a newer 2.0 version to launch the test runner and send the results back to a reporter. The newer version would be less dependent on gulp and probably utilize more of the webpack ecosystem. I'd be willing to consider adding support for jest as well if there is interest in it.

I really prefer having a real DOM to test rather than what jest currently offers.

rdy commented Nov 23, 2017

I'm a maintainer of gulp-jasmine-browser which attempts to automate headless browser controls using the techniques described in this thread except using jasmine. We just finished our support for puppeteer and also have support for phanotm and slimer. I'm thinking of building a newer 2.0 version to launch the test runner and send the results back to a reporter. The newer version would be less dependent on gulp and probably utilize more of the webpack ecosystem. I'd be willing to consider adding support for jest as well if there is interest in it.

I really prefer having a real DOM to test rather than what jest currently offers.

@jonnor jonnor referenced this issue Dec 27, 2017

Merged

React 15 / Jest #372

@bsuh

This comment has been minimized.

Show comment
Hide comment
@bsuh

bsuh Jan 1, 2018

@cpojer What about using AMD module transform? That would allow a jest AMD implementation running in the browser to asynchronously fetch the module source from a jest runtime running on node.

Aside from refactoring the require & mock implementation out of the runtime as you mentioned in #4358, several packages synchronously access the VM context, which would not be possible in a remote browser "VM".

  • jest-runner sets the VM's global.console
  • fake timers sets the VM's timer functions
  • jest-jasmine2 injects several modules into the VM and uses their API directly
    Maybe some more I missed.

The pattern I'm seeing is that the responsibility of setting up the VM context is distributed across packages and is mixed in with those packages' other responsibilities.

bsuh commented Jan 1, 2018

@cpojer What about using AMD module transform? That would allow a jest AMD implementation running in the browser to asynchronously fetch the module source from a jest runtime running on node.

Aside from refactoring the require & mock implementation out of the runtime as you mentioned in #4358, several packages synchronously access the VM context, which would not be possible in a remote browser "VM".

  • jest-runner sets the VM's global.console
  • fake timers sets the VM's timer functions
  • jest-jasmine2 injects several modules into the VM and uses their API directly
    Maybe some more I missed.

The pattern I'm seeing is that the responsibility of setting up the VM context is distributed across packages and is mixed in with those packages' other responsibilities.

@cpojer

This comment has been minimized.

Show comment
Hide comment
@cpojer

cpojer Jan 3, 2018

Contributor

@bsuh I think what you are saying makes sense, we could consider it for sure. I think your idea about centralizing all the VM interactions is a good start – are you willing to work on some PRs towards that?

Contributor

cpojer commented Jan 3, 2018

@bsuh I think what you are saying makes sense, we could consider it for sure. I think your idea about centralizing all the VM interactions is a good start – are you willing to work on some PRs towards that?

@bsuh

This comment has been minimized.

Show comment
Hide comment
@bsuh

bsuh Jan 3, 2018

I can try to tackle #4358 for starters.

bsuh commented Jan 3, 2018

I can try to tackle #4358 for starters.

@cpojer

This comment has been minimized.

Show comment
Hide comment
@cpojer

cpojer Jan 4, 2018

Contributor

Awesome! Let's continue the discussion there.

Contributor

cpojer commented Jan 4, 2018

Awesome! Let's continue the discussion there.

wbinnssmith added a commit to facebook/rebound-js that referenced this issue Feb 1, 2018

Use Jest to run tests
This ports the existing Jasmine 1.3 spec to use a modern version of
Jest.

It eliminates the hardcoded expected values and migrates them to use
Jest's snapshot functionality.

Unfortunately this also removes the browser tests as Jest cannot run in
the browser, though [there is an open
issue](facebook/jest#848)

Arguably browser testing may be better served by something more
end-to-end.

wbinnssmith added a commit to facebook/rebound-js that referenced this issue Feb 6, 2018

Use Jest to run tests
This ports the existing Jasmine 1.3 spec to use a modern version of
Jest.

It eliminates the hardcoded expected values and migrates them to use
Jest's snapshot functionality.

Unfortunately this also removes the browser tests as Jest cannot run in
the browser, though [there is an open
issue](facebook/jest#848)

Arguably browser testing may be better served by something more
end-to-end.
@trusktr

This comment has been minimized.

Show comment
Hide comment

trusktr commented Apr 29, 2018

Some more info here, #3698.

And also this: https://www.npmjs.com/package/jest-electron-runner

@trusktr

This comment has been minimized.

Show comment
Hide comment
@trusktr

trusktr Apr 30, 2018

After fiddling with Electron and NWjs, and some of others' attempted packages, with no luck, I decided to try Karma with Chrome headless.

I was able to get it running very easily based on that article, which is great because I really need all DOM APIs to be available (even WebGL).

Really hoping Jest gets headless Chrome support sometime sooner than later. It would be great!

trusktr commented Apr 30, 2018

After fiddling with Electron and NWjs, and some of others' attempted packages, with no luck, I decided to try Karma with Chrome headless.

I was able to get it running very easily based on that article, which is great because I really need all DOM APIs to be available (even WebGL).

Really hoping Jest gets headless Chrome support sometime sooner than later. It would be great!

@coleGillespie

This comment has been minimized.

Show comment
Hide comment
@coleGillespie

coleGillespie May 30, 2018

@nhunzaker did you ever move past electron?

coleGillespie commented May 30, 2018

@nhunzaker did you ever move past electron?

@nhunzaker

This comment has been minimized.

Show comment
Hide comment
@nhunzaker

nhunzaker Jun 6, 2018

Contributor

@coleGillespie Sorry to catch up so late. Hey!

I ended up shifting gears. Using Jest with Puppeteer ended up solving a lot of my use case. Still, I'm working on an electron app right now, and not having to mock out the electron interface sounds.... ideal.

Contributor

nhunzaker commented Jun 6, 2018

@coleGillespie Sorry to catch up so late. Hey!

I ended up shifting gears. Using Jest with Puppeteer ended up solving a lot of my use case. Still, I'm working on an electron app right now, and not having to mock out the electron interface sounds.... ideal.

@thymikee

This comment has been minimized.

Show comment
Hide comment
@thymikee

thymikee Jun 6, 2018

Collaborator

@nhunzaker talk to @aaronabramov about Electron integration, he's on it right now.

Collaborator

thymikee commented Jun 6, 2018

@nhunzaker talk to @aaronabramov about Electron integration, he's on it right now.

@nhunzaker

This comment has been minimized.

Show comment
Hide comment
@nhunzaker

nhunzaker Jun 6, 2018

Contributor

Oh wow. This is fantastic!

Contributor

nhunzaker commented Jun 6, 2018

Oh wow. This is fantastic!

@aaronabramov

This comment has been minimized.

Show comment
Hide comment
@aaronabramov

aaronabramov Jun 6, 2018

Member

@nhunzaker i started it as an atom test runner so i could test it on the real app, but atom has a lot of hacks/wrappers on top of electron. so before it can be generalized i'll need to clean all this hacky stuff up :)

https://github.com/facebook/nuclide/tree/master/modules/jest-atom-runner

i'm in the middle of migration from jasmine to jest with this runner (will probably take another 1-2 weeks), but once it's done i'll be moving this package out of nuclide into jest-community
also any help would be appreciated :) i don't really know much about how electron apps work so i'm just trying random things right now and see what makes the build green

Member

aaronabramov commented Jun 6, 2018

@nhunzaker i started it as an atom test runner so i could test it on the real app, but atom has a lot of hacks/wrappers on top of electron. so before it can be generalized i'll need to clean all this hacky stuff up :)

https://github.com/facebook/nuclide/tree/master/modules/jest-atom-runner

i'm in the middle of migration from jasmine to jest with this runner (will probably take another 1-2 weeks), but once it's done i'll be moving this package out of nuclide into jest-community
also any help would be appreciated :) i don't really know much about how electron apps work so i'm just trying random things right now and see what makes the build green

@nhunzaker

This comment has been minimized.

Show comment
Hide comment
@nhunzaker

nhunzaker Jun 6, 2018

Contributor

i don't really know much about how electron apps work so i'm just trying random things right now and see what makes the build green

I'm kind of in the same boat, it's my first Electron project :), but I'd be happy to help out where I can.

Contributor

nhunzaker commented Jun 6, 2018

i don't really know much about how electron apps work so i'm just trying random things right now and see what makes the build green

I'm kind of in the same boat, it's my first Electron project :), but I'd be happy to help out where I can.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment