Speed up a preparation phase of the following test #9663
Comments
daedalius
changed the title
|
Great feedback @daedalius appreciate it, and we will look at the bundling and why it takes so long to start a test, so stay tuned |
|
The support and spec bundles are extremely large. I wonder if we double pack React and other goodies
|
|
In this repo I am including testing-library/cypress for some demo tests ... and that is the cause of the giant support file, ha
|
|
Did some experiments with webpack bundling speed in https://github.com/bahmutov/webpack-bundle-experiment Then looked the current bundling speed: First load has support and spec files. So the spec is 5 seconds! and is 3 MiB On file edit, the spec is rebuilt in 1 second Notice that only the file itself is |
|
Confirmed: if we disable inline source maps with Webpack option Also see https://github.com/bahmutov/webpack-bundle-experiment#vendor-bundles If we want inline source maps, but split the vendor bundle ( |
|
Note: Webpack build timings in the project require cypress-io/cypress-webpack-preprocessor#81 |
|
Looking at timings in Material UI autocomplete-spec.js StartSupport file Spec file |
|
Moved testing-library/cypress from support file into the spec that uses it Support bundling 12 seconds -> 1.5 seconds Spec timing (same) |
|
In webpack options disable adding inline source maps const webpackOptions = {
devtool: false
}Spec bundling time 11 -> 8 seconds Touching the file to force refresh So pretty fast. |
|
Eventual Windows timings for tests in my repo:
Conclusions:
Gleb, do you have any ideas of further speed up bundling? Is it possible to bundle all tests as one or bundle them in the background? |
|
I cloned the repo and updated plugin to the latest (and removed separate webpack preprocessor, since now it is included). Find my fork in https://github.com/bahmutov/time-component-tests Running with The same stats repeat for each spec. |
|
Opening a spec runs slightly slower Acceptable in my opinion |
|
Disabling code coverage check (there was no instrumentation anyway) keeps the timing the same |
|
Running with |
|
Of course, my old Mac Pro might be faster in this situation, but so far I don't see a show stopper in terms of bundling performance. |
|
Personally I think that ~3sec to each component test is acceptable too. But still, it is kinda weird that the actual test run time takes 10 times less than the test bundling phase (and it is a sequential process). You may close the issue if there is nothing to improve. |
|
@daedalius we are working on this, I think we can do much better, but nothing 100% immediate yet, so I will keep this issue open |
|
Just wanted to check in on this thread. I recently ran the react-todo-with-hooks unit testing example, and am still seeing a lot of overhead for each spec file. I really want this project to succeed because I love the idea of having a cypress runner for effectively unit tests, since the testing experience is so much better, but the time costs are just killing me.
For our project, where we are currently running only two small specs, the overhead is even more dramatic:
Granted, our codebase is much larger, but it's not that large, we're at ~70kloc. Turning off
But we're still looking at ~15 seconds per component spec file, which is really unsustainable for us. Running on macOS, 2.9 GHz i9, 32 GB RAM. |
|
@AndrewSouthpaw appreciate the timings - we are working on speeding up the bundling step. I have created prototype system for externalizing React and ReactDOM dependencies - and that really speeds things up, since every spec only has to bundle its own "small" dependencies. The Cypress team has created a different webpack preprocessor where all 3rd party dependencies are externalized. Unfortunately they got sidetracked by moving this repo, and doing some other unrelated work. Please keep an eye on this issue for updates. |
|
Thanks so much for the update, @bahmutov! Will keep an eye out. I appreciate all the Cypress team is doing to revolutionize frontend testing. 🙏 |
|
Hej @bahmutov! I just started playing around with component testing in Cypress but quickly ran into problems with our application. It seems like bundling takes almost 30 minutes and results in a 32Mb monster bundle. A big chunk of it seems to be caused by Font Awesome being bundled in the file, which normally is split into individual chunks for each icon and lazy loaded on demand. The project is using CRA and normal prod compile time is about 1 minute. Is there anything we can do here, to make webpack behave better? |
|
I have not seen such behavior but in general right now on demand lazy loading is not possible. The team at Cypress is working on very different bundling approach similar to webpack dev server instead of making the full bundle. What I would really really would like is an example repo showing this situation so we can validate and play with it
…Sent from my iPhone
On Nov 26, 2020, at 11:20, Oscar Bolmsten ***@***.***> wrote:
Hej @bahmutov! I just started playing around with component testing in Cypress but quickly ran into problems with our application. It seems like bundling takes almost 30 minutes and results in a 32Mb monster bundle. A big chunk of it seems to be caused by Font Awesome being bundled in the file, which normally is split into individual chunks for each icon and lazy loaded on demand.
cypress:webpack finished bundling /Users/oscar/Library/Application Support/Cypress/cy/production/projects/web-d5ecf9daa93ef61ad0216aa43dc25af5/bundles/src/menu/BBBDesktopMenu.spec.tsx.js +640ms
Hash: 920ee608c7d882f4dfbb
Version: webpack 4.42.0
Time: 1528626ms
Built at: 2020-11-26 16:25:43
Asset Size Chunks Chunk Names
BBBDesktopMenu.spec.tsx.js 32.5 MiB main [emitted] main
Entrypoint main = BBBDesktopMenu.spec.tsx.js
[0] multi ./src/menu/BBBDesktopMenu.spec.tsx 28 bytes {main} [built]
***@***.***/runtime/helpers/esm/objectSpread2.js] 1.02 KiB {main} [built]
***@***.***/runtime/helpers/esm/slicedToArray.js] 397 bytes {main} [built]
***@***.***/runtime/helpers/esm/taggedTemplateLiteral.js] 225 bytes {main} [built]
***@***.***/runtime/helpers/interopRequireDefault.js] 147 bytes {main} [built]
***@***.***/runtime/helpers/interopRequireWildcard.js] 1.15 KiB {main} [built]
[../node_modules/cypress-react-unit-test/dist/index.js] 1.05 KiB {main} [built]
[../node_modules/cypress-react-unit-test/dist/mount.js] 7.61 KiB {main} [built]
[../node_modules/cypress-react-unit-test/dist/mountHook.js] 3.33 KiB {main} [built]
[../node_modules/react/cjs/react.development.js] 65.3 KiB {main} [built]
[../node_modules/react/index.js] 189 bytes {main} [built]
***@***.***/common/dist/common.es.js] 32.3 KiB {main} [built]
[./src/config/ConfigContext.tsx] 12.1 KiB {main} [built]
[./src/hooks/useApi.ts] 33.9 KiB {main} [built]
[./src/menu/BBBDesktopMenu.spec.tsx] 4.04 KiB {main} [built]
+ 3138 hidden modules
The project is using CRA and normal prod compile time is about 1 minute.
Is there anything we can do here, to make webpack behave better?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
I understand, I will see if I can lift out the icon loading (which I'm suspecting to be the culprit) into a separate repo. But it relies heavily on lazy loading, so I think the problem is just that it tries to bundle it all in one file. Using webpack dev server seems like a good approach, and should more closely match how CRA does it I guess. |
|
Yeah, I have bunch of text already explaining how specs are bundled and how we can speed it up. I got to finish and put it on my blog
…Sent from my iPhone
On Nov 26, 2020, at 11:31, Oscar Bolmsten ***@***.***> wrote:
I have not seen such behavior but in general right now on demand lazy loading is not possible. The team at Cypress is working on very different bundling approach similar to webpack dev server instead of making the full bundle. What I would really really would like is an example repo showing this situation so we can validate and play with it
I understand, I will see if I can lift out the icon loading (which I'm suspecting to be the culprit) into a separate repo. But it relies heavily on lazy loading, so I think the problem is just that it tries to bundle it all in one file. Using webpack dev server seems like a good approach, and should more closely match how CRA does it I guess.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
Also to be a bit more precise, what I'm using is webpack dynamic imports, using |
|
@bahmutov Did some testing, and the unusual long build time is due to this code in It seems webpack doesn't like that in combination with a couple of thousand dynamic imports, commenting it out and webpack builds in 17s (instead of 28m). What's the problem with serving more than one bundle? |
|
Hello @bahmutov , could you, please, update us how thing are going with fixing this issue ? |
|
Things are going great and soon it will be faster than lightning |
|
This brings me much excitement. Thanks for all your hard work! |
|
@bahmutov did you experimented with a concurrent run? I mean not cypress dashboard, but native mocha --parallel, on a single machine? I understand why Dashboard-like parallelization exists for e2e tests, but since Cypress trying to cover component, unit and api tests - probably you need to revise parallelization politic and allow it for non e2e? It's a default for Jest/Mocha/Ava and other unit-test frameworks. I have seen cypress-parallel project, but native solutions are always better :) |
|
@bahmutov do you maybe have some rough estimations regarding those improvements? maybe we can help somehow? |
|
Hey folks, this was fixed in the major rewrite we did to use your webpack dev server instead of rebuilding with the preprocessor from scratch every time. Please upgrade because the performance is significantly better. From minutes to seconds for initial startup and total runtime. In open mode it’s milliseconds to switch and run tests. It’s as fast as your local dev server for initial startup and then caches everything between specs. It also uses babel cache to make incremental runs even faster. You can also disable source maps but you don’t really need to. Headless CI will be slower than jest right now, because we bundle your application instead of mock out all of your CSS etc, but open mode is amazingly fast. Any optimizations you add for local development will speed up your Cypress test runs. If you end up using a faster dev-server in the future (like Vite, etc) then your perf can become faster than jest. It was released in early April with 7.0. You’ll need to read the migration guide here: https://docs.cypress.io/guides/references/migration-guide#Component-Testing |
|
Could you make this work for vue too? |
Hey @Thaval, can you tell me what you mean by "the preparation to test one component"? Is this before the browser launches or during the loading spinner within Chrome? In normal usage, we heavily cache the webpack builds + babel dependency graph across multiple launches. The performance of Cypress CT should be equal to that of your normal dev server setup. If you're stuck on the loading spinner in the browser while the component compiles and it's taking longer than your normal dev server, then your webpack setup is probably building for production, or is otherwise unoptimized. If the browser hasn't launched yet and you're just waiting for your terminal to print anything, then it's probably due to an issue on Big Sur (or if you're not on Big Sur, then perhaps other OSs have this electron/node issue, too). To check if you have this issue, run the command |
|
Hi, Edit: Turned out the build process is running into a "memory allocation problem". Wow. I'm stunned :D |
|
Is it possible to tell cypress which command within package.json it should use to build? |
|
Which command... what do you mean? For questions, chat on Discord |
|
@JessicaSachs hey! Could you please answer the question above? |
Hi Gleb!
Yesterday I've already experimented with the v4 version and noticed that tests run slow enough.
For example: git@github.com:daedalius/cypress-react-test.git
Here are 10 copies of the same test. I run them in a row:
On the timeline it takes:
and so on:
Total times:
Windows:
mac os:
The output says that clean test time is 3 seconds (around ~0.3 per test).
Is there any way to speed up the test preparation phase?
You know, I am in love with Cypress and this repo in particular. But it's hard to debate with colleagues who argue that Jest does 500 tests in 20 seconds.
This is maybe important:
Windows: Ryzen 2600x, NVME, Windows 10 build 1809
Mac: MacBook Pro 2019, i7 6-cores 2.6 - 4.5 GHz, SSD
The text was updated successfully, but these errors were encountered: