-
Notifications
You must be signed in to change notification settings - Fork 454
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ts-jest with cleared cache is reading package.json in node_modules over and over - start takes forever #908
Comments
I got a minimal repo working - not sure how mini it is though. Removing all the unneeded packages (like ionic) results in the problem going away so apparently they are needed after all. Update: The demo is now as small as possible [probably]. |
@Farbdose thanks for investigation and minimal repo. I have no time looking at this now, but if the read is in |
I think the broader problem is something changed in 20.10 from 20.1 such that we rolled back. |
I removed some dependencies from the demo repo that were apparently unnecessary. |
Update: I played a bit around and wrote a very basic [destructive] cache into getPackageJsonInfo
...which got the time down to around 24s again. The interesting thing though is, that after that I got a much clearer cpu profile: Update2 I tried that mini cache in a bigger project and it just crashed typescript. So apparently this only works if you aren't importing anything complex (like in the demo repo) Update 3: I found a proper place to cache the json files I'll investigate what update introduced the additional slowdown to the demo project and try to get rid of that too... after that ... it looks like I'll have to create a pullreqest for typescript. readJson patch V1 for compiled typescript.js
readJson patch V2 for typescript/src/compiler/utilities.ts
|
Update 4: so with fileSystemEntryExists patch V1 for compiled typescript.js
fileSystemEntryExists patch V2 for typescript/src/compiler/sys.ts
So it seems this slowdown is caused by non-highspeed-file-access ... apparently one needs an unencrypted SSD for this stuff. I will try to put node_modules into a ramdisk and compare the performance. I will create an issue over at typescript in the following days (after I found out how I can get the tests to pass again^^) The only thing that's left is to find out why performance dropped so much in the first place. @huafu @ericanderson But why did this get so much worse with the ts-jest update? Update 5: I put node_modules into a ram_disk and removed my caches witch got it down to 14s (no overhead for cache) tl;tr Check if your node_modules is on a fast hard drive (eg unencrypted SSD) If you are trying to optimize a CI pipeline without node_modules cache (like me) that reinstalls everything on each build, consider creating a ramdisk for node_modules before running ram disk example
Update 6: It seems like readJson is only slow because of the containing calls to |
Update 7: typescript apparently already added a cache microsoft/TypeScript#27068 Update 8: I'm giving up on fixing this. For some reason now even my baseline version that should have no extras is slow with a ramdisk - no idea what I did... I updated the demo repository to the latest ionic version and typescript to 3.2.2 and 3.3.0-dev.20181220 in another branch It now takes around 45 seconds in the gitlab build pipeline so if anybody wants to try his luck, you are welcome. I'll create an issue over at Typescript maybe they have an idea. |
I'd like to add that I also experienced a big performance regression trying to update from 23.0.x to 23.10.5. The size of this regression prevents me from upgrading. I really appreciate that this library exists and all the work that goes into it, but I may have to consider an alternative in the future. |
Also would like to inform that I tried upgrading to alpha version of jest-preset-angular (which depends on new version of ts-jest) and timing difference was impossible to work with. |
Recently, I also encountered the slowdown of tests with disabled jest cache. And here's results for my codebase: |
Same issue remains for version 24.0, as far as I can tell... |
I have a simple lib that I'm working on. It probably has about a dozen small ts files and it takes ~25 seconds to start running the tests. |
I like how you guys experience tests slowing down to N seconds... you think you have ts-jest slow... |
OH MY GOD! A week of banging my head against the wall just to find this little package.json option deep down in convoluted issue threads.... |
Thanks @VicJer that config adjustment worked brilliantly! |
Hi guys, 23.10.5 was a complete rewrite version and ts-jest switched to use typescript language service as default mode, which is be slower than isolatedModules mode. 23.1.x used isolatedModules mode so it explained why the tests ran faster. However, using this isolatedModules mode you will lose some functionalities see doc We have tried our best to do some improvements regarding to this topic. I will close this issue, please test with 25.2.1 and share your opinions and statistics about this topic in #1115. Any helps are welcome 🙏 |
Issue:
I just updated to ts-jest@23.10.5 from somewhere around ~23.1.3 (which increased the runtime of my testsuite from 10ish seconds [on the first run / with cleared cache) to over 40 seconds - with cache around 2.5s.
Update: problem was already present at 23.10.1 [not sure if earlier - the config format changed there so I have to rewrite that first if I want to go further back]
So I started investigating... and found out that something is calling readFile like mad.
To be precise: almost all calls to
ts.resolveModuleName
intypescript.js
seem to cause a readFile of the corresponding package.jsonI added some
consolge.log
tonode_modules/typescript/lib/typescript.js
andgot the following log
I would appreciate a second opinion as I'm not even sure if this is some weird edge case in ts-jest, an oversight in typescript or if its possible to cause it through misconfiguration.
Update: I think I managed to track it down to js files that need to be processed by ts-jest (as they contain import statements) being present in on of the imported modules (one of our own). Removing it causes the runtime to go back to normal. The interesting thing is that the module in question has its own jest tests and when I use the same config there I have no problems, so the problem is not the code itself but the fact that I'm importing the compiled version of it.
Also with typescript below 2.9 it's still present but only around 26 seconds [not >40]
So upgrading:
ts-jest@>=23.10.1 (23.10.0 doesn't work because of #740) causes a slowdown of around x5 and
typescript@>=2.9 causes an additional slowdown of x2
Also, removing ionic removed the problem too. So its possible this is actually an ionic issue. I created a separate issue for that though: ionic-team/ionic-framework#16703
Expected behavior :
The initial start of the test suite shouldn't take so much longer. Maybe the package.json files should be cached.
Debug log:
ts-jest.log
Full version at https://gitlab.com/Farbdose/ts-jest-multiple-fileread/raw/master/ts-jest.log
Minimal repo :
https://gitlab.com/Farbdose/ts-jest-multiple-fileread
While creating this demo repository I realized that the problem goes away if I remove ionic so this seems to be partially responsible too.
The text was updated successfully, but these errors were encountered: