Added cache for some FS operations while compile #829

timocov · 2018-08-29T08:44:36Z

Fixes #825.

Open questions:

Should it be under option?
If so, how this option should be named? Should it be something like "experimental" flag?
~~Do we need to add logs? Is so, what kind of logs (clearing caches or something else)?~~
~~Should we add the same caches to makeWatchHost too?~~
I'm not sure how to write tests for this - possible somebody has thoughts about it.

johnnyreilly · 2018-08-29T09:43:08Z

Thanks for the PR! I'm away from home right now with only my phone and dodgy connectivity and so will struggle to review this until in front of a proper screen.

To your questions:

Should it be under option? If so, how this option should be named? Should it be something like "experimental" flag?

Yes please; I like the idea of this going out under a flag called something like experimentalFileCaching. If it works well and no problems are reported we can look to promote this to the default behaviour in future.

Do we need to add logs? Is so, what kind of logs (clearing caches or something else)?

Don't feel strongly. I'd say probably not for now.

Should we add the same caches to makeWatchHost too?

I'm not sure; I think not for now.

I'm not sure how to write tests for this - possible somebody has thoughts about it.

The encouraging thing is that tests pass before you've put it behind an option. But yeah - need to think about this

johnnyreilly · 2018-08-29T09:49:05Z

Also, please could you experiment with this on your own projects and document the performance improvements you're seeing here please?

I'd like the benefits to be easily discoverable for people who want learn about this. Thanks!

timocov · 2018-08-29T10:08:13Z

Yes please; I like the idea of this going out under a flag called something like experimentalFileCaching. If it works well and no problems are reported we can look to promote this to the default behaviour in future.

Ok, I'll do it.

Also, please could you experiment with this on your own projects and document the performance improvements you're seeing here please?

Sure. But currently I am able to check this fix against 3.5.0 version only (I have fix for this version too - see https://github.com/timocov/ts-loader/tree/fix825_v3) because we are in progress to upgrade to webpack 4 for now. But I'll try to check with v4 too.

johnnyreilly · 2018-08-29T13:53:18Z

Sure. But currently I am able to check this fix against 3.5.0 version only

I'm guessing you know that ts-loader has supported only webpack 4+ since version 4?

timocov · 2018-08-29T14:31:32Z

I'm guessing you know that ts-loader has supported only webpack 4+ since version 4?

Yep 🙂

timocov · 2018-08-29T14:31:44Z

Here are cold-compilation results for our project.

Env:

PC: AMD FX-4300 Quad-Core 4x3.8GHz, 16GB RAM, HDD, Windows 10
NodeJS version: 10.3.0
TypeScript version: 3.0.1
Approx. modules count: 3000-4700 (from webpack stats)

`webpack@3.10.0` runs:

`ts-loader` version	compilation time (avg of three runs)
`2.3.3`	554s (546s, 553s, 564s)
`3.5.0`	828s (850s, 810s, 823s)
`3.5.0` (with cache)	303s (296s, 312s, 302s)

`webpack@4.16.0` runs:

`ts-loader` version	compilation time (avg of two runs)
`4.5.0`	419s (413s, 425s)
`4.5.0` (with cache)	315s (307s, 324s)

I think that we cannot compare results between webpack 3 and webpack 4 because there are some changes in our project for webpack 4 and I cannot guaratnee that the build has exactly the same input (but inside one webpack version it should be the same).

timocov · 2018-08-29T15:32:04Z

I'm guessing you know that ts-loader has supported only webpack 4+ since version 4?

If this PR will be merged, is there any chance to make the similar PR for 3 version (I'm ready to provide necessary changes for 3.5 version)?

johnnyreilly · 2018-08-29T16:32:09Z

If this PR will be merged, is there any chance to make the similar PR for 3 version (I'm ready to provide necessary changes for 3.5 version)?

You mean a PR for this branch: https://github.com/TypeStrong/ts-loader/tree/ts-loader-3?files=1

Which I'd then publish as 3.6?

Honestly I'm not keen since I've not been supporting webpack 3 since 4 shipped and I only have a certain amount of time I can devote to open source. This would take away part of that time and given people are moving away from webpack 3 it doesn't feel like time well spent.

Will that be super painful for you?

timocov · 2018-08-29T19:10:27Z

given people are moving away from webpack 3

Hm, good point 👍

Will that be super painful for you?

I guess in this case we can just publish it to local npm until we'll upgrade to webpack 4.

timocov · 2018-09-03T15:56:29Z

@johnnyreilly do I need to do something else in the PR? It is possible that you have any confused/ambiguously thoughts about it (I guess except tests)?

johnnyreilly · 2018-09-03T16:33:58Z

Sorry - I hadn't spotted that you'd added further commits. I'm back from my travels now; I'll try and take a look soon. Please bear with me!

src/instances.ts

johnnyreilly · 2018-09-03T16:41:24Z

src/instances.ts

-      "You may be using an old version of webpack; please check you're using at least version 4"
-    );
+    if (cachedServicesHost.clearCache !== null) {
+      loader._compiler.hooks.watchRun.tap(


I just wanted to check; is watchRun definitely sync not async?

It seems that watchRun is Tappable, which supports both async and sync methods (tap and tapAsync).

https://webpack.js.org/api/compiler-hooks/#watchrun
https://github.com/webpack/tapable/blob/04e52118691e0a6c77a42d15e99dc4bd2bfca07c/lib/AsyncSeriesHook.js#L21

Is there any advantage to using tapAsync instead of tap? I've an idea it doesn't really matter but I thought I'd check.

I believe that it depends on what are you doing in your handler. In this case we need to use sync because we need to just clear cache without any async operation. But if you find something interesting about it - please let me know 🙂

src/servicesHost.ts

- Added type Action - CachedServicesHost renamed to ServiceHostWhichMayBeCacheable - cachedServicesHost variable renamed back to servicesHost

src/servicesHost.ts

timocov · 2018-09-05T19:46:43Z

@johnnyreilly can you please take a look?

johnnyreilly · 2018-09-05T20:51:20Z

Thanks @timocov - I will!

johnnyreilly · 2018-09-06T19:54:29Z

Really sorry About the delay @timocov - very very busy right now. I haven't forgotten about this - just not had time

timocov · 2018-09-06T20:25:08Z

No problem 🙂

johnnyreilly · 2018-09-08T04:33:31Z

src/servicesHost.ts

-    fileExists: compiler.sys.fileExists,
-    readFile: compiler.sys.readFile,
+    fileExists: moduleResolutionHost.fileExists,
+    readFile: moduleResolutionHost.readFile,
    readDirectory: compiler.sys.readDirectory,


was there a reason that readDirectory wasn't included in the caching functionality?

Nope. I didn't see this function in the profiler before and I don't think that it can reduce perf, but if you want - I can check and provide the stats.

If you could check that'd be awesome. I'm currently working on tweaking the test packs so they can be made to work against a variety of experimental... flags - the idea being that while the cache functionality lies behind a flag we can still make sure it is tested by the existing test pack. Watch this space...

#834

If you could check that'd be awesome

Unfortunately for now I'm away from work and cannot check it. I'll do it in next couple of days and will provide here results.

@johnnyreilly I just checked readDirectory - it seems that this function is never called while compilation (at least in our case).

johnnyreilly · 2018-09-08T04:34:55Z

src/servicesHost.ts


    useCaseSensitiveFileNames: () => compiler.sys.useCaseSensitiveFileNames,

+    realpath: moduleResolutionHost.realpath,


We didn't have realpath before I think; I'm just curious; what is it used for?

To resolve path to original one. For example, if you have symlink.

Ah - we've had since we added symlink support here: https://github.com/TypeStrong/ts-loader/pull/774/files

We just didn't supply it to the LanguageServiceHost until now though. I'm kind of surprised we didn't get a compilation error previously 🤔

Hm, interesting. Perhaps microsoft/TypeScript#12020 (comment) (and the whole PR) is related to this.

1. Made variable names more descriptive 2. `cacheableFunctions` is now declared once and re-used (micro-optimisation TBH)

… fix825

johnnyreilly · 2018-09-08T22:52:41Z

Hey @timocov,

I've been beavering away on the test pack and I think I've now got the comparison test pack set up so it does 2 passes; the existing pass and then again but with the experimentalFileCaching flag set. I'm just waiting for CI to pass. Assuming it does I think we're nearly ready to merge. 😄

johnnyreilly · 2018-09-09T20:21:55Z

Released with v5.1.0 https://github.com/TypeStrong/ts-loader/releases/tag/v5.1.0

Thanks for all your work!

timocov · 2018-09-09T20:44:19Z

Wow, thank you @johnnyreilly!

johnnyreilly · 2018-09-09T20:54:56Z

My pleasure - you've been an awesome contributor! ❤️ 🌻

Added cache for some FS operations while compile (#825)

af7ab34

Added experimentalFileCaching loader option (#825)

370cf78