Question: Compiler is updating all files all the time

[sechel]

I have a question: It seems to me that the typescript compiler compiles all files even if I touch only one of the source files. Only the bundler seems to know what really changed. For a big project this seems odd as it may take a lot of time to do this. In a TDD environment this could be frustrating. Is this the expected behaviour? Here is the CLI output of me touching one of the files in the example-project:

✔ ~/workspace/karma-typescript/example-project [master|✔]
10:55 $ npm test

> karma-typescript-example-project@2.1.6 test /Users/sechel/workspace/karma-typescript/example-project
> karma start karma.conf.js

25 01 2017 10:56:06.844:INFO [compiler.karma-typescript]: Compiling project using Typescript 2.1.5
25 01 2017 10:56:08.758:INFO [compiler.karma-typescript]: Compiled 3 files in 1354 ms.
25 01 2017 10:56:10.776:INFO [bundler.karma-typescript]: Bundled imports for 3 file(s) in 153 ms.
25 01 2017 10:56:10.849:WARN [karma]: No captured browser, open http://localhost:9876/
25 01 2017 10:56:10.861:WARN [karma]: Port 9876 in use
25 01 2017 10:56:10.862:INFO [karma]: Karma v1.4.0 server started at http://0.0.0.0:9877/
25 01 2017 10:56:10.863:INFO [launcher]: Launching browser Chrome with unlimited concurrency
25 01 2017 10:56:10.914:INFO [launcher]: Starting browser Chrome
25 01 2017 10:56:12.166:INFO [Chrome 56.0.2924 (Mac OS X 10.12.2)]: Connected on socket ms021osP7Av0g3vMAAAA with id 44195702
Chrome 56.0.2924 (Mac OS X 10.12.2): Executed 1 of 1 SUCCESS (0.098 secs / 0.044 secs)
25 01 2017 10:56:50.436:INFO [compiler.karma-typescript]: Compiled 3 files in 456 ms.
25 01 2017 10:56:51.466:INFO [bundler.karma-typescript]: Bundled imports for 1 file(s) in 69 ms.
25 01 2017 10:56:51.466:INFO [watcher]: Changed file "/Users/sechel/workspace/karma-typescript/example-project/src/hello.component.spec.ts".
25 01 2017 10:56:51.471:INFO [watcher]: Changed file "/var/folders/01/6kyd1sm53xd3s96hs4nlw2740000gn/T/karma-typescript-bundle-80892IMF8Ij3NGpv7.js".
Chrome 56.0.2924 (Mac OS X 10.12.2): Executed 1 of 1 SUCCESS (0.008 secs / 0.002 secs)

[erikbarke]

This is the way the Typescript compiler works, so this is expected behavior! Typescript exposes two API's:

• The LanguageService API, which is designed for generating info for code completion, formatting etc in IDE's and editors.
• The Compiler API which is designed for generating outputs and type checking.

karma-typescriptuses the Compiler API which compiles the whole project every time something changes... However, state is cached heavily cached (program structure, source files and their AST's), which is why it took 1354 ms the first time the example project compiled on your end and only 456 ms on the second compile.

There's a great breakdown of the two API's by @mhegazy here.

Here's the output from the integration-tests@latest project on my laptop:

25 01 2017 13:58:03.071:INFO [compiler.karma-typescript]: Compiling project using Typescript 2.1.5
25 01 2017 13:58:09.171:INFO [compiler.karma-typescript]: Compiled 2056 files in 4954 ms.
25 01 2017 13:58:11.269:INFO [bundler.karma-typescript]: Bundled imports for 2050 file(s) in 1426 ms.
25 01 2017 13:58:14.988:WARN [karma]: No captured browser, open http://localhost:9876/
25 01 2017 13:58:15.014:INFO [karma]: Karma v1.4.0 server started at http://0.0.0.0:9876/
25 01 2017 13:58:15.015:INFO [launcher]: Launching browsers Chrome, PhantomJS with unlimited concurrency
25 01 2017 13:58:15.021:INFO [launcher]: Starting browser Chrome
25 01 2017 13:58:15.030:INFO [launcher]: Starting browser PhantomJS
25 01 2017 13:58:16.066:INFO [PhantomJS 2.1.1 (Mac OS X 0.0.0)]: Connected on socket bXj7nk9S3_KwubmSAAAA with id 27312835
25 01 2017 13:58:16.079:INFO [Chrome 55.0.2883 (Mac OS X 10.11.6)]: Connected on socket cV1ACwbLuuXwUwZYAAAB with id 47828327
PhantomJS 2.1.1 (Mac OS X 0.0.0): Executed 1047 of 1047 SUCCESS (1.506 secs / 1.052 secs)
Chrome 55.0.2883 (Mac OS X 10.11.6): Executed 1047 of 1047 SUCCESS (1.345 secs / 0.881 secs)
TOTAL: 2094 SUCCESS

=============================== Coverage summary ===============================
Statements   : 100% ( 7199/7199 )
Branches     : 100% ( 0/0 )
Functions    : 100% ( 2070/2070 )
Lines        : 100% ( 6166/6166 )
================================================================================

=============================== Coverage summary ===============================
Statements   : 100% ( 7199/7199 )
Branches     : 100% ( 0/0 )
Functions    : 100% ( 2070/2070 )
Lines        : 100% ( 6166/6166 )
================================================================================
25 01 2017 13:58:44.166:INFO [compiler.karma-typescript]: Compiled 2056 files in 2625 ms.
25 01 2017 13:58:44.790:INFO [bundler.karma-typescript]: Bundled imports for 1 file(s) in 41 ms.
25 01 2017 13:58:44.791:INFO [watcher]: Changed file "/Users/erikbarke/Development/npm/karma-typescript/integration-tests@latest/src/imports/style-import/style-import-tester.spec.ts".
25 01 2017 13:58:44.910:INFO [watcher]: Changed file "/var/folders/7h/mqlpsjv10694rrxdtl3v84yw0000gn/T/karma-typescript-bundle-547591S8hmo8yr4SL.js".
PhantomJS 2.1.1 (Mac OS X 0.0.0): Executed 1047 of 1047 SUCCESS (1.562 secs / 1.063 secs)
Chrome 55.0.2883 (Mac OS X 10.11.6): Executed 1047 of 1047 SUCCESS (1.559 secs / 1.031 secs)
TOTAL: 2094 SUCCESS

=============================== Coverage summary ===============================
Statements   : 100% ( 7199/7199 )
Branches     : 100% ( 0/0 )
Functions    : 100% ( 2070/2070 )
Lines        : 100% ( 6166/6166 )
================================================================================

=============================== Coverage summary ===============================
Statements   : 100% ( 7199/7199 )
Branches     : 100% ( 0/0 )
Functions    : 100% ( 2070/2070 )
Lines        : 100% ( 6166/6166 )
================================================================================

It compiled 2056 files in 4954 ms on the first run and then 2056 files in 2625 ms on the second run, not too shabby eh? Having said that, there's always room for improvement, so if you have any suggestions or want to discuss performance improvements I'm all ears!

Sorry for the wall of text and I hope i haven't gone off on a tangent here 😃

[sechel]

@erikbarke thanks for the fast reply. I have tried on one of my own projects and got the following results:
First run: Compiled 128 files in 6049 ms.
Second run: Compiled 128 files in 3547 ms.
Looks like I need a new Laptop 😄, or there is something going fundamentally wrong here. I mean you have a number of files one order of magnitude higher than me but roughly the same compile times. What do you think?

[sechel]

BTW: Is there a way to slide in constants similar to what I did with the webpack DefinePlugin?

[erikbarke]

How many seconds does it take to compile your project on the command line with tsc? It should take the same amount of time, karma-typescript uses the same api as the Typescript cli. And yeah, if my more than three years old laptop beats yours you probably want a new one 😃

Judging by the issues and comments here there's quite a few webpack users using this plugin now so maybe it's time to add some new bundler features like constants, I'll look into it!

[sechel]

tsc needs about 7 sec to finish. I still cannot believe it should be this slow. However the metrics of my project seems comparable to yours, apart from the number of files (and the coverage ratio 😉):

=============================== Coverage summary ===============================
Statements   : 76.6% ( 8234/10749 )
Branches     : 58.76% ( 3759/6397 )
Functions    : 78.62% ( 1570/1997 )
Lines        : 76.81% ( 7396/9629 )
================================================================================

Running the integration-tests@latest suite again gives comparable timings to your machine:
First run Compiled 2056 files in 6444 ms.
Second run Compiled 2056 files in 4999 ms.
So this makes me believe the compile time depends on the number of lines of code, which makes sense in a way 😄. So my laptop is not as bad as suggested.

[sechel]

I had a look into the typescript API wiki and there is a section about incrementally building a project: Incremental build support using the language services. It may really be worth it and I'm sure all TDD practitioners would be happy.

[mhegazy]

Most of the time the compiler spends in a compilation is in doing type-check. type-check is a program-wide operations, since changes to types in one file can result in errors in another. When doing incremental build either through tsc --w or when using the API, only the parsed trees can be shared. a full type check is still needed to generate errors.

If your commandline compilation with tsc --diagnostics show different numbers you are seeing here then this is a bug in the plugin. If you think your project should not take that long to build, then file a bug on the TS issue tracker.

If the issue is just incremental build, then consider 1. using --skipLibCheck this will not do a type check on your .d.ts files, saving you some time. or 2. not getting errors at all while building, and relying on your IDE to generate the errors, or another full build step. you can find similar discussion in microsoft/TypeScript#13538

[sechel]

@mhegazy thanks, I followed the various suggestions there to speed up me webpack workflow and I am quite happy now. Nevertheless the karma-typescript plugin has a few pros such as better support for source maps (VS Code debugging never really worked with the webpack setup) and coverage (only got coverage working for es5 artifact code, which is not really what I wanted).

[sechel]

@mhegazy here is the stats of my project. It seems that there are too many files processed and way too many lines too. It there a way of making typescript verbose to see which files exactly it is compiling?:

16:12 $ tsc --diagnostics
Files:           284
Lines:         69409
Nodes:        295860
Identifiers:  100240
Symbols:       93589
Types:         36512
Memory used: 189492K
I/O read:      0.14s
I/O write:     0.05s
Parse time:    1.42s
Bind time:     0.59s
Check time:    2.80s
Emit time:     1.09s
Total time:    5.90s

[mhegazy]

--listFiles

[erikbarke]

Thanks for this input guys!

I'm not sure the LanguageService api is a good fit for this plugin as it's designed mainly for creating code completion etc in IDE's and karma-typescript acts more like a glue between Karma and tsc using the Compiler api.

However, I think it would be possible to introduce some kind of "transpileMode" which would speed up compilation quite a bit for us TDD heads 😃 The plugin currently relies heavily on the Typescript compiler for creating AST's and resolving dependencies, so this functionality will have to be moved to the bundler instead, making the bundler a bit slower on startup, but incremental builds/bundling would probably be almost instant if type checking and resolving is skipped in the compilation step.

[sechel]

That would be great. And considering source maps: Is there a way to emit and serve source map files. VSCode does not support inline source code as it is included by the bundler (is it really him?) at the moment.

[erikbarke]

Off the top of my head I'd say that it should be possible to inject a dummy source map file for each .ts|.tsx file in the project into the Karma file list and output the source map there, instead of as a part of the bundle wrapper!

[erikbarke]

@sechel, the bundler relies so much on the Typescript compiler doing the grunt work with resolving dependencies that a "transpileMode" really can't be implemented without breaking the bundling process, and implementing all that logic (already implemented by the Typescript team) in the bundler would take a considerable effort so I'm closing this for now :(
Hopefully I'll be able to revisit this at some point in time.

[milimetric]

Does this new tsc --build mode help? (http://www.typescriptlang.org/docs/handbook/project-references.html#build-mode-for-typescript). Has someone figured out a way to do incremental TypeScript builds while testing with karma?