[2.8.0-rc] Segfault when running compiler with --d or --watch (out of memory)

[canonic-epicure]

Hello,

I was using 2.8.0-dev-20180216 and recently switched to 2.8.0-rc. I immediately noticed, that, when launching compiler with --watch it started to take a lot longer for the initial compilation. The subsequent compilations on file change were working fine.

But now, after some code refactoring, launching with --watch segfaults with out of memory error (after a long wait). Running w/o --watch works just fine (and fast enough).

So there's definitely some regression related to the initial compilation of the project in --watch mode.

Here's an output from the shell session. Note, how compilation w/o --watch worked fine (3rd command):

nickolay@outpost:~/workspace/Bryntum/SchedulingEngine0.x$ ./node_modules/.bin/tsc --version
Version 2.8.0-rc
nickolay@outpost:~/workspace/Bryntum/SchedulingEngine0.x$ node --version
v8.9.4
nickolay@outpost:~/workspace/Bryntum/SchedulingEngine0.x$ ./node_modules/.bin/tsc
nickolay@outpost:~/workspace/Bryntum/SchedulingEngine0.x$ ./node_modules/.bin/tsc --watch

16:40:41 - Starting compilation in watch mode...



<--- Last few GCs --->

[4089:0x23b2e20]    92678 ms: Mark-sweep 1411.0 (1459.8) -> 1411.0 (1443.8) MB, 1653.6 / 0.5 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1655 ms) last resort GC in old space requested
[4089:0x23b2e20]    94301 ms: Mark-sweep 1411.0 (1443.8) -> 1411.0 (1443.8) MB, 1622.7 / 0.6 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x43f5ef25ee1 <JSObject>
    1: getTextOfNode(aka getTextOfNode) [/home/nickolay/workspace/Bryntum/SchedulingEngine0.x/node_modules/typescript/lib/tsc.js:~60023] [pc=0xc63f1afcb97](this=0x212ba7e02311 <undefined>,node=0x27070b5c3fd9 <Node map = 0x136dd41e54c1>,includeTrivia=0x212ba7e02421 <false>)
    2: emitIdentifier(aka emitIdentifier) [/home/nickolay/workspace/Bryntum/SchedulingEngine0.x/node_modules/typescript/lib/t...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [node]
 2: 0x122563c [node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [node]
 6: v8::internal::LoadIC::LoadFromPrototype(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::JSObject>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Smi>) [node]
 7: v8::internal::LoadIC::GetMapIndependentHandler(v8::internal::LookupIterator*) [node]
 8: v8::internal::IC::ComputeHandler(v8::internal::LookupIterator*) [node]
 9: v8::internal::LoadIC::UpdateCaches(v8::internal::LookupIterator*) [node]
10: v8::internal::LoadIC::Load(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Name>) [node]
11: v8::internal::Runtime_LoadIC_Miss(int, v8::internal::Object**, v8::internal::Isolate*) [node]
12: 0xc63f178463d
Aborted (core dumped)
nickolay@outpost:~/workspace/Bryntum/SchedulingEngine0.x$

[canonic-epicure]

Correction - it actually fails with 2.8.0-dev.20180216 as well, but now, it just takes forever for the initial compilation with --watch. Compilation w/o --watch works fine with 2.8.0-dev.20180216 as well.

[canonic-epicure]

The application does a heavy use of mixins if that matters.

[DanielRosenwasser]

Potentially a duplicate of #21140, but @sheetalkamat can you take a look?

[sheetalkamat]

@samuraijack can you please share your project so i can repro and investigate this. So far we haven't gotten repro on this issue so its hard to predict without looking at it what might be going wrong. Thanks.

[canonic-epicure]

@sheetalkamat Ok, I'll prepare the reproducible test case, on Monday already..

[canonic-epicure]

Unfortunately, preparing a test case is not easy - I can't share the code.

But you should check the code, that performs the initial compilation during --watch. That initial compilation takes a lot longer than normal compilation without --watch. I guess it also uses more memory and thus crash at certain project size. So that initial compilation is done in some different mode and bug is there.

[taoqf]

I've seen this too.
I believe this may have relation with #22668 if you use cnpm too. @samuraijack

[canonic-epicure]

Nope, we don't use cnpm. I believe its some algorithm with quadratic behavior in "watch" compilation mode. At certain project size it becomes noticeable and process crashes with OOM.

@sheetalkamat To reproduce this, you could try to generate huge dummy project, based on the template like:

type Constructable<T = {}> = new (...args : any[]) => T

export const Mixin1 = <T extends Constructable>(base : T) =>

    class Mixin1 extends base {
        method1() {
        }
    }

export const Mixin2 = <T extends Constructable>(base : T) =>

    class Mixin2 extends base {
        method2 () {
        }
    }


export const Mixin3 = <T extends Constructable>(base : T) =>

    class Mixin3 extends Mixin2<Constructable>(base) {
        method3 () {
        }
    }


export const Mixin4 = <T extends Constructable>(base : T) =>

    class Mixin4 extends Mixin1(Mixin3(base)) {
        method4 () {
            this.method1()

            this.method2()
        }
    }

The idea is that Mixin(N) includes Mixin(N-1). Mixins should be in different files. This roughly corresponds to what we have in our project (of course we don't have such deep mixins "inheritance", only 2-3 levels, but there are many sibling mixins). I believe at certain N the issue will manifest itself.

[sheetalkamat]

@samuraijack can you try the nightly build that would be published tonight and see if it still repros for you. I have been playing around with various number of files with the contents from the files you shared and it seem to be comparable:
Watch Mode:

c:\temp\test>node c:\TypeScript\built\local\tsc.js --extendedDiagnostics -w
15:59:34 - Starting compilation in watch mode...


Files:                 2001
Lines:                92523
Nodes:               353922
Identifiers:         118900
Symbols:             162671
Types:               180859
Memory used:        346460K
I/O Read time:        0.64s
Parse time:           0.69s
Program time:         1.50s
Bind time:            0.31s
Check time:           5.59s
transformTime time:   4.06s
commentTime time:     0.29s
I/O Write time:       2.15s
printTime time:       8.62s
Emit time:            8.72s
Total time:          16.12s
15:59:50 - Compilation complete. Watching for file changes.


16:00:20 - File change detected. Starting incremental compilation...


Files:                 2001
Lines:                92523
Nodes:               353922
Identifiers:         118900
Symbols:             100351
Types:                  146
Memory used:        347119K
I/O Read time:        0.00s
Parse time:           0.00s
Program time:         0.06s
Bind time:            0.04s
Check time:           0.14s
transformTime time:   0.10s
commentTime time:     0.00s
printTime time:       0.16s
Emit time:            0.17s
I/O Write time:       0.00s
Total time:           0.41s
16:00:21 - Compilation complete. Watching for file changes.


16:00:30 - File change detected. Starting incremental compilation...


Files:                 2001
Lines:                92523
Nodes:               353922
Identifiers:         118900
Symbols:             100351
Types:                  146
Memory used:        348419K
I/O Read time:        0.00s
Parse time:           0.01s
Program time:         0.05s
Bind time:            0.01s
Check time:           0.00s
transformTime time:   0.01s
commentTime time:     0.00s
printTime time:       0.01s
Emit time:            0.01s
I/O Write time:       0.00s
Total time:           0.07s
16:00:30 - Compilation complete. Watching for file changes.

Without watch:

c:\temp\test>node c:\TypeScript\built\local\tsc.js --extendedDiagnostics
Files:                 2001
Lines:                92523
Nodes:               353922
Identifiers:         118900
Symbols:             162671
Types:               160859
Memory used:        438993K
I/O Read time:        0.34s
Parse time:           0.85s
Program time:         1.29s
Bind time:            0.43s
Check time:           4.37s
transformTime time:   2.68s
commentTime time:     0.36s
I/O Write time:       2.82s
printTime time:       8.30s
Emit time:            8.30s
Total time:          14.39s

I got OOM for 20000 files with and without --watch option and works for both modes with 5000 files.

[canonic-epicure]

Tried with 2.9.0-dev.20180329 - no changes, did you mean 2.9.0-dev.20180330 perhaps?

I have only 54 TS files. The internal typing references are a lot more complex than in the example I posted of course..

[canonic-epicure]

Tried with 2.9.0-dev.20180330 - no change.

[canonic-epicure]

@sheetalkamat I'm happy to do a screenshare session at your convenience if that is possible.

[sheetalkamat]

@samura Are you still able to repro this issue with typescript@next? If yes, can you please check if you see this when you run tsc --d (ensure emit is not disabled)? I am guessing this has to do with the declaration emit (which is used to build incrementally in watch mode) but would like to confirm that. Thank you.

[samura]

wrong username? I'm not @samuraijack

[canonic-epicure]

@sheetalkamat Tried with 3.0.0-dev.20180626 - seems there's no difference, I see

[11:29:03] Starting compilation in watch mode...

And then nothing.

[sheetalkamat]

wrong username? I'm not @samuraijack

Sorry about that.

@samuraijack Are you still able to repro this issue with typescript@next? If yes, can you please check if you see this when you run tsc --d (ensure emit is not disabled)? I am guessing this has to do with the declaration emit (which is used to build incrementally in watch mode) but would like to confirm that. Thank you.

Were you able to check if this repros when emit is not disabled and emitting declarations?

[canonic-epicure]

Were you able to check if this repros when emit is not disabled and emitting declarations?

@sheetalkamat How do you mean? Some particular command line switch? I'm just launching ./node_modules/.bin/tsc --watch

[sheetalkamat]

@samuraijack run it as ./node_modules/.bin/tsc --d --extendedDiagnostics

[DanielRosenwasser]

Seems like this is extremely nontrivial to fix. @weswigham may have some time to review this.

[canonic-epicure]

Any updates on this? Still present in 3.1.6