JavaScript heap out of memory for 10s of MB of source #17112
Comments
|
@sandersn , Could this be fixed? |
|
Two questions:
Afterwards, if tsc still works, then it sounds like a gulp-typescript bug. The stack is in emit when it runs out of memory. This is very unusual for vanilla tsc, so I would first suspect a configuration/setup bug in gulp-typescript. Also, @rbuckton, does that crash location mean anything to you? |
|
1 means comonjs 2 amd 3 umd |
DanielRosenwasser
added
the
Needs More Info
|
looks like the gulp typescript + typescript memory usage pushes you over the limit. I would say gulp-typescript should take the first stab at this issue. |
mhegazy
added
External
|
@mhegazy , I got this Error just now: using |
|
can you share your project |
mhegazy
added
Bug
|
+1 I have same problem.
|
|
@taoqf @swlee60 can you try If that doesn't help, is there some way you can share your project? I don't have any other starting point for debugging since eliminating gulp-typescript from the crash. I can sign an NDA if needed. |
|
@sandersn $ tsc --version PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND webpack: Compiling... <--- Last few GCs ---> 551020 ms: Mark-sweep 1288.8 (1408.1) -> 1288.8 (1412.1) MB, 876.2 / 0.0 ms [allocation failure] [GC in old space requested]. <--- JS stacktrace ---> ==== JS stack trace ========================================= Security context: 0x3f6911dcfb51 FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory |
|
@sandersn same here. $ tsc --version
Version 2.6.0-dev.20170824
$ tsc
<--- Last few GCs --->
[26137:0x3b6b020] 37623 ms: Mark-sweep 1403.3 (1445.3) -> 1403.3 (1440.8) MB, 861.8 / 0.0 ms (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 862 ms) last resort
[26137:0x3b6b020] 38487 ms: Mark-sweep 1403.3 (1440.8) -> 1403.3 (1440.8) MB, 864.1 / 0.0 ms last resort
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0xc3bfff9cef1 <JSObject>
1: declareSymbol(aka declareSymbol) [/home/taoqf/.nvm/versions/node/v8.4.0/lib/node_modules/typescript/lib/tsc.js:~14980] [pc=0xac3a15e8497](this=0xc3bfff82241 <undefined>,symbolTable=0x1eb8b183ecc9 <Map map = 0x4a1246983b9>,parent=0x29b2e1ed5e61 <Symbol map = 0x4a12469a4b9>,node=0x2cc77d79521 <Node map = 0x4a124695a79>,includes=4,excludes=0,isReplaceableByMethod=0xc3bfff82241 <undefined>)
...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [node]
2: 0x13740dc [node]
3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
5: v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [node]
6: v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object**, v8::internal::Isolate*) [node]
7: 0xac3a14040dd
Aborted |
|
@sandersn I'm totally ignorant of tsc internals, but I might have a thought worth chasing. If I understand it correctly, the fix you provided (PR #17984) seems incomplete, in at least a couple of ways:
FWIW!!! I suspect these may be subcases of broader sets of unaddressed cases. In any event, my main reason for writing is to provoke you to think along these lines for missed cases. If you're not one yourself, you might want to consult a Typescript "language lawyer". |
|
@swlee60 Out of curiosity, does your crash disappear if you disable sourcemaps ( |
|
@weswigham Do you mean webpack |
|
Interesting. Looks like @taoqf's project runs out of memory in the binder. Not much has changed in the binder recently, though. |
|
@taoqf, what is the version of Typescript that you used before 2.4.1? Your project fails on 2.3 for me as well. |
|
On master (I haven't checked with 2.3 or 2.4), the AST alone brings memory usage to 1.1 GB, which means that every byte of source results in almost 40 bytes in the parse tree. That leaves the compiler about 300 MB to bind and check with. It is unable to do so. We need to think about architectural ways to reduce parse tree overhead to make sure that Typescript can scale to very large projects. Specifically, how much overhead we incur when parsing common patterns like object literals. As a workaround, @taoqf, can you split your compilation into multiple compilations? Then you could compile it in multiple pieces. You can use .d.ts files, either generated or manually created, as interfaces between the split compilations. |
I always used next version. |
|
|
@tiger-seo can you open a new issue? There are multiple reasons for the compiler to run out of memory and your reason is likely different from @taoqf. |
|
@taoqf if compilation worked previously, what else changed? Did your project add a lot more files? Did the individual files get a lot bigger? The project you gave me fails on 2.3, 2.4 and 2.5 and I expect it would fail on earlier versions because the AST memory footprint hasn't grown much recently. |
|
It never worked , I compiled in multiple pieces in my project now. |
|
@sandersn , I tried to use gulp-typescript and --max-old-space-size=4096 to run script, it will be ok, it would be very nice if tsc and gulp-typescript could accept this option. |
|
I recently ran into the same issue using ts-loader and webpack after upgrading Typescript from 2.3.4 to 2.5.2. I downgraded to 2.4.2 and 2.4.1 and the issue persisted. With 2.4.0, however, everything works as expected. |
|
@troeggla How big is your project? If the project is less than 10 MB of source, then it's probably a different cause. Can you
|
|
Randomly getting this on my project (using nightly, tsc in watch mode), not sure how to reproduce, it just happens sometimes. 7GB/8GB of RAM ATM, but plenty of swap and pagefile (9GBish) project is not too big, 36 typescript files, but only 120kb in total |
|
@sandersn Thanks for looking into this! Yeah, my project is well below 10 MB (excluding packages though) and the resulting bundle comes to about 1.9MB (without minification). Building with If you want, I can give you access to a private repository. |
|
@troeggla If the nightly typescript is working for you then I think the recent bug fixes apply and we don't need your source. @pocesar Can you open a separate issue? For only 120k of source you do not have the same issue as @taoqf. Total RAM doesn't make a difference unless you have configured node to use more than 1.4 GB. You can try raising it to see if the problem goes away, and try lowering it to see if you can make it reproduce reliably. I think the flag is You can also try compiling with |
|
@sandersn My team also has large project, and we changed and fixed TypeScript version to 2.4.0 because upper TS 2.4.1 caused this We want to use awesome latest TypeScript's features, like 2.5, 2.6, ... or 3.0 and later! But we don't know when is this problem solved, so we can't upgrade TypeScript anymore. If you can, please let me know the progress of this issue. |
|
@cheerun Try If you have 10s of megabytes of source then you may still have a problem; that's what this issue tracks. |
sandersn
changed the title
|
The OP seems to be caused by a large object structure, @sandersn's comment in #17112 (comment) summarizes the issues. |
and others
TypeScript Version: 2.4.1 / nightly (2.5.0-dev.20170707)
Code
gulpfile.js
tsconfig.json
{ "include": [ "src" ], "compilerOptions": { "module": "commonjs", "target": "esnext", "noImplicitAny": false, "sourceMap": false, "emitDecoratorMetadata": true, "experimentalDecorators": true, "strictNullChecks": false, "noImplicitThis": true, "rootDir": "./src/", "rootDirs": [], "allowJs": false, "allowUnreachableCode": false, "allowUnusedLabels": false, "alwaysStrict": true, "baseUrl": "", "charset": "utf8", "declaration": false, "inlineSourceMap": false, "allowSyntheticDefaultImports": false, "diagnostics": false, "emitBOM": false, "forceConsistentCasingInFileNames": false, "importHelpers": false, "inlineSources": false, "isolatedModules": false, "lib": [ "es6", "es7", "dom" ], "listFiles": true, // default false "listEmittedFiles": true, // default false "locale": "zh_CN", "newLine": "CRLF", "noEmit": false, "moduleResolution": "node", "noEmitHelpers": false, "noEmitOnError": false, "noImplicitReturns": false, "noImplicitUseStrict": false, "maxNodeModuleJsDepth": 0, "noLib": false, "outDir": "./dist/", "noFallthroughCasesInSwitch": false, "noResolve": false, "noUnusedLocals": false, "noUnusedParameters": false, "paths": {}, "preserveConstEnums": false, "pretty": true, "removeComments": false, "skipDefaultLibCheck": true, // default false "skipLibCheck": true, // default false "stripInternal": false, "suppressExcessPropertyErrors": false, "suppressImplicitAnyIndexErrors": true, // default false "traceResolution": true, // default false "typeRoots": [], "types": [], "watch": false } }Expected behavior:
Actual behavior:
Exception i got:
More
tscworks fine.fp-ts,fantasylandandmonetThe text was updated successfully, but these errors were encountered: