bug(schematics): execution in a CWD w/ many directories and files causes a major delay or crash

[Splaktar]

Bug Report or Feature Request

- [x] bug report
- [ ] feature request

Area

- [ ] devkit
- [x] schematics

Versions

npm: 5.5.1
node: v8.8.1
@angular-devkit/core@0.0.20
@angular-devkit/schematics@0.0.34

Repro steps

1. cd into a directory that contains a lot of Angular CLI projects (i.e. your git/ directory where all of your projects are and where you want to create a new application)
2. run schematics application --name tmp --directory tmp

The log given by the failure

Most of the time there is no error, but the generation of a new app may take 1-3 minutes!

In extreme cases where you have a large number of projects, the heap will fill up resulting in this:

==== JS stack trace =========================================

Security context: 0x2ef8d4cfb51 <JS Object>
    1: new constructor [fs.js:150] [pc=0x344fa7d1aef4] (this=0x287332d90369 <JS Object>,dev=16777220,mode=33152,nlink=1,uid=1759047591,gid=1293641148,rdev=0,blksize=4096,ino=2858228,size=253,blocks=8,atim_msec=0x286887bfffe9 <Number: 1.50854e+12>,mtim_msec=0x286887bfffb9 <Number: 1.47277e+12>,ctim_msec=0x286887bffef9 <Number: 1.5082e+12>,birthtim_msec=0x286887bffe21 <Number: 1.47277e+12>)
    6:...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/usr/local/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/usr/local/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node]
 4: v8::internal::factory::NewByteArray(int, v8::internal::PretenureFlag) [/usr/local/bin/node]
 5: v8::internal::factory::NewCode(v8::internal::CodeDesc const&, unsigned int, v8::internal::Handle<v8::internal::Object>, bool, bool, int, bool) [/usr/local/bin/node]
 6: v8::internal::CodeGenerator::MakeCodeEpilogue(v8::internal::MacroAssembler*, v8::internal::CompilationInfo*) [/usr/local/bin/node]
 7: v8::internal::LChunk::Codegen() [/usr/local/bin/node]
 8: v8::internal::OptimizedCompileJob::GenerateCode() [/usr/local/bin/node]
 9: v8::internal::Compiler::FinalizeOptimizedCompileJob(v8::internal::OptimizedCompileJob*) [/usr/local/bin/node]
10: v8::internal::OptimizingCompileDispatcher::InstallOptimizedFunctions() [/usr/local/bin/node]
11: v8::internal::StackGuard::HandleInterrupts() [/usr/local/bin/node]
12: v8::internal::Runtime_StackGuard(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
13: 0x344fa77092a7

Desired functionality

It is understandable that the execution of most schematics would scan all of the files and directories in the CWD (current working directory) as it may assume that it is in an app and needs to know the structure.

But when creating a new app, there is no need to do this scanning in the CWD. Perhaps in the directory specified by --directory, but it is likely empty in most cases unless --force is being used.

Mention any other details that might be useful

Running the same command in an empty directory usually completes in just a few seconds (compared to 1m+ in a dir w/ Angular projects in it).

[Splaktar]

This is caused by https://github.com/angular/devkit/blob/master/packages/angular_devkit/schematics/bin/schematics.ts#L171-L172. There doesn't seem to be any way for schematics to tell at this point what kind of schematic will be executed.

Should a --root arg be added here that can override process.cwd() if supplied?

[Splaktar]

@hansl @Brocco I can make a PR for this, but I would like some feedback on the design and whether this would be an acceptable solution or not. I'd also be happy to look into an alternative solution. Thank you.

[hansl]

Please note that we don’t scan (or shouldn’t) the directories now unless the schematics requires it (e.g. list files). I think this might be coming from the sink where we apply those changes?

Basically the issue is that schematics should not care about creating an empty host for some, but the code outside the schematics itself should not force a scan. And the schematics itself should be able to listcertains dirswithout scanning the whole structure.

I need to take a quick look. But if you want I can tell you where the issue is and you can fix it.

[Splaktar]

I implemented a change locally to remove process.cwd() from the following lines https://github.com/angular/devkit/blob/master/packages/angular_devkit/schematics/bin/schematics.ts#L171-L172. I replaced it with the --directory argument used for the application schematic and the issue was resolved. The execution time for my local test case dropped from 1.2 minutes to 5 seconds. Of course that doesn't work for the rest of the schematics which depend on being executed from within a project.

Since not all schematics have a --directory option, I proposed something like --root, but I'm open to options and would love an option which didn't require an additional argument.

[Splaktar]

There is another related issue with the --directory argument of the application schematic. If you do something like schematics application --name tmp --directory ../tmp it will fail with Path "/../tmp" is invalid.. This is because you are asking it to generate files above the CWD in the file tree, which is not hooked up to the sink.

[clydin]

@Splaktar are you still experiencing the performance issues with the latest versions?