perf(ngcc): add support for parallel execution of tasks #32427

This commit addresses the review feedback from angular#32052, which was merged before addressing the feedback there.

…the overwritten one In order to prevent `ngcc`'d packages (e.g. libraries) from getting accidentally published, `ngcc` overwrites the `prepublishOnly` npm script to log a warning and exit with an error. In case we want to restore the original script (e.g. "undo" `ngcc` processing), we keep a backup of the original `prepublishOnly` script. Previously, running `ngcc` a second time (e.g. for a different format) would create a backup of the overwritten `prepublishOnly` script (if there was originally no `prepublishOnly` script). As a result, if we ever tried to "undo" `ngcc` processing and restore the original `prepublishOnly` script, the error-throwing script would be restored instead. This commit fixes it by ensuring that we only back up a `prepublishOnly` script, iff it is not the one we created ourselves (i.e. the error-throwing one).

@alxhub

…rface To persist some of its state, `ngcc` needs to update `package.json` files (both in memory and on disk). This refactoring abstracts these operations behind the `PackageJsonUpdater` interface, making it easier to orchestrate them from different contexts (e.g. when running tasks in parallel on multiple processes). Inspired by/Based on @alxhub's prototype: alxhub/angular@cb631bdb1

@alxhub

…face This change does not alter the current behavior, but makes it easier to introduce new types of `Executors` , for example to do the required work in parallel (on multiple processes). Inspired by/Based on @alxhub's prototype: alxhub/angular@cb631bdb1

Previously, `ngcc`'s programmatic API would run and complete synchronously. This was necessary for specific usecases (such as how the `@angular/cli` invokes `ngcc` as part of the TypeScript module resolution process), but not for others (e.g. running `ivy-ngcc` as a `postinstall` script). This commit adds a new option (`async`) that enables turning on asynchronous execution. I.e. it signals that the caller is OK with the function call to complete asynchronously, which allows `ngcc` to potentially run in a more efficient mode. Currently, there is no difference in the way tasks are executed in sync vs async mode, but this change sets the ground for adding new execution options (that require asynchronous operation), such as processing tasks in parallel on multiple processes. NOTE: When using the programmatic API, the default value for `async` is `false`, thus retaining backwards compatibility. When running `ngcc` from the command line (i.e. via the `ivy-ngcc` script), it runs in async mode (to be able to take advantage of future optimizations), but that is transparent to the caller.

…ty processability In the past, a task's processability didn't use to be known in advance. It was possible that a task would be created and added to the queue during the analysis phase and then later (during the compilation phase) it would be found out that the task (i.e. the associated format property) was not processable. As a result, certain checks had to be delayed, until a task's processing had started or even until all tasks had been processed. Examples of checks that had to be delayed are: - Whether a task can be skipped due to `compileAllFormats: false`. - Whether there were entry-points for which no format at all was successfully processed. It turns out that (as made clear by the refactoring in 9537b2f), once a task starts being processed it is expected to either complete successfully (with the associated format being processed) or throw an error (in which case the process will exit). In other words, a task's processability is known in advance. This commit takes advantage of this fact by moving certain checks earlier in the process (e.g. in the analysis phase instead of the compilation phase), which in turn allows avoiding some unnecessary work. More specifically: - When `compileAllFormats` is `false`, tasks are created _only_ for the first suitable format property for each entry-point, since the rest of the tasks would have been skipped during the compilation phase anyway. This has the following advantages: 1. It avoids the slight overhead of generating extraneous tasks and then starting to process them (before realizing they should be skipped). 2. In a potential future parallel execution mode, unnecessary tasks might start being processed at the same time as the first (useful) task, even if their output would be later discarded, wasting resources. Alternatively, extra logic would have to be added to prevent this from happening. The change in this commit avoids these issues. - When an entry-point is not processable, an error will be thrown upfront without having to wait for other tasks to be processed before failing.

… types Previously, `ngcc` needed to store some metadata related to the processing of each entry-point. This metadata was stored in a `Map`, in the form of `EntryPointProcessingMetadata` and passed around as needed. After some recent refactorings, it turns out that this metadata (with its only remaining property, `hasProcessedTypings`) was no longer used, because the relevant information was extracted from other sources (such as the `processDts` property on `Task`s). This commit cleans up the code by removing the unused code and types.

@alxhub

This change does not alter the current behavior, but makes it easier to introduce `TaskQueue`s implementing different task selection algorithms, for example to support executing multiple tasks in parallel (while respecting interdependencies between them). Inspired by/Based on @alxhub's prototype: alxhub/angular@cb631bdb1

This commit adds a new `TaskQueue` implementation that supports executing multiple tasks in parallel (while respecting interdependencies between them). This new implementation is currently not used, thus the behavior of `ngcc` is not affected by this change. The parallel `TaskQueue` will be used in a subsequent commit that will introduce parallel task execution.

@alxhub

`ngcc` supports both synchronous and asynchronous execution. The default mode when using `ngcc` programmatically (which is how `@angular/cli` is using it) is synchronous. When running `ngcc` from the command line (i.e. via the `ivy-ngcc` script), it runs in async mode. Previously, the work would be executed in the same way in both modes. This commit improves the performance of `ngcc` in async mode by processing tasks in parallel on multiple processes. It uses the Node.js built-in [`cluster` module](https://nodejs.org/api/cluster.html) to launch a cluster of Node.js processes and take advantage of multi-core systems. Preliminary comparisons indicate a 1.8x to 2.6x speed improvement when processing the angular.io app (apparently depending on the OS, number of available cores, system load, etc.). Further investigation is needed to better understand these numbers and identify potential areas of improvement. Inspired by/Based on @alxhub's prototype: alxhub/angular@cb631bdb1 Original design doc: https://hackmd.io/uYG9CJrFQZ-6FtKqpnYJAA?view Jira issue: [FW-1460](https://angular-team.atlassian.net/browse/FW-1460)

…tions This gives an overview of how much time is spent in each operation/phase and makes it easy to do rough comparisons of how different configurations or changes affect performance.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf(ngcc): add support for parallel execution of tasks #32427

perf(ngcc): add support for parallel execution of tasks #32427

Commits on Sep 7, 2019