ngcc crashing on every CI build #36278
Comments
|
We have seen this on our own CI builds and @gkalpak is currently investigating. We believe it might be that ngcc is creating a lot of workers to do the parallel processing, and Docker is complaining that it is using too much memory and arbitrarily killing processes. This might be due to ngcc reading that the current machine has more CPUs than the Docker instance is actually allocating. |
petebacondarwin
changed the title
|
@petebacondarwin @gkalpak - when I set |
gkalpak
added a commit
to gkalpak/angular
that referenced
this issue
Mar 27, 2020
When running in parallel mode, ngcc spawns multiple worker processed to process the various entry-points. The number of max allowed processes is determined by the number of CPU cores available to the OS. There is also currently an [upper limit of 8][1]. The number of active workers is in turn inferred by the number of [task assignments][2]. In the past, counting the entries of `ClusterMaster#taskAssignments` was enough, because worker processes were spawned eagerly at the beginning and corresponding entries were created in `taskAssignments`. Since angular#35719 however, worker processes are spawned lazily on an as needed basis. Because there is some delay between [spawning a process][3] and [inserting it][4] into `taskAssignments`, there is a short period of time when `taskAssignment.size` does not actually represent the number of spawned processes. This can result in spawning more than `ClusterMaster#workerCount` processes. An example of this can be seen in angular#36278, where the debug logs indicate 9 worker processes had been spawned (`All 9 workers are currently busy`) despite the hard limit of 8. This commit fixes this by using `cluster.workers` to compute the number of spawned worker processes. `cluster.workers` is updated synchronously with `cluster.fork()` and thus reflects the number of spawned workers accurately at all times. [1]: https://github.com/angular/angular/blob/b8e9a30d3b6/packages/compiler-cli/ngcc/src/main.ts#L429 [2]: https://github.com/angular/angular/blob/b8e9a30d3b6/packages/compiler-cli/ngcc/src/execution/cluster/master.ts#L108 [3]: https://github.com/angular/angular/blob/b8e9a30d3b6/packages/compiler-cli/ngcc/src/execution/cluster/master.ts#L110 [4]: https://github.com/angular/angular/blob/b8e9a30d3b6/packages/compiler-cli/ngcc/src/execution/cluster/master.ts#L199
|
Yes that is the current workaround. |
alxhub
pushed a commit
that referenced
this issue
Mar 27, 2020
…#36280) When running in parallel mode, ngcc spawns multiple worker processed to process the various entry-points. The number of max allowed processes is determined by the number of CPU cores available to the OS. There is also currently an [upper limit of 8][1]. The number of active workers is in turn inferred by the number of [task assignments][2]. In the past, counting the entries of `ClusterMaster#taskAssignments` was enough, because worker processes were spawned eagerly at the beginning and corresponding entries were created in `taskAssignments`. Since #35719 however, worker processes are spawned lazily on an as needed basis. Because there is some delay between [spawning a process][3] and [inserting it][4] into `taskAssignments`, there is a short period of time when `taskAssignment.size` does not actually represent the number of spawned processes. This can result in spawning more than `ClusterMaster#workerCount` processes. An example of this can be seen in #36278, where the debug logs indicate 9 worker processes had been spawned (`All 9 workers are currently busy`) despite the hard limit of 8. This commit fixes this by using `cluster.workers` to compute the number of spawned worker processes. `cluster.workers` is updated synchronously with `cluster.fork()` and thus reflects the number of spawned workers accurately at all times. [1]: https://github.com/angular/angular/blob/b8e9a30d3b6/packages/compiler-cli/ngcc/src/main.ts#L429 [2]: https://github.com/angular/angular/blob/b8e9a30d3b6/packages/compiler-cli/ngcc/src/execution/cluster/master.ts#L108 [3]: https://github.com/angular/angular/blob/b8e9a30d3b6/packages/compiler-cli/ngcc/src/execution/cluster/master.ts#L110 [4]: https://github.com/angular/angular/blob/b8e9a30d3b6/packages/compiler-cli/ngcc/src/execution/cluster/master.ts#L199 PR Close #36280
alxhub
pushed a commit
that referenced
this issue
Mar 27, 2020
…#36280) When running in parallel mode, ngcc spawns multiple worker processed to process the various entry-points. The number of max allowed processes is determined by the number of CPU cores available to the OS. There is also currently an [upper limit of 8][1]. The number of active workers is in turn inferred by the number of [task assignments][2]. In the past, counting the entries of `ClusterMaster#taskAssignments` was enough, because worker processes were spawned eagerly at the beginning and corresponding entries were created in `taskAssignments`. Since #35719 however, worker processes are spawned lazily on an as needed basis. Because there is some delay between [spawning a process][3] and [inserting it][4] into `taskAssignments`, there is a short period of time when `taskAssignment.size` does not actually represent the number of spawned processes. This can result in spawning more than `ClusterMaster#workerCount` processes. An example of this can be seen in #36278, where the debug logs indicate 9 worker processes had been spawned (`All 9 workers are currently busy`) despite the hard limit of 8. This commit fixes this by using `cluster.workers` to compute the number of spawned worker processes. `cluster.workers` is updated synchronously with `cluster.fork()` and thus reflects the number of spawned workers accurately at all times. [1]: https://github.com/angular/angular/blob/b8e9a30d3b6/packages/compiler-cli/ngcc/src/main.ts#L429 [2]: https://github.com/angular/angular/blob/b8e9a30d3b6/packages/compiler-cli/ngcc/src/execution/cluster/master.ts#L108 [3]: https://github.com/angular/angular/blob/b8e9a30d3b6/packages/compiler-cli/ngcc/src/execution/cluster/master.ts#L110 [4]: https://github.com/angular/angular/blob/b8e9a30d3b6/packages/compiler-cli/ngcc/src/execution/cluster/master.ts#L199 PR Close #36280
|
Hi, @paustint , I'm having (i think) the same problem in CircleCI. I'm running this command And I'm getting Where should I add the Thanks in advance! |
|
Unfortunately you cannot tell CLI to run ngcc sync. But as a workaround you could run |
|
@poloagustin - you will need to manually invoke Example: You could also do something like this and use The key factor is that you will want to pre-run ngcc so that it does not need to run again when you run |
|
I've added "ngcc" script @paustint commented, then ngcc crashing on circleci doesn't happen for me. |
GregorBiswanger
added a commit
to GregorBiswanger/rxjs-fruits
that referenced
this issue
Apr 13, 2020
GregorBiswanger
added a commit
to GregorBiswanger/rxjs-fruits
that referenced
this issue
Apr 13, 2020
|
FYI this is still happening for me in 9.1.1. |
gkalpak
added a commit
to gkalpak/angular
that referenced
this issue
Apr 15, 2020
TODO Jira issue: [FW-2008](https://angular-team.atlassian.net/browse/FW-2008) Fixes angular#36278
gkalpak
added a commit
to gkalpak/angular
that referenced
this issue
Apr 15, 2020
TODO Jira issue: [FW-2008](https://angular-team.atlassian.net/browse/FW-2008) Fixes angular#36278
gkalpak
added a commit
to gkalpak/angular
that referenced
this issue
Apr 15, 2020
TODO Jira issue: [FW-2008](https://angular-team.atlassian.net/browse/FW-2008) Fixes angular#36278
gkalpak
added a commit
to gkalpak/angular
that referenced
this issue
Apr 15, 2020
TODO Jira issue: [FW-2008](https://angular-team.atlassian.net/browse/FW-2008) Fixes angular#36278
|
And same here with 9.1.2 version. |
gkalpak
added a commit
to gkalpak/angular
that referenced
this issue
Apr 20, 2020
TODO Examples of ngcc being able to recover after a worker process crashed: - While idling: https://circleci.com/gh/angular/angular/682197 - While compiling: https://circleci.com/gh/angular/angular/682209 - While writing files: https://circleci.com/gh/angular/angular/682267 Jira issue: [FW-2008](https://angular-team.atlassian.net/browse/FW-2008) Fixes angular#36278
I would add something to this. If you're using CircleCI or any other CI utility that allows caching dependencies, you might want to skip the
This way, if binaries were restored from the cache, you'll save some precious time here. In our case on CircleCI that saved us around 5 mins... |
|
Thanks for the comments! in package.json scripts: Thanks for the help! PS: I'll try changing the ngcc part to |
|
@poloagustin Looks like you use an older version of Node in your circle ci build. You shouldn't have to set the |
gkalpak
added a commit
to gkalpak/angular
that referenced
this issue
Apr 22, 2020
Previously, when running in parallel mode and a worker process crashed while processing a task, it was not possible for ngcc to continue without risking ending up with a corrupted entry-point and therefore it exited with an error. This, for example, could happen when a worker process received a `SIGKILL` signal, which was frequently observed in CI environments. This was probably the result of Docker killing processes due to increased memory pressure. One factor that amplifies the problem under Docker (which is often used in CI) is that it is not possible to distinguish between the available CPU cores on the host machine and the ones made available to Docker containers, thus resulting in ngcc spawning too many worker processes. This commit addresses these issues in the following ways: 1. We take advantage of the fact that files are written to disk only after an entry-point has been fully analyzed/compiled. The master process can now determine whether a worker process has not yet started writing files to disk (even if it was in the middle of processing a task) and just put the task back into the tasks queue if the worker process crashes. 2. The master process keeps track of the transformed files that a worker process will attempt to write to disk. If the worker process crashes while writing files, the master process can revert any changes and put the task back into the tasks queue (without risking corruption). 3. When a worker process crashes while processing a task (which can be a result of increased memory pressure or too many worker processes), the master process will not try to re-spawn it. This way the number or worker processes is gradually adjusted to a level that can be accomodated by the system's resources. Examples of ngcc being able to recover after a worker process crashed: - While idling: https://circleci.com/gh/angular/angular/682197 - While compiling: https://circleci.com/gh/angular/angular/682209 - While writing files: https://circleci.com/gh/angular/angular/682267 Jira issue: [FW-2008](https://angular-team.atlassian.net/browse/FW-2008) Fixes angular#36278
gkalpak
added a commit
to gkalpak/angular
that referenced
this issue
Apr 28, 2020
Previously, when running in parallel mode and a worker process crashed while processing a task, it was not possible for ngcc to continue without risking ending up with a corrupted entry-point and therefore it exited with an error. This, for example, could happen when a worker process received a `SIGKILL` signal, which was frequently observed in CI environments. This was probably the result of Docker killing processes due to increased memory pressure. One factor that amplifies the problem under Docker (which is often used in CI) is that it is not possible to distinguish between the available CPU cores on the host machine and the ones made available to Docker containers, thus resulting in ngcc spawning too many worker processes. This commit addresses these issues in the following ways: 1. We take advantage of the fact that files are written to disk only after an entry-point has been fully analyzed/compiled. The master process can now determine whether a worker process has not yet started writing files to disk (even if it was in the middle of processing a task) and just put the task back into the tasks queue if the worker process crashes. 2. The master process keeps track of the transformed files that a worker process will attempt to write to disk. If the worker process crashes while writing files, the master process can revert any changes and put the task back into the tasks queue (without risking corruption). 3. When a worker process crashes while processing a task (which can be a result of increased memory pressure or too many worker processes), the master process will not try to re-spawn it. This way the number or worker processes is gradually adjusted to a level that can be accomodated by the system's resources. Examples of ngcc being able to recover after a worker process crashed: - While idling: https://circleci.com/gh/angular/angular/682197 - While compiling: https://circleci.com/gh/angular/angular/682209 - While writing files: https://circleci.com/gh/angular/angular/682267 Jira issue: [FW-2008](https://angular-team.atlassian.net/browse/FW-2008) Fixes angular#36278
gkalpak
added a commit
to gkalpak/angular
that referenced
this issue
Apr 29, 2020
Previously, when running in parallel mode and a worker process crashed while processing a task, it was not possible for ngcc to continue without risking ending up with a corrupted entry-point and therefore it exited with an error. This, for example, could happen when a worker process received a `SIGKILL` signal, which was frequently observed in CI environments. This was probably the result of Docker killing processes due to increased memory pressure. One factor that amplifies the problem under Docker (which is often used in CI) is that it is not possible to distinguish between the available CPU cores on the host machine and the ones made available to Docker containers, thus resulting in ngcc spawning too many worker processes. This commit addresses these issues in the following ways: 1. We take advantage of the fact that files are written to disk only after an entry-point has been fully analyzed/compiled. The master process can now determine whether a worker process has not yet started writing files to disk (even if it was in the middle of processing a task) and just put the task back into the tasks queue if the worker process crashes. 2. The master process keeps track of the transformed files that a worker process will attempt to write to disk. If the worker process crashes while writing files, the master process can revert any changes and put the task back into the tasks queue (without risking corruption). 3. When a worker process crashes while processing a task (which can be a result of increased memory pressure or too many worker processes), the master process will not try to re-spawn it. This way the number or worker processes is gradually adjusted to a level that can be accomodated by the system's resources. Examples of ngcc being able to recover after a worker process crashed: - While idling: https://circleci.com/gh/angular/angular/682197 - While compiling: https://circleci.com/gh/angular/angular/682209 - While writing files: https://circleci.com/gh/angular/angular/682267 Jira issue: [FW-2008](https://angular-team.atlassian.net/browse/FW-2008) Fixes angular#36278
AndrewKushnir
pushed a commit
that referenced
this issue
Apr 29, 2020
Previously, when running in parallel mode and a worker process crashed while processing a task, it was not possible for ngcc to continue without risking ending up with a corrupted entry-point and therefore it exited with an error. This, for example, could happen when a worker process received a `SIGKILL` signal, which was frequently observed in CI environments. This was probably the result of Docker killing processes due to increased memory pressure. One factor that amplifies the problem under Docker (which is often used in CI) is that it is not possible to distinguish between the available CPU cores on the host machine and the ones made available to Docker containers, thus resulting in ngcc spawning too many worker processes. This commit addresses these issues in the following ways: 1. We take advantage of the fact that files are written to disk only after an entry-point has been fully analyzed/compiled. The master process can now determine whether a worker process has not yet started writing files to disk (even if it was in the middle of processing a task) and just put the task back into the tasks queue if the worker process crashes. 2. The master process keeps track of the transformed files that a worker process will attempt to write to disk. If the worker process crashes while writing files, the master process can revert any changes and put the task back into the tasks queue (without risking corruption). 3. When a worker process crashes while processing a task (which can be a result of increased memory pressure or too many worker processes), the master process will not try to re-spawn it. This way the number or worker processes is gradually adjusted to a level that can be accomodated by the system's resources. Examples of ngcc being able to recover after a worker process crashed: - While idling: https://circleci.com/gh/angular/angular/682197 - While compiling: https://circleci.com/gh/angular/angular/682209 - While writing files: https://circleci.com/gh/angular/angular/682267 Jira issue: [FW-2008](https://angular-team.atlassian.net/browse/FW-2008) Fixes #36278 PR Close #36626
|
I see that this was closed some hours ago, but after taking 9.1.4 we still get the error on our CircleCI build pipeline (unless we use the ngcc async=false workaround). Maybe the fix is not part of 9.1.4? |
|
@mikeandtherest Can you provide details of your errors? Perhaps it would be cleanest to create a new issue with the detail and link to this one? For the record the changes made in the linked PR should make ngcc resilient to worker processes being killed. It may still show processes crashing but the processing should proceed successfully. |
|
I'd rather continue the discussion here, since I bet there are a lot of us already watching this thread. Regarding the actual error, this is what we get in the CircleCI pipeline: If I put back the workaround with |
|
Oh sorry! I just noticed that this PR was merged after 9.1.4 was released. See https://github.com/angular/angular/commits/9.1.x |
|
Hmm, weird. Because it says that all these were merged into master 16h ago, while the actual release happened 10h ago. But by looking at that link you posted above, it seems that indeed they came after the release. |
|
There may have been a delay between merging to master and merging to patch. |
|
9.1.5 then? |
This was referenced May 18, 2020
|
This issue has been automatically locked due to inactivity. Read more about our automatic conversation locking policy. This action has been performed automatically by a bot. |
profanis
pushed a commit
to profanis/angular
that referenced
this issue
Sep 5, 2020
…36626) Previously, when running in parallel mode and a worker process crashed while processing a task, it was not possible for ngcc to continue without risking ending up with a corrupted entry-point and therefore it exited with an error. This, for example, could happen when a worker process received a `SIGKILL` signal, which was frequently observed in CI environments. This was probably the result of Docker killing processes due to increased memory pressure. One factor that amplifies the problem under Docker (which is often used in CI) is that it is not possible to distinguish between the available CPU cores on the host machine and the ones made available to Docker containers, thus resulting in ngcc spawning too many worker processes. This commit addresses these issues in the following ways: 1. We take advantage of the fact that files are written to disk only after an entry-point has been fully analyzed/compiled. The master process can now determine whether a worker process has not yet started writing files to disk (even if it was in the middle of processing a task) and just put the task back into the tasks queue if the worker process crashes. 2. The master process keeps track of the transformed files that a worker process will attempt to write to disk. If the worker process crashes while writing files, the master process can revert any changes and put the task back into the tasks queue (without risking corruption). 3. When a worker process crashes while processing a task (which can be a result of increased memory pressure or too many worker processes), the master process will not try to re-spawn it. This way the number or worker processes is gradually adjusted to a level that can be accomodated by the system's resources. Examples of ngcc being able to recover after a worker process crashed: - While idling: https://circleci.com/gh/angular/angular/682197 - While compiling: https://circleci.com/gh/angular/angular/682209 - While writing files: https://circleci.com/gh/angular/angular/682267 Jira issue: [FW-2008](https://angular-team.atlassian.net/browse/FW-2008) Fixes angular#36278 PR Close angular#36626
torsten-simon
pushed a commit
to edu-sharing/edu-sharing-community-repository
that referenced
this issue
Dec 15, 2022
torsten-simon
pushed a commit
to edu-sharing/edu-sharing-community-repository
that referenced
this issue
Dec 15, 2022
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
🐞 bug report
Affected Package
The issue is caused by package @angular/compiler-cli - specifically `ngcc`Is this a regression?
Yes, the previous version in which this bug was not present was: 9.0 (or maybe 8.2)Description
when ngcc runs on circle-ci it fails every time, but if I SSH into the container and run ngcc it completes successfully.🔬 Minimal Reproduction
https://github.com/paustint/test-ivy-app
Example failing builds:
https://app.circleci.com/pipelines/github/paustint/test-ivy-app/1/workflows/e3719bbe-f02c-40c0-8f6f-40c7e61f09ef/jobs/1
https://app.circleci.com/pipelines/github/paustint/test-ivy-app/1/workflows/e3719bbe-f02c-40c0-8f6f-40c7e61f09ef/jobs/2
Note This fails 100% of the time, but fails while compiling seemingly different packages (e.x. it is not consistently failing while compiling any specific package.
🔥 Exception or Error
🌍 Your Environment
Angular Version:
Anything else relevant?
The text was updated successfully, but these errors were encountered: