Build doesn't generate artifacts with @angular-devkit/build-angular@12.0.1

[bjornharvold]

Description

Artifact A was built successfully and then removed from dist. If you build again, the nx cache will not do anything as it registers that there have been no changes and says "build successful" even though dist is empty.

Motivation

A CI workflow would maybe like to remove built artifacts but still keep the cache around to lower build times. If an artifact was missing but the cache registered it as built from last build, the current build will fail.

Suggested Implementation

Quick check on the existence of the artifact under dist after cache gave the artifact thumbs up. Build it again if it doesn't exist.

[bjornharvold]

I want to follow up on this ticket as it should maybe be marked as a bug.

I am building multiples SPAs with this command in my CI build:
"build:staging:fast": "node --max_old_space_size=16384 ./node_modules/.bin/nx run-many --target=build --projects=administration,booking-engine,corporate,developer,extranet,hotel-booking-engine,maintenance,seller,single-sign-on,travel-agent,web-components --configuration=staging --with-deps",

Until a few days ago, this ran flawlessly. Now, it builds only ~5 of the SPAs.

Here's what the log has to say:

build	21-May-2021 12:33:25	> nx run extranet:build:staging 
error	21-May-2021 12:33:28	- Generating browser application bundles...
build	21-May-2021 12:37:06	
build	21-May-2021 12:37:06	> nx run corporate:build:staging 
error	21-May-2021 12:37:09	- Generating browser application bundles...
build	21-May-2021 12:37:18	
build	21-May-2021 12:37:18	> nx run developer:build:staging 
error	21-May-2021 12:37:21	- Generating browser application bundles...
build	21-May-2021 12:39:05	
build	21-May-2021 12:39:05	> nx run travel-agent:build:staging 
error	21-May-2021 12:39:08	- Generating browser application bundles...
build	21-May-2021 12:39:18	
build	21-May-2021 12:39:18	> nx run single-sign-on:build:staging 
error	21-May-2021 12:39:20	- Generating browser application bundles...

If I go directly in to the build and execute the same command to see if I can build the SPA myself, I get:

> nx run extranet:build:staging [existing outputs match the cache, left as is]

———————————————————————————————————————————————

>  NX   SUCCESS  Running target "build" succeeded

But the SPA does not exist in ./dist/apps and nothing gets built. The build just skips several of my SPAs.

Project info:

Angular CLI: 12.0.1
Node: 14.15.3
Package Manager: yarn 1.22.5
OS: linux x64

Angular: 12.0.1
... animations, cdk, cli, common, compiler, compiler-cli, core
... elements, forms, language-service, localize, material
... platform-browser, platform-browser-dynamic, router
... service-worker, youtube-player

Package                         Version
---------------------------------------------------------
@angular-devkit/architect       0.1200.1
@angular-devkit/build-angular   12.0.1
@angular-devkit/core            12.0.1
@angular-devkit/schematics      12.0.0
@angular/flex-layout            12.0.0-beta.34
@nrwl/angular:                     12.3.4
@schematics/angular             10.1.7
ng-packagr                      12.0.0
rxjs                            6.6.7
typescript                      4.2.4

Log:
plan-116555777-JOB1-90.zip

[smorandi]

yes, i am having exactly the same issue. the builds are not done at all.
only on linux though. works fine on windows 10

[gmisiolek]

I have the same issue. It works fine with version 12.1.0. Stopped working after upgrade to 12.3.4.

[alx-andru]

Having the same issue and trying to narrow it down. Has anyone found some hints on it?

[jredl-va]

+1 to this issue. Works fine on mac but nothing builds on linux.

[bboehm86]

We've got a similar issue. Build works perfectly on Windows & WSL(2) but produces no output in our GitLab CI with no errors (even without any caches)

[gmisiolek]

That's really weird...

If I run npm ci or npm install and run build command within Dockerfile, build fails - there is no dist folder.

But if I remove from Dockerfile line with build (only npm ci), run container and within this container I manually run build command, dist folder if created with app...

[gmisiolek]

After some research I found that my project stopped building after update angular to 12.0.1. On angular 12.0.0 it seems to work. Could anyone else confirm that?

[bjornharvold]

@gmisiolek I'm working my way downward. Am on 12.3.0 now. Every clean build takes 3 hours so it's a bit slow to say the least.

Wondering if I go down to 12.0.0, that I'll get other errors now that Angular 12 is final. Looking at 12.0.1, there were no big changes that would affect the cache.

[bjornharvold]

If I remember correctly, my CI build has been running flawlessly until a week ago so I do think this was a recent regression error.

[gmisiolek]

@bjornharvold but I managed to make it work with nx 12.3.4 and @angular/* 12.0.0
When I upgraded @angular/* to 12.0.1 it started to fail. I have no idea why.

[jredl-va]

I can confirm that angular 12.0.1 is indeed the issue. Downgrading our workspace to angular 12.0.0 is building correctly in CI (linux based).

Running locally on my mac both versions of angular build correctly.

  Node : 12.22.1
  OS   : darwin x64
  npm  : 6.14.13

  nx : Not Found
  @nrwl/angular : 12.3.3
  @nrwl/cli : 12.3.3
  @nrwl/cypress : 12.3.3
  @nrwl/devkit : 12.3.3
  @nrwl/eslint-plugin-nx : 12.3.3
  @nrwl/express : Not Found
  @nrwl/jest : 12.3.3
  @nrwl/linter : 12.3.3
  @nrwl/nest : Not Found
  @nrwl/next : Not Found
  @nrwl/node : Not Found
  @nrwl/react : Not Found
  @nrwl/schematics : 8.12.11
  @nrwl/tao : 12.3.3
  @nrwl/web : Not Found
  @nrwl/workspace : 12.3.3
  @nrwl/storybook : Not Found
  @nrwl/gatsby : Not Found
  typescript : 4.2.4

[gmisiolek]

After further testing it looks like upgrade @angular-devkit/build-angular from 12.0.0 to 12.0.1 breaks build process.

I've updated all @angular/* packages to 12.0.1, except this one and it looks like everything works fine.
I also use nx 12.3.4.

[alx-andru]

I can confirm @gmisiolek observation. 12.0.0 runs with ubuntu-latest on Azure DevOps hosted agents

[gmisiolek]

@jredl-va But which version of @angular/* libs do you have? Especially @angular-devkit/build-angular?
I've checked nx 12.3.3 with @angular-devkit/build-angular@12.0.1 and I have the same issue.

[jredl-va]

@gmisiolek:

@angular-devkit/build-angular": "12.0.0"

Builds fine.

12.0.1 is confirmed as broken. It appears this breakage only occurs on linux based machines.

[FrozenPandaz]

Hey all, the reason it is happening is because we have a new optimization where when the dist already exists, we don't copy files when tasks are cached. However, this means that if there's another step that mutates the dist, then the original result of the cache would be lost. We'll take a look at this issue and see what we can do about it.

@leosvelperez please take a look at this issue.

[gmisiolek]

@FrozenPandaz but in my case dist folder wasn't created at all.

[bjornharvold]

Yes, I can confirm that it is indeed @angular-devkit/build-angular@12.0.1 that is the culprit.

[leosvelperez]

Hey all, as @FrozenPandaz said, we introduced an optimization to prevent copying files when they are supposed to be there already.

As an FYI, the following is the logic to decide whether to retrieve the output from the cache or not:

• If the current computation hash for the task is different from the latest computation hash, we retrieve the output from the cache.
• If the output doesn't exist, we retrieve the output from the cache.
• Otherwise, we skip retrieving the output from the cache.

So, if the output doesn't exist, it should retrieve it from the cache as it seems, from the comments in this thread, to be happening in Windows and MacOS.

I tried to reproduce the issue in a Docker container with Ubuntu but I've failed to reproduce it so far. I'm gonna need some reproduction steps so I can troubleshoot it.

This is what I've set up to try to reproduce it:

• I have all the Angular dependencies on version 12.0.1, including the @angular-devkit/build-angular.
• I have an app depending on a couple of buildable libraries.
• I build the app. It works fine, the dist artifacts are created correctly.
• I delete the dist folder. I've also tried deleting the app or libs folder inside dist.
• I build the app again. It works fine, the dist artifacts are created correctly.

I'm probably doing something wrong in the reproduction steps. Please let me know what I should do differently to reproduce the issue so I can troubleshoot this properly.

Below you can find the environment I used to run this.

  Node : 14.17.0
  OS   : linux x64
  yarn : 1.22.5
  
  nx : Not Found
  @nrwl/angular : 12.3.4
  @nrwl/cli : 12.3.4
  @nrwl/cypress : 12.3.4
  @nrwl/devkit : 12.3.4
  @nrwl/eslint-plugin-nx : 12.3.4
  @nrwl/express : Not Found
  @nrwl/jest : 12.3.4
  @nrwl/linter : 12.3.4
  @nrwl/nest : Not Found
  @nrwl/next : Not Found
  @nrwl/node : Not Found
  @nrwl/react : Not Found
  @nrwl/schematics : Not Found
  @nrwl/tao : 12.3.4
  @nrwl/web : Not Found
  @nrwl/workspace : 12.3.4
  @nrwl/storybook : Not Found
  @nrwl/gatsby : Not Found
  typescript : 4.2.4

[bboehm86]

@leosvelperez thanks for trying to reproduce this :-) you can take a look at this sample projekt (just created with nx + angular + angular material):
https://gitlab.com/benjamin.boehm/angular-nx-build-test

The build does not produce any output: https://gitlab.com/benjamin.boehm/angular-nx-build-test/-/jobs/1288607121

But when you remove any scss parts it suddenly does: https://gitlab.com/benjamin.boehm/angular-nx-build-test/-/jobs/1288614605

I hope this helps

[gmisiolek]

@leosvelperez I've created basic project with reproduction of this issue.
https://github.com/gmisiolek/nx-dist-issue-reproduction

When I was creating this reprodution repo I found that if I use ng build it works fine, but if I use nx build it fails.

Create with ng build (dist created):

docker build --no-cache -t nxtest .
docker run -ti nxtest

Create with nx build (no dist created):

docker build --no-cache -f Dockerfile-nx -t nxtest2 .
docker run -ti nxtest2

[leosvelperez]

Thanks for sharing those!

I was able to see the error happening using Docker, but it's a bit tricky to troubleshoot as it seems to only be happening while building the Docker image. As soon as I access the container and run the build within the container, it runs fine 🤷🏻 .

I'll keep digging and I'll keep you all posted if I make any progress there.

[jackdamiels]

Same problem with mac os, regardless of using --skip-nx-cache option.
Downgrading to "@angular-devkit/build-angular": "12.0.0", from `"@angular-devkit/build-angular": "12.0.1" resolves the issue.

[bboehm86]

@leosvelperez it appears to be caused by the following change in the CLI:
angular/angular-cli@b89fdc3#diff-b14bb6b4458b0b51ba8711c389cf6c73b0837d0cc8e09df93899d2db888e6ad6L117-R118
If you revert this line manually the build runs fine.

[leosvelperez]

For folks finding this issue, it has nothing to do with Nx caching. Nx is correctly trying to build the apps but due to a change in the Angular CLI packages, the SASS compilation is not doing anything, and somehow the build finish with no errors but no artifact is generated either.

This seems to be happening only in CI environments (at least Gitlab) and when building a Docker image which makes it hard to troubleshoot. As pointed out in several comments in this thread, the issue appeared with version 12.0.1 of the @angular-devkit/build-angular package. As a workaround for now, please stay on version 12.0.0.

I'll try to spend more time on this to identify what can be done on Nx side to fix this. Unfortunately, due to the nature of the environments where this is happening, it's very time-consuming to troubleshoot it, but I'll do my best to try to find a solution for it.

I renamed the issue to properly reflect what is this about and prevent confusion with Nx caching.

[literalpie]

FYI we're seeing this in Jenkins. So glad others have found the root cause - I wouldn't have known how to debug this! Using 12.0.0 is working for us.

[minijus]

@leosvelperez I was able to reproduce this behaviour (premature exit with success code with no artifact build) on Mac as well:

CI=true nx run application:build:production

It was not consistent and did not reproduce on clean Angular project, but on "real" project it worked like 90% of the time.

[pmstss]

only on linux though. works fine on windows 10

i'm able to reproduce it on win machine (success build message, no ./dist folder).
strict 12.0.1 version for @angular-devkit/build-angular workaround works fine, and it seems i found another one: enabling progress in angular.json makes it working for me (enabled it just to see what is going on)

[leosvelperez]

Hey all!

I've now submitted a PR with a workaround to solve this. The issue boils down to how the Angular CLI is managing the workers that were added as part of the new SASS compilation. The workers are unref()-ed as soon as they are created and in certain environments, where nothing else is keeping the process alive, this results in the process exiting early and no compilation is done. Running with the Angular CLI works because they have an interval running for analytics while the compilation is happening and that prevents the process from exiting.

I'll create an issue on the Angular CLI to try and get this sorted there, so the workaround is no longer needed, but in the meantime, this should solve the issue. We'll cut a patch release soon with the fix.

[leosvelperez]

We released version 12.3.5 with the fix. If you still have issues with that version, please let me know.

[jredl-va]

@leosvelperez @FrozenPandaz

I've moved our workspace to nx 12.3.5 and have moved forward @angular-devkit/build-angular to 12.0.1. Unfortunately there are still issues:

Applications in our workspace that take 10+ minutes to compile are still bailing out as described above when using build angular 12.0.1 with nx.

This fix also may have introduced a new bug: #5829

[leosvelperez]

@jredl-va that seems to be different than the previous issue. Previously, the build was exiting with an exit status code of 0, because the process was exiting prematurely with no failures. In the screenshot you're sharing, the exit status code is 128, so it seems something else is happening. It's probably better to take a look at your CI logs to figure out the issue.

I tested the 12.3.5 version with the reproduction repos that were shared in this thread and it worked correctly in them. If it's still failing in other scenarios, it might be a different issue or if it's the same I'd need a repro to be able to troubleshoot it.

[jredl-va]

@leosvelperez I'm thinking that there has been a memory leak somehow introduced with these changes. Would you like me to open a new issue? For reference our pipeline runs on 16 core machines with 128 gigs ram. Node is given 8 gigs via node_options.

The change set where we can see this error appear in our logs:

Our CI runs via codefresh pipelines where the docker image is node:12 and the command is npx nx affected:build --base=master --head=HEAD --with-deps --prod --parallel --maxParallel=5

You are right that a different error is occurring here and it I missed the output earlier:

[71:0x7fb1b8dcd8d0]     6940 ms: Scavenge 10.6 (15.4) -> 9.5 (17.4) MB, 243.1 / 0.0 ms  (average mu = 1.000, current mu = 1.000) allocation failure              
[71:0x7fb1b8dcd8d0]     7191 ms: Scavenge 12.5 (17.4) -> 11.3 (26.9) MB, 174.5 / 0.0 ms  (average mu = 1.000, current mu = 1.000) allocation failure             
[71:0x7fb1b8dcd8d0]     9028 ms: Scavenge 18.7 (27.4) -> 15.5 (29.2) MB, 179.5 / 0.0 ms  (average mu = 1.000, current mu = 1.000) allocation failure             
                                                                                                                                                                 
                                                                                                                                                                 
<--- JS stacktrace --->                                                                                                                                          
Cannot get stack trace in GC.                                                                                                                                    
FATAL ERROR: NewSpace::Rebalance Allocation failed - JavaScript heap out of memory                                                                               
12: 0xad6ad8  [/usr/local/bin/node]                                                                                                                              
 1: 0xa1a640 node::Abort() [/usr/local/bin/node]                                                                                                                 
13: 0x7fb1d56c94a4  [/lib/x86_64-linux-gnu/libpthread.so.0]                                                                                                      
 2: 0xa1aa4c node::OnFatalError(char const*, char const*) [/usr/local/bin/node]                                                                                  
14: 0x7fb1d540bd0f clone [/lib/x86_64-linux-gnu/libc.so.6]

[leosvelperez]

Yep, definitely a different issue.

Currently, there are a couple of issues in the Angular CLI repo regarding performance: angular/angular-cli#20801 and angular/angular-cli#20792. I'd recommend you first to check those, as this might be an issue with the Angular CLI and not with Nx.

[jredl-va]

@leosvelperez thank you for your time. Will redirect my comments to those issues!

[bjornharvold]

I upgraded to 12.3.5 and our CI build is no longer seeing this error.
Thank you @leosvelperez for some awesome detective work!

[github-actions]

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.