IVY writes metadata into our symlinked monorepo's non-angular library's package.json

[JonWallsten]

🐞 bug report

Affected Package

The issue is caused by package @angular/compiler?

Is this a regression?

Can't tell

Description

When enabling IVY, our symlinked monorepo packages that are imported by the app using IVY, get's metadata written in their package.json by the compiler. I guess the compiler think it's a normal package in node_modules that wouldn't mind this.

#Added as a script
"prepublishOnly": "node --eval \"console.error('ERROR: Trying to publish a package that has been compiled by NGCC. This is not allowed.\\nPlease delete and rebuild the package, without compiling with NGCC, before attempting to publish.\\nNote that NGCC may have been run by importing this package into another project that is being built with Ivy enabled.\\n')\" && exit 1"

#Added as it's own key:
"__processed_by_ivy_ngcc__": {
    "fesm2015": "9.0.0-next.13",
    "fesm5": "9.0.0-next.13",
    "es2015": "9.0.0-next.13",
    "esm2015": "9.0.0-next.13",
    "esm5": "9.0.0-next.13",
    "main": "9.0.0-next.13",
    "module": "9.0.0-next.13"
  }

🔬 Minimal Reproduction

https://github.com/JonWallsten/monorepo-repro

🌍 Your Environment

Angular Version:

Angular CLI: 9.0.0-next.15
Node: 10.16.3
OS: win32 x64
Angular: 9.0.0-next.13
... animations, common, compiler, compiler-cli, core, forms
... platform-browser, platform-browser-dynamic, router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.900.0-next.15
@angular-devkit/build-optimizer   0.900.0-next.15
@angular-devkit/core              9.0.0-next.15 (cli-only)
@angular-devkit/schematics        9.0.0-next.15
@angular/cdk                      9.0.0-next.0
@angular/cli                      9.0.0-next.15
@angular/http                     8.0.0-beta.10
@angular/material                 9.0.0-next.0
@ngtools/webpack                  9.0.0-next.15
@schematics/angular               9.0.0-next.15
@schematics/update                0.900.0-next.15
rxjs                              6.5.3
typescript                        3.5.3

Anything else relevant?

[JonWallsten]

@petebacondarwin: I saw your input in this issue AlexKhymenko/ngx-permissions#112 and thought this might be something you can explain.

[petebacondarwin]

Hi @JonWallsten - for the Angular compiler (ngtsc) to be able to build your applications in ivy mode, we need all its dependencies to be compiled in ivy mode too. We have a second compiler (ngcc) that will re-compiled dependent packages into the ivy mode. Ngcc will identify packages that have been built with view engine and recompile them (in-place effectively).

This is what is happening in your project. This has to happen for all dependencies whether they are in node_modules or not.

One way to avoid this, if the dependency is a local project and not going to be distributed on npm, is to actually compile these packages in ivy mode and then expose the build files to the downstream application rather than the view engine builds. At this stage, if the package is going to be distributed on npm then it should be built in view engine mode (i.e. non-ivy).

[JonWallsten]

@petebacondarwin First, thanks for responding! I'm not following completely, but I get the gist of it. What I don't understand though, is why my non-angular libs (they are not in the example repo) are being compiled at all at this stage. They are already compiled by Webpack/TypeScript in an earlier stage of the build process, as commonjs modules and then we import stuff in the app. They shouldn't have to be touched by angulars compiler, or am I wrong? If they do I guess it would be nice to detect if this is a installed package or a original local package, and don't write stuff in the package.json. Would that be possible?

[alxhub]

If those packages are Angular packages, they will need to be "updated" from View Engine code to Ivy code.

Another way to think about ngcc is that it's running the Angular compiler on your libraries.

[JonWallsten]

@alxhub: One of them are, and I can aceept that it's adding stuff to the package.json in that one. But the other libs are not containing any angular code. They are only guilty of being used by our angular app.

[marcj]

I think this is a fundamental design flaw in Ivy's compiler architecture, especially in a local development workflow for linked monorepos or linked libraries in general. We use Lerna for example

packages
├── cli
├── core
├── core-node
├── app
├── electron
├── server
└── website

and whenever we build the project the package.json of core gets changed (because its used by app, website). It's not clear to me how changing my actual code from a compiler could anything but unpleasant. How is the workflow supposed to be? My Git tools show now all the time changes which I explicitly need to reset before commiting changes. Tools like Jetbrains Webstorm can exclude lines in commits but not the Git CLI, which I use most of the time. I personally find that very frustrating and bad practise.

Also before publish my packages now I have to manually remove those prepublishOnly adjustments which prevent me from publishing.

Beside the bad UX of that another topic is actual a major regression: We use external libraries like https://github.com/marcj/angular-desktop-ui which is a TS-only library. We symlink it locally while development to make the workflow easy. Like so:

workspace/
├── application/
├── angular-desktop-ui/

where application/node_modules/angular-desktop-ui is linked to ../../angular-desktop-ui.

With Ivy it's not possible anymore to use that setup with more than one consumer where each uses different versions of Angular. Imagine a second application with Angular 9.0.1 that consumers angular-desktop-ui (or any other local checkedout and symlinked library), while the first application uses Angular 9.0.0.

workspace/
├── application1/ angular@9.0.0
├── application2/ angular@9.0.1
├── angular-desktop-ui/

It asks now everytime to delete the node_modules folder, however after symlinking back angular-desktop-ui the same error appears again. Removing the lines manually (super ugly) __processed_by_ivy_ngcc__ in angular-desktop-ui/package.json solves it only for one consumer. When building the other application the same error appears again and you have again to manually remove __processed_by_ivy_ngcc__.

The assumption that deleting node_modules resets all package.json and thus fixes the ngcc issue (as told in the error message The ngcc compiler has changed since the last ngcc build.) is wrong in a monorepo or symlinked libraries workflow, which leads not only to horrible UX but also major regressions regarding local development workflows.

So for me there are 2 issues:

1. Modifying package.json results in changing my actual local code (not a node_modules/ dependency which I don't care when you change their content) which is a Git repository and thus produces changes. That's a very bad idea for monorepos and alike. At least use please a dedicated file like .angular.build which I can put into .gitignore. Or better don't change anything in my actual code or its dependencies and put it in an external folder like ~/.angular/build/<project-path>/build.json.

2. Not being able to use multiple consumers for a symlinked package. This is actually the biggest deal-breaker for me as I have a couple of shared libraries that are used across multiple projects and I'm not able to use Ivy in those setups as it leads to a big headache with that ngcc compiler thing trying to build only one version at a time instead of allowing to coordinate its build artifacts in a way that includes its consumer path/version in its cache keys.

For me it seems Ivy completely ignored the legit workflow of multi-packages symlinked together using either Lerna or other local multi-package workflow patterns.

What's Angular's approach to fixing that?

[filipesilva]

@marcj I'm not really sure how you are building your monorepo in the case you described. I get the impression you're using TS paths to make consuming apps use the library TS sources directly. I think that's the case because you mention the original package.json is modified.

This is not a use case that we've ever supported in Angular CLI. Instead, we try to setup projects so that consumers use built versions of libraries. If you have libraries generated with Angular CLI, they are also updated to directly compile to Ivy in development workflows.

The symlinked package case is also not very clear to me... symlinked packages have always been a problem because they are usually broken with peer dependencies. Some setups might addressed this of course, but the devil is in the details. The setup you're describing sounds like it's using a single built (or maybe non-built) library linked to several consumers. The same built library should be able to be consumed across different Angular versions, but the same cannot be said of the particular installed version on disk for a given project.

I think you've made a pretty good description of this problem but before we can jump in and describe ways to address these issues, we'd really need to see a example reproduction. This is a very non-trivial case that hinges on how things are built, linked, and consumed. There are setups that might have worked, but not because we explicitly supported them.

[marcj]

@filipesilva sure, no problemo. Here is a full prepared and working repo with Lerna, 3 packages, and step-by-step doc on how to reproduce it: https://github.com/marcj/angular-ivy-issue1. This setup here is a very basic and common one in Lerna: Frontend and server share some code which is stored in a core package. In this case a simple User model. The result is that our packages/core/package.json is always changed when building the frontend + we can not publish that core package anymore (thanks to the inserted script hook). As already described.

A more complex scenario could be to have a package external of lerna (like described with angular-desktop-ui) and use a solution like npm-local-development to link it correctly so devDependencies and peerDependencies are correctly handled. If you use Lerna and that core package has own devDependencies (which a consumer defined as well) for its unit tests then you need the better linking approach like in npm-local-development as well.

However, that overall topic with linking external packages or packages in a mono-repo using links is a solved one and works fine (at least if you use a solution like npm-local-development for more complex setups).

There are setups that might have worked, but not because we explicitly supported them.

Which I think you should consider supporting explicitly. I worked on many TS projects especially enterprise and my experience is that in TS projects mono-repos and linked packages are pretty common. Breaking this now with Ivy pushes people away from Angular because the only alternative is not to develop on multiple packages that partially use each other locally anymore. I think https://nx.dev/ has the same issue here. If that's not supported and never will, you should at least provide a workflow that works for enterprise and bigger projects where you share code between multiple packages that does not require you to publish the shared code to a registry first all the time. But honestly, the simplest solution might just be to not modify the package.json at all or any other code and put your compiler metadata in your own folder or users home directory.

[coryrylan]

@marcj I have seen this issue as well in some private repos but also have it in our public Clarity mono repo. We distribute an Angular library as well as a handful of plain TS/CSS libraries as dependencies. I wrote a script as a temporary workaround that runs ngcc during post install removing the added properties.

vmware-archive/clarity@d160f0c

I am also interested in how NX is able to work around this issue since they follow a similar pattern I believe.

[filipesilva]

@marcj I followed the instructions on your repo:

git clone https://github.com/marcj/angular-ivy-issue1
cd angular-ivy-issue1
npm run install

This fails because it needs the lerna command somewhere. I installed it using npm i -g lerna@3.19.0 (the latest) and tried again, and it seems to work:

$ npm run install

> root@ install D:\sandbox\angular-ivy-issue1
> lerna bootstrap --nohoist \* --no-ci

lerna notice cli v3.19.0
lerna info Bootstrapping 3 packages
lerna info Installing external dependencies
lerna info Symlinking packages and binaries
lerna success Bootstrapped 3 packages

I'm not really familiar with what this command does, but I see that both packages/server/node_modules/@issue1/core and packages/app/server/node_modules/@issue1/core were created with the TS sources of packages/core/. I'm sorry to say but having TS sources in node_modules is not something we support.

Publishing TS sources is generally not safe, since the output of compiling them differs by TS version and by build system setup. If you have a separate Angular library and an app, and you should build the library using whatever build setup you have, then consume the built library from your app. Angular CLI uses @angular-devkit/build-ngpackagr to build libraries, but you might build them some other way. If you try to have the app build system consume the library TS files directly, you're no longer building the library and there's no guarantee things would work the same as if you did. In the Angular CLI case, the app and lib build pipelines are very different.

Continuing with your repro I run these commands:

cd packages/app
ng build

And see the diff you mention in the README.md. packages/core/package.json is modified.

This happens because packages/app/node_modules/@issue1/core/package.json seem to be a symlink to packages/core/package.json. When ngcc alters node_modules in this sort of setup, it will alter the source files because they are linked.

By design ngcc will always alter node_modules. We didn't cover the case of linked TS sources because it was never a supported case to begin with (for the reasons above).

I don't understand why packages/app/node_modules/@issue1/core/package.json is altered in the first place though. It's not an angular library. packages/app/node_modules/zone.js/package.json wasn't altered and it's imported by the app too. Maybe @petebacondarwin can shed some light on that.

[CDevine1]

I saw a workaround posted at #35447 which basically has you run ngcc ahead so that it won’t run at compile time.

In case it helps:

Add the "postinstall" documented in https://angular.io/guide/ivy#speeding-up-ngcc-compilation, then rm -rf node_modules, then yarn if using yarn workspaces (or lerna in your case). This fixed it for me, our project is using yarn workspaces instead of lerna.

If it fixes it for the others too, the guide's title could be changed to "Ngcc compilation with lerna, yarn workspaces, or just speeding up ngcc compilation in general.

[Nightbr]

It doesn't work with lerna.

In lerna, if you do lerna bootstrap to install, symlink, prepublish, prepare your packages together. See https://github.com/lerna/lerna/tree/master/commands/bootstrap#readme

If you do on postinstall, it will run the ngcc after the install so it will not take into account other angular packages so we get the same error & need to build twice.

I tried to put ngcc in the prepare:

{
    "prepare": "ngcc --properties es2015 browser module main --first-only --create-ivy-entry-points",
}

It compiles properly all libs installed but not other angular packages of the monorepo... So same error, need to run angular app twice to work...

Really weird, it should work but it never detect the symlinked packages...

EDIT: Found the trick, when we lerna bootstrap it creates a package-backup.json and remove all reference to monorepo packages (in order to run npm install without errors).

So when you run ngcc, it will not compile your package because it is not in the package.json...

Workaround: compile my package manually...

{
 "prepare": "ngcc --properties es2015 browser module main --first-only --create-ivy-entry-points && ngcc --source ./node_modules/@myOrg/ds-angular --properties es2015 browser module main --create-ivy-entry-points",
}

For sure we need to find a better way to handle this...

[Lonli-Lokli]

For 9.0.0 it may be that disabling ivy is required. We hope to have a solution for this by 9.1.0.

Seems like it was not fixed?
Is there any plans to fix it in the next 9.x or at least 10.0 ?

[petebacondarwin]

This is proving to be difficult to resolve. We still don't have a solution for symlinked packages. It is not even clear if we can find a resolution for v10.

[petebacondarwin]

We have two related issues here:

1. ngcc doesn't work well with sym-linked libraries - in particular we need a better story for library authors to build applications that test the libraries.
2. ngcc doesn't work well with the standard lerna setup - it seems possible that there is a workaround for lerna that will allow ngcc to process the necessary packages but no one on the Angular team has enough experience or knowledge to understand if this is valid.

[mgol]

About the Lerna workaround: Lerna only does this package.json hacky rewriting if it needs to manage workspaces by working around the package manager restrictions. I don’t recommend using Lerna like that, it only causes pain & bugs in edge cases like this one. If you use Lerna with Yarn by relying on Yarn internal workspaces support instead (there’s a Lerna setting for that) all this hacky behavior is gone as Lerna leaves managing workspaces to Yarn & it doesn’t rewrite package.json.

[Nightbr]

@mgol ok if you use yarn but if you use npm you have to stick with the "hacky" rewriting. So still need to handle that case I think.

[mgol]

@Nightbr Yes, sure! I just don’t consider npm to be suitable package manager to use with Lerna because of missing support for workspaces.

[mischkl]

We have the same problem, and there's another wrinkle to it: We have a shared library that's used by two different applications, one using Angular v8 (View Engine) and one using Angular v10 (Ivy). Up until now we relied on symlinking the library into both applications - but now we can't because as soon as the Angular 10 application is run, it modifies the shared library and thus the Angular v8 app can't deal with it any more.

It seems to me that modifying the contents of dependencies is a fundamental design flaw of the Ivy compile infrastructure, that makes it completely incompatible with symlinking, which a lot of developers depend on. In my mind it shouldn't be too hard to simply use a cache folder for the Ivy-compiled versions of View Engine packages, instead of rewriting those packages' contents. This would solve all the issues with Lerna and yarn workspaces as well.

[dennisameling]

Ran into the same issue with https://github.com/dennisameling/muuri-angular and the workaround below got me up and running at least (basically an implementation of what @petebacondarwin mentioned in #33395 (comment)). These steps are also described at https://github.com/dennisameling/muuri-angular#contributing.

Update library config
With the steps below, we'll add some additional configuration to your library so that it can more easily be consumed by projects that use your library with npm link.

• In projects/PROJECT_NAME, it's very likely you already have a tsconfig.lib.prod.json because ng generate library already defaults to the View Engine for publishing to NPM.
• Create a new tsconfig.lib.ivy.json in projects/PROJECT_NAME with the following contents:

/* To learn more about this file see: https://angular.io/config/tsconfig. */
{
  "extends": "./tsconfig.lib.json",
  "angularCompilerOptions": {
    "enableIvy": true
  }
}

• In your library's angular.json file, under projects > PROJECT_NAME > architect > build, add a new configuration ivy so it looks like this:

          "configurations": {
            "production": {
              "tsConfig": "projects/PROJECT_NAME/tsconfig.lib.prod.json"
            },
            "ivy": {
              "tsConfig": "projects/PROJECT_NAME/tsconfig.lib.ivy.json"
            }
          }

• Update your package.json for easy access to the following commands:

  "scripts": {
    ...
    "build:prod": "ng build PROJECT_NAME --prod",
    "build:ivy": "ng build PROJECT_NAME --configuration=ivy"
  },

For Angular 9+ (Ivy) projects consuming your library

• Run npm run build:ivy. This will build an Ivy-compatible library that you can use in Angular 9+ (Ivy) projects locally.
• Run cd dist/PROJECT_NAME.
• Run npm link.
• In your project, run npm link PROJECT_NAME to link to your library. Your project will now use the local PROJECT_NAME code.
• Bonus tip: Run npm run build:ivy -- --watch so that the library gets rebuilt on every code change 😃

For Angular 8 and lower (View Engine) projects consuming your library

• Run npm run build:prod. This will build the library with the legacy View Engine so that it can easily be consumed by non-Ivy projects. This would also be the build that you publish on NPM.
• Run the same steps as above for Ivy projects.

[Lonli-Lokli]

Seems like there are plans to fix this issue in future versions
#38366

[elclanrs]

I'm having the same issue with pnpm. Running ng serve a second time works as noted in previous posts. I am able to resolve this issue for now by manually compiling all dependencies in postinstall:

"postinstall": "find node_modules/{@angular,ngx-mask,ng-dynamic-component} | xargs -I% pnpx ngcc -s %"

ngx-mask and ng-dynamic-component are Angular packages that are needed to run the app.

Note that ngcc alone did not work and I had to explicitly run it on all modules. The downside is that it compiles all Angular modules, even ones I don't use or need. I suppose I could be more specific in what to compile, but it would require more maintenance.

[petebacondarwin]

This should be solved by the ng-linker that is being developed. Please follow the progress of that here: https://github.com/orgs/angular/projects/2

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}