Development workflow for library developers

[tobi-or-not-tobi]

Bug Report or Feature Request

- [x] bug report -> please search issues before submitting
- [ ] feature request

Area

- [ ] devkit
- [x ] schematics

Versions

Angular CLI: 6.0.0-rc.5

Repro steps

ng new myapp
cd myapp
ng g library mylib

import library module into main AppModule:
import { MylibModule } from 'mylib';

I'm exited by the new generation of libraries in the CLI. But I didn't got it to work initially. While the creation of build of the library seems to be taken care of, the integration into the main app seems to be missing. I believe this is caused by an incorrect reference in the path setup in the tsconfig.json, as it is configured with the following after creation of the library:

"paths": {
      "mylib": ["dist/mylib"]
}

This requires an initial build to have a generated dist/mylib folder in the first place, but would also require to continuously rebuild the library upon changes.

After I change the path to the src folder:

 "mylib": ["projects/mylib/src/public_api"]

I'm able to include the module without building, but also have a seamless experience upon any changes in my library component.

I know the public_api has been originated from the ng-packagr, but i'm wondering if we shouldn't change this into an ordinary index.ts, as it would make the import easier. in addition, if that file could live at the root of our generated library, the import could be shorten to "mylib": ["projects/mylib"]

Desired functionality

• Change the path reference to the source rather the generated build files
• (optional) simplify reference of the library by changing the public_api.ts into index.ts and move it to the root directory.

Mention any other details that might be useful

I've already logged this issue under the cli project, but after a second thought I assume it belongs to this issue tracker.

[tobi-or-not-tobi]

@dherges Can you help here?

[dherges]

I think we had some discussions how the development workflow for libraries should be. We agreed that tsconfig should point to the dist folder (depending on the library builder output). Pointing to the library sources was effectively using two separate, completely unrelated build stacks in parallel (obviously it's faster still).

[dherges]

The long-term solution here is "partial library re-builds", i.e. a watch mode that outputs only esm5 (or esm2015).

[nweldev]

So, @dherges, I think we would need to add a --library option to ng build in order to permit to only build the library project, and then a --watch option.

Correct me if I'm wrong, but I think the first part is in the angular-cli scope, will the second is both angular-devkit/build-ng-packagr and angular-cli.

[nweldev]

Task list :

• @angular-devkit: Filter project names #739
• feat(@angular/cli): build only the projects of a certain type angular-cli#10397 : add a --projectType option in angular-cli build and test
• (Development workflow for library developers #730 (comment)) add a --watch option on angular-cli build. This will need to re-use workspace.projectFilter in order to only rebuild the affected libraries each time.

I don't know yet if this --watch option will need an evolution on angular-devkit/build-ng-packagr, or even maybe on ng-packagr, for optimisation ("partial library re-builds" ?). @dherges what do you think ? Yet, I find this watch / rebuild process a little heavy for a monorepository (say you have 20 libraries ... this would take a lot of memory).

Also, if it happened on GitHub and you think you could find the discussion you talked about and share the link here, it would be great (I'm currious about it 🙂 ).

[dherges]

The first three tasks are to add a watch mode to ng-packagr, then add an option for esm-only output (skipping most of the build pipeline steps), then implement the dirty checking for invalidated build nodes.

I'd keep it low for now, because it'll mimick the bazel builds. It's a lot of work that unfortunately is only a "side project" / "free time spending" for me.

[nweldev]

@dherges then, maybe you could create an issue for that in in ng-packagr, describing this 3 steps, with an "help wanted" label. I would be very enthousiast to read more from you about this "mimick bazel builds" approach.

But, even if it's clearly the right path, I think, to begin with, that angular-cli could do with just the first step you described, using fs.watch, as the two others are ("just") for performance enhancement, and that both could be released in a next version of ng-packagr without any impact on devkit or angular-cli.

If it's ok for you, I think I could do this "dirty first step" soon enough, my related work on angular-cli (step 2) being close to PR submission.

[nweldev]

@tobi-or-not-tobi we may got carried away here, but if you think the path I'm taking would respond to your issue, you may want to change its title 😆

[tobi-or-not-tobi]

@dherges can you explain a bit more about the "completely unrelated build stacks in parallel"? I'd like to understand that better as I don't experience this in the setup described above.

The way I see and understand is that, whenever the CLI is used to generate both libraries and applications, the expectation (well at least mine) is to run the dev server and libraries that are used all together seamlessly. If I point into the source of the library, I have that experience; any changes in my library are directly visible in the app when the dev rebuilds. With this setup I'm not seeing any parallel builds, simply because I'm not building the library during development time (similar to that I'm not building the app). Only when I want to build and package the app, I'd be interested to generate the dist folder.

[dherges]

After I change the path to the src folder:
"mylib": ["projects/mylib/src/public_api"]
I'm able to include the module without building, but also have a seamless experience upon any changes in my library component.

This is linking to library sources. You're building the library with the app's build stack (ng build, webpack under the hood). ng-packagr is standalone and has an unrelated build stack (no webpack, no stylesheet loaders, etc.)

[nweldev]

Big +1. If you are using ng-packagr for the libraries in a monorepository (like I am), it should be because you need it in projects that are maintained outside this monorepository (that why you need npm). If you are not facing this, you don't need ng-packagr ... and I think we should add a new lib schematic for this kind of cases, like the one by Nrwl/nx. So, if you also need this library in an app inside your monorepository, it seems logical to keep the same build process.

But, @dherges, don't you think it could be a good solution in some cases (like the one covered by Nrwl/nx like I said above) ? Is the three shaking and AOT when using ng-packagr as good as when linking to library sources ?

[filipesilva]

I want to highlight there there is more than one library usecase. Sometimes a library is meant to be used exclusively inside the workspace/monorepo, and sometimes a library is meant to be published.

I think @tobi-or-not-tobi is interested mostly in the first usecase, and @noelmace seems to be interested in the second.

Right now we mostly cater to the second usecase because that is what ng-packagr does really well. You can build a library that is ready to be published very easily.

But it is true that the first usecase does not have a very good experience. Running ng build my-lib every time you change a file is bothersome and takes time.

Some similar setups instead add the path to the source code directly inside tsconfig. This is what @tobi-or-not-tobi proposes and it makes seeing changes in your app faster.

But doing that is risky. When you do that, the build system for your app is building the library as well. But your library is built using a different build system than your app.

Those two build systems can build things slightly different, or support completely different features.

This leads to subtle bugs where your published library behaves differently from the one in your development setup.

For this reason we decided to err on the side of caution, and make the recommended usage the safe one.

Another problem is that it doesn't scale very well. It's essentially the same as having no library at all, since it is the same as just having more app source files but in a different folder.

What we really want is to separate these compilations so that building the library is completely separate from building your app. This lets you split the compilation into smaller isolated parts that can be ran in different processes.

This also means that Angular CLI would need to know your app depends on your library, and automatically rebuilds the library when the app needs it.

At that point having a watch mode for the library is a matter of performance and usability, but everything would still work correctly even without it. But performance matters so a watch mode is important and on the roadmap.

I also want to mention @noelmace's PR (angular/angular-cli#10397) for building each project type separately. I don't think it addresses the real problem. It works for the simple case because it is common for the app to directly depend on a library. But it stops working once you have a library depending on a library.

I think the correct (temporary) solution is angular/angular-cli#10352 instead. This addresses the problem that ng build tries to build everything at once, but leaves the door open to still solve the project dependency correctly in the future. It also helps for generators.

[nweldev]

@filipesilva Thank you for this long and complete response.

So, in order to clarify, i think that :

there is 3 use cases

1. Library used only inside the mono-repository

I think this should be covered by a separated "internal-lib" schematic in schematics/angular, to begin with. For the rest, I think everything would be covered by the angular build architect, but without any build target (but I'm sure we'll find a lot of new use cases to cover later).

This is, in fact, what nrwl/nx did with it "lib" schematic and the nx command-line. I think the discussion about the migration to Angular v6 (nrwl/nx#412) could be really inspiring both ways, but we are here waiting for @vsavkin inputs (and, as a matter of fact, I wasn't at ng-conf 2018 and didn't have the time to catch up on the talks).

The main reason why this issue was created in the first place by @tobi-or-not-tobi was, I think, because "generate library" could be confusing. Maybe we should rename it npm-lib or external-lib after adding a "internal-lib" command ... or maybe it should simply be one schematic, with a --packagr option in order to add build-ng-packagr architect as a build target.

2. Library used only outside the mono-repository after publishing

This is indeed my main focus right now, as I'm migrating (for my paid job) a bunch of medium sized projects to a mono-repository, step-by-step (tried with Nrwl/Nx at first but I'm now using angular-cli v6, as a lot of work has been done on this use case, even if I there is still a lot of issues that I will try to address along the way by contributing).

Except for those who needs to use npm link (so a build-watch mode, but I don't think this is really a common use case), I don't see any issue with that. As a matter of fact, I finished this step on my work today, and everything was ok. will just need to had a bump version and publish script monday (please tell me if you think this should be in angular-cli / devkit or not, as I'm always happy to contribute).

3. both needs

This is in fact this "grey area" that could add a lot of complexity (and this is my next steps to address 😉).

In the same repository :

• some libraries may be both used inside and outside the repository. I think this is in fact the main use case, as a lot of multi-libraries projects will need some application for demo, documentation and testing. This case is covered by our discussion with @dherges, as consistency is needed.
• some others may be covered by case 1
• and some others by case 2

I think, in order to keep it simple, that we should create another separated issue for specific matters brogth by case 3, and keep this one only for case 1, as this was the goal of @tobi-or-not-tobi. I'll create other issues later about case 3.

project filter

angular/angular-cli#10397, yes, was related to this discussion
I think you're right, this is not the answer for this issue. But I think it would still be interesting for some (much more simpler) situations, especially for the CI in the third use case. As a matter of fact, #739 is the real starting point here. Let's keep the discussion about this part in this last PR.

[tobi-or-not-tobi]

Good discussions and good feedback @dherges and @filipesilva on the potential issues with building with two separate build systems, that really helps.

In our project, we build a number of shell apps and libraries in a mono repo. We build and deploy the apps our selves, however, we also publish our libraries so that customer can use those libraries to further extend and build their own apps with it. This is the use case 3, described by @noelmace and I also strongly believe that most library developers need a setup for playground/documentation reasons.

It seems that we should harmonise the build systems somehow so that we avoid those potential conflicts.

Is there anything wrong with offering multiple setups for the build, so that we can choose when (development, CI) to build what and how? I.e.:

• embedded mode; libraries are directly referenced from the mono-repo source and we rely on the cli for building. This can be useful during development, but also for those interall apps that can rely on the source directly.
• standalone mode; apps and libraries are build with the library output

If we can mix those modes even per app/project, we create a lot of flexibility.

I understand that we can have potential issues, but there seem to be scenario's where this makes sense. A warning might be useful for that to be on the save side.

[nweldev]

In order to summarise this thread, here are, in their order of difficulty, the task we (it seems) agree on.

1. "inside" libraries

@tobi-or-not-tobi "inside librairies" (like the one in nrwl/nx, and without any build target given that it would be built with the related apps) is definitively a thing that we have to address. I'll work on it (if no begin some work on it before) starting May, 5, given that I have a lot of work to do until this date.
@filipesilva, could you tell if you think it would be better to a) add this as a option to the library schematics or b) create a new schematic ?

2. global build process

As this require to work on the order of the execution of the build targets, this is something that, I think, can't be addressed in devkit, so should be in angular-cli.

a. build "outside" libraries before projects

To begin with, we could just prioritise libraries over applications when running it. I see now that ordering would be better for that than the filtering I proposed in #739.

But I don't see "default project" how angular/angular-cli#10466 could address that, especially given that we could have several applications in our repository. @filipesilva ?

b. complete build targets ordering

If we want to permit "outside" libraries to depend on other "outside" libraires, we would require to build a dependency graph of the projects. This is something I started to deal with in noelmace/packagr-for-nx#1 (but this is an unpushed work given that packagr-for-nx was just a POC).

3. watch mode

This is clearly something that would need a lot of work, and we still have for now too many issues on this area for now, but we'll need to keep that in mind.

[nweldev]

FYI : for now (ie starting yesterday), I'm using https://github.com/noelmace/ng-admin, a kind of fork of the devkit admin, to deal with the ordering / global tasks issues, and I'll do the associated PRs and issues as soon as possible in angular-cli. Another way I prototyped recently is to create some file listing all the projects in the right order, and to use this file with as a configuration for a script which run our build targets in this order with ng <target> --project <var> . This is clearly some dirty workarounds, but it may be of some inspiration for future angular-cli evolutions.

[filipesilva]

@noelmace we are still trying to figure out where the task ordering should go, and what should do it. At the moment I am leaning towards Architect doing the ordering but this still needs to be properly designed first.

Personally I see enabling the non-publishable libraries as being a subset of the watch mode... Those just need to be built fast and without all the formats, which is what the watch mode should be doing anyway.

I don't think we can avoid having a build graph. In this issue we are talking about libraries but whatever functionality goes in needs to cater for other tasks. For instance, pre-build targets (think setting up assets or generating code) should be depended on by the build target. There is no explicit code dependency but it is a dependency nonetheless. Another example is e2e tests, they should have a dependency on serving.

That is why we need to take a bit to design this system. It is not just about libraries.

[nweldev]

@filipesilva thanks for this informations. If you think I could be of any help on this, please let me know, in order to avoid that I create unnecessaries / irrelevant PRs. For example, #739.

As I understand, you think that we could, instead of creating another separated schematic for "internal" libraries, that this should be a part of the existing one, as we could need this for the watch mode, right ? If so, I understand that, as the "build architect" system isn't intended to stay "as is", that the community can't really contribute on this for now.

[alexeagle]

This issue was moved to angular/angular-cli#12119