Target aware "affected" functionality

[skynetigor]

Description

NX is an incredible tool to manage monorepo. It quite well solves problems with long CI runs. But I think it can work even better.
I'll make a few examples based on unit-tests.

Motivation

Current behavior:
Let’s imagine we have a book-store application and a book-core lib that book-store depends on.
Whenever I change only .spec.ts file in book-core, nx print-affected --target=build compares HEAD (a spec changed) with the BASE and returns book-store project even though book-store does not have files depending on any specs from book-core. Consequently, it will trigger the build of book-store even though book-core was not changed (what was changed is just its tests)

Suggested behavior:
If I change just a spec file for book-core this change should not affect book-store since these changes are just unit-tests (not changes to the lib implementation) and hence should not trigger build of book-store.

Suggested Implementation

nx print-affected --target=build should determine whether “build” related files changed in dependency, and:
if there are such files - treat dependent project with “build” target as affected
otherwise, treat the project as not affected

To let NX be aware of what exactly files relate to a specific target we could add include/exclude properties glob patterns.
For example, for project book-core in “build” target there could be added “targetUnrelatedFiles” [“**/*.spec.ts”] exclusion and when NX will build a dependency graph for book-store it will skip book-core project as there hasn’t been any “build” related files in book-core.

Targets that might need that functionality:

• Unit-tests - when changes are made only in *.spec.ts files
• Storybook - when changes are made only in *.stories.ts files

Alternate Implementations

In the current implementation, It could be achieved by adding a separate project per target.
For example, for book-core you could add book-core-unit-tests lib and make dependency to book-core lib in order to be able to test its modules. This way it will work as expected. The main downsides of it is the growing number of projects and you can’t write UTs for encapsulated services and components.

[rlmestre]

Since 14.4, you can specify inputs on targets globally or per-project. This allows you to control fileset inclusion/exclusion for the task. Does that cover your use-cases?

https://nx.dev/more-concepts/customizing-inputs

[ihor-panasiuk95]

As per the documentation, this is more about NX Cache and there is no any mentions about whether it participates in "affected" computation.
So if it does, this more a documentation issue rather than feature request.

[ihor-panasiuk95]

But if it doesn't, I guess reusing of this inputs for "affected" computation along with caching looks good.
Thanks for the reply!

[evanliomain]

I made some experiments with inputs, and I confirm it only control the Nx Cache detection.

I am also interested in this feature to launch only libs with significant changes.

Today, Nx executor relies on affected libs computation and cache system. Only the cache can be fine tune with inputs. It would be great is affected libs will be too.

[chrisui]

What does the --target X option do within nx print-affected command?

[skynetigor]

What does the --target X option do within nx print-affected command?

https://nx.dev/packages/nx/documents/print-affected#targets
it prints affected apps by a specific target (build, test, storybook, e2e, etc)

[tomdglenn91]

Posting for visibility as googling if this was possible brought me here. Nx's greatest strength is in saving time in CI, and this would be a huge huge boost. If I make a PR that only updates tests, and my CI server not running build, would be incredible.

[ianldgs]

Posting for visibility as googling if this was possible brought me here. Nx's greatest strength is in saving time in CI, and this would be a huge huge boost. If I make a PR that only updates tests, and my CI server not running build, would be incredible.

In theory that already happens because of build cache. Test files are typically not inputs of the build target. Module is affected but the build runs instantly.

[tomdglenn91]

That's only the case if I have a cache to read from. We're currently not set up for nx cloud (early days of integrating nx) so i've got a cold app on all fronts.

[mirobo]

We have 170+ projects and each of the project also has e2e tests as well as Angular code that is tested with these e2e tests. Since "nx affected" detects only if a project (in its whole) is affected, we have the problem that changing e2e tests results in a rebuild of the app.
We could create a separate project for each library to account for that. But it will not resolve the underlying problem, since we also have MD-files in the project that need to be linted, if they were changed, unit tests that need to be run, if they were changed (without a rebuild of the app). As previously mentioned we could boost CI build times massively if NX affected is "target-aware".

[FrozenPandaz]

I do agree that affected could be smarter. However, I don't know when we will get to this. There's a lot of design which will have to go into this.

I know it's not the same but tasks being cached is a huge mitigation here. Though there will be affected tasks, if they haven't changed, they should be cached. Hence we've prioritized other things first. Again, not the same.. but very much yields fast pipelines.

[skynetigor]

The problem with relying only on cache is that you will anyway have to run CI job for unaffected project, you will need to setup dependencies, do some other prerequisites (sometimes they might take long and hence expensive, depending on the project) and only on build step NX will understand that nothing affected relatively to build and it can restore build for the project from cache.

Maybe, if NX could rely on Cache during "affected" functionality (maybe it's the easiest way to achieve what described in this issue), like check in cache if cache exists for project (in cloud I think) then project is not "affected", otherwise, if cache does not exist, the project is affected.

In our project we have 12 MFE applications, cypress project per each app, and dozens of shared libraries with unit tests.
We managed to achieve the desired behavior but leveraging "hash" property that NX provides.
In CI workflow, we have one Job called "Compute affected projects", that computes what projects are affected and then distribute affected projects though matrix jobs.
Then, each CI job that builds/tests a project, saves output of build/test to GCP bucket with this "hash". In the next workflow run, "Compute affected projects" computes NX hash again and checks in GCP bucket if cache exists. If so, project is not affected. Otherwise - affected. Additionally, we have ability to restore all our project and glue up all our MFE build results into one artifact by just restoring all build caches relative to the current run. We use restore multiple times: during deployment and to run Cypress tests in separate jobs.

NOTE: we don't use "affected" at all. We always rely on Cache in GCP bucket.

Hope I described it clear and it would help to improve your build process.
I think it's worth writing a blog about this process.

Nevertheless, NX is doing great job, I really love this concept and it brings monorepo on a new level. Don't like Micro-repo since I've got familiar with NX. 😌