Support for dependency-aware "watch" tasks on nx run-many

[emilio-martinez]

Description / Motivation

nx run-many ... works really great for processes that actually end, such as test, lint, build, and so on. However, "watch"-like jobs processes don't exit until the process is either explicitly terminated, errored, or otherwise. It would be great to have a mechanism for these kinds of situations where Nx could either detect when a watch cycle completes (harder/brittle, in my opinion) or helps manage concurrent processes for dependencies, triggering subsequent builds when changes are detected for a specific project. This is particularly critical for projects that are not linked via tsconfig's paths feature.

This could be incorporated/interop with into the current "dependsOn" and "--with-deps" features.

Example:
App X depends on Lib Y, which in turn depends on Lib Z. When running a "serve" job for App X:

• Lib Y and Lib Z would need to be built in "watch" mode
• Lib Z would need to complete at least one build cycle before Lib Y's build cycles start.
• Once at all Lib dependencies have completed at least one build cycle, App X can serve.

Suggested Implementation

It would be hard to work with normal "watch" jobs because they're indeterminate, but I believe Nx could do something like spawn build processes for each dependency graph, triggering subsequent builds when changes are detected for a specific project. The change detection mechanism could use something like chokidar. All of this would follow the topological dependency graph, the same way "dependsOn" and "--with-deps" do today.

Alternate Implementations

Another approach, although I'm worried about its brittleness, would be to have "wait for" parameters and arguments to detect when indeterminate jobs end a build cycle. This however, would require a configurable understanding of what is logged at the end of each watch build cycle.

[github-actions]

This issue has been automatically marked as stale because it hasn't had any recent activity. It will be closed in 14 days if no further activity occurs.
If we missed this issue please reply to keep it active.
Thanks for being a part of the Nx community! 🙏

[emilio-martinez]

I guess this was missed 🤷 opened this a while ago..

[luxaritas]

I've been playing around with a setup using Nx core + NPM workspaces, and have just gotten to looking at builds - and watching/HMR/etc.

A couple considerations I'm making:

• I'm exploring Vite - being significantly slimmer and significantly faster than webpack is a big draw.
• I'd like to reduce the amount of "magic" going on. I'd like to avoid doing things like build-time config rewriting, managed execution of commands, etc - eg I want to just have a package script that runs eslint and know it runs exactly that, using the extract configs I have in my workspace.

That being said, I'm left to wonder what the best approach is if I have an app I want to watch alongside its dependencies. You could just ignore any build artifacts of packages and build from source, but that means 1) bypassing the nx cache 2) some sort of aliasing or build-time config rewriting to point to the source instead of dist if you have the library's package.json main point to dist (important if it's going to be distributed Eg on NPM) and 3) you can't have isolated build configurations for each package, which could lead to issues with reproducibility, or limitations if you need a slightly special config for one package, or issues with run-time asset loading or imports in some scenarios. Ok... so we could run our build using watch mode in parallel for all packages that are dependencies. Would that have a significant overhead when you have a lot of packages vs running one vite/tsc/webpack process? Or you could have nx watch for file changes and then kick off the build script for the package. But would that then negate the incremental build speed benefits of Vite?

I mean, I dunno, is this a scenario where there would need to be some additional support from the build tool to make one of the latter two options not run into issues with overhead? This isn't wholly an Nx-specific problem, but I'd be interested to hear thoughts from others who are more knowledgeable on the subject.

[walec51]

would be great if this could be solved in time, its the root cause of the problem reported in #5003

[luxaritas]

Cross-referencing other relevant issues: #2429 and #3501, which link to https://github.com/nrwl/nx-incremental and https://github.com/nrwl/nx-incremental-large-repo. It seems like the suggested approach is to basically spin up a file watcher at the workspace root that runs the build command when anything changes.

[luxaritas]

I wound up building my own little utility to handle this in our codebases: https://github.com/eternagame/workspace-helpers/tree/main/packages/nx-spawn

Walks through the project graph and uses concurrently to run watch scripts in parallel. One notable limitation is that since I have things configured so that dependencies (including types) are resolved from the dist directory, in order for this to work consistently (ie, without my application complaining that it can't resolve some modules), I need to run a regular build before running the watch scripts since there's no way to know when the watch scripts of the downstream dependencies have finished their first build.

[erdysson]

Cross-referencing other relevant issues: #2429 and #3501, which link to https://github.com/nrwl/nx-incremental and https://github.com/nrwl/nx-incremental-large-repo. It seems like the suggested approach is to basically spin up a file watcher at the workspace root that runs the build command when anything changes.

Could anyone make this work for angular libraries as well? Cause does not seem to work with @nrwl/angular:package

[luxaritas]

Opened a PR implementing this functionality - input is welcome: #10706

[FrozenPandaz]

Crossposting from the opened PR.

We discussed this internally and we'd need to design it properly.

Thank you for posting use cases here in this issue. We will continue to monitor this thread for inspiration for the design. If you have any ideas please share as well. 🙏

[luxaritas]

@FrozenPandaz I definitely appreciate the need for more design work here - could you share the specific issues you came up with internally? I'd like to help move this along further (there are a few folks here where this is important for our use case), but in order to do that I feel like I'd need to have a better idea of what the problems/edge cases are that need to be addressed.

[luxaritas]

I guess one potential use-case to consider is something like running cypress in CI, where you want to spin up a preview server, wait for it to be online, run cypress tests, and then exit

[luxaritas]

What about adding another attribute to dependsOn? Could be something like completion, which could be an array of things to wait for before considering the task completed. I could imagine it working like wait-on:

file: - regular file (also default type). ex: file:/path/to/file
http: - HTTP HEAD returns 2XX response. ex: http://m.com:90/foo
https: - HTTPS HEAD returns 2XX response. ex: https://my/bar
http-get: - HTTP GET returns 2XX response. ex: http://m.com:90/foo
https-get: - HTTPS GET returns 2XX response. ex: https://my/bar
tcp: - TCP port is listening. ex: 1.2.3.4:9000 or foo.com:700
socket: - Domain Socket is listening. ex: socket:/path/to/sock
For http over socket, use http://unix:SOCK_PATH:URL_PATH
like http://unix:/path/to/sock:/foo/bar or
http-get://unix:/path/to/sock:/foo/bar

Another interesting one would be something like "wait until the console output matches this regex" - that way you could wait until the message from your build tool that the process has finished building in watch mode. You could also make it an object like {"type": "file", "path": "path/to/file"} - perhaps support both syntaxes similarly to how dependsOn works (though without any additional options this seems a bit redudant). The default would probably be a special value like "exit" which just waits for the task process to exit, but if I instead do something like http://localhost:8080, it wouldn't wait for the process to exit to mark the task as complete, it would wait until you get a 2XX HEAD from that URL (ie, dev server is up). Then from there, whether or not you keep those processes active just depends on whether or not the root task is long-running or not.

Does this solve all the edge cases? This seems much more robust to me, and provides the flexibility to add additional ways to wait in the future if any come up. The only other thing I could think of that would need to be worked out would be the output format - the way I had implemented it is probably fine? It would be neat to "summarize" all processes (eg, you see a line item for each task with whether or not they completed and when they last wrote to stdout), but you'd probably want to actually see process output, particularly errors. You could theoretically do some regex magic to pull out errors, but that would be heavily dependent on the build tool and likely to be brittle. The other thing would be if the console was actually interactive and you could expand/fold different processes to see their contents, but I'm not sure if that would be favorable. Similar thing would be if output for a process was shown under it for a limited amount of time (or limited number of lines), but I wouldn't want to do that unless there was a way to support recall as for a variety of reasons you may not be able to read it fast enough.

[luxaritas]

More thoughts from the cypress use case: We'd want to be able to specify the host/port of the web server in one place - that is, our vite config or node server can pick up on it, our cypress config can pick up on it, AND Nx can use the same information to know what URL to wait for. I imagine the answer there would be environment variables. Do something like set packages/app/.env with HOST=lcoalhost and PORT=1234, use process.env.HOST/process.env.PORT in the server and cypress configs (cypress config would need a bit of extra work to know where it should look - would need to think through that more), and then set the completion to something like http://$HOST:$PORT, with nx being smart enough to know that HOST and PORT are defined in the environment for the app being waited on.

[luxaritas]

One additional thing I just thought of: In the case of waiting for a watch mode build to start, it may clear the outdir before writing. Either you'd need to add a synchronous task to do that manually before executing the long-running task OR you'd need separate options for waiting for some file/set of files to exist vs actually being added (which means the file watcher would need to be spun up before the task starts)

[karfau]

We had a very fruitful discussion in Slack about this topic
https://nrwlcommunity.slack.com/archives/C03J3SCCHJ9/p1664520709486369

[ianldgs]

Another use case for this: A react app with some codegen, like graphql.

As a workaround, it should be possible to wrap the nextjs executor and pass all options down, while at the same time spawning a process for the codegen with a watch flag.
However, I would fear this not being picked up by some future migration that would look for the nextjs (or other) executor.

[karfau]

Fyi/to make it more explicit:
The main idea that we concluded with to solve my use case in a reliable way is to let nx watch the input and only rerun the required tasks, otherwise reuse the stuff from cache.

There might be a performance impact for large/huge projects.
The person from the nx team stated that the next major release (I think v16?) will make the related internal API much more accessible to customize the behavior. So it is not a simple option as of now.

[isaiahdahl]

Been searching for this kind of solution for a while. Currently, my monorepo is powered by turborepo but they also have a bug similar to this waiting on the internal team to come up with some solution for this.

My use case is a music chord chart editor component library. Where I have web components that depend on libraries to power some features like parsing the text to render a chord chart, and syntax highlighting.

My dream is to have a DX where I can just run one command at the root level. Then work in any of the packages and it'll propagate the changes up through a pipeline. ie I make a change in the library that powers syntax highlighting, and it rebuilds that library, and the web component package that depends on it so that the changes are reflected in the browser for that web component. NX knows the dependency graph, it should be able to figure this out.

I can't just run them all in parallel because the component library I'm using Stencil, uses webpack to build all the dependencies into a loader dir it serves the components from. So the component package requires a fresh build cycle to get the changes to show up.

[luxaritas]

Crossreference to some additional comments I've made on the Nx 16 roadmap wrt some current plans for addressing this: #12836 (comment)

[dominikfeininger]

Also interesting for me. Running a typescript monorepo containing multiple SPAs and libraries.

Would be nice to have a project.json with defined dependencies and a targets property which can run multiple dependencies scripts in parallel.

run-many only works for me with an additional workaround since script names differ in dependencies.

[VanCoding]

Hi guys

I want to throw my protocol into the mix as a possible solution to this as well.

It's a very simple way how task-runners like (Nx, turborepo, you name it) could talk to non-terminating watch-processes and tell them when to rebuild, and get the results of those builds.

The idea is that we settle on one protocol, that then can be implemented by a lot of task runners and build tools. I'd really like to get your feedback on this! I hope to get the discussion started here

[DibyodyutiMondal]

I think that the true solution would something like a parallel+tree approach

We make a graph/tree of the dependencies (which is already done anyway), and then we run all the watch-mode tasks in parallel.
However, for any given task, the success/error is not broadcast to all the other tasks, but only to the tasks which depend on it, and they trigger a rebuild for those projects. This rebuild will then notify the task further on in the chain and this carries on.

It's like making a custom incremental watch mode.

[VanCoding]

@DibyodyutiMondal That's what the watch-task-protocol is all about. In your scenario, you need a standard way to tell your tasks that are running inparallel when to rebuild, and a way to know when they're finished.

[DibyodyutiMondal]

this is probably too early, but maybe we can check out if using angular signals or rxjs observables tied to child process event listeners can help implement this?

[datner]

I can't imagine why this thread is so lackadaisical, isn't this an issue that around exactly 100% of all monorepos have to handle?
The current and not that great solution is to force the running task (lets say an app) transpile/compile/build the dependent libraries for example like nextjs transpileModules.
What tasks manage to out-rank this requirement? I'm so confused how this wasn't tackled in a serious manner for 2 years..

[billiegoose]

I, too, would love a --watch mode that actually worked for a graph or tree of projects instead of hanging.

[EthanML]

Just adding on to the pile that it'd be amazing to see some sort of support for this added. Not really sure how to make a good dev setup work with my setup of a Next app being served that depends on a buildable/publishable package, otherwise.

[datner]

@EthanML the current way to go around it is to put the build steps of the dependencies as a pre-req for the dev script, so when you run dev for your next projects it waits for a build of the dependencies, while they build in watch mode (which is their dev script, assuming rollup, tsup, tsc-watch, what-have-you) It's not elegant but it works for me..

[EthanML]

@EthanML the current way to go around it is to put the build steps of the dependencies as a pre-req for the dev script, so when you run dev for your next projects it waits for a build of the dependencies, while they build in watch mode (which is their dev script, assuming rollup, tsup, tsc-watch, what-have-you) It's not elegant but it works for me..

@datner When you say pre-req do you mean by using the dependsOn option in my Next app's serve target? Because that's the thing that doesn't work for me on account of the dev task of the lib I depend on being a tsup --watch script which is long lived and thus won't "finish".

My Next app's serve target looks something like the following FWIW:

"serve": {
      "executor": "@nx/next:server",
      "defaultConfiguration": "development",
      "options": {
        "buildTarget": "demo:build",
        "dev": true
      },
      "dependsOn": [
        "@myorg/dependency-lib:dev"
      ],
      "configurations": {
        "development": {
          "buildTarget": "demo:build:development",
          "port": 3001,
          "dev": true
        },
        "production": {
          "buildTarget": "demo:build:production",
          "dev": false
        }
      }
    },

While the dependency lib in question has a package.json with the following dev script:

"dev": "tsup src --watch --clean --dts --format esm",

Is there some alternative approach here that works better as you mention?

---

EDIT: Ahh, after reading a few more times I think I see what you're suggesting now. You mean to have the lib's build (not dev) target as a dependency of my Next app. Then in addition the lib itself will also run its own independent dev process, assuming I am kicking all of this off via something like run-many -t dev. Right?

As you say it's not the most elegant but I think it solves the issue?

[datner]

@EthanML you got it, yeah 👍

[EthanML]

I think though that this brings me back to the original issue that started me on this path. That being that having a --watch process running as one of my nx run-many tasks permanently consumes one of my available parallel tasks. So right now I have two next apps to run + a few libs. I basically have to set --parallel=4 to allow the dev command to function, otherwise one of the next apps will not start. And I suppose this is also something that will have to be increased in future as similar apps/libs with the same requirements are added 😬

I guess that's the point of this post though. So again, re-iterating that it'd be great to find a true solution here 😅

[datner]

@EthanML I feel your pain. Sometimes I feel like nx is built for 2 things:

1. tutorials
2. big orgs that dedicate a team to build their nx with custom everything

anything between is extremely painful. From opaque errors to lack of support for basic features (tagging a task as persistent or long-lasting to not count it in the parallel limit????) and layers upon layers of configurations for the most trivial of things. When I worked with 100 other people on the same thing, I liked nx, didn't need to bother with most of the build process. But the last year convinced me that nx is just irrelevant or even harmful for individuals or small teams.

I migrated my stuff to turborepo and suddenly everything Just Works. Not only that, my monorepo got slimmer and less unruly (less, not completely). Not saying it's a perfect solution, it's plenty flawed. But at least I mostly focus on developing my product instead of my monorepo tooling.

So yeah, next time use turborepo if you want a monorepo that works. Or Bazel/Rush. You'll waste the same amount of time customizing them as you do nx and slim down the "have need -> use plugin -> should work -> doesn't -> customize -> rage reimplement the entire thing -> have need" cycle to "have need -> implement -> have need"

[tazo90]

I resolved app reloading using nodemon which watches app and library files.
For faster app bootstrapping, explore options like vite with esbuild/swc.

Structure:

/apps
 /api
   /src
/libs
 /shared
   /src

/apps/api/nodemon.json

{
 "watch": ["src", "../../libs/shared/src"], <--- enables app restart on changes in the shared library
 "ext": "ts",
 "delay": 2,
 "ignoreRoot": [".git"],
 "ignore": ["src/**/*.spec.ts"],
 "exec": "ts-node -r tsconfig-paths/register src/main.ts"
}

/apps/api/package.json
"start:dev": "cross-env NODE_ENV=development nodemon",