--parallel Does not honor build dependencies

[VanTanev]

Describe the feature you'd like to request

In the canonical example, turbo run dev --parallel --no-cache is used for development.

However, --parallel will build dependent tasks in parallel too, even though those might need to be executed in dependency order.

Given a graph like: A -> B and A -> C

And a turborepo definition like:

    "turbo": {                                                                                                                                                                                                                  
        "pipeline": {
            "build": {
                "dependsOn": ["^build"]
            },
            "dev": {
                "dependsOn": ["build"]
                "cache": false
            }
        }
    }

Running turbo run dev --parallel --no-cache will break on the build step, because the builds of B and C depend on the build of A completing.

Describe the solution you'd like

One possible solution is to add a parameter like --parallel-depth=1 (default: Infinite), where only the dev commands are executed in parallel, but deeper dependent commands are not.

Describe alternatives you've considered

Removing build from dev: { dependsOn }, and executing 2 commands works:

turbo run build
turbo run dev --parallel --no-cache

[vladfrangu]

We've also encountered this issue in discordjs/discord.js#7175, where the packages/rest module doesn't await the building of packages/collection before starting its own, causing CI and local build errors. It seems that parallel doesn't account for the dependencies correctly (please correct me if wrong)

CI where it errors: https://github.com/discordjs/discord.js/runs/4674304974?check_suite_focus=true
After removing --parallel from all our scripts: https://github.com/discordjs/discord.js/runs/4674651684?check_suite_focus=true

Would love to help debug this further, let me know how I can help 🎉

[VanTanev]

@jaredpalmer I've created a repository to reproduce the issue: https://github.com/VanTanev/turborepo-reproduce-460

I would expect when running in parallel, the leafs are run in parallel, but they still honor their build deps;

Instead, leaf commands will execute in parallel with commands they depend on.

[IPWright83]

The docs do specifically say this ignores the dependency graph, but I'm struggling to see how this is useful?

In the meantime is there a workaround for local development where you build your dependencies with a tsc -w or similar? If you remove the parallel flag, as the processes never exit (watch mode) the build just hangs :(

[jaredpalmer]

Leafs automatically run in at max parallelism with turbo. It sounds like what you want is dev to depend on its dependencies being built? ^build

[IPWright83]

@jaredpalmer does that work when the "build" operation however is a watch and therefore never terminates? It seemed to hang when I tried it, which would make sense as if the process doesn't exit how does it know when to move on in the dependency graph?

[IPWright83]

So I just tested some more bits out... I've got a dev task defined as follows in turbo.json:

"dev": {
    "cache": false,
    "dependsOn": ["^dev"]
},

The dev task does different things:

"dev": "react-scripts start" // Apps
"dev": "tsc -w" // Libs

I believe this is exactly the same setup as the Turborepo kitchen-sink example uses.

Here's the graph for context:

If I don't use the parallel flag (turbo run dev --no-cache --continue) then running things gets stuck:

09:44:47 - Starting compilation in watch mode...
@iw/title:dev:
@iw/title:dev:
@iw/title:dev: 09:44:49 - Found 0 errors. Watching for file changes.
@iw/log:dev:
@iw/log:dev: 09:44:49 - Found 0 errors. Watching for file changes.

If I add the parallel flag back in (turbo run dev --no-cache --parallel --continue) then things run, but I get those errors during the build process (which sometimes manifest in the web application too):

app2:dev:
@iw/log:dev:
@iw/log:dev: 09:57:44 - Found 0 errors. Watching for file changes.
@iw/title:dev:
@iw/title:dev: 09:57:44 - Found 0 errors. Watching for file changes.
@iw/button:dev: src/components/Button/Button.tsx(2,21): error TS2307: Cannot find module '@iw/log' or its corresponding type declarations.
@iw/button:dev:
@iw/button:dev: 09:57:45 - Found 1 error. Watching for file changes.
...
app:dev: TS2307: Cannot find module '@iw/button' or its corresponding type declarations.
app:dev: 2 | import logo from './logo.svg';
app:dev: 3 | import './App.css';
app:dev: > 4 | import { Button } from '@iw/button';
app:dev: | ^^^^^^^^^^^^
app:dev: 5 | import { Title } from '@iw/title';
app:dev: 6 |
app:dev: 7 | function App() {

(Note that this app is actually running correctly, and hot-reloading correctly. But always displays this error over the top).

[jaredpalmer]

Sounds like what you want to do isn't the parallel flag, but rather increase the allowed concurrency so that you are able to run through all build processes first, and then accumulate long running dev processes without running out of the 10 default go-routines.

So have dev depend on ^build, but then run with turbo run dev --concurrency=50

[IPWright83]

@jaredpalmer I just tried that, and unfortunately it doesn't work either (though I was using ^dev to keep the watch mode). The concurrency is going to max out at 2 at the start (due to the lowest level dependencies) and because they never finish it can't start the next tasks, regardless of the concurrency limit.

[IPWright83]

You're welcome to use https://github.com/IPWright83/pnpm-monorepo as another example if needed. Requires pnpm, but using pnpm run dev_ideal illustrates the issue. I've added a dev script that also does a build to workaround it for now... turbo run build --scope=@iw* --no-deps && turbo run dev --no-cache --parallel --continue

[weyert]

Yeah, would be great to have this fixed. Especially, if you want to execute a collection of applications during development, e.g 10-15 watchers

[robertcepa]

We have a similar use case, we have a monorepo of libraries (currently 9) and need to run all of them in watch mode. Libraries depend on each other and their dev process is defined as tsc --watch & esbuild ... --watch. tsc only emits declaration files, and esbuild emits runtime code. Both watchers include initial build step.

Our dev task in turbo.json is defined as:

"dev": {
   "cache": false
}

and we run it as yarn turbo run dev --parallel.

Our watchers do not hang, but once all watchers are started, package A that depends on package B doesn't not seem to be aware of changes in package B, even though the declaration file definitely exists on the filesystem.

packageA: src/file.ts: error TS7016: Could not find a declaration file for module 'packageB'. '/Users/users/monorepo/packages/packageB/lib/index.js' implicitly has an 'any' type.

[daividh]

Since this feature is IMHO a very basic need, I don't understand how we can use Turborepo dev mode when some packages relies on others. Did someone found any workaround on this ?

[VanTanev]

My understanding is that this currently works /fine/.

You must not pass --parallel flag, because that breaks deps, but by default turbo runs with 10 concurrent tasks, and if you have tasks that "hang" and never exit like dev, those will work as long as they fit within the allotted concurrency slots.

[daividh]

Ow ! I didn't understand the same thing as you. I thought that Turborepo was "waiting" for jobs to end to start the dependents one.
Since dev is a non ending tasks, dependents tasks are not executed. Any thought ?

EDIT : without the parallel mode

[VanTanev]

dev cannot depend on a process that never exits, but it can depend on eg build of another package

[IPWright83]

I fixed by doing a build first (that does exit) followed by a build-watch. Then you know all the dependencies should be ready

[luwes]

this would still be a cool feature to have, pseudo parallel by topological order maybe with a configurable timeout when the dependencies dev commands are seen as "completed", or some other way to detect it's just in idle watching mode.

in some use cases this beats first running the build command to ensure the fresh dependencies are built.

[fengkx]

Since dev is a non ending tasks, dependents tasks are not executed. Any thought ?

I also realized this, So I try to let the dev task to depend on ^build as the like whats said here.
#460 (comment)

But if you have a dependency graph like A -> B. For example A is a typescript package, its build script is tsc.
If you are developing A and B at the same time which is a common case in monorepo. Once A has a type error during development, tsc will failed and exit then the whole turbo run dev exited.

Nx seems planing to support a long runing task as dependency. nrwl/nx#10706

It would be nice if turborepo also supported this.

[codepunkt]

I have watch modes with incremental rebuilds for all packages as their dev task, one-off optimized builds as their build task.

Because I want to run all dev tasks in parallel and the incremental watch builds output to the same directories the one-off optimized builds would also output, I need to have all packages built before I start their dev modes. In the repo's root, I have turbo run dev --continue as my dev task at the root and this in my turbo.json:

"dev": {
  "dependsOn": ["^build"],
  "cache": false
}

This starts all builds and after they finish, starts all watch modes. So far, so good.

However, if application A depends on package B which in turn depends on package C, what I'd like to see when I save a file in package C is that C does an incremental rebuild, followed by an incremental rebuild of B, followed of an incremental rebuild of A, followed by a browser refresh (which is out of turborepo scope).

What happens instead is an incremental rebuild of package C. That's it. Is there any way to get what I want?

[IPWright83]

@codepunkt what tech are you using here, as this is working for me...

If I update @iw/log then @iw/title builds and app refreshes correctly, logging the new output. This is with create-react-app's all the way down.

[georeith]

How do you do this with filters? I have a monorepo set up with two different folders:

• packages (typescript compiled libraries)
• apps (individual sites/servers that depend on packages)

Every package and app has a dev script which in the case of packages runs a watched build, in the case of apps runs a dev server or whatever.

I want to work on one app at a time, so I want to run turbo run dev --filter=app. The problem is adding the filter means it won't run dev in my packages. I cant tell turborepo that dev depends on ^dev because dev commands never exit and it will never run. I have tried turbo run dev --filter=...app but that does not run the dev scripts in my dependent packages either.

Is there really no inbuilt way of doing this without running ALL the dev scripts in my monorepo?

Edit: I had the ... round the wrong way, turbo run dev --filter=app... does the trick.

[rwieruch]

I ran into the same issue where package A depends on the build (and its type definitions) of package B. So I followed the advice given here and used:

"dev": {
  "dependsOn": ["^build"],
  "cache": false
}

and the script without the parallel flag:

"dev": "turbo run dev --continue",

So far everything seems to start and update accordingly.

[IPWright83]

@rwieruch I think you might end up with a race (if dependencies don't build in the correct order). That's why I've opted to do a pre-build before hand.

[steida]

So if you ended here as me, let's reiterate how Turborepo dev works (and how not) because it's confusing.

The workflow a developer would expect is to install modules and then run the dev script, but that's now how Turborepo works. A dev task needs two things: generate code and run watchers. From Turborepo docs:

--parallel
Default false. Run commands in parallel across workspaces and ignore the dependency graph. This is useful for developing with live reloading.

It's confusing because Turborepo runs everything parallel, if possible. This configuration should be named --ignoreDependecyGraph

--parallel is required to run watchers, but before that, we need to build code in order so we can't use --parallel. It's a chicken-egg problem.

I bet many developers just run a dev task, then they see some errors, then they will rerun it, and then it magically works. That's not ideal.

Turborepo suggests that we should use dependsOn https://turborepo.org/docs/handbook/dev#running-tasks-before-dev, but it solves nothing because --parallel is required for watchers.

As I see it, there have to be two tasks run separately from the command line. Something like buildForDev and then dev.

Because my build does not do anything special, the recommended way (for me) is always to run build before dev or start.

[anthonyshew]

The title and description here describes the intended behavior of --parallel.

What's being described here is more like several feature requests put together. Something like --watch and some health checking across tasks. We have an issue that we're actively working on in #986 and have ideas about depending on long-running tasks.

I'm going to close this issue since it's describing the intended behavior - but rest assured that we hear the pains!