[WEB-880] [turborepo] Persistent task concurrency validation not taking into account whether package has the script defined

[shawnmcknight]

What version of Turborepo are you using?

1.8.5

What package manager are you using / does the bug impact?

pnpm

What operating system are you using?

Windows

Describe the Bug

I believe this is related to #4205. If you have a task which is designated as persistent, it will validate that concurrency limits are not breached. However, the calculation of how many parallel tasks will be executed is not taking into account that not every package in the defined filter has the script defined.

For instance, I have a turbo task dev which is flagged as persistent. This is launched as:
turbo run dev --filter=@scope/app.*...

The count of tasks gets indicated as 24 in my example, which breaches the concurrency limit of 10. The count of 24 is the total count of packages matching the filter and all of its dependencies. However, only 4 of the packages in the filter's scope actually have dev scripts defined. This same issue does not occur if a filter is not defined. Effectively, the validation of concurrency limits should verify if a package in the filter's scope actually has the script defined before erroring. Packages without the script defined should not be counted in the validation.

Expected Behavior

Since in my example only 4 dev scripts exist within the filter's scope, the turbo task should be permitted to run as it will not breach concurrency limits.

To Reproduce

• Create a workspace with 2 packages -- a and b
  • Make b a dependency of a
• Define a persistent turbo task (e.g. dev) in turbo.json
• Add a script to package a matching the persistent task's name
• Execute turbo run <task_name> --filter=a... --concurrency=1

In this example, only package a will run a script because it's the only one with the script defined. However, the count will be detected as exceeding the limit of 1 parallel task because it's using a count of all packages in scope and not the number of scripts which will be executed.

Reproduction Repo

No response

_WEB-880

_TURBO-1141

[shawnmcknight]

Actually, upon further testing, the filter isn't a necessary step here. If I have a dev script defined in a small number of workspace packages, but have more than 10 total workspace packages, I will get the error without supplying a filter as well.

[rottbers]

Running into the same with npm. In my case only 4 out of 10 workspaces have the persistent script (dev) present in their package.json but yet all 10 workspaces seems to be targeted resulting in the following error:

ERROR run failed: error preparing engine: Invalid persistent task configuration:
You have 10 persistent tasks but `turbo` is configured for concurrency of 10. Set --concurrency to at least 11

[connorjs]

I just hit this too. Current workaround: (1) Run any dependsOn steps manually (e.g. build) and (2) use npm run directly.

npm run dev --workspaces --if-present

Swap dev for the appropriate run script name.

[connorjs]

Better workaround: Downgrade turbo to 1.8.3. The bug started appearing in 1.8.4. (Assuming you do not need 1.8.4+ features.)

Also consider disabling the update notifier in this case? https://turbo.build/repo/docs/reference/command-line-reference#--no-update-notifier.

[unindented]

Experiencing the same issue on macOS, with 1.8.5-1.8.8.

[hsuabina]

+1

We had to disable the persistent flag on the root dev script because of this issue.

[trappar]

I'm also seeing issues as a result of this on 1.8.8 and it's causing confusion for people onboarding onto a monorepo project I recently built.

[mehulkar]

Hey all, thanks for reporting, we'll fix this up shortly!

[pencilcheck]

Better workaround: Downgrade turbo to 1.8.3. The bug started appearing in 1.8.4. (Assuming you do not need 1.8.4+ features.)

Also consider disabling the update notifier in this case? https://turbo.build/repo/docs/reference/command-line-reference#--no-update-notifier.

still having this issue tonight, reverting back to 1.8.3 seems to fix this issue.

[ph55]

• npx create-turbo -e kitchen-sink
• npm run dev
• Get this error:

❯ npm run dev

> dev
> turbo dev --no-cache  --continue

 ERROR  run failed: error preparing engine: Invalid persistent task configuration:
You have 10 persistent tasks but `turbo` is configured for concurrency of 10. Set --concurrency to at least 11

[hBulldozer]

Better workaround: Downgrade turbo to 1.8.3. The bug started appearing in 1.8.4. (Assuming you do not need 1.8.4+ features.)
Also consider disabling the update notifier in this case? https://turbo.build/repo/docs/reference/command-line-reference#--no-update-notifier.

still having this issue tonight, reverting back to 1.8.3 seems to fix this issue.

Same here... Cant update

[nedis-code]

same issue

yarn dev
yarn run v1.22.19
$ turbo dev --no-cache  --continue
 ERROR  run failed: error preparing engine: Invalid persistent task configuration:
You have 10 persistent tasks but `turbo` is configured for concurrency of 10. Set --concurrency to at least 11
Turbo error: error preparing engine: Invalid persistent task configuration:
You have 10 persistent tasks but `turbo` is configured for concurrency of 10. Set --concurrency to at least 11
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[Tobjoern]

I just bumped into this issue at v1.9.3.

[alikleitcr7]

Well, do what they say until resolved:

 "dev": "turbo run dev --concurrency 11" 

I guess it can be increased based on the number of projects in the workspace.

[mehulkar]

Hey all, I'm sorry I wasn't able to get to this yet. Taking pat leave for some time, but I've passed this on to the team. I've let the team know about this open issue, but if anyone is able to make a PR even if just to revert #4205, it may be worth merging until we can find a real fix. @nathanhammond is also on leave for some time, which is why you haven't heard from him in a while.

[geovanesantana]

I just got this issue in v1.9.3

yarn run v1.22.19
$ turbo dev --no-cache  --continue
 ERROR  run failed: error preparing engine: Invalid persistent task configuration:
You have 10 persistent tasks but `turbo` is configured for concurrency of 10. Set --concurrency to at least 11
Turbo error: error preparing engine: Invalid persistent task configuration:
You have 10 persistent tasks but `turbo` is configured for concurrency of 10. Set --concurrency to at least 11
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[hongfaqiu]

Same issue in v1.10.12:

pnpm dev

> turbo dev --filter=!doc --no-cache  --continue

 ERROR  run failed: error preparing engine: Invalid persistent task configuration:
You have 13 persistent tasks but `turbo` is configured for concurrency of 13. Set --concurrency to at least 14
Turbo error: error preparing engine: Invalid persistent task configuration:
You have 13 persistent tasks but `turbo` is configured for concurrency of 13. Set --concurrency to at least 14
 ELIFECYCLE  Command failed with exit code 1.

When I set the concurrency to 14, the command takes effect:

"dev": "turbo dev --filter=!doc --concurrency 14"

[Nilegfx]

I got the same error with 1.10.14.

[jpreynat]

Same here on 1.10.3 and after trying to upgrade to 1.10.15 hoping it'd been fixed.
This is really painful... 😓

[mehulkar]

Just an update, we talked about this at the team meeting and we want to fix it. PRs welcome

[ilyagru]

Hey, any updates on this one? I have 10+ packages overall but the "dev" script is defined in one of them only. I guess it shouldn't fail in this case.

"turbo": "^1.10.16"

[levino]

Is it possible to set the concurrency in turbo.json globally? To say 1000 or something? Kinda annoying to have to tell all developers to run turbo [...] --concurrency 1000.

[arlyon]

The linked PR fixes the issue. Should be released in the next minor. Thanks for the patience as we were working towards the rust port ❤️