Repository navigation
Weighted task parallelism: let a target declare how many --parallel slots it uses
#37740
muhqu
started this conversation in
Feature Requests
Replies: 2 comments 1 reply
|
Die vorgeschlagene Gewichtung trifft das Problem einer überlasteten CI-Maschine gut. Besonders bei Vitest und Browser-Tests kann ein einzelner Nx-Task intern viele Worker starten. Ein Vergleich derselben Merge-Queue mit festen Task-Gewichten und einem niedrigeren globalen --parallel-Wert wäre interessant: Wie unterscheiden sich Laufzeit und Fehlerrate? Ein leichter E2E-Check einer externen Seite wie https://revolutioncasinos.at/ könnte dabei zeigen, ob einfache Prüfungen trotz schwerer Tests zeitnah starten. |
1 reply
|
I opened a PR implementing option B (a separate |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Summary
Allow a target to declare a weight: how many of the
--parallelslots it uses while running. A target that runs its own worker pool (Vitest/Jest with many workers, Playwright, a multi-threaded bundler) could say, for example, "I use 8 slots". The scheduler would then start tasks only while the total weight of running tasks fits within--parallel.Today, every task counts as one slot, whatever it runs internally. The only built-in control is the all-or-nothing
"parallelism": false.Motivation
We run a single
nx run-many --target=cion one CI machine (36 vCPUs). The workspace has a mix of targets:On a typical PR run only the affected heavy targets run, so the machine has plenty of headroom. In the merge queue all of them run at once. Nx starts them together because each counts as one task, and the machine ends up far oversubscribed. In our case the heavy unit-test target took 65% longer, and a timing-sensitive Vitest browser test failed all its retries, blocking the merge queue. The same code passed in PR CI on the same machine type.
None of the existing controls fits:
--parallel(number or%)"parallelism": falseon heavy targetsrun-manyinvocations by weightProposal
Add an optional per-target weight, settable in
project.json,package.jsonnxconfig,targetDefaultsand by inference plugins.Option A: extend
parallelismto also accept a positive integer:Option B: a separate field, keeping
parallelismboolean:{ "targets": { "test": { "weight": 8 } } }Either way:
1(true/ unset), so existing workspaces behave exactly as before.parallelism: false), so nothing can get stuck.--parallel=<n>%/NX_PARALLEL. For example,--parallel=100%plus weights equal to each target's worker count gives "use all cores, never oversubscribe".@nx/vitestcould derive one frommaxWorkers), but that's optional and can come later.Implementation notes
The local scheduler already has the hook for this.
TasksSchedule.canBeScheduled()inpackages/nx/src/tasks-runner/tasks-schedule.tschecksparallelism === falseagainst the set of running tasks. A weighted version would track the total weight of running tasks and compare it to the parallel limit, replacing the plain task count the orchestrator uses now. Other changes needed:parallelism);parallelism: false;Out of scope for a first version: distributed execution / Nx Agents assignment. A weight could later feed agent packing, but local scheduling is useful on its own.
Related
--parallel(now supported via%)parallelismscheduling fix intasks-schedule.tsI'd be happy to implement this if the team agrees on the approach (A or B).
All reactions