[RFC] Decouple Turbo tasks from package.json scripts And allow more automation

[zaripych]

Describe the feature you'd like to request

I've been thinking about this for some time now. The idea keeps nagging me. Imagine - I have a dts task that I can run for every package in my monorepo, but that task doesn't have any value for the end user (read developer in a team using the monorepo) by itself - meaning user will never run that task manually via turbo run or pnpm run. The task is needed to run another dts-bundle task - essentially dts produces an intermediate artefact that allows dts-bundle that depend on it to succeed without errors. Now the problem is that dts-bundle needs to run dts for all dependencies.

This seem to work nicely:

  "pipeline": {
    "dts": {
      "inputs": [...],
      "outputs": [".tsc-out"]
    },
    "dts-bundle": {
      "dependsOn": ["^dts"],
      "inputs": [...],
      "outputs": ["dist/**/*.d.ts"]
    },
  }

But little issues creep up - mostly scalability:

• Every package has to have that dts task declared in package.json scripts - it probably wouldn't make sense to have to copy-paste it everywhere since the task is basically single script that can work in every package sub-folder and produces output in that package sub-folder, but has no input other than source code and doesn't need any configuration on per package basis
• The dts task must be run for every dependency of a package where we about to run dts-bundle, but a developer might forget to add a script to package.json for one of those dependencies - when adding that dts script they might also make a mistake and pass wrong parameters or things like that
• When a developer decides that they want to start using dts-bundle for their package they have to go and check every dependency and whether that dependency has dts task declared. Hopefully it's very easy change and a quick PR away, but those packages could be owned by other teams.

In addition to that it's also possible that inputs of some other build task in one package is different to another package's inputs. But the fact that those differences should be specified in the root of the monorepo in a single file is kind of limiting. If we look at monorepos with huge number of packages I feel single turbo.json might become a bottleneck (with some merge issues maybe even?). Sometimes a script can determine those inputs/outputs programmatically and can operate in the entire monorepo for any sub-package. E.g. I imagine a script could determine all the dependent *.ts files imported and build a graph of files that it depends on (inputs) - and that would probably be more efficient than using globs like src/**/*.ts that could hit hundreds of files.

Another script could determine which dependencies need to have dts task run and which don't. There is generally a lot of things that can and should be automated and turbo doesn't provide too much here. The CLI could be considered as API and turbo.json can be generated - but it makes it harder to use hybrid approach - use ability of the turbo to cache and also allow devs to use turbo.json without having to deal with a script.

Describe the solution you'd like

1. Allow specifying commands right in the turbo.json, example:

  "pipeline": {
    "dts": {
      "command": "node ./monorepo-scripts/generateDts.js",
      // ^-- no need to copy-pasta in every package
      "inputs": [...],
      "outputs": [".tsc-out"]
    },
  }

turbo would assume the command exists in every package as if it was added to every package's scripts config.

2. Allow passing location of a custom turbo.dts.json file OR preferably - allow it to be passed as stdin. If location of the turbo.json implies root of the monorepo we could also add extra parameter or assume it's cwd. This will make turbo super powerful tool that would allow caching any task - whether it can be described by turbo.json manually or generated by a script.

Both of the above seem like a simple change on the surface. But there must be some caching implications. E.g. we would need to store hash of the stdin config somewhere which could serve as a cache key for the rest of the data associated with the run ...

Describe alternatives you've considered

Generating turbo.json is possible, but writing scripts to every package.json - seem like a less desired path as it's often modified manually.

[jlarmstrongiv]

@zaripych in the meantime, here’s a workaround for you.

In my projects, I have been duplicating package scripts just as you described. But, I try to abstract away all of the details. I have many package.json files that look like:

{
  "scripts": {
    "turbo:build": "../../../bin/nextjs/build",
    "turbo:clean": "../../../bin/clean/all",
    "turbo:dev": "../../../bin/nextjs/dev",
    "turbo:test:ci": "../../../bin/jest/ci",
  }
}

The only thing is, most packages have 40+ scripts, not 4.

I find that abstracting the scripts into a common bin folder helps a lot (I also like prefixing shared scripts with turbo:).

As my monorepo has grown, I have different types of packages. Some for nodejs, some for reactjs, some for apps, some for configs. There’s only a few scripts that are actually run in every single repo due to turbo disallowing circular dependencies. This abstraction allows me to group sub-packages and sub-apps like packages/reactjs/* or apps/nextjs/*. Plus, I can exclude config packages altogether by just not including the script. Lastly, it still lets me run scripts very easily in just the package or two that I’m working on via npm run <script>, without having to specify filter each time in the package root.

As for ensuring every package has a particular script, I find that using a template generator is awesome (see plop and hygen). Or, writing a short script in bash or the language of your choice is straightforward.

Anyway, the developer experience of turbo is interesting to explore. Reminds me of:

• Execute turbo from anywhere in a monorepo #1412
• Execute Arbitrary package.json Scripts #1254
• Zero-config #749

While I prefer convention over configuration too, it’s important not to become overspecialized and ensure exceptions can be specified.

[zaripych]

@jlarmstrongiv Actually, already doing that. I have a dev package that is mounted as devDependencies to the root of monorepo and has bin's exposed and available to every sub-package. So no need to "../../thing" I just add it as "thing". It's all same shared scripts for every package that use same bespoke monorepo configs/conventions/tools.

The only thing is, most packages have 40+ scripts, not 4.

That's a lot.

Plus, I can exclude config packages altogether by just not including the script. Lastly, it still lets me run scripts very easily in just the package or two that I’m working on via npm run <script>, without having to specify filter each time in the package root.

Ok. I'm talking about different tasks that do not belong in scripts of a package but which I still want to cache and benefit from turbo. Developers won't ever care to run them manually.

As for ensuring every package has a particular script

Agreed. Code generation is ok as long as you do it within a monorepo and can regenerate it a little differently at any time 👍
Meaning results of the previous generation can be thrown away and changed at any time to something else. This becomes hard and dangerous for files which are also hand-edited or when you need to take backward compatibility into consideration. E.g. you can go and overwrite somebody else's script in package.json ... so - I'm doing code generation but making sure to avoid mixing hand-edited code and generated code.

#1412

FYI - I was annoyed and fixed that by wrapping standard turbo bin with my own. Now I can run it in sub-packages.

Anyway, the developer experience of turbo is interesting to explore.

It's still a little raw and a bit buggy for now. (E.g. at the moment running a canary because previous version was hitting the cache too much) I'm very enthusiastic though to see how this will evolve further and really considering using it in more projects at work.

it’s important not to become overspecialized

Agree. Customisation is needed at times, but those also might mean that turbo.json needs changing. If we want to over-blow this - imagine you need to customise every script - so multiply number of packages you have (I've seen people talking 200+ in some threads here) by the number of scripts you have (40+ ? :D ) and put it all into turbo.json pipeline ... will end-up containing 1k of task configs with entries like package1#build, ... package200#build ... fun 😅

[jlarmstrongiv]

Actually, already doing that. I have a dev package that is mounted as devDependencies to the root of monorepo and has bin's exposed and available to every sub-package. So no need to "../../thing" I just add it as "thing". It's all same shared scripts for every package that use same bespoke monorepo configs/conventions/tools.

FYI - I was annoyed and fixed that by wrapping standard turbo bin with my own. Now I can run it in sub-packages.

Very cool! I like those workarounds.

I wonder if it should be a broader discussion on turbo UX. Discussion points include:

• avoid defining scripts in the turbo.json (Zero-config #749 and Execute Arbitrary package.json Scripts #1254)
• avoid defining duplicate scripts in the package.json (Decouple Turbo tasks from package.json scripts And allow more automation #1495)
• ability to run common turbo tasks anywhere (Execute turbo from anywhere in a monorepo #1412)
• ability to run a common turbo task in a package without defining it in scripts
• configure the cache and exceptions to particular commands (https://turborepo.org/docs/core-concepts/caching and https://turborepo.org/docs/core-concepts/filtering and package#build)
  • group common turbo tasks by subfolder (like apps/nextjs/* or packages/reactjs/*)
  • group common turbo tasks by npm name prefix (@apps/*)
• keep turbo fast by avoiding js/ts config files Request for having a turbo.js or/and turbo.ts to configure turbo #899
• support scripts from other languages Add engines.turbo Validation #1361 (comment)

How would that look? Curious what your thoughts are on it. There’s a lot to think about and there are tradeoffs in choosing what to support too.

---

Ok. I'm talking about different tasks that do not belong in scripts of a package but which I still want to cache and benefit from turbo. Developers won't ever care to run them manually.

I think there are use cases where devs would want to run some common scripts manually in a package. Examples https://www.npmjs.com/package/sort-package-json and https://www.npmjs.com/package/typesync

[dsilvasc]

As a shorter term solution (until turbo supports additional config for this in turbo.json), when turbo doesn't find a task in a package's package.json scripts, can it then check if a command with that name exists in the context of that package?

If making it possible to declare this for all packages in turbo.json is far away in the turbo roadmap, then maybe a quicker change might be for turbo to do something similar to yarn bin $TASK_NAME within each package to discover commands provided by devDependencies for tasks when corresponding scripts are omitted.

For example, suppose your turbo.json defines a test task that depends on test-unit and test-ui, but you don't want to repeat this for every package.json:

"scripts": {
  "test": "true",
  "test-unit": "jest --passWithNoTests",
  "test-ui": "our-custom-ui-testing-script",
  ...
}

Today you can already define a package like @zaripych mentioned providing executables with the same names as those scripts/tasks, so your package.json files become:

"scripts": {
  "test": "test",
  "test-unit": "test-unit",
  "test-ui": "test-ui",
  ...
}

Running yarn bin test-ui yields $PWD/node_modules/.bin/test-ui, or $REPO_ROOT/node_modules.bin/test-ui (when hoisted), or a virtual path (when using yarn berry/v3 pnp), or some other place. Either way, turbo should be able to use that, since yarn already does: you can run yarn test-ui in the example above without the entry in package scripts.

[dobesv]

Another use case to throw in here is in our project in each package.json I have a script "build": "yarn build:node && yarn build:browser && yarn build:types".

It would be nice if I could tell turbo that build depends on build:node, build:browser, and build:types but that it shouldn't actually run build.

Currently I just use a different fake name for the build task for each package, _build.

[connorjs]

../../../bin/some-script

I wanted to share a new approach I am using for these cases: bin scripts.

Turbo already recommends having “config” packages (example: eslint-config). I then have the root package depend on the config as a dev dependency. Thus, instead of having a root bin directory and using ../../ to access it, I have the config package publish a bin script itself.

After some self-bike-shedding, I arrived at *x naming convention: something simple (one extra character), somewhat valid naming (x for eXtended), and still showcases the underlying script (which I found helpful, because I have multiple linter tools).

Then, in each workspace/sub-package I can replace eslint <args> or ../../bin/lint <args> with eslintx <args>.

---

Code example.

• eslint-config’s package.json

  {
  	// normal package.json stuff...
  	"directories": { "bin": "bin" }
  }

• eslint-config’s bin/eslintx (swap the command invocation with whatever you want)

  #!/bin/sh
  eslint -f pretty "$@"

• some other package’s package.json

  {
  	// normal package.json stuff...
  	"scripts": {
  		// other scripts...
  		// before
  		"lint": "eslint -f pretty ."
  		// after
  		"lint": "eslintx ."
  	}
  }