Caching codegen tasks whose inputs come from a *remote* source (git SHA / introspection)?

[bitttttten]

Goals

tl;dr we have a task that is cacheable, but the inputs come from a remote source.

Let's say.. an async input. In the context of this issue, I'm making a "codegen" task that can depend on a remote graphql schema file/introspection query.. or with buildbuf or protobuf, you can point your codegen to run against a "git_repo" that can change. both of these are async inputs, and so our codegen task cannot be cached as turbo doesn't know about this "remote input". If it did, this our codegen could be broken up into 2 steps: 1. fetch the remote input (i.e. git sha). 2 feet git sha into codegen task as an input - since our codegen, if run against the same git sha, produces the same output.

Goals:

1. Let a codegen task (that build depends on) be a cache hit when its real input, an external/remote source (e.g. a schema service queried via introspection, or a separate git repository of definitions), has not changed without every machine having to run the slow/rate-limited generator just to discover that nothing changed

2. Allow a small, fast "resolve the identifier" step (e.g. git ls-remote for the default-branch SHA, or a cheap introspection check) to run inside the Turbo graph and have its result become the cache key for the expensive codegen task, so turbo run build "just works" and caches identically on developer machines and in CI.

3. Have this remote-derived input compose correctly with turbo query affectedTasks, so it invalidates codegen/build` when the remote actually changes, but does not spuriously mark unrelated packages as affected.

Non-goals

1. Changing how Turbo hashes local file inputs.. local source files already work well.. this is only about inputs whose source of truth lives "outside the working tree" (I consider them as "remote inputs").

2. Prescribing the transport for the resolved identifier (file vs. env var vs. anything else).. I don't mind which mechanism, only that codegen's cache key reflects it. I could write this to a file? I could export it as an env var?

3. Caching the remote content itself (we're not asking Turbo to mirror a remote repo or schema).. a cheap identifier such as a git SHA or response hash is sufficient. It's up to me to make this async lookup as fast or as slow as I want to.

Background

We have a codegen task that build depends on:

{
  "tasks": {
    "codegen": { "outputs": ["src/gen/**"] },
    "build": { "dependsOn": ["codegen", "^codegen", "^build"], "outputs": ["dist/**"] }
  }
}

The catch: part of codegen's input doesn't live in our repo. It's derived from external/remote sources that change independently of our working tree. So for example, if our GraphQL deploy with a new schema - we want to bust the codegen task (imagine we are using GraphQL Code Generator).

• One codegen step runs a GraphQL introspection query against a schema service and generates types/hooks from the result. This time, it can download a graphql-schema.json file - this is fast! Or even it can hash the introspection query and export that to an env var or a .graphql-schema-hash file.
• Another example of codegen, is a protobuf plugin using git_repo.. it consumes a separate git repository (protobuf/schema definitions) and generates code from it. Again, if the remote schemas repo changes, we want our codegen busted.

The natural cache key for "did the remote change?" is cheap to obtain.. the git SHA of the remote's default branch, or a hash of the introspection response. What we want is simply: a developer (or CI) runs turbo run build (which triggers codegen), and codegen is a cache hit whenever the remote source is unchanged.. without paying the full generation cost just to learn that. For example, buildbuf codegen can take 1 to 2 mins every time, so this is a real opportunity for caching.

Current alternative / prior art (our workaround): the identifier has to be known before Turbo hashes codegen's inputs, but it's only produced by running something.. a chicken-and-egg between "task that computes the key" and "task that's keyed by it." Today we resolve the SHA / run introspection outside the Turbo graph in a pre-step (a shell script that writes a generated identifier file), then feed that file in as an input. It works, but:

• It's CI-shaped.. local runs either skip it (no codegen cache hits locally) or re-implement the dance, so the local experience is inconsistent.
• It splits one logical operation ("generate code from the remote") across a bootstrap script and a Turbo task, exactly the kind of glue Turbo otherwise lets us delete.
• We must hand-verify it composes with turbo query affectedTasks, which we use to gate which apps deploy.
• We must opt out of caching if you run codegen locally, so all local codegen scripts are slow. Otherwise we need to force developers into running a special script to run codegen, but devs can easily just run turbo run codegen themselves without the setup and now we have a) a cache HIT as they may have not run the pre-step recently which leads to b) cache poisoning

We also noticed $TURBO_JIT$ (deferred input hashing, v2.9.17 / #13045) and suspect it may be the intended primitive for "compute the input, then hash it," but we're unsure of the intended shape and how it interacts with the affected graph.

Proposal

Ideally we could express a fast producer step whose output becomes the input hash of the slow consumer task, while staying cacheable:

{
  "tasks": {
    // fast: a few-hundred-ms `git ls-remote` / cheap introspection HEAD check
    "codegen:setup": { /* produces the remote identifier */ },

    // slow/expensive: only runs when the identifier changes
    "codegen": {
      "dependsOn": ["codegen:setup"],
      "inputs": [ /* the identifier produced above */ ],
      "outputs": ["src/gen/**"]
    },

    "build": { "dependsOn": ["codegen", "^codegen", "^build"] }
  }
}

The above syntax is "todays turbo" - I have no idea how to propose this as part of a new schema. Maybe "async": true? { "tasks": { "codegen:setup": { "async": true, "outputs": [".codegen-hash"] } } }. I'm not sure I'm a fan, what if the output was an env var?

1. Is there a blessed pattern for "an input that must be resolved by running something first", a producer task whose result becomes a downstream task's cache key, that stays cacheable? Maybe it exists already?
2. Is $TURBO_JIT$ intended to cover exactly this "compute the input, then hash it" case? If so, what's the recommended shape for a producer → consumer chain, and does it compose with $TURBO_ROOT$?
3. How should any such pattern interact with turbo query affectedTasks so that a remote-derived input correctly invalidates the chain when the remote changes, without marking unrelated packages as affected?
4. Would "remote source as a first-class input" (a git ref / URL response) be something Turbo would consider supporting natively, rather than each repo reinventing a bootstrap script?

Environment

• Turborepo 2.9.17
• pnpm workspaces monorepo, build → codegen dependency
• Codegen sources: a GraphQL schema service (introspection) and an external git repo (protobuf)