Per-package codegen with a root prepare task: 9 of 23 task hashes change between two runs of the identical commit while every reported hash ingredient is identical

[bitttttten]

Summary

We run per-package codegen (codegen:buf, codegen:gql) in a large monorepo, scoped so that
building one app only generates for that app and the packages it actually uses. That shape works,
with one exception: on Vercel, a subset of codegen:buf task hashes changes between two
runs of the identical commit, while every hash ingredient the run summary reports for those
tasks is byte-identical.

Two questions, one about the layout and one about the hashes:

1. Is the layout below the intended way to model "generated code derives from an external schema
   repository, per package"?
2. When a task's hash changes but its entire run-summary entry (inputs and their hashes,
   resolvedTaskDefinition, environmentVariables, hashOfExternalDependencies, dependencies,
   framework, directory, command) is identical, what else feeds the hash, and how do we
   inspect it?

All names below are anonymized. Structure, counts, task config and hashes are real.

Environment

• Turborepo 2.10.6
• pnpm 10.24.0, Node 24.13.0
• ~16 apps, ~160 packages; ~30 of them define codegen:buf
• Remote cache enabled; each app deploys as its own project from one repository
• envMode: strict

What we are trying to do

Generated RPC clients derive from a separate schema repository (protobuf), pinned by commit.
Each package that consumes protos owns a buf.gen.ts naming the proto packages it generates from.
Building apps/a should run codegen:buf for apps/a plus the packages in its dependency
closure, and nothing else.

One root task fetches the schema repo once per run and writes a per-package input pin
(.codegen-schemas-inputs-hash) next to each buf.gen.ts. That pin is a content hash of only the
protos that package generates from, plus their transitive proto import closure. Each package's
codegen:buf then declares its own pin as a JIT input, so a schema commit only invalidates the
packages whose protos actually changed:

{
  "tasks": {
    "//#codegen:prepare:buf": {
      "cache": false,
      "inputs": ["scripts/prepare-codegen-inputs.mjs"],
      "passThroughEnv": ["CI", "SCHEMA_REPO_HASH", "SCHEMA_GIT_URL", "GIT_SSH_COMMAND", "HOME"]
    },
    "codegen:buf": {
      "dependsOn": ["^codegen:buf", "//#codegen:prepare:buf"],
      "env": ["!VITE_*", "!NEXT_PUBLIC_*"],
      "passThroughEnv": ["SCHEMA_GIT_URL", "GIT_SSH_COMMAND"],
      "inputs": [
        "./buf.gen.ts",
        "$TURBO_ROOT$/packages/codegen-tool/bin/**",
        "$TURBO_ROOT$/packages/codegen-tool/plugins/**",
        "$TURBO_ROOT$/packages/codegen-tool/src/shared-packages.ts",
        "$TURBO_ROOT$/packages/codegen-tool/src/prune-rewrite.ts",
        { "mode": "jit", "globs": ["./.codegen-schemas-inputs-hash"] }
      ],
      "outputs": ["./src/gen/service/**", "./gen/service/**"]
    },
    "codegen": {
      "dependsOn": ["codegen:gql", "codegen:buf", "codegen:translations", "^codegen"],
      "cache": false
    }
  }
}

codegen:gql is the same shape with its own root prepare task. Builds run
turbo run build --filter=<app>, and build depends on codegen.

The pin is deliberately fail-safe: if a proto reference cannot be resolved, or the schema fetch
fails, the package falls back to the whole-repo schema SHA — over-invalidates, never stale.

If there is a more idiomatic way to express "external input, scoped per package", we would rather
adopt it than keep this.

The hash observation

Deployment B is a redeploy of deployment A: same commit, same lockfile, same schema pin, no
input file differs. In each build, after the primary codegen:buf graph finished, we ran a nested
turbo run codegen:buf --filter=<app> --summarize and logged each task's hash, its per-input file
hashes, and the global cache inputs.

Of 23 tasks in that graph, 14 hash identically across A and B and 9 change:

task	A	B
packages/p1#codegen:buf	0f1c828f98793f4f (HIT)	0c25d63aaa8e96ef (MISS)
packages/p2#codegen:buf	8c1aa79318b64687 (HIT)	812a118e917198a0 (MISS)
packages/p3#codegen:buf	e861d9a04eb49502 (HIT)	4731177494c1c619 (MISS)
packages/p4#codegen:buf	e57190a7ca97d29a (HIT)	ea821254525b1e85 (MISS)
packages/p5#codegen:buf	e071a679d31d3147 (HIT)	984acfa873b845c5 (MISS)
packages/p6#codegen:buf	d23ed1bdbf8c39d0 (HIT)	cbeef30e7dbefe34 (MISS)
packages/p7#codegen:buf	3d520244a9fd4b9f (HIT)	62884172f1232d0c (MISS)
packages/p8#codegen:buf	4bfad31c74d982b9 (HIT)	0c2f47fd68d7990d (MISS)
apps/a#codegen:buf	ecc5431972f521d4 (HIT)	c2b94805fe00f92f (MISS)

Only three of those nine are roots: p1, p2 and p3 have no dependency whose hash also moved.
The other six are downstream, which is expected once a dependency hash moves.

For the roots, comparing the full summary entry field by field between A and B, the only fields
that differ are hash and cache. Identical: every declared input path and its hash (11 files),
resolvedTaskDefinition, environmentVariables (configured: [], inferred: [],
passthrough: []), hashOfExternalDependencies, dependencies, dependents, command,
directory, framework, outputs, envMode, cliArguments.

globalCacheInputs is identical too, except for the hashed values of three pass-through variables
(VERCEL_DEPLOYMENT_ID, VERCEL_URL, VERCEL_OIDC_TOKEN) — which by definition should not enter
a cache key, and cannot be the cause anyway, since a global change would move all 23 tasks rather
than 9.

Within a single build, two consecutive nested --summarize runs hash all 23 tasks identically, so
this is variance between runs, not state mutating as a run proceeds. Locally, across two
different schema pins and across three different --filter roots, all 23 hashes are stable.

What we ruled out

• JIT input rewrite by the nested run. The nested run re-runs the cache: false prepare task.
  We snapshot every pin file before it and abort the comparison if any content changed; none did.
• Schema pin. Both builds resolve the same schema commit, and the pins are byte-identical.
• Lockfile / external deps. hashOfExternalDependencies is identical per task, and identical
  globally.
• A single unstable transit node. Each root's dependencies list contains many
  <NONEXISTENT> nodes from the ^codegen:buf edge (15–27 per root; 76 for the app task). We
  looked for a transit node shared by all three roots and absent from every stable task's
  dependency set, which would localize the movement: there are 9 shared transit nodes and all 9
  also appear under at least one task whose hash did not move. Also, the task with the second
  largest transit-node count in the graph (42) is stable. So neither "has transit nodes" nor "has
  many transit nodes" predicts the movement.

Questions

1. Is a run-summary task entry exhaustive with respect to hash ingredients? If it is, two
   entries that are byte-identical apart from hash should be impossible. If it is not, which
   ingredients are omitted?
2. How do we inspect the hash of a <NONEXISTENT> dependency node? Those nodes appear in
   dependencies but get no entry of their own, and a parent hash includes its dependency hashes,
   so today they are the one ingredient we cannot see or diff. Is there a flag or log level that
   prints them?
3. Does a cache: false root task contribute anything to a dependent's hash, and if so, is
   that contribution visible anywhere?
4. Can --summarize report the resolved file set and hashes for mode: "jit" inputs as
   evaluated at execution time? --dry=json necessarily defers them, which leaves exactly this
   class of problem un-diffable without the nested-run workaround above.
5. For the layout in the first half: is a root prepare task writing per-package pins the intended
   pattern for an external schema dependency, or is there a first-class mechanism we are missing?

Happy to extend the public repro at
https://github.com/bitttttten/turborepo-vercel-broad-dependency-cache-repro to whatever shape is
most useful — it already covers the ^task / <NONEXISTENT> graph and JIT inputs, and previously
did not reproduce a related invalidation, which is why this is a Help discussion and not an
issue. A previous discussion (#13590) covered the ^task expansion question and is closed; this is
the follow-up with the identical-commit redeploy evidence.

[bitttttten]

Follow-up: a stronger reproduction — identical git tree, different task hash

Since posting, I ran a cleaner control that removes the last bit of ambiguity in the original report. Previously I could only show that every ingredient Turborepo reports in the run summary was identical. Now I can show that the git tree object is identical and the task hash still changes.

Setup. Three consecutive Vercel preview deployments of the same branch, same lockfile, same turbo version, same remote cache:

Run	Commit	Tree object	Notes
A	991f6b1	40f194e2e0b90462842c54ca8789b4c5da7b4f3b	warms the cache
B	9b2f67e9	(adds one comment line to a .tsx file that is not in the task's inputs)	control
C	0369b19a	40f194e2e0b90462842c54ca8789b4c5da7b4f3b	commit B reverted, so byte-identical tree to run A

git diff 991f6b1 0369b19a is empty and git rev-parse <c>^{tree} returns the same object for runs A and C.

Result for the 10 codegen:buf tasks in this project's graph:

• The 9 library-package tasks produced the same hash in all three runs and were restored from cache in runs B and C. Those same 9 hashes also reproduce exactly on my laptop.
• The single application-package task produced three different hashes: a3d988aa95078761 → 4d8cd7dfcda57615 → 850fee0a52394eb8.

So runs A and C built the same tree object, with the same dependency-task hashes (verified in the logs — every upstream task reports the identical hash), and the app task still computed a third distinct hash and missed the cache.

The task hashed unstably is the only one in the graph with an inferred framework. From --summarize, the 9 stable tasks report framework: "" with configured: [] and inferred: []; the unstable one reports framework: "nextjs", also with configured: [] / inferred: [] (the task declares "env": ["!VITE_*", "!NEXT_PUBLIC_*"]). That pattern held in the original 9-mover report too: the movers were the deepest packages plus the app task.

It does not reproduce locally. Applying the exact same source edit as run B locally leaves all 11 hashes untouched, including the app task, and a second local run restores 10/10 from cache. The instability only appears between Vercel deployments.

Relevant task definition (inputs are declared explicitly, and the only jit input is a file this project's prepare step writes deterministically):

{
  "dependsOn": ["^codegen:buf", "//#codegen:prepare:buf"],
  "env": ["!VITE_*", "!NEXT_PUBLIC_*"],
  "inputs": [
    "./buf.gen.ts",
    "$TURBO_ROOT$/packages/rpc-codegen/bin/**",
    "$TURBO_ROOT$/packages/rpc-codegen/plugins/**",
    "$TURBO_ROOT$/packages/rpc-codegen/src/shared-packages.ts",
    "$TURBO_ROOT$/packages/rpc-codegen/src/prune-rewrite.ts",
    "$TURBO_ROOT$/packages/rpc-codegen/git-repo.ts",
    { "mode": "jit", "globs": ["./.codegen-schemas-inputs-hash"] }
  ],
  "outputs": ["./src/gen/service/**", "./gen/service/**"]
}

If framework inference contributes anything to the hash beyond the configured/inferred env names that --summarize reports, that would explain both this and the original report — and it would mean the run summary is not a complete record of the hash inputs, which is what made this so hard to diagnose. Happy to run any additional instrumentation on Vercel if that would help narrow it down.

[bitttttten]

Follow-up: a structurally identical task type in the same runs does not move

Earlier in this thread I reported 9 of 23 codegen:buf task hashes changing between runs of an identical tree, with every ingredient in the run summary identical. I now have a second task type in the same runs that stays stable, which rules out several explanations.

Setup

Two Vercel builds of the same branch, turbo 2.10.6:

• Run A — commit c0c87ec
• Run B — a redeploy of commit 402c60a

The two commits differ by exactly one added comment line in a .tsx file, which is not an input to either task type below.

Both runs resolved identical values for the two external inputs codegen depends on (logged by our own prepare tasks):

• graphql-schema.json sha256 a0b633b6… in both runs
• schemas git repo HEAD 9e528dbf… in both runs

Run B was a redeploy, which forces execution, so every task logged cache bypass. The comparison below is therefore on task hashes, not on hit/miss.

The two task types are structurally identical

"codegen:buf": {
  "dependsOn": ["^codegen:buf", "//#codegen:prepare:buf"],
  "env": ["!VITE_*", "!NEXT_PUBLIC_*"],
  "inputs": [ /* … static globs … */
    { "mode": "jit", "globs": ["./.codegen-schemas-inputs-hash"] } ]
},
"codegen:gql": {
  "dependsOn": ["^codegen:gql", "//#codegen:prepare:gql"],
  "env": ["!VITE_*", "!NEXT_PUBLIC_*"],
  "inputs": [ /* … static globs … */
    { "mode": "jit", "globs": ["$TURBO_ROOT$/graphql-schema.json"] } ]
}

Both //#codegen:prepare:buf and //#codegen:prepare:gql are root tasks with "cache": false, and each writes the file named by its task's JIT glob.

Result

Task type	Tasks	Identical hash across A and B	New hash
codegen:gql	13	13	0
codegen:buf	22	13	9

The 9 movers:

@repo/lib-1                                      068e6f2ddd5aa8b6 -> d0b71dcfb48057d2
@repo/lib-2                                      16459faea84a27f0 -> 8925213ad65263f7
@repo/lib-3                                      310db3e20992ac4a -> 28e2913d2a7c78dd
@repo/lib-4                                      11ca0f81d3742464 -> b3e68bbae5e0bc6d
@repo/lib-5                                      6658c8c2e4c6133a -> 9bb6f0854dad8c50
@repo/lib-6                                      a0e01d53eb58311a -> e2c93f35970cfbc6
@repo/lib-7                                      0c0219b642876794 -> 2ac5aaf471d4d473
@repo/lib-8                                      624b820c2317521b -> 1cd6685184695dae
a                                                875063a70f186d80 -> 3166454009a64ccf

This is the same set of 9 packages in every pair of runs we have compared.

Why the control matters

codegen:gql reproduced all 13 of its hashes exactly — on the same machine, in the same graph, in the same invocation as the 9 codegen:buf tasks that did not. So none of these explains the movement:

• "JIT hashing is nondeterministic." The gql tasks use a JIT input and are stable.
• "The environment differed between the two runs." Then gql would have moved too.
• "A cache: false root dependency poisons downstream hashes." gql depends on one as well.

Whatever changes those 9 hashes is specific to those tasks, and there is now a stable sibling in the same run to diff against.

Previously established in this thread

• Within a single build, running turbo run codegen:buf --summarize twice produced identical hashes for all 23 tasks. The movement appears only between builds.
• For the movers, globalCacheInputs, resolvedTaskDefinition, the per-input file hashes, externalDepsHash, and the dependency lists in the run summary are byte-identical across runs whose task hashes differ.
• All 9 movers reproduce a single stable hash locally, across repeated runs and across different --filter roots.

Happy to run further experiments on this graph if there is a specific field or debug flag worth dumping.

[bitttttten]

Narrower repro: the entry package's own task never reproduces its hash, while all of its dependencies do

Follow-up to the two comments above. I have a much smaller reproduction now — a graph of 17 tasks in which 15 are byte-stable across builds and the only 2 that move both belong to the entry package.

Setup

Vercel builds each app by running turbo from the app directory, so the entry package is inferred from cwd:

Detected Next.js version: 16.1.4
Running "turbo run build"
• Packages in scope: a

turbo 2.10.6. The entry package (apps/a below) is a Next.js app; every other package in scope is a plain library.

Both codegen task types declare explicit inputs plus one JIT input written by a "cache": false root task:

"codegen:buf": {
  "dependsOn": ["^codegen:buf", "//#codegen:prepare:buf"],
  "env": ["!VITE_*", "!NEXT_PUBLIC_*"],
  "inputs": [ "./buf.gen.ts", /* … root globs … */
    { "mode": "jit", "globs": ["./.codegen-schemas-inputs-hash"] } ]
},
"codegen:gql": {
  "dependsOn": ["^codegen:gql", "//#codegen:prepare:gql"],
  "env": ["!VITE_*", "!NEXT_PUBLIC_*"],
  "inputs": [ "graphql/*.graphql", "src/**/*.graphql", /* … root globs … */
    { "mode": "jit", "globs": ["$TURBO_ROOT$/graphql-schema.json"] } ]
}

Three consecutive builds of the same branch

Build	Commit	Change
B1	d862874	adds one comment line to a .tsx component in apps/a
B2	6fa22b66	edits that same comment line
B3	5743f2c	deletes the comment line, so the git tree is byte-identical to the commit before B1

That component file matches no glob in either task's inputs. All three builds resolved identical external pins, logged by our own prepare tasks:

• graphql-schema.json sha256 a0b633b6… in all three
• schemas repo HEAD 9e528dbf… in all three

Result

In every build, all 15 library-package tasks reproduced identical hashes and were restored from cache. The entry package's own two tasks produced a new hash in every build and never repeated one:

a:codegen:buf   e169a67b95fd9cfb -> e7818e5fce3ea146 -> 6391d1a7a4152c83
a:codegen:gql   805f29d562a40806 -> 56de5b1472037a6b -> 18c818bb10db0deb

B3's tree is byte-identical to a tree that had already been built, so no input difference can account for its hash.

A second project, apps/b, in the same push behaves identically — its 7 library tasks reproduced their hashes, and only its own entry-package task moved:

b:codegen:gql   79d34af4d3820234 -> 70048caaa660afce

What this rules out

• Not content. B3's tree is identical to an already-built tree, and the intermediate diffs are a comment in a file that is not an input.
• Not the environment in general. VERCEL_GIT_*, VERCEL_URL and VERCEL_DEPLOYMENT_ID are declared in globalPassThroughEnv, so they are not hashed — and if any of them were leaking into the hash, the 15 library tasks in the same run would have moved too.
• Not the cache: false root prepare dependency. The library tasks depend on the same prepare task and are stable. The prepare tasks' own hashes were identical across all five builds and across both projects (3f8b0a67… for buf, f9ea3eee… for gql).
• Not JIT input mode. Both task types use it, and the library tasks that use it are stable.

The remaining asymmetry

The only structural difference we can find between the movers and the stable tasks is that the movers belong to the entry package — the package turbo infers from cwd, and the only package in scope that is a framework. We opt out of framework inference on these tasks with "env": ["!VITE_*", "!NEXT_PUBLIC_*"].

If any per-deployment value can still enter the hash through framework inference for the entry package, that would produce exactly this pattern: framework package moves every build, non-framework dependencies stay stable. I have not been able to confirm that from the run summary, since the reported ingredients are identical between runs whose hashes differ.

Happy to dump anything specific for these two tasks if there is a field or debug flag that would settle it.