^task and <NONEXISTENT> nodes: negative Vercel hash repro; what graph state is hashed?

[bitttttten]

Summary

Can maintainers clarify the cache-hash semantics of a broad ^task edge when it expands through packages that do not implement the task?

I suspected that a source-only change outside an affected task's explicit inputs could change its hash on Vercel when dependsOn contained ^codegen:buf, and that replacing the caret with explicit real prerequisites would stabilize it. I built a public, Git-integrated, two-project Vercel reproduction, but the suspected invalidation did not reproduce.

Because the claimed bug criteria failed, I am opening a Help discussion rather than a bug issue.

Environment

• Turborepo: 2.10.6
• pnpm: 10.24.0
• Vercel build Node: v24.15.0 (project setting 24.x)
• Vercel build CLI: 58.1.0
• Two Vercel projects from the same public Git repository

Task under test

{
  "dependsOn": ["^codegen:buf", "//#codegen:prepare:buf"],
  "inputs": [
    "./buf.gen.ts",
    "$TURBO_ROOT$/scripts/fake-generator.mjs",
    "$TURBO_ROOT$/fake-schema-pin.txt"
  ],
  "outputs": ["./src/gen/**"],
  "env": ["!VITE_*", "!NEXT_PUBLIC_*"]
}

The Vercel dry runs report configured: [] and inferred: []. The affected tasks' hashOfExternalDependencies is 459c029558afe716 throughout.

The broad app task has seven direct dependencies. Two are:

• @repro/demo-app-package#codegen:buf — <NONEXISTENT>, stable hash 1923ead99452d765
• @repro/dependency-without-task#codegen:buf — <NONEXISTENT>, stable hash a376a8dafab31b48

The explicit package-local override replaces the app's broad edge with five real prerequisites. Its baseline miss is expected because the new package-local turbo.json becomes an effective input; subsequent marker-only commits hit cache.

Control results

Graph	app-a	app-b
Broad baseline	1dc42cf857ea798f / miss	08bf754d7525a57d / miss
Broad marker 1	1dc42cf857ea798f / hit	08bf754d7525a57d / hit
Broad marker 2	1dc42cf857ea798f / hit	08bf754d7525a57d / hit
Explicit baseline	19635477346426a7 / miss	d3e8b7e45853d9e0 / miss
Explicit marker 1	19635477346426a7 / hit	d3e8b7e45853d9e0 / hit
Explicit marker 2	19635477346426a7 / hit	d3e8b7e45853d9e0 / hit

Validity checks

• Each control diff changes only apps/app-a/src/cache-control.ts and apps/app-b/src/cache-control.ts.
• Schema pin, generator, lockfile, manifests, task config, and package-local config are frozen within each graph's control set.
• The marker is absent from --dry-run=json task inputs.
• Exact input fingerprints remain stable.
• Generated output SHA-256 remains 71041ea...e42 for app-a and 96d846e...738 for app-b across all controls.
• Local broad and explicit hashes also remain stable.

Questions

1. Are <NONEXISTENT> transit/direct task nodes intentionally included in the dependency-hash chain using their effective package/task configuration?
2. Besides the dry-run fields captured here, can any package-graph or Vercel integration state make a parent task hash change after a file outside explicit inputs changes?
3. Is graph size alone expected to affect sensitivity, or does this negative result imply the original observation had another changing input/environment/manifest condition?

Reproduction and evidence

• Minimal repository
• Consolidated Vercel evidence
• README control matrix with commits, comparison diffs, task hashes, cache statuses, and deployment links

git checkout e1756e1ffed6adadedc19acd6ca819cdb7356d55
pnpm install --frozen-lockfile
pnpm experiment:broad

git checkout main
pnpm turbo run codegen:buf --filter=@repro/app-a --dry-run=json
pnpm turbo run codegen:buf --filter=@repro/app-a --force

One direct CLI source-upload pilot was deliberately excluded: without a Git-backed checkout, Turbo omitted buf.gen.ts from effective inputs, so it failed the validity gate.

Related: #10109 and merged documentation correction #10865 clarify that ^ selects the matching task in direct package dependencies, with further traversal coming from those tasks' own dependencies. This repro follows that model and records the resulting task graph.

[bitttttten]

Correction and final deferred-JIT follow-up

I need to correct one implication in the original report: in the non-public observation, the changed application hash was not localized to a change in the application's own effective input set. The first changed real task visible in the available logs was an intermediate dependency task. Because a parent task hash includes dependency-task hashes, the changed application hash was downstream propagation, not direct evidence that the unused application source marker entered the application's own hash.

I extended the public repro to cover the two largest missing conditions:

• a preparation task now writes a deterministic ignored schema-pin file immediately before codegen;
• the codegen task declares that file with structured JIT inputs;
• the workspace now has 105 projects;
• the broad app closure has 103 dependency-task nodes: 18 real tasks and 85 <NONEXISTENT> nodes;
• env: ["!VITE_*", "!NEXT_PUBLIC_*"] remains in place from the start;
• a normal dry run and --framework-inference=false report identical target inputs and environment configuration.

Public implementation and evidence:

• Repository
• Large-JIT results, exact input fingerprints, deployment IDs, and sanitized env configuration
• Broad baseline to marker-only diff
• Explicit baseline to marker-only diff

Results

Graph	app-a baseline → marker	app-b baseline → marker
Broad JIT, 103 dependency nodes	fffda372338b7c06 miss → same hash, remote hit	c4f518a5a92bc984 miss → same hash, remote hit
Explicit JIT, 6 dependency nodes	8e0e570053faf870 miss → same hash, remote hit	b075068ba28013d8 miss → same hash, remote hit

Local final JIT hashes were also stable: 6209219ef056085c for the broad control and 87b0201db2fb6e94 for the explicit control.

In every within-graph comparison:

• only the two unused source-marker files changed;
• the marker was absent from the target task's effective inputs;
• schema source, prepared JIT pin, generator, lockfile, manifests, and task configuration were unchanged;
• hashOfExternalDependencies stayed 459c029558afe716;
• configured and inferred environment key lists were empty;
• generated output remained byte-identical.

Some broad runs logged HTTP 429 responses for intermediate remote-cache artifact requests. That did not change the result under test: both target application tasks kept their baseline hashes and produced verified remote hits from the baseline commits.

Remaining observability question

The dry run shows all 85 <NONEXISTENT> nodes, but because hashing is deferred it reports hash: null. The final .turbo/runs/*.json summary contains all 18 real dependency-task records and zero of the 85 nonexistent records. I therefore still cannot find a supported CLI/run-summary view that exposes the final resolved hash contribution of each <NONEXISTENT> node. A maintainer pointer would still be useful if such a view exists.

Conclusion

The substantially larger deferred-JIT public control still does not reproduce the suspected invalidation. I am closing this discussion as inconclusive/project-specific and will not open a bug issue. The evidence is now more consistent with a missing condition in the original project or an unobserved intermediate dependency input than with broad ^task expansion alone.

[bitttttten]

Resolved: it was VERCEL_DEPLOYMENT_ID reaching <NONEXISTENT> nodes through framework inference

Closing the loop, because my earlier posts here described this as unexplained hash instability and that framing was wrong. The hashes were deterministic all along; I was changing an input I could not see. Posting the mechanism because the invisibility part is, I think, still worth your attention.

The mechanism

1. Our builds run on Vercel with skew protection on, so VERCEL_SKEW_PROTECTION_ENABLED=1.
2. frameworks.json gives nextjs an envConditional: when that flag is 1, include VERCEL_DEPLOYMENT_ID.
3. Framework inference in calculate_task_hash_with_file_hash is per task, keyed off the package's external_declarations(). Any package depending on next therefore hashes VERCEL_DEPLOYMENT_ID for every task in that package, including a codegen task that has nothing to do with Next.js.
4. TaskHashable.resolved_env_vars hashes key=value, and the deployment id is unique per deployment. New hash every build, guaranteed miss.

Our exclusions were !VITE_* and !NEXT_PUBLIC_*, which do not match VERCEL_DEPLOYMENT_ID. Adding !VERCEL_DEPLOYMENT_ID to the task's env fixes it completely (0 movers, and the variable no longer appears in any task's hashed env).

Why this took weeks to find, and the part that ties back to this discussion

Two of the affected packages have no codegen:buf script. They exist in the graph only because of the broad ^codegen:buf dependency, as <NONEXISTENT> nodes — the exact nodes this discussion is about. Those nodes:

• are hashed, including inferred env from framework detection, and
• are absent from run summaries, so their inputs cannot be inspected.

For their dependents, the only visible trace is a changed entry inside taskDependencyHashes. That is why every field I diffed across identical-tree redeploys was byte-identical while the hash moved: the moving input lived on a node the summary does not emit.

Turborepo's own debug log was the thing that finally showed it (TURBO_LOG_VERBOSITY=turborepo_task_hash=debug):

auto detected framework for @repo/lib-a
framework: nextjs, env_prefix: ["NEXT_PUBLIC_*", "VERCEL_DEPLOYMENT_ID"]
task hash env var names for @repo/lib-a:codegen:buf
 vars: ["VERCEL_DEPLOYMENT_ID"]

Correlation was exact: running the task twice with only VERCEL_DEPLOYMENT_ID changed, 9 of 23 tasks moved, and those 9 are precisely the tasks whose graph closure reaches one of the two <NONEXISTENT> Next.js nodes. The other 14 do not reach them and were stable. It also explains the app-level cases I posted earlier (apps/a, apps/b): those packages depend on next directly, so no ghost hop is involved and both their codegen tasks missed on every deploy.

Suggestions, in priority order

1. Emit <NONEXISTENT> nodes in run summaries (or at least their hash, environmentVariables, and framework). Today a ghost node can inject an input into a whole subtree with zero representation in --summarize, which makes cache-miss debugging a dead end. This is the concrete gap behind my earlier reports.
2. Reconsider inferring framework env for tasks that are not the framework's build. A codegen task in a package that happens to depend on next picking up VERCEL_DEPLOYMENT_ID is surprising, and it silently makes every cached non-build task in every Next.js package uncacheable on Vercel when skew protection is on.
3. Warn when an inferred variable is known-volatile. VERCEL_DEPLOYMENT_ID changing per deployment means "this task can never hit remote cache across deployments," which is worth a one-line warning rather than silence.

Happy to open a separate issue for the run-summary gap in (1) if that's useful — that's the piece that seems clearly actionable.

[bitttttten]

Closing this as resolved — the mechanism is understood (see the comment above: VERCEL_DEPLOYMENT_ID entering <NONEXISTENT> nodes via the nextjs envConditional, then propagating through taskDependencyHashes). Nothing here was actually nondeterministic.

The forward-looking part is better tracked elsewhere, so as not to keep a resolved thread alive:

• Config ergonomics — a per-task "frameworkInference": false would have prevented this entirely, including the conditional variable that no negative wildcard can express: Per-task opt-out of framework inference in turbo.json #13494 (I've added the full repro there).
• Run-summary visibility — <NONEXISTENT> nodes are hashed, including inferred framework env, but are not emitted in --summarize, so an input that invalidates a whole subtree has no representation in the summary. That's the piece that made this undiagnosable from summaries alone. Happy to open a separate issue for it if a maintainer thinks it's worth one.

Thanks to anyone who read through the earlier reports here.