migrate turbo-tasks to scattered collect

[lukesandberg]

What

This uses scatter_collect from @mmastrac to register all values, traits and value impls of traits. now all data can be consistently accessed when constructing the VALUEs registry, eliminating a number of racy access opportunities.

Why

Scatter collect uses link sections to gather registries at link time, this resolve some inherent brittleness in constructor functions since the relative execution order is not guaranteed and it is also to even detect execution that is too early.

We have a single report that appears to be due to this

web:dev: thread 'tokio-rt-worker' (41574444) panicked at turbopack/crates/turbo-tasks/src/macro_helpers.rs:274:13:
web:dev: no trait impl registered for value type turbopack_core::module_graph::ModuleGraphImportTracer
web:dev: note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Which implies a race condition somewhere in our value registration. Unfortunately initialization logic is brittle since it relies on

• an inventory of all value types
• an inventory of all trait types
• a set of ctor functions to register trait implementations (basically value, trait pairs)

All of these need to be initialized before the VALUES registry is accessed, but the above panic implies that the ctor function for ModuleGraphImportTracer to register its implementation of ImportTracer hasn't occurred yet.

By moving to link time registration we completely eliminate all startup races since the linker does all the 'gathering'

[lukesandberg]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• migrate turbo-tasks to scattered collect #94503 👈 (View in Graphite)
• [tubopack] migrate rcstr! to use scattered collect #94498
• canary

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Stats cancelled

Commit: 5d27669
View workflow run

[github-actions]

Failing test suites

Commit: 5d27669 | About building and testing Next.js

pnpm test test/unit/image-optimizer/lru-disk-eviction.test.ts (job)

• LRU disk eviction > should evict old entries when new entries are set (DD)

Expand output

● LRU disk eviction › should evict old entries when new entries are set

expect(received).toEqual(expected) // deep equality

- Expected  - 0
+ Received  + 1

  Array [
+   "new-a",
    "new-b",
    "new-c",
  ]

  118 |     await setTimeout(100)
  119 |     const contents = await promises.readdir(cacheDir)
> 120 |     expect(contents).toEqual(['new-b', 'new-c'])
      |                      ^
  121 |   })
  122 |
  123 |   it('should promote entries on get() to prevent eviction', async () => {

  at Object.toEqual (unit/image-optimizer/lru-disk-eviction.test.ts:120:22)