Skip to content

Early PoC: scattered-collect#427

Draft
mmastrac wants to merge 1 commit intomasterfrom
scattered-collect
Draft

Early PoC: scattered-collect#427
mmastrac wants to merge 1 commit intomasterfrom
scattered-collect

Conversation

@mmastrac
Copy link
Copy Markdown
Owner

@mmastrac mmastrac commented May 5, 2026

This is the PoC for a new crate built on top of ctor and link-section: scattered-collect.

The idea is:

#[gather]
pub static COLLECTION: ScatteredSlice<T>;

#[scatter(COLLECTION)]
pub static ITEM: T = { ... };

Why?

Distributed slice is awesome and nearly perfect but it's a static snapshot out of the linker and downstream code occasionally needs more functionality on top of it, ie:

  • Sorting/ordering
  • Lookups by something other than ordinal index

Also, it doesn't support WASM which is a tougher to fix.

Inventory supports pretty much everything, but it's just a #[ctor] with a linked list and doesn't offer easy random access.

TODO

I'll spin some PRs off of this one and slowly land some pieces:

  • PR for placeholder crate (& publish a 0.0.0)
  • Land the top-level scatter/gather macro implementations
  • Implement and land the slice/referenced slice trivial types
  • Implement and land sorted versions of slices
  • Implement and land iterable
  • Implement and land map

Data types

  • (TODO) Iterable: collected items, only available via iterator (analogous to inventory crate)
  • Slice: collected items, not available at original definition site
  • Referenced slice: collection items, available at original definition site (analogous to distributed slice crate)
  • Sorted slice: collected and sorted items, not available at original definition site
  • Sorted referenced slice: collected and sorted items, available at original definition site
  • Map: similar to hashmap's swisstable implementation, but space is allocated at link time and hashes are precomputed to make construction much cheaper

There are "referenced" and "unreferenced" versions of the slices - the unreferenced versions may be faster as the code is free to re-organize the data more easily.

Const hashing

  • New trait: ConstHasher (should be able to derive this for custom structs as well)

Platform challenge notes

  • WASM: link sections are pretty much useless unless we do some sort of post-processing on the binary, which might be an option. We can, however, at least count the number of items using link-section and then place all the data via alloc + ctor copies where needed (the scattered iterator won't need this, and the map needs at least one iteration anyways, so we might be able to have a custom impl there).
  • All other platforms: generally will work via slices as expected. Similar single-threaded init with a late ctor to ctor's static will allow for guaranteed init before main without ordering issues.

@mmastrac mmastrac force-pushed the scattered-collect branch from 2a79cd3 to a330cf8 Compare May 5, 2026 13:57
mmastrac added a commit that referenced this pull request May 5, 2026
Broken off from #427 - this is the basic crate structure and supporting
macros.
@mmastrac mmastrac force-pushed the scattered-collect branch 2 times, most recently from 9f686bd to 02c00d5 Compare May 5, 2026 14:07
@mmastrac mmastrac force-pushed the scattered-collect branch from 02c00d5 to 0e9e658 Compare May 7, 2026 01:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant