Strata: Layered Data & Asset Loading System

[Rongmario]

Summary

A runtime-reloadable subsystem that loads both server-side data (data/) and client-side assets (assets/) from an ordered stack of layered sources such as mod JARs, resource packs, and the active world.

Goals

• Provide a single, generic pipeline for discovering, loading, merging, and reloading data from multiple sources.
• Establish an ordered layered source model: the vanilla jar first (assets), mod JARs by explicit priority, user resource packs in their configured order (assets), and the active world always last (data).
• Server data/ and client assets/ flow through the same source model, the same IO layer, and the same cached indexes.
• Allow features to register as data-category consumers. Strata feeds each consumer its files in layer order and lets the consumer parse, merge, and bake an immutable result.
• Provide a framework that consumers use to declare typed schemas once. The same schema parses JSONs from disk and serializes the baked result over the network, file format and wire format will be closely linked.
• Make loading fast and reload non-blocking: memory-mapped zip access with one central-directory parse per JAR per run, a zero-copy JSON scanner, and an async prepare phase (virtual threads?) with a synchronous game-thread apply.
• Support runtime reload (server start, world load, and the /reload command) with per-consumer failure isolation, atomic swap of baked results, and a completion event for cache invalidation. Previous point also means that: /reload should never freeze the tick loop.
• Attribute every contribution to its originating source, and emit logging whenever one source overrides or removes another source's contribution.
• Replace the existing resource/data manager internals behind a compatibility shim: SimpleReloadableResourceManager becomes a deprecated adapter over Strata with identical class, constructor, and method signatures, so the existing 1.12 mod ecosystem keeps working unmodified... including reload listeners and Forge's selective reload.

Non Goals

• Strata is not a tag system. Although our version of tags will be the first consumer of Strata, it will be proposed separately and depends on this CFP.
• Not a re-implementation of Mojang's DataFixerUpper: no DataResult, no DynamicOps, no dynamic backend plugging.
• Strata does not persist baked/derived data across restarts. Results are rebuilt from sources on each load.
  • Although this maybe up for contention if we have a reliable system to check if mods/configs/scripte etc has differed between runs
• No changes to the public resource API surface: IResource, IResourceManager, IResourcePack, SimpleResource, the pack repository, and the metadata (.mcmeta) system all remain untouched.

Motivation

The 1.12 OreDictionary is the only widely-used "shared classification" mechanism, and it is ItemStack-only, mutation-based, not reloadable, and not data-driven. Modernizing it touches on file discovery, layered merging, override/removal semantics, attribution, runtime reload, and client sync. Every one of those concerns is generic and this CFP serves to cover it.

1.12 has no meme version "datapack" infrastructure: there is no server-side resource manager, so data cannot ride any existing system. Even worse is that every feature that reads files out of mod JARs reinvents JAR walking from scratch, just take a look at : CraftingHelper.findFiles for example... Opens and closes a zip FileSystem per mod per walk. Strata aims to solve the plumbing once, with an IO layer that maps each JAR exactly once per run and shares that index between the data and asset sides.

On the client, SimpleReloadableResourceManager/FallbackResourceManager re-enumerate packs eagerly per domain and predate every modern IO facility. Since Strata indexes every mod JAR anyway, making it the real backend behind the vanilla client resource system removes the duplication instead of adding a parallel one.

Description

The system is Strata. Its unit of contribution is a stratum: a single source's root. A mod ships a stratum (the data/ and assets/ roots of its JAR), a resource pack ships a stratum (assets only), the active world ships a stratum (<world>/strata/, data only). The ordered stack of all active strata forms the layering that determines override precedence.

Source model

Kind	data	assets	Order
Vanilla (jar, asset index)	Y	Y	First
Mod (JAR or dev directory)	Y	Y	Mod registration order? Manifest order?
Resource pack (zip, folder, or the resource-loader esque folder)	N	Y	After mods, in the user's configured pack order
World	Y	N	Last

One stratum exists per physical source (deduplicated), and both the data stack and the assets stack view the same cached file indexes. A mod may declare its priority in some optional way in their jars. Layers will see what preceded them and opt to manipulate by overriding, merging etc.

File discovery

For each data stratum, Strata walks:

(data)(resources)/<namespace>/<category>/**.json

• <namespace>: modid of the data. Independent of which source provided it, any stratum may write to any namespace.
• <category>: which consumer receives the file (e.g. tags). Routes the file to a registered consumer.

Discovery is per-stratum, so each consumer receives files grouped and ordered by source layer. Enumeration is a prefix-range query against the source's sorted name index, no need to walk directories or reopen zips.

Consumer SPI

Features register a consumer during a bootstrap event fired once at load completion. Because registration is code present identically on client and server, the consumer set is deterministic across sides which is the prerequisite for safe synchronization.

Something like:

public interface StrataConsumer<Entry, Baked> {
    String category();
    Scribe<Entry> scribe();
    Baked prepare(List<StrataEntry<E>> entries); // off-thread: merge in layer order, bake immutable
    void apply(Baked baked); // game thread: atomic install
    default Scribe<Baked> syncScribe() { return null; } // non-null => synced on login + after reload
    default void applySynced(Baked baked) { apply(baked); }
}

Each StrataEntry carries the file's ResourceLocation id, its already-decoded value E, and the stratum it came from, so the consumer can implement its own override/merge semantics with full attribution.

Scribes

A scribe is a typed schema that does the work: it reads JSON (or any data type that has valid serder backends) from disk and writes/reads the network form for its type. Consumers declare scribes by explicit composition over record classes. The canonical record constructor is resolved once via a caller-supplied MethodHandles.lookup(), so there is no per-parse reflection:

For example:

record Tag(boolean replace, List<ResourceLocation> values) {
    static final Scribe<Tag> SCRIBE = Scribe.record(MethodHandles.lookup(), Tag.class,
        Scribe.optional("replace", Scribes.BOOL, false, Tag::replace),
        Scribe.field("values", Scribes.RESOURCE_LOCATION.listOf(), Tag::values));
}

• The same scribe drives data reading and network write/read. The wire form is positional in declared field order with presence bytes for optionals.
• Data parsing is a streaming bind: field-name tokens are matched against cached (UTF-8?) bytes without allocation, values fill an argument array, and one MethodHandle invocation constructs the record. Unknown keys are skipped, duplicates are last-wins with the aforementioned logging, and missing required fields produce errors attributed to source and file, perhaps even byte offsets...
• Declared fields are validated against the record's components at consumer registration. Schema mistakes fail fast at startup, not at first parse.
• Discriminator-keyed dispatch scribes support polymorphic entries, with a streaming fast path when the type key comes first and a tree-capture fallback for any key order. Say for JSONs, an immutable JsonValue tree remains available as an escape hatch for fully dynamic consumers.

IO layer

Each zip source is memory-mapped once per run as a single MemorySegment (FFM - FileChannel.map). Its central directory is parsed once into a sorted name table supporting exact lookup and prefix-range enumeration. Stored entries are served as true zero-copy slices - deflated entries are inflated with ThreadLocal Inflaters directly from the mapped segment. A process-wide cache keyed on path, size, and mtime shares these indexes between the data and asset sides and invalidates stale files on reload. Directory sources (dev workspaces, folder packs, the world stratum) read files conventionally, mmap of many small files is syscall churn for nothing. ZIP64 archives fall back to java.util.zip.

A custom pull-mode JSON scanner operates directly over memory segments with no intermediate DOM and no Gson in the hot path will easily win out the old performance w/ Gson, it can also parse errors carrying the source id, stratum, and byte offset.

Reload lifecycle

Trigger => game thread snapshots the strata
PREPARE (off-game-thread executor):
  - Refreshes stale file indexes (Parallel across each source)
  - Consumer: enumerate its category across data strata in layer order
    - Decode files w/ Scribe (Parallel across each file? Worth?)
    - `consumer.prepare(entries)` => Baked result
APPLY (game thread):
  - `consumer.apply(baked)` per successful consumer
  - Atomic swap => reload-complete event => sync to clients

• Failure isolation: a malformed file is logged with full attribution, skipped, and summarized, and won't cause the whole category to become errored. A consumer that throws during prepare can choose to NO-OP or keep its previous baked state, other consumers are unaffected.
• Coalescing: concurrent reload triggers into a single rerun rather than queueing up and waiting.
• Triggers: server start, world load, and /reload. 1.12 has no vanilla /reload; Strata introduces it. The command replies immediately and reports completion when apply lands. The tick loop never blocks on IO or parsing.

Attribution

Each contribution is stamped with its source stratum. Strata does not retain a full in-memory audit trail. Instead, whenever a source overrides or removes another source's contribution, logging is emitted for both sources.

Sync

Consumers with a non-null syncScribe() are synchronized automatically: on player login and after each reload, the baked result is serialized with the scribe and sent over a dedicated channel with format-version framing and packet splitting. Payloads over a soft threshold will warn and payloads that would exceed the transport's hard limit are skipped with an error and should never result in desyncs. The system will try its hardest to not exceed the thresholds.

Client Assets

On the client, Strata becomes the actual storage and lookup backend behind the vanilla resource system, with a hard-capped patch surface:

File	Change
SimpleReloadableResourceManager	Deprecated adapter: resource lookup delegates to Strata's asset stack; reloadResources forwards the pack list into Strata, then runs the untouched listener notification (Forge's selective reload preserved). Same class, constructor, and signatures.
FallbackResourceManager	Deprecation javadoc only - remains functional standalone(?).

Lookups return genuine vanilla SimpleResource instances (the pack name preserved from the originating pack for bug-compatibility), which defuses the common downcasts in the ecosystem. Known pack types (zip, folder, vanilla defaults) are unwrapped to their underlying files and memory-mapped, unknown third-party IResourcePack implementations are wrapped in a passthrough source with no existence caching (and this will be noted in logging and debugging). Strata still ultimately aims for correctness over speed. Stay compatible with dynamic packs. Mods that reflect into the old per-domain manager internals find the fields present but empty, this is the one documented behavioral break and a needed one.

Compatibility and Risks

• Inflation cost can dominate mmap wins... a per-reload timing log will measure this from the beginning of implementation. However, the structural wins from one central-directory parse per JAR per run, no Gson churn, parallel prepare still stand regardless.
• Foreign resource-pack implementations may have side effects on lookup, the passthrough source deliberately does not cache existence checks.

Dependencies

N/A

References

• Minecraft datapack infrastructure (1.13+).
• Mojang DataFixerUpper: considered and rejected.
• 1.12's OreDictionary which is the limitation Strata's first consumer replaces and the main motivation to kickstart this proposal.
• JDK Foreign Function & Memory API and virtual threads: platform facilities the IO layer and reload pipeline are built on.

Guidelines

• I have and will continue to adhere to the guidelines

[Rongmario]

This proposal aims to deliver on

Embed Resource Loader

and perhaps

Simplify Data Synchronization

[kappa-maintainer]

Is the future tag system planned to be data-driven?

[Rongmario]

Indeed, but the aim is to allow data to submitted not only via jsons. More details to follow.