Cooked Volumes

Cooked Volumes

NWBLot runtime assets are stored in cooked volume segments. The loader mounts the graphics volume from the runtime res directory through IFilesystem and uses it as the binary source for typed assets.

Why Volumes Exist

Volumes give the engine one runtime loading path for cooked assets:

• shader bytecode
• materials
• textures
• meshes
• skins
• skeletons
• models
• shader archive index records
• generated graphics/cook payloads

The runtime does not scan source asset directories. It reads virtual paths from a mounted volume.

Build-Time Publication

asset_builder writes standalone .nwba artifacts and an assets.list manifest. asset_gatherer then publishes a volume through staged publication:

1. gather .nwba payloads and cached object payload references into a pack manifest
2. build a staged volume in an empty staging directory
3. validate artifact identities and write payload data plus metadata through VolumeFileSystem
4. flush and unmount the staged volume
5. promote staged volume segments into the output directory
6. back up and restore existing segments if promotion fails

This avoids leaving partially written runtime assets in the final res directory when a cook fails midway.

Filesystem Volume Domains

core/filesystem keeps the volume layers separate so runtime loading, low-level storage, and publication do not share one oversized interface:

• filesystem.h defines IFilesystem and resource-free file cursors.
• factory.h owns FilesystemFactory and CreateFilesystem.
• volume_types.h defines the shared mount and build descriptions.
• volume_file_system.h is the default segmented backend. It owns the mounted segment list, metadata index, direct file operations, and compaction (segment-compaction itself lives in core/filesystem/volume_segment_compact.cpp with I/O in core/filesystem/volume_io.cpp).
• volume_build.h provides BuildVolume for the all-in-memory case: write a supplied file map into an isolated staged volume, flush it, then publish it.
• volume_staging.h owns stage-directory cleanup plus staged-segment promotion, backup, and rollback behavior.

The loader mounts through CreateFilesystem. The gatherer constructs VolumeFileSystem directly for CookWrite staging because a pack manifest can stream both in-memory payloads and validated object-cache payloads without first materializing one generic file map. See Filesystem.

Cook-Write Call Semantics

IFilesystem distinguishes immediate metadata publication from deferred metadata publication without deferring ownership of the payload bytes:

• writeFile writes the payload and publishes metadata for that write.
• writeFileDeferred copies the payload into the staged volume during the call, but postpones the metadata flush. Caller-owned or scratch-backed payload buffers therefore only need to remain valid until the call returns.
• flush compacts the staged file layout, writes the final metadata index, and trims unused trailing segments. A staged volume is not ready for promotion until this succeeds.

This lets cook workers reuse temporary serialization buffers while still batching metadata work before the staged volume is promoted.

Pack Manifest And Object Cache

The graphics volume cook separates payload preparation from volume writing with AssetVolumePackManifest.

Manifest entries have two source kinds:

• PayloadBytes: bytes already held by the cook, with a size/hash identity.
• ObjectFilePayload: a content-addressed object file in the cook cache, with the same size/hash identity plus a cook-key hash.

Registry-backed typed assets use the object-cache path. Each serialized asset is written under:

<cache>/<configuration>/objects/<asset-type-hash-lanes>/<payload-size>_<asset-type-hash-lanes>/v2_<cook-key-hash><payload-hash>.nwbobj

Compact paths encode asset-type hash lanes directly across a two-level directory; the cook key folds every lane and the complete payload identity. The header validation checks the complete asset-type hash before reusing a cached file.

The object file contains a small header followed by the serialized asset payload. The header records the asset type hash, payload size, payload hash, and cook-key hash. Repeated cooks reuse the same object path when the serialized bytes and cook key are unchanged; changed payloads produce a new object path. The volume writer reads the object file back, verifies the identity against the manifest entry, and only then pushes the payload into the staged volume.

Standalone builder outputs are .nwba files plus assets.list. Artifact names combine a compact hexadecimal slot with payload and cook hashes. The builder reuses byte-identical artifacts and advances past occupied slots with different contents, preserving artifacts referenced by an earlier manifest. assets.list contains relative filenames for only the current successful build and is published after all artifacts have been written, so failed builds leave the previous manifest intact.

Volume Segment Naming

Volume segment file names are derived from:

• volume name
• segment index
• stable hash naming

The loader does not need to know authored asset roots. It only needs the runtime resource mount directory and the volume name graphics.

Runtime Mounting

The loader mounts the volume with:

VolumeMountDesc graphicsMount(frame.projectObjectArena());
graphicsMount.volumeName = "graphics";
graphicsMount.mountDirectory = resourceMountDirectory;
UniquePtr<IFilesystem> graphicsFilesystem = CreateFilesystem(
    frame.projectObjectArena(), graphicsMount, startupContext.filesystemFactory
);

On success, it creates a binary source backed by that filesystem:

readAssetBinary(virtualPath, reusedBytes) -> graphicsFilesystem.readFile(virtualPath, reusedBytes)

The asset manager then performs typed deserialization on those bytes.

Virtual Paths

Components and assets store references by typed virtual path. Examples from Testbed include:

• project/characters/female/model
• project/materials/mat_skinned_uv
• project/meshes/ground_plane/model
• project/materials/mat_white_opaque

The virtual path is stable across runtime loading. It does not need to match a physical file path one-to-one after cooking.

Asset Manager Flow

Runtime loading flow:

1. AssetManager::loadSync(assetType, virtualPath) -> Expected<UniquePtr<IAsset>>
2. IAssetBinarySource::readAssetBinary(virtualPath, bytes)
3. IFilesystem::readFile(virtualPath, bytes)
4. AssetRegistry::deserializeAsset(assetType, virtualPath, bytes) -> Expected<UniquePtr<IAsset>>
5. typed codec constructs and loads the asset

The manager, registry and codec return the freshly constructed owning asset through Expected; callers check admission before moving it into runtime state. The binary source and whole-file read retain the supplied AssetBytes arena and reusable capacity. Async loading uses request IDs and AssetLoadResult; tryPopResult(requestId) returns Expected<AssetLoadResult>, while the same binary-source and codec path applies.

The mounted volume is therefore only the byte source; type selection and object construction stay in the asset registry and codec layer:

flowchart LR
    Request["Typed asset request<br/>type + virtual path"]
    Manager["AssetManager"]
    Source["IAssetBinarySource"]
    Volume["IFilesystem<br/>graphics volume"]
    Registry["AssetRegistry"]
    Codec["Typed asset codec"]
    Asset["Loaded TypedAsset"]

    Request --> Manager --> Source --> Volume --> Source --> Manager --> Registry --> Codec --> Asset

Loading

Cook Cache

The Testbed build writes cooker cache data under:

__build_obj/c/<platform>/<arch>/<domain>

The cache is separate from the runtime res output. Runtime only consumes the cooked volume.

The cache can include registry object files and intermediate .nwba artifacts. These are cook acceleration artifacts, not a runtime mount format.

Determinism

Cook determinism matters because the same source roots should produce byte-identical cooked payloads when inputs and configuration are unchanged.

Important deterministic behaviors:

• stable virtual path keys
• stable metadata serialization
• sorted or canonical shader dependency handling
• deterministic shader variant naming
• deterministic typed material layout and packed bytes
• deterministic meshlet stream and skeleton serialization
• staged volume promotion
• stable registry object cache paths for unchanged serialized typed assets

Determinism is verified by the Linux convergence procedure (isolated two-root pipeline cooks), not by an automated in-tree byte-diff test. That procedure reports cook success separately from byte-level determinism, because a cook can succeed while still producing non-identical bytes — different failure modes that are easier to diagnose when tracked apart.