Skip to content

Architecture

Jump to bottom
Nanook edited this page Sep 24, 2026 · 1 revision

How NKit and NKDS are designed, what principles guide them, and how the pieces fit together. Useful for anyone who wants to understand the system before embedding the libraries, contributing, or just knowing why things work the way they do.

Note: This documentation is a first draft and will be improved over time.

Design Principles

1. Forward-reading streams, no temp files

NKit processes disc images as a forward-reading stream. It never seeks backwards in the source, which means it can read directly from RAR, ZIP, 7-Zip, and GZip archives without extracting them first. No temp files, no double the disk space, no waiting.

This is achieved with a BufferStream layer — a cache-backed forward-only stream that sits between the archive reader and the format decoders. If a decoder needs to re-read a region it already passed, the cache serves it without going back to the source.

2. Filesystem parsed up front and frozen before parallel processing

When NKit reaches a filesystem area in a disc image (the region containing the game's internal files), it parses the entire filesystem before emitting any data from that area. This is done as a forward-looking peek — the read cursor does not advance — so subsequent sequential reads are served from cache.

Once parsed, the filesystem view is frozen and published as an immutable snapshot (AreaFileSystemView). The parallel processing workers read from this frozen snapshot. There is no locking on reads, no risk of the filesystem changing mid-process.

This is why NKit can extract individual files from encrypted Wii images without ever decrypting the whole image first: it knows where every file is before it reads a single data block.

3. Four-stage pipeline with clear serial/parallel boundaries

Every NKit task runs sections of the disc image through four stages:

Input (serial) → PreProcess (serial) → Process (parallel) → Output (serial, ordered)
  • Input reads one section at a time from the source image, drives the area state machine, and queues sections for preprocessing. Single-threaded — the source stream has exactly one reader.
  • PreProcess handles per-section setup that must run in order: decryption decisions, filesystem marker discovery, file-list population from the frozen snapshot.
  • Process is where the CPU-heavy work happens — format conversion, extraction, scanning, CRC computation, compression. This runs on N workers in parallel.
  • Output collects completed sections back into source order and writes them. Single-threaded — the output stream has exactly one writer.

The key invariant: because the filesystem is frozen before Process begins, all parallel workers read stable data. Because Output is serial and ordered, the result is always deterministic.

4. Content-addressed storage with no trust assumptions

NKDS uses (XxHash64, CRC32) as the block identity — approximately 96 bits of collision resistance. Every block read from the store is verified against its stored key before being returned. If the bytes don't match, the read fails loudly rather than silently returning corrupt data.

5. AOT-safe, no reflection

All NKit binaries are compiled with .NET Native AOT — no JIT, no runtime, single self-contained executable. This means no reflection anywhere in the read/write/parse path, no runtime code generation, and no external serialisation libraries for I/O. All binary format parsing is hand-written with explicit endian helpers.

6. Explicit endianness everywhere

All multi-byte fields in disc formats and in NKDS's own binary formats are stored big-endian on disk. Every field is read and written through explicit endian helpers (ReadUInt32B, WriteUInt64L, etc.) rather than relying on host byte order. This ensures identical behaviour on any CPU architecture.

Library Structure

NKit/                   ← Core processing engine
NKitCore/               ← Generic async section pipeline (StreamBlockCore)
NKitDataStore/          ← Block storage engine (shards, index, compaction, crash safety)
NKitStream/             ← BufferStream and streaming primitives
NKDS/                   ← DataStore product (sets, VFS mount, session management)
NKDS.UI/                ← Avalonia GUI for NKDS
NKit.UI/                ← Avalonia GUI for NKit
NKitApp/                ← nkit CLI
NKDSApp/                ← nkds CLI
Shared/                 ← Shared CLI utilities (SpectrePrompter, interactive prompter)
Tmds.Fuse/              ← FUSE mount support for Linux

Dependency flow

NKitApp ──────────────────────────────► NKit
NKit.UI ──────────────────────────────► NKit
                                         │
                                         ▼
NKDSApp ──────┬──────────────────────► NKDS ──► NKit
NKDS.UI ──────┘                         │
                                         ▼
                                   NKitDataStore
                                         │
                                         ▼
                                     NKitCore
                                     NKitStream

NKit knows disc formats, not storage. It converts, extracts, scans, fixes, verifies, and deduplicates by streaming through its four-stage pipeline.

NKitDataStore knows storage, not disc formats. It provides the content-addressed block store, the binary shard format, crash-safe commits, and the block index. It has no knowledge of GameCube vs Wii vs PS3.

NKDS composes both and adds set management, VFS mounting (Dokan on Windows, FUSE on Linux, FUSE-T on macOS), session lifecycle, and the nkds/nkds-ui applications.

How a Disc Image Flows Through NKit

Taking nkit convert game.rvz --format iso as an example:

1. Source discovery
   NKit scans the input path, detects file types, groups multi-file images
   (disc + cue, multi-part archives, etc.)

2. System identification
   The first few bytes identify the disc system (GameCube, Wii, PS3, Xbox, etc.)
   The format container (RVZ, ISO, CHD, archive) is detected separately

3. Container decode
   The RVZ decoder wraps the source stream as an IAsIso — a transparent
   layer that presents the compressed RVZ as a raw ISO stream to the reader above

4. IImage reader
   The system-specific reader (e.g. WiiGc.Image) drives the area state machine:
   - Header area → reads boot.bin, FST, apploader
   - FileSystem area → parses FST up front (the forward peek), then emits sector by sector
   - Audio/Other areas → emitted as raw blocks

5. Four-stage pipeline
   Each section (typically 0x8000 bytes) flows through:
   Input → PreProcess (decrypt Wii partition if needed) → Process (convert, CRC) → Output (write ISO)

6. Results
   NKit reports per-image results: system, format, verify outcome, output path, CRC

The same flow applies to Extract (Output writes individual files instead of a disc image), Scan (Output writes a .nkit.yaml fingerprint), Verify (Output checks checksums and dat entries), and Dedupe (Output writes blocks to NKDS).

How a Disc Image Flows Through NKDS

Taking nkds add --datastore /data/wii.nkds game.rvz as an example:

1. NKit pipeline runs (same as above, task = Dedupe)
   The NKit engine reads and normalises the source image

2. Block chunking
   Each internal file in the disc's filesystem is divided into 64 KiB blocks

3. Dedup check
   Each block's (XxHash64, CRC32) key is looked up in the index
   Existing blocks are referenced; new blocks are compressed and appended to the shard

4. Image metadata
   The block map (which blocks make up this image, in order) is written to the index
   Image name, system, format, checksums, and size are recorded

5. Atomic commit
   The dual-header commit makes the new image visible in one atomic step
   Before this step, nothing has changed from a reader's perspective

Reading back (VFS mount or export):

1. Image block map is loaded from the index
2. Blocks are read from shards, decompressed, and verified against their keys
3. The disc image is reconstructed in memory and streamed to the caller
   (emulator, file system, export target)

The reconstructed image is byte-identical to the original — not "close enough", exactly the same.

Virtual Filesystem (VFS)

NKDS mounts a virtual drive using:

  • Windows: Dokan (kernel-mode filesystem driver)
  • Linux: FUSE 3 via Tmds.Fuse
  • macOS: FUSE-T

The VFS layer translates filesystem operations (open, read, readdir) into NKDS block reads. Multiple view modes let you see the store as image files, as browsable disc contents, or both simultaneously.

The mount and the DataStore session are independent instances — a session (used by the UI to list/manage images) and a mount (used by emulators) hold separate handles to the same set. This lets you browse and add images in the UI while an emulator is actively reading from the mount. The session close path always unmounts first and waits for the mount host thread to finish before disposing — so closing the UI never leaves a stale mount.

Configuration and Path Variables

NKit uses a YAML config file (nkit.yaml) for defaults. Configuration paths support variables:

Variable Expands to
$configPath$ Config directory (portable: exe dir; system: %APPDATA%\nkit / ~/.config/nkit)
$userPath$ User data directory
$appPath$ Executable directory
$system$ Current system name (lowercase: wii, wiiu, ps3, …)
$task$ Current task (lowercase: convert, scan, …)
$date$ Current date as yyyymmdd

Portable vs system mode: if nkit.yaml exists next to the executable, all paths resolve locally (portable mode). Otherwise the OS user-config directory is used. The full directory tree (dats/, keys/, fix/, out/, logs/, temp/, dedupe/) is created automatically on first run.

Further Reading

Clone this wiki locally