Skip to content

NKDS

Jump to bottom
Nanook edited this page Sep 24, 2026 · 2 revisions

A deduplicated game-storage and archiving format designed for efficient long-term preservation, storage optimisation, and fast access.

NKDS Launch Video

Watch the NKDS introduction video

What is NKDS?

NKDS is a content-addressed block store that understands what's inside disc images — not just raw bytes — and uses that knowledge for massive space savings. It builds on NKit's processing engine to provide a modern, mountable data store for game collections and archival workflows.

The problem: A full Wii set is ~16 TB. Compressed to RVZ that's still ~6 TB.

The solution: NKDS stores it in under 3 TB — an 83% reduction. This isn't better compression. It's filesystem-level deduplication — shared system updates, engine libraries, and textures stored once instead of thousands of times, across your entire collection.

⚠️ NKDS is under active development. While the storage format is stable and crash-safe, it is good practice to keep source files until you have verified stored copies with nkds verify. See NKDS Data Safety for the full picture.

Applications

App Type Description
nkds / nkds.exe CLI DataStore management — see NKDS CLI
nkds-ui / nkds-ui.exe GUI (Avalonia) Full visual interface with storage analysis, dat verification, per-image stats

NKDS sets can also be created via NKit directly using nkit -task dedupe.

The GUI includes additional features not available via the CLI, such as dat verification (load dats, classify entries as Correct/Missing/Badly Named/Wrong CRC/Unmatched, auto-rename), storage analysis graphs, and per-image analysis bars.

Key Features

  • ~80% space savings — Deduplication + Zstandard compression
  • Mountable virtual drive — Emulators see regular ISOs, no extraction needed (Dokan/FUSE)
  • Byte-perfect reconstruction — Every image verifiable, no data loss
  • Crash-safe — Dual-header atomic commits, automatic recovery from power loss
  • Flexible storage — Single-file (portable per-game) or multi-file (scalable to hundreds of TB)
  • Any source format — ISO, RVZ, WBFS, WUX, CHD, archives — NKDS reads them all
  • Multiple workflows — Use nkds / nkds-ui directly, or use NKit's dedupe task (nkit -task dedupe) to create and populate NKDS sets
  • Folder storage — Store non-disc content too (MAME sets, versioned ROM collections)
  • Export on demand — Reconstruct to any format directly from the store
  • Sequential I/O — Performs well over SMB/NFS network shares

Core Concepts

DataStore

A DataStore is a directory on disk containing one or more sets.

D:\NKitData\
  wii.nkds           ← index for the "wii" set
  wii_0000.nkds      ← shard 0 (compressed block data)
  wii_0001.nkds      ← shard 1 (auto-created when shard 0 fills up)
  gamecube.nkds      ← index for the "gamecube" set
  gamecube_0000.nkds

Set

A named collection of images stored together. Images within a set deduplicate against each other — keep related images in the same set for best results.

Image

A single stored disc image or folder. Disc images are deconstructed at the filesystem level — internal files are chunked into blocks with deduplication and Zstandard compression.

Block

The deduplication unit (default 64 KiB). Two images sharing any 64 KiB region of identical data share that block's physical storage.

Crucially, deduplication happens at the internal file (filesystem) level rather than raw disc sectors. Because formats like GameCube, Wii, and Wii U don't always align files to sector boundaries, scanning at the filesystem level dramatically improves duplicate detection.

Shard

Binary data files holding compressed blocks. A shard grows until it reaches the configured size limit (default 50 GiB), then a new one is created.

Storage Modes

Mode Shard Size Description
Separate > 0 (default 50 GiB) Multi-file: index + numbered shards. Recommended for large collections.
Embedded 0 Single file: block data + index + footer. Perfect for per-game OGMR sets.

How Deduplication Works

Take 4 versions of the same game — Europe, USA, Japan, and a revision. Each is 1 GB.

  • ~10% of each image is truly unique (region-specific text, assets)
  • ~90% is shared (engine, libraries, textures, system files)

Traditional compression stores all 4 GB separately. NKDS stores the shared 900 MB once, plus 400 MB of unique data = 1.3 GB instead of 4 GB.

On top of that, NKit identifies filler data and encryption that can be perfectly recreated on demand. Only truly irreplaceable data gets stored. Then it compresses everything.

Scale that across thousands of games sharing system updates, SDK libraries, and media files — that's where the 80%+ savings come from.

Virtual Filesystem (VFS)

Mount your entire collection as a virtual drive. Multiple view modes:

View Flag Shows
Image -i Each image as a single file (e.g. Wii/Game.iso)
Filesystem -fs Disc contents as browsable folders
System -s Hidden system entries (headers, filesystem.yaml)
Update -u Enable rename/delete from file explorer

Default (no flags): image + filesystem views.

Images are always served as full pristine ISOs regardless of the format they were added in. The VFS reconstructs them on-the-fly from the deduplicated store.

Requirements:

  • Windows: Dokan must be installed
  • Linux: FUSE 3
  • macOS: FUSE-T

Folder Storage

Not limited to disc images. Store arbitrary directory structures with the same block-level deduplication. Identical files (or identical regions within files) across multiple additions share physical storage.

Use cases: Versioned ROM sets (MAME, GoodTools, Tosec), photo/music archives, development snapshots, any collection with shared content.

Folder contents are browsable via VFS mount in filesystem view.

Auxiliary (Aux) Stores

Companion sets providing fallback block data. Primary use: shared Wii/WiiU update partitions stored once, referenced by many per-game OGMR sets.

If wii.aux.nkds exists in the DataStore directory, update partition blocks are automatically routed to the aux store during add and ogmr operations. Game data blocks go to the primary set.

Soft-Delete, Restore, and Compact

NKDS uses a non-destructive deletion workflow:

  1. Remove — Flags images as deleted. Data remains in shards. Images disappear from list/mount.
  2. Restore — Reverses a remove (only before compact).
  3. Compact — Permanently deletes block data for removed images and reclaims space.

💡 Best practice: If replacing a bad dump with a good dump, don't compact in between. The new dump will deduplicate against the still-present blocks from the old dump. Remove → Add → Compact.

Crash Safety

  • Dual-header atomic commit protocol
  • Append-only semantics (no in-place modification)
  • Copy-on-write for shard compaction
  • Automatic recovery from interrupted operations

If power is lost mid-write, the set recovers automatically on next access.

Thread Safety

  • Multiple concurrent readers per set (lock-free caches)
  • Single writer per set (enforced)
  • Concurrent read + write (readers see last committed state)
  • Per-set isolation (operations on one set don't block another)

Configuration

Parameter Default Description
shardSize 50 GiB Max shard file size (0 = embedded mode)
blockSize 64 KiB Deduplication block size

These are immutable after set creation.

Dependencies

  • GrindCore — XXHash64, CRC32, Zstandard compression (statically linked in AOT builds)

Further Reading

Clone this wiki locally