-
Notifications
You must be signed in to change notification settings - Fork 7
NKDS User Guide
⚠️ IMPORTANT NOTICE: NKDS is currently in an alpha / Proof of Concept (PoC) release. Do not use it to persist data that is not backed up elsewhere. Currently, only GameCube, Wii, and Wii U are supported. More systems are coming soon.
nkds is the NKit DataStore command-line tool. It lets you:
- Store disc images from multiple gaming systems into a single deduplicated, compressed archive. Crucially, deduplication happens at an internal file level rather than raw disc sectors, allowing for better duplicate detection when formats like GameCube, Wii and WiiU do not align files to sectors.
- Mount the archive as a virtual drive so you can browse every image as a folder tree without extracting anything
- List the images and sets currently stored
- Verify that stored images are intact
- Export images back to their original format, or convert them to a different format
It is a companion to nkit. nkit handles format conversion and image processing; nkds manages
the long-term storage of the results.
A DataStore is a directory on disk. Everything lives inside it. You point nkds at this
directory with --datastore (or -ds).
D:\NKitData\
wii.nkds ← index database for the "wii" set
wii_0000.nkds ← shard 0 (raw compressed block data)
wii_0001.nkds ← shard 1 (created automatically when shard 0 fills up)
gamecube.nkds ← index database for the "gamecube" set
gamecube_0000.nkds
A set is a named collection of images stored together. Images across sets share nothing — a block
in the wii set is never deduped against a block in the gamecube set. Keep related images in
the same set to get the best deduplication.
A set name is just the base filename of its .nkds index, e.g. wii, gamecube, multisys.
When you pass --datastore D:\NKitData\wii.nkds (pointing directly to the .nkds file), nkds
scopes all operations to the wii set only. When you pass --datastore D:\NKitData (the
directory), operations apply to all sets in that directory.
An image is a single disc or application — Wii, GameCube, Wii U, etc. Disc images are deconstructed and their internal files are stored block-by-block with deduplication and Zstandard compression when over the configured block size (e.g. 64 KiB). This allows them to be seeked and recreated on request during verify, export, or mount reads.
The deduplication unit. By default each block is 64 KiB. Two images that share any 64 KiB region of identical data will share that block's physical storage.
Crucially, deduplication happens at the internal file (filesystem) level rather than raw disc sectors for supported systems. Because formats like GameCube, Wii, and Wii U do not always align their internal files to strict CD/DVD sector boundaries, scanning at the filesystem level dramatically improves duplicate detection. A shared compiled library, sound file, or texture hidden deep inside a disc image will be perfectly deduplicated regardless of its exact sector offset. Across a large library of games from the same engine or SDK, this yields significant space savings beyond simple file compression.
The binary data file that holds the compressed blocks. A shard grows until it reaches the configured shard size limit (default 50 GiB), then a new one is created. Shard files are never split — a block always lives entirely within one shard.
nkds stores a filesystem.yaml for every image. It records the internal directory tree of the
disc image, which is what lets the virtual filesystem (nkds mount) show you the contents of a
disc as folders without ever extracting the image. This ensures folder navigation is always
available when mounting.
Not everything stored in NKDS has to be a disc image. The ImageFormat.Folder format lets you store arbitrary directory structures — loose files, homebrew collections, save-game backups, or any other folder of data — directly in a set.
Files within a folder are chunked into blocks and compressed with the same block-level deduplication used for disc images. Identical files (or identical regions within files) across multiple folder additions share physical storage, just as they would across disc images. No disc image processing pipeline is involved — files are stored as flat block sequences without any detection, normalisation, or conversion steps.
A filesystem.yaml is automatically generated for each folder, recording the directory tree along with file offsets and hashes for every stored file. This means folder contents are browsable via nkds mount -fs just like disc image filesystems. Because there is no disc image to reconstruct, folder images appear only in the filesystem view (-fs) — they have no image view (-i).
The primary way to ingest folder content is the nkds adddir command.
When Wii U TmdApp images are processed via nkds add, the source folder may contain multiple TMD index files (tmd.0, tmd.1, tmd.2, …). Each index file is processed as a separate child image with a disambiguated name — for example, "Game Title [tmd.0]", "Game Title [tmd.1]", and so on. After all child images have been committed, NKDS automatically creates a TmdAppFolder image that aggregates them into a single browsable structure.
The TmdAppFolder's filesystem.yaml has two sections:
-
fs— real files stored as blocks in the set (e.g.tmd,tik,cetk,h3). These are the metadata and ticket files that accompany the title. -
ifs— image file references that point to content inside the child[tmd.X]images by image ID. Rather than duplicating the child image data, the TmdAppFolder references it in place.
How TmdAppFolder images appear when mounted depends on the view mode:
- In image mode (
-i), the TmdAppFolder appears as a single browsable folder. The individual[tmd.X]child images are hidden, giving you a unified view of the title. - In filesystem mode (
-fs), each[tmd.X]child image appears as its own folder containing its decrypted filesystem contents, letting you explore each content partition independently.
nkds create --datastore D:\NKitData wii
This creates D:\NKitData\wii.nkds. You only need to do this once per set. The add command
will also auto-create the set if you use nkds add pointing at a .nkds path that does not yet
exist.
nkds add --datastore D:\NKitData\wii.nkds D:\WiiISOs
nkds add runs the full NKit pipeline (detect system, normalise, deduplicate, verify) and stores
the result. You can add individual files, a folder, or a glob mask. Add is always incremental —
images already present in the set are skipped.
nkds list --datastore D:\NKitData\wii.nkds
nkds stats --datastore D:\NKitData\wii.nkds
nkds verify --datastore D:\NKitData\wii.nkds --mask *.iso
Windows
nkds mount --datastore D:\NKitData --mount N:\
Linux
nkds mount --datastore /var/lib/nkitdata --mount /mnt/nkit
You can now open N:\ in Explorer (or /mnt/nkit in your file manager) and see all your images
as folders. Navigate inside a disc image the same way you would browse a folder of files.
nkds create <set-name> [options]
nkds create --datastore <path.nkds> [options]
| Option | Default | Description |
|---|---|---|
--datastore, -ds |
current dir | DataStore directory or explicit .nkds path |
--shard-size, -ss |
50GiB | Max size of each shard file (0 = single embedded file) |
--block-size, -bs |
64KiB | Deduplication block size (2KiB–2MiB) |
Examples
# Create a "wii" set using defaults
nkds create --datastore D:\NKitData wii
# Create a "psp" set with smaller shards
nkds create --datastore D:\NKitData psp --shard-size 10GiB
# Create a portable single-file set (no shard rotation)
nkds create --datastore D:\NKitData portable --shard-size 0
Size syntax — Sizes accept suffixes: KiB, MiB, GiB, TiB, KB, MB, GB, TB,
or bare bytes. Suffix matching is case-insensitive.
nkds add --datastore <path> [options] <input ...>
| Option | Description |
|---|---|
--datastore, -ds |
DataStore directory or explicit .nkds path |
--input, -in |
Add an input file, folder, or wildcard mask (repeatable) |
--recursive, -r |
Scan input folders recursively |
--no-archives |
Skip scanning inside archives (zip/7z/rar) |
--config, -cfg |
Path to an NKit YAML config file |
Inputs may also be provided as bare positional arguments after the command.
nkds add always runs with -task dedupe. Any extra options you pass after the known flags are
forwarded directly to the NKit pipeline (e.g. -v n to disable verification, or -system wii
to restrict to one system).
Examples
# Add all images from a folder
nkds add --datastore D:\NKitData\wii.nkds D:\WiiISOs
# Add recursively
nkds add --datastore D:\NKitData\wii.nkds D:\WiiISOs --recursive
# Add with a custom NKit config and skip verification
nkds add --datastore D:\NKitData\wii.nkds D:\WiiISOs -cfg nkit.yaml -v n
# Add a single file
nkds add --datastore D:\NKitData\multisys.nkds "D:\Games\Example Game.wbfs"
# Add, skipping archives
nkds add --datastore D:\NKitData\wii.nkds D:\WiiISOs --no-archives
nkds adddir --datastore <path> [options] <directory>
Store the contents of a directory directly into a set using the ImageFormat.Folder format. Unlike nkds add, this command bypasses the NKit processing pipeline entirely — no disc image detection, normalisation, or conversion occurs. Files are chunked into blocks and compressed with the same block-level deduplication used for disc images, so identical files (or identical regions within files) across multiple additions share physical storage.
A filesystem.yaml is automatically generated recording the directory tree, file offsets, and hashes for each stored file. This means folder contents are browsable via nkds mount -fs just like disc image filesystems.
| Option | Description |
|---|---|
--datastore, -ds |
DataStore directory or explicit .nkds path |
--recursive, -r |
Include subdirectories recursively |
The image name defaults to the directory name being added.
Examples
# Store a folder of loose files into the set
nkds adddir --datastore D:\NKitData\extras.nkds D:\MyFiles\Homebrew
# Store recursively
nkds adddir --datastore D:\NKitData\extras.nkds D:\MyFiles\Saves --recursive
nkds 1gmr <1gmr-yaml> [options] <input ...>
Batch-import disc images into a DataStore directory, routing each input file to a dedicated per-game nkds set based on regex mask matching defined in a 1GMR YAML file. This implements the 1G*R (1 Game, many ROMs) workflow where each game and all its regional variants are stored in their own set.
| Option | Description |
|---|---|
--datastore, -ds |
DataStore directory (not a .nkds file) |
--1gmr |
Alternative to the positional argument for specifying the YAML file path |
--input, -in |
Add an input file, folder, or wildcard mask (repeatable) |
--recursive, -r |
Scan input folders recursively |
--no-archives |
Skip scanning inside archives (zip/7z/rar) |
--shard-size, -ss |
Max shard size for newly created sets (default 50 GiB) |
--block-size, -bs |
Deduplication block size for newly created sets (default 64 KiB) |
--config, -cfg |
Path to an NKit YAML config file |
Note:
--datastoremust be a directory path, not a.nkdsfile. The1gmrcommand creates multiple per-game sets inside this directory, so pointing at a single set file is not valid.
The YAML file contains a games: list. Each entry has a name (the game title used as the set name) and a masks list of regex patterns used to match input filenames to that game:
games:
- name: "Example Racer"
masks:
- '^Example Racer(?= \(|\.iso|$)'
- name: "Example Fighter"
masks:
- '^Example Fighter(?= \(|\.iso|$)'
- '^Example Fighter II(?= \(|\.iso|$)'
- name: "007: Quantum"
masks:
- '^007 - Quantum(?= \(|\.iso|$)'Each mask is a .NET regular expression. A game entry can have multiple masks to cover different naming conventions for the same title. The set name is derived from the name field — filesystem-unsafe characters (: ? * < > | " / \) are replaced with _ to produce a valid filename.
When processing input files, the command extracts each file's name (without directory path) and tests it against the masks in YAML order:
- Game entries are evaluated in the order they appear in the YAML file.
- For each game entry, its masks are tested against the filename.
- First-match-wins — the file is assigned to the first game entry whose mask matches. Evaluation stops immediately.
- Matching is case-insensitive.
- If a filename does not match any mask in the YAML, it is skipped and added to the unmatched list.
After processing completes, the command displays a distinct "Unmatched files" section listing every input file that did not match any mask. This lets you identify gaps in your YAML, amend the masks, and reprocess the unmatched files in a subsequent run.
Aux store routing works automatically during 1gmr import — no extra flags are needed. If an aux store file (e.g. wii.aux.nkds) exists in the DataStore directory, update partition blocks are routed to the aux store and game data blocks go to the primary per-game set, exactly as with nkds add.
The 1GMR YAML files should be vetted against dat files (e.g. No-Intro, Redump) to ensure each mask uniquely identifies one game. If a mask inadvertently matches filenames belonging to multiple games, the first-match-wins rule applies — the file is routed to whichever game entry appears first in the YAML. Validate your YAML against your dat source to avoid misrouting.
Examples
# Batch import using a 1GMR YAML file
nkds 1gmr wii-1gmr.yaml --datastore D:\NKitData D:\WiiISOs
# Recursive scan with custom shard size (embedded single-file sets)
nkds 1gmr wii-1gmr.yaml --datastore D:\NKitData D:\WiiISOs --recursive --shard-size 0
# Using --1gmr option instead of positional argument
nkds 1gmr --1gmr wii-1gmr.yaml --datastore D:\NKitData --input D:\WiiISOs --recursive
# With a custom NKit config
nkds 1gmr wii-1gmr.yaml --datastore D:\NKitData D:\WiiISOs -cfg nkit.yaml
nkds list [options]
| Option | Description |
|---|---|
--datastore, -ds |
DataStore directory or .nkds path |
--set, -s |
Filter by .nkds index path |
--system, -sys |
Filter by system name (e.g. Wii, GameCube) |
--search, -sch |
Filter by image name substring |
-r |
Show soft-deleted (removed) images in the listing |
--format, -f |
Output format: text (default) or json
|
Examples
# List all images in a DataStore
nkds list --datastore D:\NKitData
# List only Wii images
nkds list --datastore D:\NKitData --system Wii
# Search by name
nkds list --datastore D:\NKitData --search "Example"
# Include soft-deleted images in the listing
nkds list --datastore D:\NKitData -r
# JSON output for scripting
nkds list --datastore D:\NKitData --format json
nkds remove --datastore <path> --mask <mask> [options]
Mark one or more images as removed by setting a soft-delete flag (removed=1) in the set index. The block data for removed images is not physically deleted — shard files remain unchanged. This means you can undo a removal with nkds restore at any time, as long as you have not yet run nkds compact on the set.
To actually reclaim disk space after removing images, run nkds compact (see below).
| Option | Description |
|---|---|
--datastore, -ds |
DataStore directory or explicit .nkds path |
--mask, -m |
Image name wildcard mask (e.g. *.iso, Mario*) |
Examples
# Remove all images matching a wildcard pattern
nkds remove --datastore D:\NKitData\wii.nkds --mask "Example Shovelware*"
# Remove a specific image by exact name
nkds rm --datastore D:\NKitData\wii.nkds --mask "Example Game (USA).iso"
Note: Removed images no longer appear in
nkds listornkds mount, but their block data still occupies space in the shard files. Runnkds compactto permanently delete the data and reclaim storage.
nkds restore --datastore <path> --mask <mask> [options]
Reverse a soft-delete, recovering previously removed images. When you run nkds remove, images are only flagged — their block data remains intact in the shard files. nkds restore clears that flag and brings the images back into the active set so they appear in nkds list and nkds mount again.
⚠️ Important: Restore only works on images that have been soft-deleted and not yet compacted. Once you runnkds compact, the block data for removed images is permanently deleted and can no longer be recovered.
| Option | Description |
|---|---|
--datastore, -ds |
DataStore directory or explicit .nkds path |
--mask, -m |
Image name wildcard mask (e.g. *.iso, Mario*) |
Examples
# Restore a previously removed image by name
nkds restore --datastore D:\NKitData\wii.nkds --mask "Example Game (USA).iso"
# Restore all images matching a wildcard
nkds restore --datastore D:\NKitData\wii.nkds --mask "Accidentally Removed*"
nkds compact --datastore <path> [options]
Permanently delete the block data for all soft-deleted images in a set and reclaim shard storage space. When you run nkds remove, images are only flagged — their blocks still occupy space in the shard files. nkds compact rewrites the shard files, stripping out every block that is no longer referenced by an active image, and shrinks the shards accordingly.
⚠️ Warning: Compacted images cannot be restored. Oncenkds compacthas run, the block data for removed images is permanently deleted. Make sure you no longer need the removed images before compacting.
| Option | Description |
|---|---|
--datastore, -ds |
DataStore directory or explicit .nkds path |
Examples
# Compact a set to reclaim space after removing images
nkds compact --datastore D:\NKitData\wii.nkds
# Compact using the short alias
nkds cp --datastore D:\NKitData\wii.nkds
💡 Best practice: Don't compact between remove and re-add. If you're replacing a bad dump with a good dump of the same game, do not compact in between. The bad dump and good dump share almost all their blocks. If you compact immediately after removing the bad dump, NKDS destroys all its blocks — including the ones the good dump would have deduplicated against. Instead: remove the bad dump, add the good dump (which instantly deduplicates against the still-present blocks), then compact. NKDS will only erase the small percentage of blocks that the new dump didn't reuse.
nkds rollback --datastore <path> <image-identifier>
Revert a set to a specific image state. This command rolls the set back so that it reflects the state as of the specified target image, undoing any changes made after that point. This is useful when you want to return to a known-good state after a series of additions or modifications.
| Option | Description |
|---|---|
--datastore, -ds |
DataStore directory or explicit .nkds path |
<image-identifier> |
The target image name or ID identifying the state to roll back to |
Examples
# Roll back the set to the state at a specific image
nkds rollback --datastore D:\NKitData\wii.nkds "Example Game (USA).iso"
nkds verify --datastore <path.nkds> --mask <mask> [options]
| Option | Description |
|---|---|
--datastore, -ds |
Explicit .nkds index path |
--mask, -m |
Image name wildcard mask (e.g. *.iso, Mario*) |
--config, -cfg |
Path to an NKit YAML config file |
nkds verify always runs with -task verify -v y. Extra options are forwarded to NKit.
Examples
# Verify all images in the wii set
nkds verify --datastore D:\NKitData\wii.nkds --mask *.iso
# Verify a specific game
nkds verify --datastore D:\NKitData\wii.nkds --mask "Example Game*"
# Verify with a custom config
nkds verify --datastore D:\NKitData\wii.nkds --mask *.iso -cfg nkit.yaml
nkds export --datastore <path.nkds> --mask <mask> --output <folder> [options]
| Option | Description |
|---|---|
--datastore, -ds |
Explicit .nkds index path |
--mask, -m |
Image name wildcard mask |
--output, -out |
Destination folder |
--format, -f |
Output format (e.g. wux, rvz:zstd:19:128k:16). Omit for full expand |
--config, -cfg |
Path to an NKit YAML config file |
If --format is omitted, nkds export uses -task expand (full restore to original). If
--format is specified, it uses -task convert -convert <value>.
Examples
# Export all images as ISO
nkds export --datastore D:\NKitData\wii.nkds --mask *.iso -out C:\Temp
# Export as WUX (Wii U)
nkds export --datastore D:\NKitData\wiiu.nkds --mask *.iso -out C:\Temp --format wux
# Export as high-compression RVZ
nkds export --datastore D:\NKitData\wii.nkds --mask *.iso -out C:\Temp --format rvz:zstd:19:128k:16
# Export a specific title
nkds export --datastore D:\NKitData\wii.nkds --mask "Example Game*" -out C:\Temp
nkds sets [options]
| Option | Description |
|---|---|
--datastore, -ds |
DataStore directory |
--format, -f |
Output format: text (default) or json
|
nkds sets --datastore D:\NKitData
nkds sets --datastore D:\NKitData --format json
nkds stats [options]
| Option | Description |
|---|---|
--datastore, -ds |
DataStore directory or .nkds path |
--set, -s |
Target .nkds index path |
--details |
Include per-image statistics |
--format, -f |
Output format: text, json, or yaml
|
The stats output shows image count, total logical size, physical shard file size, and the shrink ratio — how much smaller the stored data is compared to the uncompressed originals.
nkds stats --datastore D:\NKitData
nkds stats --datastore D:\NKitData\wii.nkds --details
nkds stats --datastore D:\NKitData --format yaml
nkds mount [options]
| Option | Default | Description |
|---|---|---|
--datastore, -ds |
— | DataStore directory or .nkds path |
--mount, -m |
— | Mount point (drive letter on Windows, directory on Linux) |
--image, -i |
— | Show images as files (e.g. GameTitle.iso) |
--filesystem, -fs |
— | Show disc filesystem as folders |
--system, -s |
— | Show system entries (headers, filesystem.yaml) |
--update, -u |
off | Enable write operations (rename and delete images) |
--uid |
— | Set the owner UID for mounted files (Linux only) |
--gid |
— | Set the owner GID for mounted files (Linux only) |
--allow-other |
off | Allow other users to access the mount (Linux only) |
When none of -i, -fs, or -s is given, the default is to show both images and filesystem
views. When one or more flags are explicitly given, only the selected views are shown.
Views explained
| View | Flag | What you see |
|---|---|---|
| Image | -i |
Each stored image as a single file (e.g. Wii/Game Title.iso). While you can add compressed formats like .wbfs, .rvz, or .ciso to the DataStore, the VFS always reconstructs and presents the full, pristine disc image. |
| Filesystem | -fs |
Contents of the disc as subfolders (e.g. Wii/Game Title/DATA/) |
| System | -s |
System entries: partition headers, filesystem.yaml, etc. |
Update mode (-u)
By default, the mount is strictly read-only to protect your archives. Adding --update (or -u)
enables limited write semantics directly from your OS file explorer:
- Rename images — rename a game file or folder in the mount and NKDS updates the image name in the database. This applies to the top-level image entries only, not files within the filesystem view.
-
Delete images — delete (or send to trash) a game in the mount and NKDS runs a soft-delete
(
remove) on that image. The image disappears from the mount immediately but its block data remains intact until you runnkds compact.
Update mode does not allow modifying the contents of disc images or filesystem views — only renaming and deleting the images themselves.
Examples
# Mount all sets (default view: images + filesystem)
nkds mount --datastore D:\NKitData --mount N:\
# Mount a single set
nkds mount --datastore D:\NKitData\wii.nkds --mount N:\
# Show only the filesystem folder view (no ISO files)
nkds mount -ds D:\NKitData -m N:\ -fs
# Show images and system entries, no filesystem folders
nkds mount -ds D:\NKitData -m N:\ -i -s
# Mount with update mode (allows renaming and deleting images from Explorer)
nkds mount -ds D:\NKitData -m N:\ -u
# Linux
nkds mount --datastore /var/lib/nkitdata --mount /mnt/nkit
# Mount with specific ownership and multi-user access (Linux)
nkds mount --datastore /var/lib/nkitdata --mount /mnt/nkit --uid 1000 --gid 1000 --allow-other
Unmounting — Press Ctrl+C in the nkds mount console window, or use your OS unmount
command. After unmount, nkds checkpoints the SQLite WAL files automatically.
| Alias | Full Command | Description |
|---|---|---|
ls |
list |
List images in a DataStore |
rm |
remove |
Soft-delete images |
cp |
compact |
Permanently delete removed image data |
vfy |
verify |
Verify stored images |
Use either the alias or the full command name interchangeably.
When mounted, the virtual drive is organised as:
<mount>/
GameCube/
Example Game A.iso ← image view (-i)
Example Game A/ ← filesystem view (-fs)
DATA/
opening.bnr
sys/
boot.bin
Wii/
Example Game B.iso
Example Game B/
DATA/
main.dol
Race/
Course/
...
WiiU/
...
System folders (-s) contain entries prefixed with / in the stored filesystem.yaml. These
are hidden unless --system mode is active.
You can store all systems in a single set (multisys is a common convention):
nkds create --datastore D:\NKitData multisys
nkds add --datastore D:\NKitData\multisys.nkds D:\WiiISOs D:\GCISOs D:\PS2ISOs
nkds mount --datastore D:\NKitData\multisys.nkds --mount N:\
The virtual filesystem groups images by system automatically. System folders appear at the root of the mount point regardless of whether the images came from separate input directories.
If you point --datastore to a directory (e.g., D:\NKitData) rather than a specific .nkds database, nkds allows all images across all sets in that directory to be mounted and listed together.
This is incredibly powerful for managing 1G*R (1 Game, * ROMs / 1 game, many ROMs) collections. A 1G*R collection means keeping all the different regional releases, revisions, and language variants of each game. Because of NKDS's internal file-level deduplication, storing all these overlapping versions of the same game takes almost no additional space compared to storing just one.
When you mount a folder containing many smaller sets (for instance, organized by genre, region, or alphabetical chunks), the virtual drive merges them on the fly. The filesystem groups images by system automatically, perfectly reconstructing a single, unified view of your entire 1G*R library regardless of which underlying .nkds file holds the data.
In a 1G*R collection you keep every regional release and revision of every game. Wii and Wii U disc images all contain large update partitions that are nearly identical across every title in a region — often the exact same system menu update bundled into hundreds of discs. NKDS's block deduplication already collapses these into shared storage, but the update blocks still inflate your primary set's index and shard files. When your goal is a lean, portable primary set that holds only the unique game data, the update partitions are dead weight.
An Aux DataStore is a secondary set that holds update partition data separately from your primary game data. This keeps your primary set lean and focused on unique game content, while the shared update data lives in a single, highly-deduplicated aux store.
Key benefits:
- Smaller primary sets — game-only data is significantly smaller without update partitions. For a full 1G*R Wii library, the aux store can reduce primary set size substantially because every regional variant shares the same update partition
- Massive aux deduplication — hundreds of games sharing the same update partition compress down to nearly nothing in the aux store
- Transparent fallback — mounting, exporting, and verifying work identically whether or not the aux store is present
- Graceful degradation — without the aux store, games still mount and play; update partition regions simply read as zeros
- Portable primary — carry just the primary set on a small drive for gaming; keep the aux store on a larger archive drive for full 1:1 reconstruction when needed
This pairs naturally with the 1G*R workflow: your primary set holds the unique game content (which deduplicates heavily across regional variants thanks to filesystem-level dedup), and the aux set holds the update partitions (which deduplicate almost perfectly since they're identical across all titles in a region). Together they give you a complete, verifiable 1:1 archive at a fraction of the raw storage cost.
Aux stores use a simple naming convention. No CLI flags, config files, or special paths are
needed. NKDS auto-discovers the aux store by looking for a .aux.nkds file in the same
directory as your primary sets.
The aux store is typically named by system — for example, wii.aux.nkds holds update partition
data for all Wii sets in that directory. All primary sets in the same directory share the same
aux store automatically. You choose the aux name when you create it (e.g. wii.aux,
wiiu.aux, or simply update.aux if you prefer a single shared store for all systems).
If the aux file exists in the same directory, NKDS automatically enables aux routing (on writes) or aux fallback (on reads). If it doesn't exist, everything works exactly as before — no behaviour change whatsoever.
The ideal 1G*R layout uses shard size 0 (embedded single-file sets) so each game and all
its regional variants live in one portable .nkds file, with a shared aux store holding the
update partitions that are common across all of them.
# 1. Create per-game sets with shard size 0 (one file per game, all variants inside)
nkds create --datastore D:\NKitData "Example Racer Wii" --shard-size 0
nkds create --datastore D:\NKitData "Example Fighter Wii" --shard-size 0
# 2. Create the shared aux set (holds update partitions for all games)
nkds create --datastore D:\NKitData wii.aux
# 3. Add all regional variants of a game — aux routing happens automatically
nkds add --datastore "D:\NKitData\Example Racer Wii.nkds" D:\WiiISOs\ExampleRacer\
nkds add --datastore "D:\NKitData\Example Fighter Wii.nkds" D:\WiiISOs\ExampleFighter\
Because wii.aux.nkds exists in the same directory, NKDS automatically routes update partition
blocks to the aux store and game data blocks to the primary store. Each game's primary set
contains only the unique game content (which deduplicates heavily across regional variants),
while the aux store holds the update partitions shared across all titles.
Resulting file layout:
D:\NKitData\
Example Racer Wii.nkds ← all regional variants (USA, EUR, JPN, etc.) in one file
Example Fighter Wii.nkds ← all regional variants in one file
wii.aux.nkds ← shared aux index (update partition blocks)
wii.aux_0000.nkds ← aux shard
wii.aux_0001.nkds ← aux shard (grows as needed)
Each primary .nkds file is self-contained and portable — copy it to another machine and the
game works (update regions read as zeros without the aux store). When the aux store is present,
full 1:1 reconstruction is available for verification and export.
| System | Aux Content | Primary Content |
|---|---|---|
| Wii | Update partition (PartitionHeader, FileSystem, Other areas) | Game partition, Channel partition, all other data |
| Wii U | Update partition content areas | Game partition, SI partition, all other data |
No special flags needed. If the aux store exists, it's used automatically:
nkds mount --datastore D:\NKitData\wii.nkds --mount N:\
When a block is requested, NKDS checks the primary store first. If the block isn't there (because it's an update partition block), it falls back to the aux store. If the block isn't in either store, zeros are returned. This three-tier resolution (primary → aux → zeros) is completely transparent to emulators and tools.
The aux store is entirely optional. You can move or delete it at any time:
- Games still mount and play perfectly — game data is entirely in the primary store
- Update partition regions read as zeros (emulators typically don't need them)
- Verification and export will report missing blocks but won't crash
The aux store must use the same block size as the primary set. This is validated automatically
when the aux store is opened. If you need to recreate the aux store, use the same --block-size
value you used for the primary set.
The aux store is a standard NKDS set. All normal commands work on it:
# Check aux store stats
nkds stats --datastore D:\NKitData\wii.aux.nkds
# List images in the aux store
nkds list --datastore D:\NKitData\wii.aux.nkds
# Compact the aux store
nkds compact --datastore D:\NKitData\wii.aux.nkds
💡 TIP: When you remove and compact images from your primary set, remember to also compact the aux set to reclaim the corresponding update partition blocks.
| Use case | Shard size | Block size |
|---|---|---|
| Large local library (HDD) | 50 GiB (default) | 64 KiB (default) |
| NAS with small volume quota | 10–20 GiB | 64 KiB |
| Portable / optical distribution |
0 (single embedded file) |
64 KiB |
| Very large images (Wii/PS3) | 50–100 GiB | 128 KiB |
| Maximise deduplication across small files | 50 GiB | 32 KiB |
Smaller block sizes improve deduplication granularity at the cost of a larger block table index.
Larger block sizes compress more efficiently but may miss deduplication opportunities when two
images differ by small edits. The 64 KiB default is a good balance for disc images.
Block size cannot be changed after a set is created. If you need a different block size, create a new set and re-add your images.
The .nkds index file (separate-mode) uses a compact binary layout with two 256-byte headers, a reserved directory region, and a data region.
┌──────────────────────────────────────────────────────────────┐
│ Primary Header (256 bytes, 0x000–0x0FF) │
├──────────────────────────────────────────────────────────────┤
│ Secondary Header (256 bytes, 0x100–0x1FF) │
├──────────────────────────────────────────────────────────────┤
│ Directory Region (variable capacity, starts at 0x200) │
│ [zstd-compressed Image Directory] │
│ [unused padding up to DirectoryRegionCapacity] │
├──────────────────────────────────────────────────────────────┤
│ Data Region (starts at 0x200 + DirectoryRegionCapacity) │
│ Image sections, Block Index, Deltas, etc. │
└──────────────────────────────────────────────────────────────┘
The primary and secondary headers are identical copies for crash-safe atomic commits. The secondary header acts as a backup — if the primary is corrupted mid-write, the secondary provides a known-good state.
| Offset | Size | Field | Description |
|---|---|---|---|
| 0x00 | 4 | Magic |
0x4E4B4453 ("NKDS") |
| 0x04 | 2 | MajorVersion | 1 |
| 0x06 | 2 | MinorVersion | 0 |
| 0x08 | 8 | ImageDirectoryOffset | File offset to compressed directory |
| 0x10 | 4 | ImageDirectoryCompressedSize | Compressed size in bytes |
| 0x14 | 4 | ImageDirectoryUncompressedSize | Uncompressed size for buffer pre-allocation |
| 0x18 | 8 | BlockIndexOffset | File offset to main Block Index |
| 0x20 | 4 | BlockIndexSize | Size of main Block Index |
| 0x24 | 4 | BlockIndexDeltaCount | Number of deltas |
| 0x28 | 8 | BlockIndexDeltaHeadOffset | Head of delta linked list |
| 0x30 | 8 | FileEndOffset | Logical end of valid data |
| 0x38 | 8 | ShardSize | Shard rotation size (0 = embedded) |
| 0x40 | 4 | BlockSize | Block size in bytes |
| 0x44 | 4 | MaxOffsetBlocks | Max block keys per offset |
| 0x48 | 4 | ImageCount | Total images (including removed) |
| 0x4C | 4 | DirectoryRegionCapacity | Byte capacity of reserved directory region |
| 0x50 | 8 | HeaderChecksum | XXHash64 of bytes 0x00–0x4F |
| 0x58–0xFF | 168 | reserved | Zero-filled |
The Image Directory is serialized and then compressed with Zstandard (zstd). A reserved region starting at offset 0x200 holds the compressed directory. When the compressed directory fits within DirectoryRegionCapacity, it is written in-place — no append or file growth is needed. This makes commits for small-to-medium sets (up to ~500 images) very fast.
When the compressed directory grows beyond the reserved capacity, it overflows to the end of the file. The ImageDirectoryOffset header field always points to the current location of the directory, whether in-place or appended.
Running nkds compact rewrites the index file and restores the in-place directory layout. The new DirectoryRegionCapacity is set to the current compressed directory size plus 4 KiB of growth headroom, so subsequent image additions don't immediately overflow again.
When you create a set with shardSize=0, the entire set is stored as a single .nkds file. There is no shard rotation — block data, the binary index, and a small footer are all packed into one file. This makes the set trivially portable: copy one file and you have everything.
┌─────────────────────────────┐
│ Block data (variable) │ ← compressed, deduplicated blocks
├─── Shard Boundary ──────────┤
│ Binary Index (variable) │ ← headers, directory region, image sections, block index
├─────────────────────────────┤
│ 8-byte Index_Size (BE) │ ← byte length of the binary index region
├─────────────────────────────┤
│ 4-byte "NKDS" magic │ ← footer identifying embedded format (0x4E4B4453)
└─────────────────────────────┘
Reading the file works from the tail: read the last 4 bytes to confirm the "NKDS" magic, then read the preceding 8 bytes (big-endian int64) to get the index size. The binary index starts at file_size - 12 - Index_Size (the "Shard Boundary"). The total footer is 12 bytes.
The binary index is read in-place — no extraction or decompression to a temporary file is needed. The index uses the same format as separate-mode .nkds files (see "Binary Index File Layout" above), just positioned at an offset within the embedded file rather than at offset 0.
NKDS automatically detects whether a .nkds file is in embedded mode or separate mode by checking the last 4 bytes for the "NKDS" magic. No configuration flags or naming conventions are needed — the file format is self-describing.
The binary index cannot be modified in place while it sits in the middle of the file (between block data and footer). Every write operation follows a rename-based state machine:
- Extract — The binary index is extracted to a standalone file, and the embedded file is renamed to a plain shard file. After this step, the set looks like a normal separate-mode set (index file + shard file).
- Write — All write operations (add, remove, compact, etc.) proceed normally against the standalone index and shard, exactly as they would with a non-zero shard size. New blocks are appended to the shard.
- Re-embed — The updated index and a new footer are appended to the shard, and the file is renamed back to the embedded filename, restoring the single-file format.
Each intermediate state is unambiguously detectable by which files exist and their names, enabling perfect crash recovery at any point.
The extract/re-embed cycle is lightweight compared to the old SQLite-based approach:
- No compression/decompression — The binary index is not compressed as a whole, so extraction and re-embedding are simple file copies and renames.
- Zero-copy reads — Reading the embedded index is a seek to the Shard Boundary followed by in-place reads. No temporary files are created for read operations.
- Block data untouched — The block data region (bytes 0 to Shard Boundary) is never modified during writes or compaction. Only the index region at the end of the file changes.
For most workflows (including 1G*R collections with frequent adds), embedded sets perform comparably to separate-mode sets. The overhead is limited to a few file renames per write transaction.
The rename-based state machine ensures that a crash at any point leaves the file system in a recoverable state:
| Files on disk after crash | Recovery action |
|---|---|
Only test.nkds with valid footer |
Normal embedded file — no recovery needed |
test.nkds + test_0000.nkds
|
Mid-write state — treated as separate-mode set, writing can continue |
test_0000.nkds.tmp with footer magic |
Re-embed was complete, rename interrupted — rename to test.nkds
|
test_0000.nkds.tmp without footer magic |
Append interrupted — re-append index from test.nkds, then rename |
test.nkds.tmp alongside test.nkds
|
Extraction interrupted — delete .tmp, embedded file is intact |
Recovery is automatic on next open — no manual intervention required.
Embedded sets support the same concurrency model as separate-mode sets:
- Multiple concurrent readers — Readers use position-independent I/O and can access block data and the index simultaneously without locking.
- Single writer — During writes, the set is temporarily in separate-mode layout. Readers with open file handles continue reading from the old file until the re-embed rename completes.
- Atomic visibility — The final rename is the commit point. Readers either see the old complete state or the new complete state, never a partial update.
- Portable distribution — a single file is easy to copy, move, or share.
- 1G*R collections — each game and all its regional variants in one portable file.
- Small to medium collections that benefit from single-file simplicity.
- Archival snapshots where the set will not be modified after initial creation.
Shard size 0 works well for most use cases. For very large sets (hundreds of GB of block data) where you want to limit individual file sizes, use a non-zero shard size to enable shard rotation.
NKDS mounts are designed to be lightweight. Metadata is loaded eagerly when the mount starts, but actual block data is loaded on demand — only when a file is read. This means you can mount one set or many sets simultaneously with a minimal memory footprint. The system never loads an entire disc image into memory; it reads only the blocks you actually access.
When nkds mount starts, it calls DescribeSetsWithImages() to load image metadata from one or multiple sets. This loads image records — names, sizes, formats, and systems — and optionally pre-loads filesystem.yaml data so that folder navigation is available immediately.
No actual block data is read at this stage. Only the SQLite index databases are queried. For embedded sets (shardSize=0), the database is extracted to a temporary directory for reading.
When a user opens (accesses) a specific image, the mount system acquires three shared resources from the MountResourceManager:
-
ImageReader — loads all block offset locations for that image into an in-memory cache. The reader knows exactly where every block lives in the shard files, so subsequent reads are simple seeks rather than database lookups.
-
OffsetsManager — a pre-built index of area records, offset records, and per-section segment maps for the image. This is constructed once (querying SQLite for areas and offsets, then building section/segment indexes) and cached for all subsequent file handles on the same image. Without this cache, every file open would repeat the same expensive SQLite queries and index construction — roughly 1 second per open for images with many areas and thousands of offset records (common with TmdAppFolder child images).
-
Buffer Cache — a per-image pool of decompressed section buffers shared across all concurrent file handles reading the same image.
All three resources use the same lifecycle pattern: reference-counted with a TTL grace period. Each resource is created on first access, shared across all concurrent file handles for that image, and only disposed after the last handle releases it and a configurable TTL expires without re-acquisition. This avoids expensive teardown/rebuild churn from filesystem drivers that rapidly open and close handles during normal browsing.
Block offsets are small (a few bytes per block), so even large images consume minimal memory for their offset table.
When a file within an image is read, the reader fetches only the blocks needed for that file region. The per-image buffer cache provides:
- 2 MB default buffer size (one section's worth of decompressed data)
- Maximum 16 buffers per image
- LRU (Least Recently Used) eviction — frequently accessed regions stay cached, rarely accessed regions are evicted
Decompression buffers are reused across reads to minimize memory allocation churn. This means the system does not allocate and free memory for every block decompression — it recycles a pool of buffers instead.
Because the buffer cache is shared across all file handles for the same image, two applications reading different files from the same disc image benefit from each other's cached sections.
All shared mount resources (readers, OffsetsManagers, buffer caches) follow the same lifecycle:
- First access — resource is created and cached with ref count = 1.
- Additional handles — ref count increments. No new construction occurs.
- Handle close — ref count decrements. While ref count > 0, the resource stays alive indefinitely.
- Last handle closes — ref count reaches zero. A TTL timer starts (default 30 seconds).
- Re-opened before TTL — timer is cancelled, ref count goes back up. No reconstruction cost.
- TTL expires — resource is disposed and removed from the cache. Next access will reconstruct it.
- Shutdown — all resources are immediately disposed regardless of ref count or TTL state. The OffsetsManager cache is cleared first, then buffer caches, then readers — ensuring no resource holds references to an already-disposed dependency.
This design means:
- Rapid open/close cycles (common with FUSE readdir → open sequences) reuse cached resources with zero reconstruction cost.
- Idle images gradually release memory after the TTL window.
- Multiple concurrent readers of the same image share a single set of resources rather than each building their own.
graph LR
subgraph "Mount Start"
A[DescribeSetsWithImages] -->|image metadata| B[VfsModel]
end
subgraph "Image Open"
B -->|user accesses image| C[MountResourceManager]
C -->|acquire| D[ImageReader<br/>ref-counted + TTL]
C -->|acquire| E[OffsetsManager Cache<br/>ref-counted + TTL]
C -->|acquire| F[Buffer Cache<br/>ref-counted + TTL]
end
subgraph "File Read"
D -->|seek to block| G[Shard File]
G -->|decompress| F
F -->|bytes| H[FUSE/Dokan response]
end
- Mounting thousands of images uses only the memory needed for image metadata — not the images themselves.
- Reading a single file from a disc image touches only the blocks that contain that file.
- OffsetsManager cache eliminates repeated SQLite queries — opening the same image 100 times queries the database only once.
- Buffer cache serves repeated reads of the same region from memory — common with media players and emulators that re-read headers or small files frequently.
- Shared resources mean multiple concurrent file handles on the same image add negligible overhead beyond the first open.
- TTL grace period prevents expensive teardown/rebuild cycles during normal filesystem browsing patterns.
- Multiple sets can be mounted together with no additional overhead beyond their metadata.
nkds add is a convenience wrapper around nkit -task dedupe. They produce identical results.
Use nkds add when you want a simple command without managing an NKit YAML config file. Use
nkit directly when you need full control over the pipeline (custom fixup files, DAT lookups,
specific conversion options).
NKit recognises and correctly processes images for:
| System | Formats |
|---|---|
| Nintendo GameCube | ISO, GCM, NKit.ISO, RVZ, CISO, WBFS |
| Nintendo Wii | ISO, NKit.ISO, RVZ, CISO, WBFS, GCZ, WIA |
| Nintendo Wii U | WUX, WUD, APP/TMD |
| Sony PlayStation 1 | BIN/CUE |
| Sony PlayStation 2 | ISO, CSO, ZSO, BIN/CUE |
| Sony PlayStation 3 | ISO, CSO |
| Sony PSP | ISO, CSO |
| Microsoft Xbox | ISO, XISO |
| Microsoft Xbox 360 | ISO |
| Sega Dreamcast | GDI, BIN/CUE, CHD |
| Sega Saturn | BIN/CUE |
| Sega CD / Mega CD | BIN/CUE |
| Philips CD-i | BIN/CUE |
| NEC PC Engine CD | BIN/CUE |
Standard ISO 9660 images not matching any of the above are also supported via the default
system handler.
nkds-ui is the graphical companion to the nkds command-line tool. It provides a visual interface for browsing, managing, and operating on DataStores and sets.
The UI includes a Settings panel (accessible from the toolbar) that lets you register OS-level file associations and context menu entries. These integrate NKDS directly into your file manager (Windows Explorer or Linux file managers).
Directory (folder) context menu:
| Entry | Action |
|---|---|
| NKDS Open | Opens the folder as a DataStore |
| NKDS Mount | Opens the folder as a DataStore and shows the Mount dialog |
| NKDS Add Dir | Prompts for a .nkds set, then adds the folder contents to it |
| NKDS Add Dir New | Prompts to create a new set, then adds the folder contents to it |
File context menu (.nkds files):
| Entry | Action |
|---|---|
| NKDS Open Set | Opens the .nkds file as a set |
| NKDS Mount Set | Opens the .nkds file and shows the Mount dialog |
File context menu (all files):
| Entry | Action |
|---|---|
| NKDS Add | Prompts for a .nkds set, then adds the selected file(s) to it |
| NKDS Add New | Prompts to create a new set, then adds the selected file(s) to it |
Double-click handler (.nkds files):
| Entry | Action |
|---|---|
| Open as Set | Opens the .nkds file as a set |
- Files: When multiple files are selected and you click "NKDS Add" or "NKDS Add New", all selected files are batched into a single operation. The app prompts once for the target set, then processes all files sequentially with progress tracking.
- Directories: Windows Explorer does not support multi-select invocation for directory context menu entries (this is a Windows shell limitation). Single directory selection works correctly.
On Linux, the Settings panel includes a toggle between client-side decorations (custom title bar) and native OS title bar. Changes require an application restart to take effect.
nkds-ui accepts command-line arguments for automation and OS integration. These are the same arguments used by the registered context menu entries.
# Open a directory as a DataStore
nkds-ui --datastore "D:\NKitData"
nkds-ui -ds "D:\NKitData"
# Open a specific .nkds set file
nkds-ui --set "D:\NKitData\wii.nkds"
nkds-ui --datastore "D:\NKitData\wii.nkds"# Open a DataStore and immediately show the Mount dialog
nkds-ui --action mount --datastore "D:\NKitData"
# Open a set and show the Mount dialog
nkds-ui --action mount --set "D:\NKitData\wii.nkds"# Add one or more files — prompts for the target .nkds set
nkds-ui --action add --input "D:\ISOs\game.iso"
nkds-ui --action add --input "D:\ISOs\game1.iso" --input "D:\ISOs\game2.iso"
# Add a directory's contents — prompts for the target .nkds set
nkds-ui --action adddir --input "D:\MyFolder"# Add files to a new set — prompts for folder location and set name/options
nkds-ui --action addnew --input "D:\ISOs\game.iso"
nkds-ui --action addnew --input "D:\ISOs\game1.iso" --input "D:\ISOs\game2.iso"
# Add a directory to a new set — prompts for folder location and set name/options
nkds-ui --action adddirnew --input "D:\MyFolder"| Argument | Description |
|---|---|
--datastore, -ds <path> |
DataStore directory or .nkds file to open |
--set <path> |
.nkds set file to open |
--action <action> |
Action to perform after opening: mount, add, adddir, addnew, adddirnew
|
--input <path> |
Input file or folder path (repeatable for multiple inputs) |
nkds-ui enforces single-instance execution. When multiple instances are launched (e.g., from multi-selecting files in Explorer), subsequent instances forward their arguments to the already-running instance via a named pipe and exit immediately. The running instance accumulates all input paths and processes them in a single batch operation.
This means selecting 10 files → right-click → "NKDS Add" results in one set picker prompt followed by all 10 files being added sequentially with progress tracking.
No images found in DataStore set '…' matching name '…'
Verify is looking for an image that does not exist in the set. Check that nkds add completed
successfully for that image (nkds list --search "partial name").
Set '…' does not exist. Call CreateSet before AddImage.
The .nkds file does not exist yet. Run nkds create first, or point --datastore at an
existing .nkds file.
Database schema version mismatch.
The .nkds index file was created by a different version of NKitDataStore. It cannot be opened by this version. Re-create the set and re-add
your images.
Mount fails on Windows
nkds mount on Windows requires Dokan or
WinFsp to be installed. Install either driver and retry.
Mount fails on Linux
Ensure FUSE is available: sudo apt install libfuse-dev (Debian/Ubuntu) or the equivalent for
your distribution.
Images were added but the shard files seem too small
If many of your images share large regions (same game engine, same SDK libraries, update partitions
shared across multiple titles) the deduplication ratio can be very high. Use nkds stats to see
the actual shrink ratio.
Images were removed but disk space was not reclaimed
nkds remove only sets a soft-delete flag — the block data for removed images remains in the shard files. To permanently delete the data and reclaim space, run nkds compact on the set. Until you compact, the shard files will stay the same size.
Writes to an embedded set (shardSize=0) seem slow
Every write to an embedded set requires an extract-write-re-embed cycle: the binary index is
extracted from the single file, the write is performed, and the updated index is re-appended.
For very large sets with frequent modifications, this overhead adds up. Consider creating a new
set with a non-zero shard size (e.g., --shard-size 50GiB) and re-adding your images.