Skip to content

NKDS Data Safety

Jump to bottom
Nanook edited this page Sep 24, 2026 · 1 revision

Plain-language answers to "is my data safe?" — what protections NKDS has, what compaction does, what changed in v3, and what you should do to be careful.

For the technical detail behind these protections, see NKDS Storage Model.

The Short Version

  • Adding images is safe. NKDS appends new data and commits atomically. A power cut mid-add leaves your previously stored images intact. The new image either fully appears or doesn't appear at all — there is no partial-add state.

  • Removing (soft-delete) is safe and reversible. nkds remove marks images as deleted but does not touch the data. You can undo it with nkds restore at any time before compacting.

  • Compaction was the problematic operation in earlier versions. It was fixed in v3.0.0. If you used NKDS before v3 and ran compaction, run nkds verify to confirm your data is intact.

  • Do not delete your source files immediately after storing them. This is good practice with any storage tool until you have verified the stored copies. Once you have run nkds verify and are happy, you can safely rely on the store.

What Each Operation Does to Your Data

nkds add / nkds ogmr

  • Runs the full NKit pipeline: reads the source, normalises, deduplicates, compresses, stores
  • Appends new block data to the shard — nothing previously stored is touched
  • Atomically commits using the dual-header protocol — the new image either fully appears or not at all
  • A power cut at any point during add leaves all previously stored images completely intact
  • Source files are never modified or deleted by add

nkds remove

  • Marks the image as soft-deleted in the index
  • Block data is not touched — it remains in the shard
  • The image disappears from nkds list and from VFS mounts
  • Fully reversible with nkds restore until you run nkds compact

nkds restore

  • Reverses a soft-delete
  • Can only be used before nkds compact has run
  • After compaction, the block data for removed images is gone and restore is not possible

nkds compact

  • This is the only operation that permanently deletes data
  • Removes block data for images that have been soft-deleted
  • Rebuilds shards using only the live (non-deleted) block data
  • Writes to temp files first, validates them, then atomically replaces the originals
  • Only runs if you explicitly call it — it never runs automatically

Best practice for replacing a bad dump:

nkds remove  (soft-delete the old image)
nkds add     (add the new image — it will deduplicate against the old blocks)
nkds compact (now safe to reclaim the space)

Do not compact between remove and add — the new image will deduplicate better if the old blocks are still present.

nkds verify

  • Reads every stored block, decompresses it, and checks it against its stored checksum
  • Reports any block that does not match — nothing is modified
  • Safe to run at any time, including while a mount is active
  • Does not modify or delete anything

nkds export

  • Reads from the store and reconstructs the image to a file
  • Does not modify the store
  • The reconstructed image is byte-identical to the original

Crash Safety

NKDS uses three layers of protection against power loss or crashes:

1. Append-only writes. New block data is always appended to the end of the shard. Existing data is never overwritten. A crash mid-write leaves the previously committed data intact.

2. Dual-header atomic commit. The index header is written twice — secondary first, primary last. If a crash happens between the two writes, the secondary (which was fully written) is used as authoritative. The primary is the commit point; the secondary is the fallback.

3. Temp-file compaction with promote-or-delete recovery. Compaction writes to .nkds.tmp files first. On recovery (next time you access the set), NKDS checks the committed index to decide whether the compaction reached its commit point — if yes, the temp files are promoted (renamed over the originals); if no, they are deleted and the originals are used. There is no ambiguous state.

What Changed in v3.0.0

Earlier versions of NKDS had a compaction bug that could result in data loss under specific conditions. This happened because compaction was writing directly over existing shard data without first validating the new shard, and without the promote-or-delete recovery protocol.

v3.0.0 fixes:

  • Compaction now writes to temp files first and validates them before replacing originals
  • If validation fails, the originals are left untouched
  • On next open, promote-or-delete recovery resolves any interrupted compaction deterministically
  • The dual-header atomic commit protocol is used for the index commit during compaction
  • These mechanisms are covered by automated tests

If you used NKDS before v3 and ran compaction: Run nkds verify on your sets to confirm the block data is intact. Images stored before compaction ran are most likely fine — the bug was in the compaction path, not the add path.

If you used NKDS before v3 but never ran compaction: Your data is intact. The add path was always safe.

Recommendations

While NKDS matures:

  • Keep your source files until you have verified the stored copies with nkds verify
  • Use nkds verify after any compaction, and after upgrading from an older version
  • If you are storing a large collection, consider staggering compaction — add in batches, verify, then compact

General good practice:

  • Store your .nkds and shard files on reliable storage
  • Back up the index file (.nkds) separately from the shards — it is small and contains the full block map needed to understand what is stored
  • If you move your DataStore, move all the files (index + all shards) together

Can I Use NKDS for Long-Term Archival?

Yes, with normal storage hygiene. NKDS is designed for long-term archival use — content-addressed blocks, verified reads, crash-safe commits, and byte-perfect reconstruction are all archival-quality properties.

The things to be aware of:

  • NKDS is under active development — keep the version you used to add images until a stable release is declared, in case a future format change requires migration tooling
  • Shard files are large binary blobs — store them on reliable media with your normal backup strategy
  • The index file is small and critical — back it up separately

Further Reading

  • NKDS Storage Model — Technical deep-dive into how writes and compaction work
  • NKDS — Core concepts and quick start
  • NKDS CLI — Full command reference

Clone this wiki locally