Skip to content

Processing Tasks

Nanook edited this page Mar 20, 2023 · 15 revisions

NKit is able to perform a various tasks when processing an image. Currently there's only limited functionality available:

System Currently Supported Tasks
Convert ConvertSwap Dedupe Expand Extract Fix Scan Verify
GameCube
Wii
WiiU -
PS1 - - - - -
PS2 - - - - -
PS3 - -
PSP - - - - -
Dreamcast - - - - -
Saturn - - - - -
SegaCD - - - - -
CD-i - - - - -
PCEngine - - - - -
Default - - - - -

Scan

Process an entire image and output an xml nkit scan. Scans are a logical hybrid map of the image and filesystems. All the scan data and more is exposed to the nkit engine as it processes images. They also serve as a fingerprint of an image. As they're xml they can be compared easily to determine differences between similar images.

Convert

Convert an image to another format.

Wii / GameCube

Read Write
ISO GCZ WBFS CISO RVZ WIA ISO.DEC NKit.ISO/GCZ
ISO/GCM - - - -
GCZ - - - -
WBFS - - - -
CISO - - - -
RVZ - - - -
WIA - - - -
ISO.DEC - - - -
NKit.ISO/GCZ - - - -

When converting to ISO, NKit will swap to the Expand Task.

RVZ is a fresh implementation from the spec. It's well tested and fully compatible with Dolphin. It is not binary equal to the RVZ created by Dolphin due to NKit inserting the source checksums as XXHash64, MD5, SHA-1 and CRC32. There are also minor (non breaking) differences with the RVZ encoding and zstd implementation.

WBFS and CISO have an option to be lossless or lossy (scrubbed). NKit is the first app to create lossless WBFS and CISO meaning that when expanding back to full ISO no data is lost, the output ISO is exactly the same that was converted to lossless WBFS/CISO.

Verifying lossy WBFS/CISO is not possible due to gap data being removed. If verify is specified it will fall back to DatLookup mode where it will attempt to look up the output image checksum in a dat. For this to yield success you would require a dat of lossy wbfs/ciso files. Otherwise, do not use verify with lossy conversions.

NKit.ISO/GCZ expands to a temp iso before converting to WBFS, CISO or RVZ.

WiiU

Read Write
ISO WUX CDN/tmd* App/*.tmd
ISO/WUD -
WUX -
CDN/tmd* - - -
App/*.tmd - - -

When converting to ISO, NKit will swap to the Expand Task.

Dreamcast

GDI <=> Cue + Bin is planned, but not implemented.

PS3

Encrypted (iso) => Decrypted (dec.iso). With 3K3y header support.

Expand

Expands images to their full state. Currently only implemented for:

System Format
GameCube ISO
Wii ISO
WiiU ISO (a.k.a. WUD
PS3 ISO

ConvertSwap

ConvertSwap switches between the convert task and the expand task. If the image already matches the format specified by the convertFormat setting then it will run Expand, else it will convert it from the current format. This is to mimic behaviour in other apps.

Included with the app are:

  • rvzconvert
  • wuxconvert

These apps are clones of the nkit launch app. They have their own yaml file set up to convertSwap when images / directories are dropped on or passed to them.

Extract

Extract file systems from images. A folder is created based on the source image name and files from the filesystem are extracted to it.

GameCube

Gamecube files are extracted with system files. The typical structure is as follows:

/sys
  apploader.img
  bi2.bin
  boot.bin
  fst.bin
  main.dol         # if not present then there is normally a default.dol in ../files
/files
  opening.bnr
  ...              # all other directories / files

Wii

Wii files are extracted with system files. There will be one folder per partition, typically UPDATE and DATA (sometimes there are others). The typical structure is as follows:

/DATA
  /disc
    header.bin
    region.bin
  /files
  ...              # all other directories / files
  /sys
    apploader.img
    bi2.bin
    boot.bin
    fst.bin
    main.dol       # if not present then there is normally a default.dol in ../files
  cert.bin
  h3.bin
  ticket.bin
  tmd.bin
/UPDATE
  ...              # Same as /DATA - /disc, /files, /sys etc.
/CHANNEL
  ...              # Same as /DATA - /disc, /files, /sys etc.
/P?                # ? being the partition type ID when < 255
  ...              # Same as /DATA - /disc, /files, /sys etc.
/P-????            # ???? being the 4 character ID
  ...              # Same as /DATA - /disc, /files, /sys etc.

WiiU

WiiU extraction will include the data from all partitions. The typical structure is as follows:

/SI
  /02
    title.cert
    title.tik
    title.tmd
  /03              # included when there's a supplementary GM partition
    ...            # same as above
/UP0000000000000000000000000000
  /??????          # date
    ...            # update files
/GM0005000000000000000000000000     # Title ID will differ - sometimes more than one
  /code
    app.xml
    ????.rpx       # title specific boot
    cos.xml
  /content
    ...            # files
  /meta
    ...            # files

ISO9660 Based Systems (Everything else)

The extracted files from these images will match the content of the image as if mounted / extracted by other tools. PS3 will be decrypted if the key is present

ISO9660 Form2 sectors used by PS1 / CD-i that have interleaved audio etc are not fully supported. This will be addressed soon. The Current behaviour is no different from a generic ISO tool

Fix

Fix will recover an image to attempt to match a dat file. This feature existed in NKitv1 for GameCube and Wii. Currently only supports Wii and GameCube and works best with the recovery files from nkit 1.

PS3 IRD recovery will be added at a later date.

Verify

The Verify task will attempt to test the image using various methods. There are 2 modes:-

  • y: The default for the Verify Task (even when v is n)
  • datLookup: Look up the image checksums in a Dat file. If no Dat is specified an error is thrown

When not in datLookup mode NKit will use the following methods to verify the image (in order of preference)

  • Internal Source Image Checksums - NKit will display the methods and checksums
  • DatMatch - checksums located in the dat where the source image name matches a Dat Item
  • NKitScan - matches a .nkit scan. Used when inScan is set and a scan has the same name as the source image
  • DatLookup - use the checksums from the scanned image and look up a match in the Dat

NKit will display the method and any verify checksums being used

See the Verify page for detailed information

Dedupe (Experimental)

Dedupe takes a set of images and extracts all the files from multiple images in a set puts them in a single file structure. Other non-recreatable data is also stored. This greatly reduces the size of a full set of images.

An NKit scan (xml) is produced for each image that is deduped. The scans can then be used as images by nkit. NKit can process the image using the scan and the deduped file set allowing all NKit Tasks and features to be used.

There is also a currently unreleased tool that can mount the NKit scans to a Virtual File System (VFS).

This task is experimental as there are changes still to be made. Deduped sets also bring more complicated management scenarios that need to be addressed, E.g:

  • Removing unused files if a scan is removed
  • Uplifting the deduped file stores when there's breaking changes
  • Validating all the files are still present and valid

There is still work on the file store to be completed. When using this task, DO NOT DELETE YOUR SOURCE IMAGES as it is currently an experimental feature.