SQPack Archive Access

Relevant source files

The following files were used as context for generating this wiki page:

Purpose and Scope

This page documents the implementation of the clarity.ffxiv.sqpack and clarity.ffxiv.tex modules ([clarity/ffxiv/sqpack.py:1-199](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/sqpack.py?plain=1#L1-L199), [clarity/ffxiv/tex.py:1-123](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/tex.py?plain=1#L1-L123)). The system provides a lightweight, dependency-free, read-only reader for FFXIV (Win32, DX11) archive files (.win32.index, .win32.index2, and .win32.dat*) without relying on external libraries or game installations. It handles path hashing, index traversal, block decompression, and texture header inspection.

Path Parsing and Hashing

FFXIV organizes assets into categories and optional expansion repositories. Path resolution relies on specific bitwise hash functions matching the game engine's internal structure.

crc32(b: bytes) computes the bitwise-NOT of the standard zlib CRC-32, seeding with 0xFFFFFFFF and omitting the final XOR ([clarity/ffxiv/sqpack.py:42-44](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/sqpack.py?plain=1#L42-L44)).
split_hash(path) isolates the directory and file components by splitting at the final slash and hashing each part separately ([clarity/ffxiv/sqpack.py:46-49](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/sqpack.py?plain=1#L46-L49)).
full_hash(path) hashes the entire normalized lower-case path ([clarity/ffxiv/sqpack.py:52-53](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/sqpack.py?plain=1#L52-L53)).
parse_path(path) extracts the category ID, expansion ID, and chunk ID ([clarity/ffxiv/sqpack.py:56-63](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/sqpack.py?plain=1#L56-L63)). Categories are mapped via the global CATEGORIES dictionary ([clarity/ffxiv/sqpack.py:20-36](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/sqpack.py?plain=1#L20-L36)), and expansions via EXPANSIONS ([clarity/ffxiv/sqpack.py:37](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/sqpack.py?plain=1#L37)).

Sources: [clarity/ffxiv/sqpack.py:20-64], [tests/test_sqpack.py:15-44](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/tests/test_sqpack.py?plain=1#L15-L44)

Index and Repository Architecture

Indices map path hashes to physical .dat file identifiers and byte offsets.

Index parses both standard .index and .index2 files ([clarity/ffxiv/sqpack.py:66-87](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/sqpack.py?plain=1#L66-L87)). Standard indices use 64-bit folder/file composite hashes (<QII), while .index2 uses 32-bit full-path hashes (<2I).
Index.decode(d) extracts the data file ID (dataFileId) and byte offset (offset) from the index entry integer ([clarity/ffxiv/sqpack.py:88-90](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/sqpack.py?plain=1#L88-L90)).
Repository manages a specific expansion directory (ffxiv, ex1, etc.) and lazily loads Index instances per category, chunk, and index variant ([clarity/ffxiv/sqpack.py:93-111](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/sqpack.py?plain=1#L93-L111)).

Sources: [clarity/ffxiv/sqpack.py:66-111], [tests/test_sqpack.py:57-87](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/tests/test_sqpack.py?plain=1#L57-L87)

GameData Accessor and Handle Caching

GameData serves as the primary facade for archive queries ([clarity/ffxiv/sqpack.py:113-168]).

GameData.locate(path) resolves a virtual game path to a physical (dat_path, offset) tuple ([clarity/ffxiv/sqpack.py:124-158](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/sqpack.py?plain=1#L124-L158)). It queries index1 files across chunks (0 through 7) using composite hashes. If no index1 is present (common in benchmark clients), it falls back to querying index2 with full_hash() ([clarity/ffxiv/sqpack.py:143-157](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/sqpack.py?plain=1#L143-L157)).
GameData.read(path) retrieves the raw bytes for a given asset path ([clarity/ffxiv/sqpack.py:163-167](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/sqpack.py?plain=1#L163-L167)).
_DAT_HANDLES and _dat(path) implement a process-wide file handle cache ([clarity/ffxiv/sqpack.py:186-199](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/sqpack.py?plain=1#L186-L199)). Caching .dat file handles eliminates redundant file opening overhead during bulk asset operations.

graph TD
    A["GameData.locate(path)"] --> B["parse_path(path)"]
    B --> C["Repository.index(cat, exp, ch)"]
    C --> D{"index1 exists?"}
    D -->|Yes| E["Lookup composite hash (dh<<32 | fh)"]
    D -->|No| F["Fallback to index2 full_hash()"]
    E --> G["Index.decode() -> (dat_file_id, offset)"]
    F --> G
    G --> H["Repository.dat() -> (dat_path, offset)"]

    sub: "Sources: [clarity/ffxiv/sqpack.py:113-168](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/sqpack.py?plain=1#L113-L168)"

Sources: [clarity/ffxiv/sqpack.py:113-199]

DAT Block Decoding

Data files (.win32.dat*) store assets in structured blocks containing header metadata and compressed or uncompressed payloads ([clarity/ffxiv/sqpack.py:170-183](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/sqpack.py?plain=1#L170-L183)).

_block(f, offset, out) seeks to the block offset, reads the block header (<4I), and extracts the payload ([clarity/ffxiv/sqpack.py:174-183](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/sqpack.py?plain=1#L174-L183)).
If block type equals 32000, the payload is uncompressed ([clarity/ffxiv/sqpack.py:177-178](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/sqpack.py?plain=1#L177-L178)).
Otherwise, the block is decompressed using raw DEFLATE (zlib.decompressobj(-15)) ([clarity/ffxiv/sqpack.py:179-182](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/sqpack.py?plain=1#L179-L182)).

graph TD
    A["SQPack Binary Parser System"] --> B["GameData"]
    B --> C["Repository"]
    C --> D["Index"]
    C --> E["_dat (Cache)"]
    D --> F["Index.decode"]
    B --> G["_block"]
    G --> H["zlib.decompressobj"]

    sub: "Sources: [clarity/ffxiv/sqpack.py:66-199](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/sqpack.py?plain=1#L66-L199)"

Sources: [clarity/ffxiv/sqpack.py:170-183]

Texture Header Parsing (.tex)

The clarity.ffxiv.tex module parses .tex metadata headers ([clarity/ffxiv/tex.py:1-123](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/tex.py?plain=1#L1-L123)).

TexHeader unpacks attributes, format identifiers, dimensions, mip counts, LOD offsets, and surface offsets ([clarity/ffxiv/tex.py:46-74](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/tex.py?plain=1#L46-L74)).
FORMATS maps 16-bit format codes to human-readable surface formats (e.g., 0x1450 to B8G8R8A8, 0x6432 to BC7) ([clarity/ffxiv/tex.py:5-29](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/tex.py?plain=1#L5-L29)).
mip_dimensions(mip) computes mip level dimensions using floor division and clamping at 1, matching Direct3D and Lumina specifications ([clarity/ffxiv/tex.py:94-106](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/tex.py?plain=1#L94-L106)).
surface_size(mip) calculates byte sizes per mip level, incorporating block-compression padding rules ([clarity/ffxiv/tex.py:108-113](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/ffxiv/tex.py?plain=1#L108-L113)).

Sources: [clarity/ffxiv/tex.py:1-124], [tests/test_tex_header.py:1-61](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/tests/test_tex_header.py?plain=1#L1-L61)

Home · Repository · Migrated from DeepWiki