Manifest Schema and Row Lifecycle

Relevant source files

The following files were used as context for generating this wiki page:

Purpose and Scope

This section details the design and implementation of the SQLite manifest database in clarity/manifest.py. The manifest serves as the single source of truth for the upscaling pipeline, tracking every vanilla FFXIV texture targeted for processing, its classification into families and roles, its current execution status, and its metadata.

Sources: [clarity/manifest.py:1-22](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/manifest.py?plain=1#L1-L22)

Manifest Database Schema and Migrations

The manifest is backed by a SQLite database containing two primary tables: tex for tracking individual texture assets and fpsnap for audit trails of patch fingerprints.

The core schema definition establishes the tex table with a primary key on asset paths, along with an index optimized for family, role, and status queries. Post-ship schema expansions are handled via incremental migrations that check for column existence.

-- SQLite Schema Definition in clarity/manifest.py
CREATE TABLE IF NOT EXISTS tex (
    path TEXT PRIMARY KEY, family TEXT, part TEXT, role TEXT,
    w INTEGER, h INTEGER, fmt TEXT, mips INTEGER, bytes INTEGER, ttype TEXT,
    status TEXT DEFAULT 'planned', tiers TEXT DEFAULT '', note TEXT DEFAULT '', updated REAL
);
CREATE INDEX IF NOT EXISTS tex_family ON tex(family, role, status);

Schema migrations dynamically introduce columns for execution recipes and source hashing without destroying existing state:

-- Migrations executed during Manifest initialization
ALTER TABLE tex ADD COLUMN recipe TEXT DEFAULT ''
ALTER TABLE tex ADD COLUMN srchash TEXT DEFAULT ''
ALTER TABLE tex ADD COLUMN srcver TEXT DEFAULT ''

---
title: Manifest SQLite Schema and Migration Mapping
---
flowchart TD
    subcode[Code Entity Space]
    subcode --> DB[sqlite3.Connection]
    DB -->|Executes SCHEMA| TEX[Table: tex]
    DB -->|Executes SCHEMA| FPS[Table: fpsnap]
    DB -->|Executes MIGRATIONS| COL1[Column: recipe]
    DB -->|Executes MIGRATIONS| COL2[Column: srchash]
    DB -->|Executes MIGRATIONS| COL3[Column: srcver]
    TEX --> IDX[Index: tex_family on family, role, status]

Sources: [clarity/manifest.py:102-129](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/manifest.py?plain=1#L102-L129)

Row Statuses and Lifecycle

Each row in the manifest progresses through distinct operational statuses managed during planning, execution, and maintenance commands.

Status	Meaning
`planned`	Discovered and classified, awaiting encoding/upscaling.
`done`	Successfully processed and written to the Penumbra mod destination.
`failed`	Encountered an error during decoding, inference, or encoding.
`skipped`	Evaluated as unsupported or lacking a valid processing path.

Row notes and timestamps track patch deltas. When a vanilla file changes between game patches, a fingerprint pass marks the row with a changed at <version> note. Requeue operations clear failure states while explicitly preserving patch-delta markers to ensure incremental --since runs process updated assets.

---
title: Manifest Row Lifecycle and Status Transitions
---
flowchart TD
    subcode[Code Entity Space]
    subcode --> PLAN[Manifest.add / Enumeration Entrypoints]
    PLAN -->|Status: planned| ST_PLANNED[Row State: planned]
    
    subcode --> RUN[clarity.cli.cmd_run]
    ST_PLANNED -->|Inference & Encoding| RUN
    RUN -->|Success| ST_DONE[Status: done]
    RUN -->|Error| ST_FAILED[Status: failed]
    
    subcode --> SYNC[Manifest.sync_skipped]
    ST_PLANNED -->|No Processing Path| ST_SKIP[Status: skipped]
    
    subcode --> REQUEUE[clarity.cli.cmd_requeue]
    ST_FAILED -->|Reset State| ST_PLANNED
    ST_SKIP -->|Reset State| ST_PLANNED

Sources: [clarity/manifest.py:102-107], [clarity/cli.py:22-42], [tests/test_manifest_requeue.py:13-32](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/tests/test_manifest_requeue.py?plain=1#L13-L32)

Manifest Core API (`Manifest` Class)

The Manifest class abstracts all SQLite interactions. Key methods include:

add(): Inserts or updates texture records while evaluating skip reasons.
rows(): Queries rows filtered by family, role, status, or update timestamps.
set_status(): Updates processing statuses and writes descriptive notes.
snapshot(): Records fingerprint execution metrics in the fpsnap table.
summary(): Aggregates counts and source byte sizes grouped by family, role, and status.

Sources: [clarity/cli.py:44-92], [clarity/manifest.py:300-450](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/manifest.py?plain=1#L300-L450)

Enumeration Entrypoints

To populate the manifest without external dependencies for core assets, clarity provides generative enumeration functions alongside pathlist ingestion:

gen_chara(man, gd): Walks character sets (equipment, accessory, weapon, monster, demihuman, human) using .mdl material paths and .imc variant tables to resolve .mtrl references and their associated textures [clarity/cli.py:25-26](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/cli.py?plain=1#L25-L26).
gen_icons(man, gd): Probes UI icon ID ranges directly via the game data accessor [clarity/cli.py:27-28](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/cli.py?plain=1#L27-L28).
from_pathlist(man, gd, path): Parses ResLogger text files or arbitrary lists containing one game path per line for environment assets (bg/, bgcommon/, ui/uld/) [clarity/cli.py:29-30](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/cli.py?plain=1#L29-L30).

Sources: [clarity/cli.py:22-42], [clarity/manifest.py:1-22](https://github.com/off-cmd/XIVUpscaler/blob/5c9360ae/clarity/manifest.py?plain=1#L1-L22)

Home · Repository · Migrated from DeepWiki