Pipeline Lifecycle

Pipeline Lifecycle: plan → run → pack → qa → release

Relevant source files

The following files were used as context for generating this wiki page:

Purpose and Scope

This document provides a technical walkthrough of the stateful pipeline execution stages within XIVUpscaler (clarity). It details how texture assets move through the pipeline lifecycle, the internal mechanics of each stage, and how row states within manifest.sqlite transition across operations.

1. The Stateful Manifest and Row Lifecycle

The pipeline operates on a SQLite database (manifest.sqlite) acting as the single source of truth. Every texture discovered from the game data or pathlists is recorded as a row with a specific status: planned, done, failed, or skipped.

Pipeline Stages to Code Entities

Title: Pipeline Stage Architecture Mapping

graph TD
    subgraph "Natural Language Space"
        PlanStage["Plan Stage"]
        RunStage["Run Stage"]
        PackStage["Pack Stage"]
        ReleaseStage["Release Stage"]
    end

    subgraph "Code Entity Space"
        PlanCode["clarity/cli.py:cmd_plan\nclarity/manifest.py:gen_chara\nclarity/manifest.py:from_pathlist"]
        RunCode["clarity/cli.py:cmd_run\nclarity/cli.py:check_encoder\nclarity/cli.py:check_disk"]
        PackCode["clarity/packaging/penumbra.py:pack"]
        ReleaseCode["clarity/packaging/release.py:export"]
    end

    PlanStage --> PlanCode
    RunStage --> RunCode
    PackStage --> PackCode
    ReleaseStage --> ReleaseCode

Sources: clarity/cli.py:22-42, clarity/packaging/release.py:140-183

Row State Transitions

Title: Manifest Row State Machine

stateDiagram-v2
    direction LR
    [*] --> planned : clarity/cli.py:cmd_plan
    planned --> done : clarity/cli.py:cmd_run (Success)
    planned --> failed : clarity/cli.py:cmd_run (Error)
    planned --> skipped : clarity/manifest.py:sync_skipped
    failed --> planned : clarity/cli.py:cmd_requeue
    skipped --> planned : clarity/cli.py:cmd_requeue
    done --> planned : clarity/cli.py:cmd_requeue (--old-recipe)

Sources: clarity/cli.py:22-42, tests/test_manifest_requeue.py:35-55

Current State	Transition Trigger	Target State	Notes
None	`clarity/cli.py:cmd_plan` clarity/cli.py:22-42	`planned`	Inserted via enumeration functions.
`planned`	`clarity/manifest.py:sync_skipped` clarity/cli.py:36-40	`skipped`	Rows without a valid processing role or policy path.
`planned`	`clarity/cli.py:cmd_run` clarity/cli.py:1-172	`done`	Successfully inferred, scaled, and encoded.
`planned`	`clarity/cli.py:cmd_run` clarity/cli.py:1-172	`failed`	Encountered an exception or encoder failure; records error in `note`.
`failed` / `skipped`	`clarity/cli.py:cmd_requeue` tests/test_manifest_requeue.py:35-55	`planned`	Clears error notes unless protected by a patch marker (`changed at ...`).

Sources: clarity/cli.py:22-40, tests/test_manifest_requeue.py:35-55

2. Stage 1 — `plan` (Enumeration and Classification)

The plan subcommand initializes or updates the manifest database by parsing game archives and pathlists.

Implementation and Execution Flow

Database Initialization: cmd_plan instantiates manifest.Manifest(a.db) clarity/cli.py:22-23.
Enumeration: Depending on CLI flags, it invokes enumeration helpers against the game accessor (kb.game()):
- mf.gen_chara(man, gd) clarity/cli.py:25-26
- mf.gen_icons(man, gd) clarity/cli.py:27-28
- mf.from_pathlist(man, gd, a.pathlist) clarity/cli.py:29-30
Skip Synchronization: mf.sync_skipped(man) evaluates rows lacking a processing path, transitioning them to skipped and summarizing counts clarity/cli.py:36-40.
Estimation: Automatically calls cmd_estimate(a) to print anticipated storage and workload metrics clarity/cli.py:41.

Sources: clarity/cli.py:22-42

3. Stage 2 — `run` (Batch Encoding and Inference)

The run command processes planned rows using thread pools and GPU model inference slots.

Pre-flight Safety Checks

Before executing inference, cmd_run performs mandatory environmental validation:

check_encoder(a): Validates whether texconv is available and running via DirectCompute GPU acceleration rather than CPU emulation clarity/cli.py:95-122.
check_disk(a, man, families): Compares estimated output mod sizes against available filesystem free space to prevent mid-run disk exhaustion clarity/cli.py:154-172.

Budget and Resumability

Executions can be constrained using --budget <seconds>. The loop pulls batches from the database, processes them via model slots (Engine), writes out multi-tier texture hierarchies, and updates row statuses to done or failed with diagnostic strings stored in note.

Sources: clarity/cli.py:95-172, README.md:20-27

4. Stage 3 — `pack` (Penumbra Mod Generation)

Once textures are marked done, the pack command organizes processed assets into Penumbra-compatible folder structures.

Packaging Mechanics

Tier Option Groups: Generates mod option groups allowing users to toggle between native, 2x, and 4x resolutions per family.
File Hierarchy: Maps input file paths to family-specific mod subdirectories using pack.mod_for_family() and pack.file_rel() clarity/packaging/release.py:125-133.
Reserved Placeholders: Injects shared dummy textures (RESERVED_PATHS) where required clarity/packaging/release.py:135-137.

Sources: README.md:20-27, clarity/packaging/release.py:125-137

5. Stage 4 — `qa` (Quality Assurance and Validation)

The qa stage inspects processed assets for integrity issues.

Normal-Length Preservation: Ensures normal maps maintain unit length after scaling.
Colour Drift Checks: Verifies that utility maps (masks, specular maps) do not incur unintended hue or channel shifts.
Contamination Audits: Validates output directories against expected family boundaries.

Sources: README.md:20-27

6. Stage 5 — `release` (Versioned Export)

The final release export converts packed assets into immutable, versioned distributions without re-running neural inference.

Implementation Details

Version Management: Validates semantic versions via version_key() and constructs release tags using version_for() clarity/packaging/release.py:60-78.
Staging Output: Writes to a .building temporary directory before renaming into place upon successful validation clarity/packaging/release.py:140-151.
Catalog Taxonomy: Iterates through the predefined CATALOG structure, mapping families (e.g., human-face, equipment, ui-icon) to export profiles clarity/packaging/release.py:17-56, clarity/packaging/release.py:140-183.