Run Loop and Batch Encoding

The Run Loop and Batch Encoding

Relevant source files

The following files were used as context for generating this wiki page:

Purpose and Scope

This page details the core execution loop (cmd_run) in clarity/cli.py, which drives the transition of items from a planned state in the manifest to processed, encoded assets on disk clarity/cli.py:1-172. It covers model slot initialization via Engine, multi-threaded batch encoding (ThreadPoolExecutor), performance tuning parameters (--encode-batch, --encode-workers, --encode-mb), time budgeting, crash resilience, and row-level failure recording tests/test_run_pipeline.py:1-182.

1. Execution Orchestration and Model Slot Selection

The run loop is initiated through the CLI subcommand handler for run. Before processing texture rows, cmd_run performs critical pre-flight checks:

Encoder Verification (check_encoder): Probes texconv.exe to ensure GPU-accelerated DirectCompute acceleration is active, avoiding slow CPU-fallback encoding unless explicitly overridden clarity/cli.py:95-123.
Disk Space Validation (check_disk): Estimates total output bytes across all target tiers and compares them against available volume capacity to prevent mid-run storage exhaustion clarity/cli.py:125-172.
Model Slot Resolution: Instantiates the Engine class with target model profiles, mapping functional roles (e.g., normal, color, ui) to specific model weights registered in the configuration.

Below is a diagram bridging the execution orchestrator to its concrete code entities:

graph TD
    CLI["cli.py:cmd_run"] -->|Reads rows| MAN["manifest.Manifest"]
    CLI -->|Fetches source bytes| GAME["ffxiv.sqpack.GameData"]
    CLI -->|Loads models| ENG["processing.engine.Engine"]
    ENG -->|Delegates role processing| ROL["processing.roles"]
    ROL -->|Produces tensors| TEX["texio.encode_tiers_many"]
    TEX -->|Invokes subprocess| CONV["texio._texconv_many"]

    subgraph "Code Entity Space"
        CLI
        MAN
        GAME
        ENG
        ROL
        TEX
        CONV
    end

Sources: clarity/cli.py, tests/test_run_pipeline.py

2. The Texture Processing and Staging Loop

Once initialization succeeds, the runner iterates through uncompleted manifest rows. For each row:

The raw texture bytes are fetched via the SQPack accessor (game.read()) tests/test_run_pipeline.py:28-42.
The image header and pixel buffer are parsed using texio.read() tests/test_run_pipeline.py:56-58.
The processing role pipeline (roles.process_top()) applies neural upscaling or algorithmic filters (such as Lanczos or normal-map unit-length preservation) tests/test_run_pipeline.py:126.
Processed arrays are placed into a staging buffer awaiting batch encoding.

3. Batched Encoding via `ThreadPoolExecutor` and `--encode-*` Parameters

Invoking texconv.exe per individual texture incurs a severe fixed process-creation and D3D11 shader-compilation overhead (~0.3s per invocation) scripts/bench_texconv.py:1-8. To mitigate this, XIVUpscaler aggregates textures into memory batches before writing them to the scratch directory and invoking the encoder.

Configuration Parameters

--encode-batch <N>: Maximum number of textures grouped into a single texconv invocation tests/test_run_pipeline.py:103-106.
--encode-workers <K>: Number of concurrent worker threads managed by a ThreadPoolExecutor executing _texconv_many scripts/bench_texconv.py:105-125.
--encode-mb <M>: Maximum accumulated uncompressed payload size (in MiB) allowed in a single batch before triggering an early flush tests/test_run_pipeline.py:123-130.

The following sequence details how the batch encoder coordinates concurrent multi-file subprocessing and fallback retries:

sequenceDiagram
    autonumber
    participant Loop as "cli.py:run_loop"
    participant TPool as "concurrent.futures.ThreadPoolExecutor"
    participant Many as "texio.encode_tiers_many"
    participant Conv as "texio._texconv_many"
    participant Subp as "subprocess.run [texconv.exe]"

    Loop->>TPool: "Submit texture batch (size <= --encode-batch)"
    TPool->>Many: "Dispatch item collection"
    Many->>Conv: "Write DDS files to SCRATCH, call _texconv_many"
    Conv->>Subp: "Execute batched texconv.exe -gpu <ID> file1.dds file2.dds ..."
    Subp-->>Conv: "Return exit code (rc=0 or rc=1 if subset skipped)"
    alt "Batch contains corrupted or incompatible texture"
        Conv->>Subp: "Retried individually up to TEXCONV_TRIES times"
        Subp-->>Conv: "Return individual failure or success"
    end
    Conv-->>Many: "Return encoded bytes or RuntimeError per item"
    Many-->>Loop: "Update row statuses (done / failed)"

Sources: clarity/texio.py, tests/test_texio_batch.py, scripts/bench_texconv.py

4. Resilience, Budgets, and Failure Isolation

The run loop is designed around strict failure isolation and resumability:

Row-Level Granularity: If a single texture inside a batch of 32 fails encoding (e.g., due to invalid dimensions or codec limitations), only that specific row is marked as failed with the exact error traceback recorded in its note column tests/test_run_pipeline.py:132-148. Neighboring textures in the same batch successfully transition to done.
Wall-Time Budgets (--budget): Operators can restrict run duration using --budget <hours>. When the budget elapses, the engine completes active batches, flushes processed assets to disk, and safely terminates. A subsequent invocation of clarity run picks up precisely where it left off via manifest state inspection tests/test_run_pipeline.py:161-182.