Role Specific Processing Pipelines

Role-Specific Processing Pipelines

Relevant source files

The following files were used as context for generating this wiki page:

This page details the implementation of per-role image processing modules within clarity.processing. These modules handle pre-processing, model dispatching, and post-processing mathematical corrections tailored to specific FFXIV texture types (color maps, normal maps, material masks, and UI elements).

1. Core Processing Contract and Utilities (`clarity/processing/utils.py`)

The clarity/processing/utils.py module defines the abstract interface expected of the inference engine (Upscaler) and provides shared tensor conversion and manipulation routines clarity/processing/utils.py:1-79.

Upscaler Protocol: Defines the structural interface required by role functions (has, enabled, run, run_batch), decoupling the processing logic from the concrete PyTorch Engine implementation and allowing testing with fakes clarity/processing/utils.py:11-31.
Data Conversions: _f() casts uint8 arrays to float32 tensors in the [0.0, 1.0] range clarity/processing/utils.py:33-35. _u8() clips, rounds, and converts float32 arrays back to uint8 clarity/processing/utils.py:38-40.
gray(): Replicates a single-channel scalar across RGB to pass through a neural network model, then collapses the result back via channel averaging clarity/processing/utils.py:48-55.
average_color_fix(): Corrects low-frequency color shifts introduced by neural upscalers by comparing downscaled box filters of the source and output, adding back a smoothed correction term clarity/processing/utils.py:58-74.
has_alpha(): Checks if any texel in the alpha channel falls below 255 clarity/processing/utils.py:77-79.

Sources: clarity/processing/utils.py:1-79

2. Material Control Masks (`clarity/processing/masks.py`)

Material control masks in FFXIV typically combine independent scalar properties (such as specular power, roughness, and ambient occlusion) into separate color channels.

do_mask(): Splits RGBA/RGB input into individual single-channel planes, replicates each plane into an RGB tensor, and executes independent model inferences via engine.run_batch() to prevent channel cross-bleed clarity/processing/masks.py:9-43.
If the source format is BC1 and scale > 1, bc1clean is optionally applied prior to mask upscaling clarity/processing/masks.py:36-37.

Sources: clarity/processing/masks.py:1-43

3. UI Elements and Batch Upscaling (`clarity/processing/ui.py`)

UI textures and icons require special handling around transparent boundaries to prevent dark fringes caused by bilinear filtering against zeroed-out RGB values under transparent alpha channels.

do_ui(): Performs alpha-premultiplication on input images before upscaling. It processes straight color, pre-multiplied color, and alpha channels independently, then reconstructs valid edge texels where a_up > 0.02 using pre_up / a_up, falling back to straight_up for fully transparent pixels clarity/processing/ui.py:9-43.
do_ui_batch(): Stacks multiple small UI elements into a single tensor batch to optimize inference throughput for large batches of icons clarity/processing/ui.py:46-97.

Sources: clarity/processing/ui.py:1-97

4. Color and Albedo Maps (`clarity/processing/color.py`)

Color maps store diffuse and albedo data, often requiring model specialization based on asset family (e.g., human faces vs. monster models).

do_color(): Dynamically selects model slots based on resource family and compression format (e.g., routing "human-face" to "face", "monster" with BC7 to "color_v3", or defaulting to "color") clarity/processing/color.py:10-40.
Applies bc1clean if the source format is BC1 clarity/processing/color.py:42-43, and applies skin-specific models if the family matches human body parts clarity/processing/color.py:44-50.
Combines neural upscaling with average_color_fix() to preserve low-frequency lighting accuracy clarity/processing/color.py:51-53.

Sources: clarity/processing/color.py:1-57

5. Tangent-Space Normal Maps and Unit-Length Preservation (`clarity/processing/normals.py`)

Tangent-space normal maps store $X$ and $Y$ vector components in the Red and Green channels. Because neural upscaling alters pixel values, normal vectors must be mathematically renormalized to unit length.

do_normal(): Isolates the Red and Green channels, routes inference to "normal_bc1" or "normal" depending on source compression and alpha presence clarity/processing/normals.py:9-40.
Unit-Length Renormalization: After inference, $X$ and $Y$ are mapped back to [-1, 1], and the $Z$ component is mathematically reconstructed via $\sqrt{\max(0, 1 - nx^2 - ny^2)}$. The vector is then normalized across all three components to ensure unit length even if the neural network overshoots clarity/processing/normals.py:52-55.
Preserves or processes the Blue (opacity/skin influence) and Alpha channels depending on family rules clarity/processing/normals.py:56-64.

Sources: clarity/processing/normals.py:1-65

6. Architecture & Data Flow

The following diagrams bridge the natural language description of the pipeline to the underlying Python code entities, illustrating how role modules dispatch tasks through the Upscaler interface.

Diagram: Role Pipeline Dispatch Architecture

graph TD
    subgraph "NaturalLanguageSpace"
        A["ColorAlbedoPipeline"] -->|RoutesTo| B["ColorModelSlot"]
        C["NormalMapPipeline"] -->|Renormalizes| D["UnitLengthVector"]
        E["MaskPipeline"] -->|BatchProcess| F["IndependentChannels"]
        G["UIPipeline"] -->|Premultiply| H["AlphaEdgeFix"]
    end

    subgraph "CodeEntitySpace"
        I["clarity/processing/color.py:do_color"] -->|Calls| J["clarity/processing/utils.py:Upscaler.run"]
        K["clarity/processing/normals.py:do_normal"] -->|Computes| L["nz = np.sqrt(...)"]
        M["clarity/processing/masks.py:do_mask"] -->|Calls| N["Upscaler.run_batch"]
        O["clarity/processing/ui.py:do_ui"] -->|Applies| P["has_alpha()"]
    end

    A -.-> I
    C -.-> K
    E -.-> M
    G -.-> O

Sources: clarity/processing/color.py:10-57, clarity/processing/normals.py:9-65, clarity/processing/masks.py:9-43, clarity/processing/ui.py:9-43, clarity/processing/utils.py:11-31

Diagram: Normal Map Mathematical Renormalization Flow

graph TD
    Input["rgba uint8 input"] --> Cast["_f(rgba)"]
    Cast --> Slice["Extract RG channels"]
    Slice --> Inference["engine.run(slot, ...)"]
    Inference --> Map["nx, ny = rg * 2 - 1"]
    Map --> Recalc["nz = sqrt(clip(1 - nx^2 - ny^2))"]
    Recalc --> Norm["n = sqrt(nx^2 + ny^2 + nz^2) + 1e-6"]
    Norm --> Output["Reconstructed Unit Normal [0, 1]"]

Sources: clarity/processing/normals.py:33-65

Home · Repository · Migrated from DeepWiki