texconv Integration and Batching

Relevant source files

The following files were used as context for generating this wiki page:

Purpose and Scope

This section covers the integration of Microsoft's texconv.exe (DirectXTex) tool within the clarity texture I/O pipeline (clarity/texio.py). It details how the pipeline discovers the binary, detects unoptimized debug builds, probes hardware capabilities via direct GPU compression test runs, manages scratch directories, implements multi-file batching through _texconv_many, handles per-item fallback retries, and leverages the benchmarking script scripts/bench_texconv.py to optimize throughput.

Sources: clarity/texio.py:1-194(), scripts/bench_texconv.py:1-167()

Discovery, Probing, and Debug Build Detection

The pipeline determines whether to use texconv.exe based on platform constraints, environment variables, and file availability clarity/texio.py:42-47.

Discovery Candidates and Resolution

The search order checks variables and fallback paths in _CANDIDATES clarity/texio.py:26-33:

The CLARITY_TEXCONV environment variable.
paths.TEXCONV (vendor-tools\texconv\texconv.exe).
<project>/tools/texconv.exe.
The toolbox copy (ffxiv_7_0_toolbox/scripts/texconv.exe).

The selected binary path is stored in TEXCONV, and the target DXGI adapter index is configured via TEXCONV_GPU (defaulting to "0") clarity/texio.py:34-37.

Debug Build Detection

is_debug_build() inspects the executable binary bytes for references to the debug C runtime DLL (ucrtbased.dll) clarity/texio.py:50-60. Debug builds trigger the Direct3D debug layer, fall back to unoptimized CPU codecs, and significantly degrade performance (e.g., taking ~54 seconds for a single 512×1024 texture) clarity/texio.py:23-25.

Probing

probe_texconv() executes a dry-run encoding pass on a 64×64 generated block, checking for strings such as "Using DirectCompute" in the output to confirm hardware acceleration status clarity/texio.py:62-92.

graph TD
    A["use_texconv"] --> B{"Platform == nt?"}
    B -->|No| C["Return False (Fallback to numpy)"]
    B -->|Yes| D{"CLARITY_TEXCONV != 'none'?"}
    D -->|No| C
    D -->|Yes| E{"TEXCONV isfile?"}
    E -->|No| C
    E -->|Yes| F["Return True"]
    
    F --> G["probe_texconv"]
    G --> H["Encode 64x64 Block via _texconv"]
    H --> I{"Output contains 'Using DirectCompute'?"}
    I -->|Yes| J["Set gpu = True"]
    I -->|No| K["Set gpu = False"]
    J --> L["Return _probe Dictionary"]
    K --> L

Figure 1: Discovery and Probing Logic (use_texconv, probe_texconv)

Sources: clarity/texio.py:26-92()

The Batched Texture Encoder (`_texconv_many`)

Invoking texconv.exe per texture introduces heavy process-creation and device-initialization overhead (~0.3 seconds per invocation) scripts/bench_texconv.py:1-6. To mitigate this, clarity implements multi-file batching through _texconv_many, passing lists of files in a single invocation.

Data Flow and DDS Wrapping

Uncompressed DDS Input: Raw RGBA numpy arrays are packed into uncompressed 32-bit RGBA DDS headers via _dds_rgba() clarity/texio.py:114-135 and written to scratch files.
Command Construction: _texconv_cmd() builds arguments including -nologo, -y, -f <format>, -m <mips>, -o <out_dir>, and -gpu <adapter> clarity/texio.py:156-173. For BC7_UNORM, it adds -bc x to enable 3-subset modes on the GPU codec clarity/texio.py:171-173.
Subprocess Invocation and Signal Isolation: Subprocesses are spawned using subprocess.run() with CREATE_NEW_PROCESS_GROUP flags. This shields the active encoding job from immediate termination if the operator issues a Ctrl+C interrupt clarity/texio.py:176-186.
Parsing Outputs: _dds_split() extracts block payloads per mip level from the resulting DX10 DDS container clarity/texio.py:137-150.

Error Recovery and Per-Item Fallback

If texconv encounters an invalid or malformed input texture, it skips that specific file, processes the remainder of the batch, and exits with a non-zero return code (rc=1) tests/test_texio_batch.py:31-59.

_texconv_many inspects the output directory for successfully written files.
For missing outputs (failed items), it triggers single-file fallback retries up to TEXCONV_TRIES times clarity/texio.py:153.
If a file repeatedly fails, a RuntimeError is recorded specifically for that item without failing the entire batch tests/test_texio_batch.py:89-102.

graph TD
    A["_texconv_many"] --> B["Write uncompressed DDS to Scratch Dir"]
    B --> C["_texconv_cmd & subprocess.run"]
    C --> D{"texconv exit code == 0?"}
    D -->|Yes| E["Read all output DDS files"]
    D -->|No| F["Identify missing outputs in out_dir"]
    F --> G["Retry missing items individually up to TEXCONV_TRIES"]
    E --> H["Return list of byte payloads or exceptions"]
    G --> H

Figure 2: Batch Execution and Fallback Routing (_texconv_many, _texconv)

Sources: clarity/texio.py:114-194(), tests/test_texio_batch.py:31-102()

Scratch Directories and Temporary File Management

Batching requires high-performance temporary storage to hold uncompressed input DDS files and encoded output DDS files.

Paths are resolved through clarity/paths.py via the CLARITY_SCRATCH environment variable.
Using a fast local scratch disk (such as NVMe or RAM disk) rather than network shares or archive drives minimizes I/O bottlenecks during batch staging scripts/bench_texconv.py:11-14.
tempfile.TemporaryDirectory scopes file creation, ensuring automatic cleanup upon completion or failure of a batch tier.

Sources: clarity/texio.py:10-15(), scripts/bench_texconv.py:11-23()

Benchmarking Script (`scripts/bench_texconv.py`)

The repository includes a dedicated diagnostic utility (scripts/bench_texconv.py) to empirically evaluate process startup overhead, batch scaling, and concurrency limits against local hardware scripts/bench_texconv.py:1-23.

Key Benchmarking Functions

make_source(h, w, seed): Generates synthetic uncompressed RGBA DDS byte structures matching texio input requirements scripts/bench_texconv.py:43-50.
run_texconv(paths_in, out_dir, fmt): Measures pure wall-clock execution time for a single subprocess invocation across an arbitrary list of input paths scripts/bench_texconv.py:53-65.
bench_batch(dds, counts, repeats, scratch): Evaluates batch sizes ranging from 1 to 64 files per invocation, tracking subprocess execution times alongside input/output file I/O overhead scripts/bench_texconv.py:68-102.
bench_concurrency(dds, levels, repeats, scratch): Measures multi-threaded concurrency levels using ThreadPoolExecutor scripts/bench_texconv.py:105-125.