Planning Estimation and Probing

Planning, Estimation and Probing

Relevant source files

The following files were used as context for generating this wiki page:

Purpose and Scope

This page documents the planning, estimation, and diagnostic probing subsystems of XIVUpscaler. Specifically, it covers the CLI command handlers cmd_plan and cmd_estimate, pathlist ingestion via the manifest engine, and hardware/texconv diagnostics (check_encoder, check_disk, and probe_texconv). These modules ensure that texture processing batches are accurately enumerated, resource footprints and output sizes are mathematically bounded before execution, and invalid hardware/codec environments are caught prior to multi-day runs.

1. The Planning Stage (`cmd_plan`) and Pathlist Ingestion

The planning phase initializes the SQLite manifest database, discovers target game assets through various enumeration routines or user-provided pathlists, synchronizes unprocessable rows to a skipped state, and chains directly into output estimation clarity/cli.py:22-42.

---
title: Planning Subsystem - Natural Language to Code Entity Mapping
---
graph TD
    CLIInput["User CLI Command\n('clarity plan --chara --pathlist')"] --> Dispatch["cli.cmd_plan(a)"]
    Dispatch --> ManifestInit["mf.Manifest(a.db)"]
    Dispatch --> GameInit["kb.game()"]
    
    GameInit --> CharaEnum["mf.gen_chara(man, gd)"]
    GameInit --> IconsEnum["mf.gen_icons(man, gd)"]
    GameInit --> PathlistEnum["mf.from_pathlist(man, gd, a.pathlist)"]
    
    CharaEnum --> SyncSkipped["mf.sync_skipped(man)"]
    IconsEnum --> SyncSkipped
    PathlistEnum --> SyncSkipped
    
    SyncSkipped --> EstimateChain["cli.cmd_estimate(a)"]

Sources: clarity/cli.py:22-42

Enumeration Entrypoints and Queue Synchronization

cmd_plan parses incoming arguments to select asset scopes clarity/cli.py:22-30. Rows inserted during planning evaluate their processing verdict immediately via Manifest.add, which consults skip reasons clarity/cli.py:31-32.

To prevent stale queue counts caused by outdated classifiers or previous runs, mf.sync_skipped(man) inspects all planned rows, transitions unprocessable entries to the 'skipped' state, and reports summary statistics clarity/cli.py:33-40. Once queue synchronization completes, execution flows directly into cmd_estimate(a) clarity/cli.py:41.

Sources: clarity/cli.py:22-43

2. Output Estimation (`cmd_estimate` and `estimate_output_bytes`)

Because upscaling generates multi-tier mipmapped texture sets across diverse compression formats, sizing the output correctly before encoding is critical.

---
title: Estimation Flow - Calculating Tier Storage Requirements
---
graph TD
    EstimateCmd["cli.cmd_estimate(a)"] --> SummaryQuery["man.summary()"]
    SummaryQuery --> RowLoop["Iterate Families, Roles, Status"]
    RowLoop --> RoleCheck{Role in id, skip, other?}
    RoleCheck -- Yes --> LogRaw["Print Unscaled Stats\n(Source MB Only)"]
    RoleCheck -- No --> FetchRows["SELECT w,h,fmt FROM tex"]
    FetchRows --> CalcTiers["roles.top_tier() & roles.tiers_below()"]
    CalcTiers --> ScaleMath["w * h * s * s * bpp * 1.33"]
    ScaleMath --> Aggregate["Accumulate Totals per Tier\n(native, 2x, 4x)"]
    Aggregate --> PrintSummary["Print Final GB Estimates"]

Sources: clarity/cli.py:44-93, clarity/cli.py:125-151

Mathematical Sizing Logic

cmd_estimate iterates over manifest summary groups clarity/cli.py:51. For upscale-eligible roles, it queries individual row dimensions and formats (w, h, fmt) clarity/cli.py:59-62.

Top Tier Resolution: Computes the highest scaling tier allowed for the texture dimensions using roles.top_tier(family, w, h, a.top) clarity/cli.py:65.
Bit Depth Scaling: Assigns 4 bytes per pixel for uncompressed formats (B8G8R8A8, B8G8R8X8) and 1 byte per pixel for block-compressed formats clarity/cli.py:68.
Mip Overhead Factor: Multiplies raw pixel surface areas by 1.33 to account for BC7 mipchain storage overhead (approx. 33% additional space) clarity/cli.py:71.
Target Deltas: estimate_output_bytes(...) applies similar logic restricted to active families or --since patch-delta timestamps clarity/cli.py:125-151.

Sources: clarity/cli.py:44-93, clarity/cli.py:125-151

3. Hardware, Codec and Disk Diagnostics (`check_encoder`, `check_disk`)

Before initiating queue processing, the pipeline runs strict diagnostic checks to verify that hardware acceleration is available and that local storage is sufficient.

---
title: Diagnostics Subsystem - Encoder Probing and Disk Validation
---
graph TD
    CheckEncoder["cli.check_encoder(a)"] --> UseTexconv["texio.use_texconv()"]
    UseTexconv -- False --> FallbackCPU["Warn: numpy bc7enc (slow)"]
    UseTexconv -- True --> Probe["texio.probe_texconv()"]
    
    Probe --> DebugCheck{"texio.is_debug_build()?"}
    DebugCheck -- Yes --> DebugWarn["Warn: Debug build imports ucrtbased.dll"]
    DebugCheck -- No --> GPUCheck{"DirectCompute in output?"}
    
    GPUCheck -- Yes --> GPUOk["Encoder ready on GPU"]
    GPUCheck -- No --> GPUErr["Stop or require CLARITY_ALLOW_CPU_BC7"]
    
    CheckDisk["cli.check_disk(a, man, families)"] --> EstimateBytes["estimate_output_bytes()"]
    EstimateBytes --> DiskUsage["shutil.disk_usage()"]
    DiskUsage --> SpaceCheck{"Free Space < Required Output?"}
    SpaceCheck -- Yes --> DiskWarnAdvisory["Advisory Warning / --strict-disk Fatal"]

Sources: clarity/cli.py:95-172, clarity/texio.py:42-92

Texconv and DirectCompute Probing (`check_encoder`)

check_encoder prevents multi-day CPU fallback bottlenecks by invoking texio.probe_texconv() clarity/cli.py:95-100, clarity/texio.py:62-92.

Debug Build Detection: texio.is_debug_build() inspects binary headers for references to ucrtbased.dll. Debug builds fail to initialize D3D11 device creation without SDK layers, silently falling back to unoptimized CPU execution clarity/texio.py:50-59 (approx. 54 seconds per texture).
DirectCompute Verification: If a release build is active, the probe verifies that text output contains "Using DirectCompute" clarity/cli.py:105-106, clarity/texio.py:84. Unless CLARITY_ALLOW_CPU_BC7=1 is set, non-GPU execution halts execution clarity/cli.py:114-122.

Disk Capacity Validation (`check_disk`)

check_disk calculates the total byte volume required for planned outputs and compares it against free volume space using shutil.disk_usage clarity/cli.py:154-172. Because running out of disk space near the end of a multi-day encode leaves corrupted manifest states and half-written files, this check guards against catastrophic storage exhaustion clarity/cli.py:154-162. By default, it issues a severe advisory warning, which becomes a fatal abort if --strict-disk is asserted clarity/cli.py:167-170.

Sources: clarity/cli.py:95-172, clarity/texio.py:42-92

4. Environment Probing Summary

The following table summarizes the diagnostic probes executed during pipeline initialization:

Diagnostic Function	Target Component	Success Condition	Failure Action
`texio.use_texconv()`	`texconv.exe` binary path	File exists on Windows host	Fallback to pure-numpy `bc7enc` clarity/texio.py:42-48
`texio.is_debug_build()`	DirectXTex binary headers	Absence of `ucrtbased.dll` string	Warns about slow CPU fallback codec clarity/texio.py:50-59
`texio.probe_texconv()`	GPU Acceleration	`"Using DirectCompute"` present in stdout	Halts unless `CLARITY_ALLOW_CPU_BC7=1` clarity/cli.py:105-122, clarity/texio.py:84
`cli.check_disk()`	Filesystem Storage	Free space > estimated output bytes	Advisory warning (fatal with `--strict-disk`) clarity/cli.py:154-172