feat(layoutlm): add QNN NPU document QA recipes

[Qiong Wu (qiowu) (DingmaomaoBJTU)]

Summary

This change adds QNN NPU recipes for impira/layoutlm-document-qa and fixes the class-wide LayoutLM and build-pipeline behavior needed to reproduce them. The shipped Outcome-L1 includes generalized code, regression tests, and exact PASS recipes for fp16 and W8A16. Goal-L2 passes for those two tuples; required fp32 and W8A8 tuples remain explicit EXHAUSTED-FAIL results and have no recipe.

Model metadata

What the model does

This English LayoutLM v1 checkpoint performs extractive question answering over documents. Its neural forward path consumes tokenized question/document text, OCR-derived normalized token bounding boxes, an attention mask, and token-type IDs, then emits per-token start and end logits for an answer span. Image or PDF decoding, OCR, word-to-box alignment, and final answer decoding are outside the model forward path.

• Evidence/confidence: the pinned checkpoint, its model card/config, and LayoutLMForQuestionAnswering source (verified).

Primary user stories

• A user supplies an invoice image and asks a question such as "What is the invoice number?" to obtain the matching answer span for invoice field extraction (verified; pinned model-card widget/example).
• A user supplies a contract or financial document and asks a question such as "What is the purchase amount?" to obtain the matching amount span (verified; pinned model-card widget/example).

Supported tasks

• document-question-answering: declared by the checkpoint/model card and exposed by the Transformers document-QA pipeline (verified).
• question-answering: the neural span-scoring/export task exposed by Transformers, Optimum ONNX, and WinML. Optimum provides LayoutLMOnnxConfig but does not vendor-register layoutlm/question-answering; WinML supplies LayoutLMQAIOConfig (verified).

Model architecture

LayoutLMForQuestionAnswering
├── LayoutLMModel backbone
│   ├── Document embeddings
│   │   └── word + 1D position + 2D x/y/h/w layout + token type (hidden 768)
│   ├── Encoder layer x 12
│   │   ├── Self-attention (12 heads, head size 64) + output residual/LayerNorm
│   │   └── Feed-forward (768 -> 3072 -> 768, GELU) + residual/LayerNorm
│   └── First-token pooler (computed, not consumed by QA head)
└── QA span head (hidden 768 -> start/end logits)
    ├── start_logits [batch, sequence]
    └── end_logits [batch, sequence]

• Source/confidence: pinned config and Transformers 4.57.6 concrete LayoutLM source (verified).

Validation and support evidence

1. Baseline

• Current base: f5ee1267a9a6ec32639c15f3cac3006f56d5b46b; WinML version 0.1.0.
• The original evidence base was 6b16162816121bec4f1610726dc0769b2df5a703. A hash-gated partial refresh verified the candidate-owned files and retained fp16/W8A16 recipes/artifacts were unchanged; the two intervening commits affect perf/eval/monitor surfaces, not the export, optimization, quantization, build, compile, recipe-parser, or custom fixed-input session paths used by the retained evidence.
• Recipe-free baseline build: FAIL, exit 2. Automatic task resolution selected AutoModelForNextSentencePrediction, which rejects LayoutLMConfig; no ONNX or latency result was produced.
• Explicit-task baseline config emitted sequence length 514, token-type range [0,2), no compile target, and did not freeze analyzer-discovered graph changes. The shipped recipes are compared against that starting config below.
• Optimum probe: vendor tasks are feature extraction, fill-mask, text classification, and token classification; question-answering appears only after WinML registration (WINML-ONLY).
• Functional document-QA eval was already unavailable at baseline and remains a separate CLI capability gap.

2. Goal

• Effort: L2. The contribution needs class-wide LayoutLM input-bound fixes plus shared autoconf-precedence and compile-config-preservation fixes.
• Goal ceiling: L2. A passing tuple must build one QNN EPContext, run fixed-input performance, and pass four-sample tensor/span parity.
• Outcome: L1. Ship generalized code, tests, and recipes only for exact passing QNN/NPU/precision tuples; no new task-family runtime/evaluator is added.
• Success definition: preserve four-input/two-output LayoutLM QA I/O, use valid sequence/token-type/bbox bounds, retain recipe-owned QNN options through explicit CLI precision overrides, honor explicit optimization vetoes, and pass fixed-input correctness before performance.

3. Outcome

• Highest reached: L2 PASS for QNN/NPU fp16 and W8A16.
• Coverage: required-tuples-contain-exhausted-failures, not full or partial. Passed tuples are qnn/npu/fp16 and qnn/npu/w8a16; reachable exhausted tuples are qnn/npu/fp32 and qnn/npu/w8a8; deferred tuples: none.
• Shipped recipes:
  • examples/recipes/impira_layoutlm-document-qa/qnn/npu/question-answering_fp16_config.json
  • examples/recipes/impira_layoutlm-document-qa/qnn/npu/question-answering_w8a16_config.json
• Shipped generalized code/tests: src/winml/modelkit/{build/common.py,commands/build.py,models/hf/layoutlm.py} and focused tests under tests/unit/{build,commands,export}.
• Quality gates: mypy reported no issues in 441 source files; Ruff passed; models matrix 1542 passed, 6 skipped, 2 xfailed; commands matrix 3653 passed, 7 skipped; license check passed.
• Model knowledge layoutlm-009 through layoutlm-017 and methodology findings _meta-111 through _meta-114 were shipped separately in ModelKitArtifacts draft PR #285, keeping skill changes out of this PR.

4. Per-EP/device/precision results and Functional smoke Eval

Overall Goal-ladder roll-up is EXHAUSTED-FAIL at L0/L1/L2 because two reachable required tuples exhausted bounded repairs. This does not erase the exact fp16 and W8A16 PASS rows.

Tier	EP / Device	Precision	Verdict	Mean	p50	Throughput	RAM RSS Δ
L0-L2	QNNExecutionProvider / NPU	fp32	EXHAUSTED-FAIL	—	—	—	—
L0-L2	QNNExecutionProvider / NPU	fp16	PASS	30.8760 ms	29.8871 ms	32.3876 samples/s	+368.9 / +370.7 MiB
L0-L2	QNNExecutionProvider / NPU	W8A8	EXHAUSTED-FAIL	—	—	—	—
L0-L2	QNNExecutionProvider / NPU	W8A16	PASS	16.0221 ms	15.8387 ms	62.4138 samples/s	+278.4 / +278.9 MiB

• Protocol for each PASS row: two fresh sessions, 10 warmups plus 100 measured iterations per session, 200 measured samples total, fixed valid sample_02 input, QNN burst mode, graph-finalization mode 3. fp16 p95 is 34.9445 ms; W8A16 p95 is 17.1269 ms.
• fp16 L2: four samples; minimum cosine 0.999634322962559, maximum normalized RMSE 0.027264006657588, all outputs finite and all start/end argmax values exact. The artifact has one EPContext and 208 FLOAT16 tensors; payload ratio is 0.500167 relative to fp32.
• W8A16 L2: four samples; minimum cosine 0.996144127518274, maximum normalized RMSE 0.097549940490650, all outputs finite and all start/end argmax values exact. The artifact has one EPContext with UINT8 learned weights and UINT16 activations.
• fp32 exhaustion: native-GELU and Erf graphs produced byte-identical QNN outputs; enable_htp_fp16_precision=0 also produced the same outputs. The CPU precompile graph preserved the reference argmax and 0.000403165817 top-two margin, while QNN changed the ordering by -0.0009765625. No fp32 recipe is shipped.
• W8A8 exhaustion: random/default calibration reached minimum cosine 0.7091183631850756 and maximum normalized RMSE 0.7649870221470522; calibration using all four valid NPZ inputs still failed all four samples, with minimum cosine 0.7372705169071898 and maximum normalized RMSE 0.7570158892054824. No W8A8 recipe is shipped.

Functional smoke Eval: CLI-BLOCKED, exit 2. winml eval does not support document-question-answering; it reports the supported task list without that task. Text-only question-answering was not substituted because it has no image/OCR/bbox/word-alignment contract. No task metric was produced, so the L2 parity evidence proves the neural span scorer, not representative document-QA accuracy.

5. Delta

The two retained recipes differ from the explicit-task starting config at exactly 25 RFC 6901 paths. _note fields were excluded from the comparison.

Common compile delta for both recipes:

{
  "execution_provider": "qnn",
  "provider_options": {
    "device_type": "NPU",
    "htp_performance_mode": "burst",
    "htp_graph_finalization_optimization_mode": "3"
  },
  "provider_option_file_keys": [],
  "enable_ep_context": true,
  "embed_context": false,
  "compiler": "ort",
  "qnn_sdk_root": null,
  "device": "npu",
  "ep_device": null,
  "validate": false
}

Recipe	JSON pointer	Baseline	Shipped
fp16	/compile	null	QNN compile object above
fp16	/export/input_tensors/0/shape/1	514	512
fp16	/export/input_tensors/1/shape/1	514	512
fp16	/export/input_tensors/1/value_range/1	1	1000
fp16	/export/input_tensors/2/shape/1	514	512
fp16	/export/input_tensors/3/shape/1	514	512
fp16	/export/input_tensors/3/value_range/1	2	1
fp16	/optim/clamp_constant_values	missing	true
fp16	/optim/gelu_fusion	missing	false
fp16	/optim/matmul_add_fusion	missing	false
fp16	/quant/activation_type	uint16	uint8
fp16	/quant/fp16_keep_io_types	missing	true
fp16	/quant/fp16_op_block_list	missing	null
fp16	/quant/mode	static	fp16
W8A16	/compile	null	QNN compile object above
W8A16	/export/input_tensors/0/shape/1	514	512
W8A16	/export/input_tensors/1/shape/1	514	512
W8A16	/export/input_tensors/1/value_range/1	1	1000
W8A16	/export/input_tensors/2/shape/1	514	512
W8A16	/export/input_tensors/3/shape/1	514	512
W8A16	/export/input_tensors/3/value_range/1	2	1
W8A16	/optim/clamp_constant_values	missing	true
W8A16	/optim/gelu_fusion	missing	true
W8A16	/optim/matmul_add_fusion	missing	false
W8A16	/quant/seed	null	42

Bug fix explanation: LayoutLM bounds

1. Symptom/trigger: explicit layoutlm/question-answering config used sequence length 514 and token-type range [0,2) for a checkpoint whose usable sequence capacity is 512 and type_vocab_size=1, causing out-of-bounds export/runtime inputs.
2. Root cause: LayoutLMQAIOConfig exposed raw max_position_embeddings, and the dummy generator hardcoded all-zero token types instead of deriving the high-exclusive bound from model metadata.
3. Fix mechanism: LayoutLMQAIOConfig reuses the established RoBERTa position-capacity adjustment, adds normalized type_vocab_size, and uses LayoutLMTextInputGenerator to generate token types within that configured range.
4. Why general: behavior is derived from max_position_embeddings, pad_token_id, and type_vocab_size; there is no checkpoint-ID branch.
5. Compatibility/blast radius: BERT-style and RoBERTa-style LayoutLM capacities, explicit sequence overrides, single-type, and multi-type configs are covered; existing public I/O names stay unchanged.
6. Regression evidence: 16 LayoutLM/position compatibility tests passed, including distinct BERT/RoBERTa capacities and high-exclusive token-type ranges.

Bug fix explanation: analyzer optimization precedence

1. Symptom/trigger: a recipe's matmul_add_fusion=false was overwritten when analyzer discovery returned that option as true, producing a Gemm representation that changed a span argmax.
2. Root cause: _run_analyze_loop merged every discovered optimization without preserving explicitly disabled sparse-config keys.
3. Fix mechanism: the loop snapshots explicit-false keys and filters only matching discoveries before re-optimization and persistence.
4. Why general: the rule applies to every registered optimization key; it contains no model or operator special case.
5. Compatibility/blast radius: unspecified discoveries and explicit true values still apply; mixed sparse configs retain unrelated discoveries; auto=true/false loop behavior is preserved.
6. Regression evidence: four TestAutoconfOptimPrecedence cases cover explicit false, unspecified discovery, mixed sparse config, and auto=false; the affected build/common tests pass.

Bug fix explanation: compile config preservation

1. Symptom/trigger: the mandatory --precision fp16 override replaced the recipe's compile config, dropping QNN provider options and changing validate=false back to invalid random-input validation.
2. Root cause: build._patch_device replaced the entire WinMLCompileConfig with a provider-factory result when an explicit device/precision was supplied.
3. Fix mechanism: it now retargets provider/device fields in place, merges target provider defaults, preserves recipe-owned compile behavior/options, and clears stale resolved target bindings.
4. Why general: the preservation rule applies to existing compile configs for single and module-config-list builds; there is no LayoutLM branch.
5. Compatibility/blast radius: explicit target retargeting remains effective, no-override configs remain unchanged, and compile=null remains null.
6. Regression evidence: the full TestBuildFlagPassthrough class passed 24 tests; focused cases cover fp16, field preservation, stale binding, module arrays, no override, and null compile.

The production examples/recipes/README.md is unchanged.

6. Analyze summary — component level and op level

Static analysis for both artifacts is ANALYZE-PARTIAL-SUCCESS (exit 1), with complete payloads for 12 targets. The evidence is hash-verified reuse from 6b16162816121bec4f1610726dc0769b2df5a703; it is compatibility-rule analysis, not runtime execution.

Component-level summary

Artifact	Architecture coverage	Mapping	Actionable EP findings
fp16	backbone; document embeddings; 12x attention; 12x FFN intermediate/output; QA span head; start/end split	442/444 mapped; 2 graph-output Cast nodes outside frozen scopes; confidence partial; pooler absent by expected dead-code elimination	QNN NPU partial: Add, Div, Erf, Mul across backbone/embeddings/attention/GELU/output/head. OpenVINO NPU partial: Slice in attention. No unsupported types.
W8A16	backbone; document embeddings; 12x attention; 12x FFN intermediate/output; QA span head; start/end split	1156/1374 mapped; 218 exported nodes outside frozen scopes; confidence partial; pooler absent by expected dead-code elimination	QNN NPU partial: MatMul (QDQ) across attention, FFN intermediate/output, and QA span projection. No unsupported types.

Op-level summary

Artifact	Graph	Dominant ops	EP roll-up
fp16	444 ops / 16 types	Add 129; MatMul 97; Reshape 48; Transpose 48; Mul 37; LayerNormalization 25	QNN NPU partial Add/Div/Erf/Mul; OpenVINO NPU partial Slice
W8A16	1374 ops / 17 types	DequantizeLinear 593; QuantizeLinear 387; Add 117; MatMul 97; Reshape 48; Transpose 48; LayerNormalization 25	QNN NPU partial MatMul (QDQ)

CPU, CUDA, DML, MIGraphX, TensorRT, and VitisAI rows emitted no static classifications. NvTensorRTRTX, OpenVINO CPU/GPU, and QNN GPU are rule-backed but retain material unknown classifications; W8A16 OpenVINO NPU is also in the largely-unknown QDQ group.

7. Reproduce commands

The fixed-input parity/performance evidence requires the sealed tester input and helper scripts. The tester-provided portable command block is reproduced unchanged:

$OUT='temp/layoutlm-document-qa-qnn-npu'
$INPUT='path/to/sample_02.npz'
$TOOLS='path/to/tester-tools'
winml build -c examples/recipes/impira_layoutlm-document-qa/qnn/npu/question-answering_fp16_config.json -m impira/layoutlm-document-qa -o $OUT/fp16 --precision fp16 --rebuild --no-color
winml build -c examples/recipes/impira_layoutlm-document-qa/qnn/npu/question-answering_w8a16_config.json -m impira/layoutlm-document-qa -o $OUT/w8a16 --rebuild --no-color
python $TOOLS/benchmark_fixed_input.py --model $OUT/fp16/model.onnx --input $INPUT --label fp16 --session 1 --warmup 10 --iterations 100 --protocol-reference path/to/perf.json --output $OUT/fp16/perf-session-01.json
python $TOOLS/benchmark_fixed_input.py --model $OUT/w8a16/model.onnx --input $INPUT --label w8a16 --session 1 --warmup 10 --iterations 100 --protocol-reference path/to/perf.json --output $OUT/w8a16/perf-session-01.json
winml eval --schema --task document-question-answering --no-color

[Qiong Wu (qiowu) (DingmaomaoBJTU)]

APPROVE

Reviewed PR: #1369
Reviewed SHA: 8caab36
Base SHA: f5ee126
Merge base observed locally: f5ee126
Remote metadata: DRAFT=true, label model-scale-by-skill present, head stable before/after final gates.

Exact diff files (8):

• examples/recipes/impira_layoutlm-document-qa/qnn/npu/question-answering_fp16_config.json
• examples/recipes/impira_layoutlm-document-qa/qnn/npu/question-answering_w8a16_config.json
• src/winml/modelkit/build/common.py
• src/winml/modelkit/commands/build.py
• src/winml/modelkit/models/hf/layoutlm.py
• tests/unit/build/test_common.py
• tests/unit/commands/test_build.py
• tests/unit/export/test_onnx_config_overrides.py

Checklist outcome: all applicable reviewer.md gates passed.
README gate: git diff --name-only f5ee1267a9a6ec32639c15f3cac3006f56d5b46b...HEAD -- examples/recipes/README.md returned empty.
PR body hierarchy gate: remote PR body contains Summary, Model metadata, and Validation and support evidence in the required order; local source file PR_impira_layoutlm_document_qa.md matches that structure.
Thread/comment gate: issue comments=0, line comments=0, total review threads=0, open review threads=0.

Engineering observations:

• src/winml/modelkit/models/hf/layoutlm.py derives usable sequence length from max_position_embeddings after the same RoBERTa-style position adjustment already used elsewhere and derives token-type dummy/input bounds from type_vocab_size. This is data-driven and avoids checkpoint-specific branching while preserving explicit sequence overrides.
• src/winml/modelkit/build/common.py limits analyzer autoconf to discovered options not explicitly set false in the incoming config, which addresses the reported Gemm-producing matmul_add_fusion=true override without suppressing unrelated discoveries.
• src/winml/modelkit/commands/build.py retargets existing compile configs in place instead of replacing them, preserving recipe-owned provider options, validation mode, context embedding flags, compiler choice, and SDK paths while clearing stale resolved target bindings.
• Regression coverage is focused and proportionate: LayoutLM sequence/token-type cases in tests/unit/export/test_onnx_config_overrides.py; autoconf precedence in tests/unit/build/test_common.py; compile-retarget preservation in tests/unit/commands/test_build.py.
• No scope leakage observed. The change remains within the claimed Outcome-L1 / Effort-L2 surface and leaves production recipe index documentation untouched.

Environment classification:

• Local workflow-parity review environment on PR head: uv selected Python 3.11.16 on Windows; uv run completed Ruff, mypy, and affected non-hardware CI partitions successfully.
• Sealed tester Environment A: Python 3.11.16 AMD64, Transformers 4.57.6, fixed-input tuple execution with packaged Qualcomm QNN EP resolved.
• Sealed tester Environment B quality gate classification: package-version-set-equivalent; venv is not all-extras sync-exact. Gate-relevant versions matched lock expectations for Ruff, mypy, pytest, Transformers, and type stubs.

Command and check summary:

• Local exact-head checkout created in detached review worktree; git rev-parse HEAD = reviewed SHA above.
• uv run ruff check src/ tests/: PASS.
• uv run mypy -p winml.modelkit: PASS, Success: no issues found in 441 source files.
• uv run pytest tests/unit/models tests/unit/loader tests/unit/datasets tests/unit/export --tb=short --no-cov -m "not e2e and not npu and not gpu": PASS, 1542 passed, 6 skipped, 2 xfailed.
• uv run pytest tests/unit/commands tests/unit/config tests/unit/build tests/unit/compiler tests/unit/session tests/unit/eval --tb=short --no-cov -m "not e2e and not npu and not gpu": PASS, 3653 passed, 7 skipped.
• uv run python -m winml.modelkit eval --schema --task document-question-answering --no-color: reproduced the documented CLI blocker, exit 2, with no unsupported substitution to text-only QA.
• uv run python -c <artifact structural audit> directly inspected retained fp16 and W8A16 model.onnx files: both preserve the four expected int32 inputs and two logits outputs at [1,512], and each contains exactly one EPContext node.
• uv run python -c <probe> against the live checkout showed the raw task registry depends on registration timing; the frozen optimum-probe.json remains the authoritative classification artifact for the charter and still supports the claimed WINML-ONLY target for question-answering.

Artifact and evidence checks:

• Sealed tester-v5 manifest verification passed: 304 declared files, 0 mismatches, 0 unexpected unsealed files.
• tester-v5/final_validation.json passed all declared checks, including exact four-tuple accounting, PASS tuple safety/perf, one-EPContext final artifacts, exhausted-failure omission for fp32/W8A8, and quality-gate closure.
• Candidate-to-PR-head audit found byte drift in common.py, test_common.py, test_build.py, and both recipe JSON files. Independent semantic comparison showed the Python files are AST-identical and the recipe files are JSON-object-identical; direct diff with ignored end-of-line whitespace produced no semantic changes. I therefore classify the post-seal drift as format-only: Ruff/newline normalization, not artifact-producing logic change. No fresh PR-head rebuild was required beyond the mandated local workflow/test reruns.
• Direct artifact audit of retained models:
  • fp16 artifact: one EPContext, input contract input_ids, bbox, attention_mask, token_type_ids at [1,512], outputs start_logits and end_logits at [1,512].
  • w8a16 artifact: same I/O contract, one EPContext, quantized initializer types present.
• Direct sealed tuple evidence audit:
  • fp16 safety: 4/4 samples passed, minimum cosine 0.9996343229625595, maximum normalized RMSE 0.027264006657588272, all argmax exact.
  • fp16 perf: 2 sessions, 200 measured samples, mean 30.8760255 ms, p50 29.8871 ms, p95 34.9445 ms, throughput 32.38758822763636/s.
  • w8a16 safety: 4/4 samples passed, minimum cosine 0.9961441275182744, maximum normalized RMSE 0.09754994049065031, all argmax exact.
  • w8a16 perf: 2 sessions, 200 measured samples, mean 16.022102500000003 ms, p50 15.838650000000001 ms, p95 17.12688 ms, throughput 62.413781212546844/s.
• Provider snapshot verified from sealed environment identity: QNN packaged provider resolved for NPU target; benchmark helper imported the candidate qnn_monitor.py transitively.
• Analysis reuse check passed: component/op analysis for fp16 and W8A16 remained explicitly reused from base 6b16162816121bec4f1610726dc0769b2df5a703 with unchanged graph-dependency and artifact hashes.

Tuple split:

• Freshly review-validated shipped PASS tuples: qnn/npu/fp16, qnn/npu/w8a16.
• Reachable exhausted-failure tuples, correctly absent from shipped recipes: qnn/npu/fp32, qnn/npu/w8a8.
• Deferred tuples: none.
• Coverage classification confirmed: required-tuples-contain-exhausted-failures.

Current GitHub statusCheckRollup for reviewed SHA:

• Analyze (Python) / CodeQL: COMPLETED SUCCESS
• lint / Lint: COMPLETED SUCCESS
• test (analyze) / WinML CLI CI: COMPLETED SUCCESS
• test (models) / WinML CLI CI: COMPLETED SUCCESS
• test (optim) / WinML CLI CI: COMPLETED SUCCESS
• test (commands) / WinML CLI CI: COMPLETED SUCCESS
• test (remaining) / WinML CLI CI: COMPLETED SUCCESS
• CodeQL: COMPLETED SUCCESS
• license/cla: COMPLETED SUCCESS

Conclusion:

• I independently reviewed all changed source, tests, and recipe lines; reran the required current-workflow static and affected CI partition commands on exact PR head; rechecked remote PR metadata, comments, threads, and status checks; and directly audited the sealed tuple artifacts and manifests.
• The retained fp16 and W8A16 evidence remains admissible because the only post-seal changes were formatting-only and the exact-head workflow/test gates are green.
• The PR should merge as drafted without adding fp32 or W8A8 recipes.