[WS1][Ascend] Qwen3-8B Dense train-inference closeout for BF16 execution vs FP32 reference

[zhangj1an]

Latest Status [12 Sep 2026]

Ready to be included in v0.1.0. No further PR will be created for Ascend WS1. The discrepancy listed below stems from BF16-execution-vs-FP32-reference quantization drift (a control chain built entirely from gold ops reproduces the same failure class), not from a defect in any individual Ascend kernel.

Forward accuracy: Against the same FP32 reference, the full-model selected_logp error is max_abs 0.0388 on CUDA (H20) versus 0.0857 on Ascend, about 2.2× larger on Ascend.

Gradient accuracy: On the same judgment, the CUDA candidate passes all 798 weight-gradient rows with a worst-case max_abs 0.1034, while the Ascend candidate fails 44 rows with a worst-case max_abs 1.88 — about 18× larger at the near-zero gradient elements.

Details

This PR closes #266 for the Ascend profile by mirroring the protocol and requirement of #315 (the WS1 full Qwen3-8B Dense train-inference closeout), adding ascend_bf16 (BF16 execution vs FP32 reference) as the third required profile.

Passing:

• Forward invariance 7/7 bitwise, gradient invariance 2394/2394 bitwise (max_abs 0.0); train/infer parity aggregates 0.0 / 0.0 / 0.0 (exact).
• The attention backward is a dedicated Ascend C kernel (deterministic_attention_backward_ascend.asc) with fixed keyBegin-anchored reduction orders, closing the cross-physical-length padding-invariance follow-up (Sv=63/64/65/96/128 with pad=5: 0.0 bitwise, both pad sides). FP16 backward falls back to the torch VJP.
• The FP32 reference now runs on this 64 GB host via CPU-resident weight paging (Qwen3DenseWeightsOffloaded), verified bitwise against a resident-FP32 model.

Not passing (accuracy judgments):

• Forward accuracy: selected_logp max_abs 0.0857 vs atol 0.06.
• Gradient accuracy: 44/798 rows fail (max_abs up to 1.88), all at near-zero reference-gradient elements. Swapping any single Ascend kernel for its gold twin does not eliminate them, and a control chain built entirely from gold ops with the candidate's BF16 dtypes/cast positions still fails (24 tensors, max_abs 2.19, relL2 12.6%) — so the failures come from the BF16-vs-FP32 quantization drift of the execution strategy itself, not from any single Ascend kernel. Thresholds stay unchanged; formal recalibration would require multi-workload evidence.

---

## Validation Identity

Field	Value
Commit	1fe03e7 (head)
NPU	Ascend 910B (910_9362)
HBM	64 GB
CANN	9.0.0 (Bisheng)
PyTorch / torch_npu	2.7.1+cpu / 2.7.1.post4
Execution dtype	BF16
Reference and accumulation dtype	FP32 (CPU-resident weights paged per access — see C10)
Workload	ws1-qwen3-8b-dense-primary-v6
Workload seed	20260812
Manifest version	ws1-c2-v8
Weight snapshot	Qwen/Qwen3-8B @ b968826d9c46dd6066d109eabc6255188de91218, content hash fc664a19c52c82b6f5ddb33d4fe2723181daeb93a344b16fee6369963e5a13a5

## C1 contract (tolerance contract, dtype policy, aggregates)

• tolerance_contract.json declares ascend_bf16 -> backend_family "ascend" as a third required profile; thresholds, roles and aggregates unchanged.
• CPU contract/workload/wiring tests: 107 passed, 1 skipped.
• TF32 disabled holds by construction (Ascend has no TF32 mode).

## C2 candidate evidence (manifest-pinned, random inputs, --check-grad)

python scripts/ws1_candidate_evidence.py --profile ascend_bf16 --all --check-grad

Result	Count
Pinned Ascend cases	23
Passed	23 / 23

The deterministic GEMM's accuracy gold is the leaf-space tree reference (DetGemmTreeReferenceOp) and its backward is the canonical FP32-accumulation rowwise VJP; both match the unrounded FP32 reference to ULP.

## C3 forward invariance / C4 gradient invariance

• C3/C4 smoke (silu): all pairs bitwise, max_abs=0.0 (chunk, batch, permutation, left/right padding).
• C4 for det_gemm / rms_norm / qk_norm: green via the canonical parameter_vjp_contributions_fp32 hooks — chunk/permute/padding/singleton pairs bitwise.

## C5 elementwise inventory

• tests/test_elementwise_inventory.py passes (CPU classification); on-NPU verdicts flow through C3/C4.

## C6 direct decode-prefill consistency

python scripts/check_decode_prefill.py --backend-profile ascend_bf16

decode-bn-varlen          attn_pass=True max_abs=0.0 logp_pass=True
decode-bn-padded-right    attn_pass=True max_abs=0.0 logp_pass=True
decode-bn-padded-left     attn_pass=True max_abs=0.0 logp_pass=True
decode-b1-primary-s3      attn_pass=True max_abs=0.0 logp_pass=True
cases=6 (all include direct decode)

6/6 passed, max_abs 0.0.

## C7 stateful KV + generate-rescore

python scripts/check_stateful_kv.py --backend-profile ascend_bf16

profile=ascend_bf16 candidate=ascend b1=True rescore=True b2=absent passed=True

passed.

## C8 four-judgment matrix sweep

python scripts/sweep_ws1_four_judgments.py --execute --profile ascend_bf16 --json

Count
Green	88
Red	0
N/A	8 (declared pack layout-helper boundary)

## C9 full-model assembly

• Pinned weights load and verify (content hash matches the manifest); the full model assembles and runs on the NPU (all eight C10 cells execute with real backward).
• Runtime provenance records the NPU name and SoC as the architecture key.

## C10 full Qwen3-8B Dense chain gate

python scripts/ws1_chain_gate.py --backend-profile ascend_bf16 --model qwen3-8b-dense --dtype bfloat16 --weights required --weights-path ...

The FP32 reference runs on this host with CPU-resident weights paged onto the NPU per access (bitwise-identical to a resident-FP32 model); the gate runs all eight cells with real backward:

Check	Result
Cells executed	8/8 (BN/full, BN/chunked, BN/permuted, BN/padded×2, BN/packed, B1/full, B1/chunked), all with real backward and all 350+ parameter gradients
Train/infer parity aggregates	✅ max_abs_dlogp = 0.0, approx_kl0 = 0.0, clipfrac0 = 0.0 (exact)
selected_logp invariance	✅ all pairs bitwise (7/7, max_abs 0.0)
gradient accuracy vs reference	754/798 pass; 44 fail (max_abs up to 1.88, near-zero elements; BF16-vs-FP32 quantization drift — all-gold BF16 control fails the same way)

## C11 CI

Not runnable on this host (requires the self-hosted Ascend runner and the GitHub workflow). The CI script ci/run_ws1_ascend_ci.sh was exercised step-by-step locally; its gates are the sections above.

## Regression tests

pytest -q \
  tests/test_tolerance_contract.py \
  tests/test_ws1_workload.py \
  tests/test_four_judgment_matrix.py \
  tests/test_elementwise_inventory.py \
  tests/test_ws1_ascend_closeout.py
# 107 passed, 1 skipped

pytest -q tests/test_det_gemm_ascend.py tests/test_silu_ascend.py
# 57 passed

pytest -q tests/test_attention_ascend.py
# 26 passed

pytest -q tests/test_rms_norm.py
# 91 passed, 32 skipped

pytest -q tests/test_kv_consistency.py tests/test_ws1_qwen3_dense.py tests/test_ws1_chain_integration.py
# 37 passed

## Notes

• The Ascend vector unit's reduction orders differ from CUDA's, so cross-platform bitwise parity is not claimed; the guarantee is the one each platform provides for itself — batch-invariant determinism under the shared contract.
• The three selected_logp invariance failures found on-device were: (1) the fused-logp wrapper falling back to the native log-softmax for non-contiguous logits (batch-layout-dependent); (2) the attention kernel's softmax denominator reduced per physical 64-key tile (left padding changed the FP32 addition grouping — tiles now anchored to the first valid key); (3) the RMSNorm rstd's torch mean/sum selecting shape-dependent reduction kernels (52/140 rows flip between batch layouts) — all three paths now share a fixed-order shape_invariant_rstd. All verified bitwise at the cell level.
• Attention backward: the valid-token-compaction VJP was replaced by the dedicated Ascend C backward kernel (three stream-ordered launches, keyBegin-anchored fixed reduction orders), closing the cross-physical-length padding-invariance follow-up; FP16 backward (optional contract row) falls back to the torch VJP.
• C10 reference: the FP32 reference runs on-device with CPU-resident weights paged per access (bitwise-identical to the resident-FP32 model); its linear nodes are the C2 contract tree, whose BF16 leaf/merge rounding makes the accuracy judgments sensitive to the candidate's own rounding at near-zero gradient elements.
• Single-device operator; no CP/SP involvement, consistent with the CUDA implementation in issue [WS1][kernels] Batch-invariant matmul / GEMM #146.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 58892c20-7ac6-4e46-a21d-c2772cea8305

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[Flink-ddd]

LGTM