Skip to content

SuperNPUBench ops-20260914

Choose a tag to compare

View all tags
@lvhao7896 lvhao7896 released this 14 Sep 12:56
· 123 commits to main since this release
ops-20260914
085b3bc

SuperNPUBench ops-20260914

Summary

  • 579 compiled ELFs, 486 PASS, 93 FAIL, 0 TIMEOUT (83.9% pass rate)
  • Excludes HIF4/HIF8 ELFs (require assemble functionality)
  • gfrun (asl): fix/gfrun-shared-tmatmul-layout-257 ad972d21
  • PTO ISA v0.58.6, clang 15.0.4 (linx64v5-musl)

Environment

Component Branch Commit
gfrun / SuperScalarModel-asl fix/gfrun-shared-tmatmul-layout-257 ad972d21
llvm-project dev-llvm15_56 4a3e0bdb5
Linx-TileOP-API linx 987d034
linx-toolchain-build main e6a31ef
SuperNPUBench main 085b3bc

Key Changes (09-11 → 09-14)

gfrun branch switch

  • codex/gfrun-pto-0586-asl e7d883c9 → fix/gfrun-shared-tmatmul-layout-257 ad972d21
  • Shared TMATMUL layout adaptation
  • Caused solution gather_v2 ×3 + view_copy ×3 regression (R2=1, result mismatch)

TileOP-API stricter matrix shape validation

  • e93ef98 → 987d034: new AValidCols==BValidRows, D valid shape, MX ScaleB shape assertions
  • Broke 19 single-thread ELF compilations (09-11 all PASS): fa_2d_unroll ×8, sfa ×2, matmul MASK_FP16 ×3, MASK_FP8 ×3, A16W4 ×3
  • Only MASK_FP32 ×3 + fa_softmax_pto ×2 survive

New operators

  • fa_gmma_kchains (+2 PASS): GMMA k-chain FA variant (QK_CHAIN_K=32)
  • matmul_blockM (+13 PASS): shared blockM matmul, FP32/BF16/FP8/MXFP4
  • matmul_kchains (+1 PASS): k-chain matmul
  • matmul_lowp_blockM (+5 PASS): low-precision blockM, FP8/MXFP4
  • dynamic_mx_quant (+14, 2 PASS / 12 FAIL): dynamic MX quantization

Output cap 500MB → 2GB

  • A16W4 matmul, concat_scatter, group_token_old_mt no longer need individual verification
  • 6 operators still exceed 2GB (topk, group_token_old, group_token_vec, matmul_test ×2, mega_moe_sim)

compile.all fix

  • Single-thread matmul/fa compile.all: set -euo pipefail → set -u
  • HIF4 compilation failure no longer blocks subsequent A16W4/MASK variants

Per-suite results

Suite ELF PASS FAIL Rate
microbenchmark 439 405 34 92.3%
one-level (kernel) 108 72 36 66.7%
solution 32 9 23 28.1%
Total 579 486 93 83.9%

FAIL root causes (93)

Root cause Count Category
validCol TSTORE assertion (fa 2048B) 28 multi_thread/fa
MX scale dataType assertion 16 micro/fixp
dynamic_mx_quant priorSources 12 solution/quant
R2=1 result mismatch (regression) 6 solution/gather_v2 + view_copy
Output >2GB 6 solution + sort
vector TSTORE dtype/descriptor 13 micro/vector
vector compare/select TEPL 4 micro/vector
control MGATHER operand 6 one-level/control
reduction validCol/TROWSUM 2 one-level/reduction
broadcast COPY 1 one-level/broadcast
reserved/deleted tile selector 1 micro/vector

Tarballs

  • supernpubench_microbenchmark_20260914.tar.gz — 405 PASS ELFs (903K)
  • supernpubench_one-level_20260914.tar.gz — 81 PASS ELFs (1.6M)
Assets 4
Loading