Skip to content

b9951

Choose a tag to compare

View all tags
@github-actions github-actions released this 10 Jul 16:15
b9951
082b326
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

ggml-et: Initial ET backend (#24179)

  • ggml-et: Add performance logging

  • ggml-et: Quants helpers

  • ggml-et: Add MUL_MAT kernel

  • ggml-et: Add ROPE kernel

  • ggml-et: Add RMS_NORM kernel

  • ggml-et: Add GLU kernel

  • ggml-et: Add SOFT_MAX kernel

  • ggml-et: Add GET_ROWS kernel

  • ggml-et: Add CONT kernel

  • ggml-et: Add SET_ROWS kernel

  • ggml-et: Add MUL_MAT_ID kernel

  • ggml-et: Build et kernels as part of ggml

  • ggml-et: Embed kernels with fs fallback

  • ggml-et: Build fixes

  • ggml-et: Add MUL_MAT F32xF32 op

  • ggml_et: Add MUL_MAT_ID op

  • ggml-et: Disable offloading for debug

  • ggml-et: Refactor out block ops

  • ggml-et: ggml backend API changes

  • ggml-et: Add RESHAPE/TRANSPOSE to supported

  • ggml-et: Add CONT_F16

  • ggml-et: Add supported ops doc

  • gglm-et: Initial doc

  • ggml-et: Remove runtime import hacks

We can now import the runtime by a simple find_package(), so we
can cleanup the CMakeLists.txt.

  • ggml-et: Fix GET_ROWS kernel

Fix lost batch dimension.

Also clean vibe-comments.

  • ggml-et: Fix SET_ROWS kernel

Remove incorrect broadcasting guard.

  • ggml-et: Use custom instruction for fp32->fp16

  • ggml-et: Vectorize set_rows fp32->fp16

  • ggml-et: Fix ROPE kernel (yarn)

ggml-et: fix et_logf

WIP: Fix ramp

WIP: fix ROPE!

  • ggml-et: Better sinf

  • ggml-et: Fix SOFT_MAX

Add max_bias and sink support.

  • ggml-et: Fix CONT

Reorder from contiguous write to read with atomic stores.

  • ggml-et: Fix elmap kernel

Remainder handlin

  • ggml-et: Fix MUL_MAT MUL_MAT_ID remainders

  • ggml-et: Fix ET-SOC reference

  • ggml-et: Fix embed kernels scripts for old python

This allows GGML-ET to build on pre-3.8 python.

  • Add sysemu support with compile time flag -DGGML_ET_SYSEMU=ON (#6)

  • Example using ET-Soc-1 emulator configuration

Example usage:

cmake -B build -DGGML_CUDA=OFF -DGGML_ET=ON -DLLAMA_CURL=OFF -DGGML_CCACHE=ON
cmake --build build --config Release -j $(nproc)

time ./build/bin/test-backend-ops

./build/bin/llama-server \
    --model Qwen3-0.6B-Q8_0.gguf \
    --alias Qwen3-0.6B-Q8_0 \
    -fa 0 \
    --ctx-size 1024 \
    --no-warmup \
    --host 127.0.0.1 \
    --port 8080

  • build: proper dep tracking for kernels

  • support host using MOLD linker

  • initial multi core GET_ROW F32 implementation

  • vectorized q8 dequant

  • wip: cland warning clenaups and initial logging refactor

  • wip: message default message cleanup

  • chore: message cleanups

  • cmake cleanup

  • migrate to use platform provided functions

  • cmake back into subdir

  • support et_print() in kernels

  • fix: repair kernel building

  • perf: operations run async by default

  • debug: proper kernel dep tracking and error detection on kenrel launch

  • fix: kernel binary dep tracking and fixing get_rows_f32 erroring

  • perf: back to doing async kernel runs by default

  • perf: vectorize and parallel device memset

  • merge matmul work

  • misc: align allocation and enable all offload

  • misc: delete deadcode and respect memory limits

  • fix: repair tensor debug print

  • fix: loosen RMS_NORM op percision

  • feat: Q4_0 GET_ROWS

  • perf: FP32 MUL_MAT using TensorFMA

  • update limitations

  • perf: redue L1 load in compute_block_dot_product_q8_0

  • feat: save kernel mapping (name to id) when profiling is enabled

  • chore: memops cleanup

  • perf: parallelize softmax by rows

  • perf: vectorize 2nd phase of softmax

  • perf: ban GET_ROWS from offloaded

  • perf: vectorize and non-atomic for eltwise ops and sub support

  • perf: vectorize normal rope

  • perf: glu runs in parallel

  • merge: manually merge saqib's work on kernel fixes

  • perf: more vectorized RoPE

  • perf: parallelize mul_mat_id

  • perf: parallelize set_rows_f32

  • perf: vectorize softmax

  • feat: support kernel fusion and fuse RMS_NORM + MUL

  • fix: mostly resolve test-backend-ops failure in SOFT_MAX and ROPE

  • fix: bump max rope dims for gemma

  • feat: GeGLU and SCALE support to fully offload Gemma

  • perf: faster device memset

  • feat: get_rows supporting Q4_K and avoid cont cache coherent issues

  • better F32 MM

  • feat: NORM for ET backend

  • feat: SQR for ET backend

  • feat: UNARY on ET

  • feat: el_map support broadcasting for ET

  • feat: SUM_ROWS in ET backend

  • feat: more ops in ET backend

  • feat: WKV* operators in ET backend

  • perf: parallelize operators across cacheline instead of row

  • perf: parallelize get_rows on cacheline

  • wip: baseline FlashAttention for ET backend

  • wip: enough FA and CPY f32->f16 to run llama 3.1 fully offloaded with FA on

  • feat: f16 x f16 -> f32 MM using matrix engine

  • wip: f16 FlashAttention using matrix engine

  • wip: clean up

  • feat: barriers

  • perf: optimize FA_F16 in ET

  • perf: vectorize pack_k_for_transpose16

  • perf: prefetch next loop matrix tile

  • perf: FlashAttention 2nd MM uses TensorFMA and optimizations

  • cleanup: flashattention reorg

  • perf: optimizations and fixes

  • feat: L2SCP API and make FlashAttention support DV = 256 for gemma

  • perf: parallelize norms beyond single row

  • feat: GATED_DELTA_NET support and relaxed L2_NORM requirment

  • feat: loosen RMS_NORM, NORM, ROPE contingous req too

  • feat: repeat supports brocasting on dim 0 and loosen cont check

  • feat: FILL and DIAG operator

  • feat: loosen UNARY support chcek

  • feat: TRI support

  • feat: SOLVE_TRI support

  • feat: basic SET support

  • feat: loosen CONT req

  • perf: fp16_to_fp32 use ASM

  • feat: IMROPE support

  • feat: PAD support

  • feat: global barrier

  • fix: view must live on the same backend as backing tensor

  • feat: relax CONCAT in ET backend

  • feat: dead simple CUMSUM implementation

  • feat: basic SSM_CONV support

  • feat: loosen CONCAT req

  • feat: relax GATED_DELTA_NET and add SET support proper

  • cleanup: cleanup LCM math

  • feat: SWIGLU single input

  • feat: SSM_SCAN support

  • feat: el_map supports non aligned tensors in best effort

  • feat: basic GROUP_NORM support

  • feat: loosen MUL_MAT capablities slightly

  • feat: loosen MUL_MAT and GET_ROWS and add IM2COL

  • feat: special case for softmax 1x1x1x1

  • feat: loosen SOFT_MAX req in ET backend

  • fix: el_map unaligned acse fixes

  • perf: optimize zero_acc_vec in flash_attn_ext_f16_me

  • perf: use hart 1 for packing in MM and FA for FP16

  • feat: kernel semaphore

  • perf: better instruction sequence in FlashAttention

  • fix: gated_delta_net with proper masking

  • perf: better parallelization for GATED_DELTA_NET

  • perf: parallelize SSM_CONV over nr

  • perf: vectorize SSM_CONV

  • perf: optimize MUL_MAT for q8

  • feat: support Gemma 4

  • fix: support multi-device

  • feat: broader GLU support

  • feat: unary ops supports view

  • fix: repair fp16 MM using matrix engine

  • perf: handle large N GEMV better

  • perf: better q8_0 MM

  • perf: better set_rows

  • add back deleted files

  • fix: repair after merge

  • feat: POC version of uberkernel

  • feat: RMS_NORM in uberkernel

  • feat: add more kernels into usage

  • chore: clean up uberkernel compilation

  • perf: faster flash attention

  • perf: opt flash attention for large seq length

  • feat: loosen op bounds. clamp and mean support

  • perf: vectorize ssm_scan

  • perf: slightly faster FA

  • perf: FlashAttention parallel MM and load

  • perf: fuse Q8 MM and ADD

  • feat: basic conv kernel for ET

  • softMAx_test

  • set_rows_f32

  • get_rows and cont

  • testing

  • set_rows_exp

  • Junk addition

  • Narrowing the issue

  • Update flash_attn_ext_f16_me.c

Focusing FA_ext_f16_me

  • test

  • Eviction updated

  • Detailed cache eviction debug

  • mulmat

  • removeal of BUILD_FOR_UBERKERNEL flag

  • cleaning...

  • fix: balance FCC0 count

  • feat: implement mul_mat and mul_mat_id for Q4_0 type

  • optimize uberkernel plan upload

  • add mul_mat q4 into uberkernel

  • enable gating flush to just uberkernel

  • update docs for ET

  • update op support for ET

  • et-backend: optimize Q4_0 and Q8_0 mul_mat_id row accumulations

  • et-backend: specialize mul_mat_id kernels for Q4_0 and Q8_0

  • et-backend: fix RoPE YaRN corr_dim formula and handle degenerate inputs

  • test-backend-ops: add DeepSeek-V2-Lite RoPE test coverage

  • et-backend: add Q4_0 mul_mat matrix-engine kernel using TensorFMA32

  • et-backend: vectorize Q4_0 matrix-engine dequantization

  • et-backend: support hybrid matrix/vector engine execution for Q4_0 mul_mat tail

  • et-backend: run partial-N tiles on matrix engine for Q4_0 mul_mat

  • et-backend: route Q4_0 mul_mat N < 53 to vecdot for better prefill latency

  • Update uberkernel.c

  • Update unary_f32.c

  • gemma 4

  • bisect gemma4: enable scale_f32 only

  • bisect gemma4: +rms_norm_f32

  • bisect gemma4: +rms_norm_mul_f32

  • bisect gemma4: disable rms_norm_mul_f32 -- BREAKS OUTPUT

  • bisect gemma4: +rope_f32 (skip rms_norm_mul)

  • bisect gemma4: +el_map_f32

  • bisect gemma4: +softmax_f32

  • bisect gemma4: +get_rows_f32

  • bisect gemma4: +glu_f32

  • bisect gemma4: +mul_mat_f32 +mul_mat_f32_matrix_engine

  • bisect gemma4: +mul_mat_f16 +mul_mat_f16_matrix_engine

  • bisect gemma4: +mul_mat_Q8_0 +mul_mat_Q4_0

  • bisect gemma4: +flash_attn_ext_f32 +flash_attn_ext_f16_me

  • bisect gemma4: +mul_mat_id_f32

  • bisect gemma4: +sum_rows_f32

  • bisect gemma4: +cont_f16

  • bisect gemma4: +fill_f32

  • bisect gemma4: +unary_f32 (all ops re-enabled except rms_norm_mul)

  • Update rms_norm_mul_f32.c

  • bisect2 gemma4 n64: +scale_f32 only

  • bisect2 gemma4 n64: +rms_norm_f32 +rope_f32

  • bisect2 gemma4 n64: +rms_norm_mul_f32 (with ET_UBERKERNEL eviction fix)

  • bisect2 gemma4 n64: +el_map +get_rows +glu +softmax (skip rms_norm_mul)

  • bisect2 gemma4 n64: all ops enabled except rms_norm_mul

  • bisect2 n64: test unary+cont+fill+sum_rows (no mul_mat/flash_attn)

  • bisect2 n64: +mul_mat_f32 +mul_mat_f32_matrix_engine

  • bisect2 n64: +mul_mat_f16 +mul_mat_f16_matrix_engine

  • bisect2 n64: +mul_mat_Q8_0 +mul_mat_Q4_0

  • bisect2 n64: +mul_mat_Q8_0 only (disable Q4_0)

  • bisect2 n64: +mul_mat_Q4_0 only (Q8_0 breaks)

  • bisect2 n64: +mul_mat_id +flash_attn_ext (skip Q8_0)

  • run-3: matmul + rms_norm_mul

  • run-4

  • Revert "run-4"

  • run5

  • changes after cleanup

  • cleanup before upstream

  • restrict changes into ET backend

  • move kernel embedding from Python to CMake

  • move uberkernel gen into CMake

  • apply clang format

  • update CMake style

  • update to match C and C++ style

  • use source ggml and quant headers instead of ET's

  • MROPE support

  • absorb view ops into same branch as none

  • fix bad rebase

  • add marty1885 to codeowners

  • oops

  • remove redundant newline

  • fix CI editor warnings

Co-authored-by: Vidas vidas@nuolat.lt
Co-authored-by: Gianluca Guida glguida@tlbflush.org
Co-authored-by: Gianluca Guida gianluca@nekko.ai
Co-authored-by: ubergarm leimgrub@gmail.com
Co-authored-by: SaqibAkram-10xE saqib.akram@10xengineers.ai
Co-authored-by: Rehan Qasim rehan.qasim@10xengineers.ai

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI:

Assets 27
Loading