OpenCL: OP_GATED_DELTA_NET

[ymcki]

I was the author of the backend agnostic KDA support, so I am highly familiar
with the math and algorithms of GDN and KDA.
#18755

Since support is not here for OpenCL, so I might as well give it a try to implement it.

pp gain is about 9.5%. tg gain is about 22.3%.

Master

Details

model	size	params	backend	ngl	threads	cpu_mask	cpu_strict	poll	n_ubatch	fa	mmap	test	t/s
qwen35 4B Q4_0	2.44 GiB	4.21 B	OpenCL	99	6	0xfc	1	1000	1024	1	0	pp64	86.91 ± 0.54
qwen35 4B Q4_0	2.44 GiB	4.21 B	OpenCL	99	6	0xfc	1	1000	1024	1	0	pp128	98.48 ± 0.34
qwen35 4B Q4_0	2.44 GiB	4.21 B	OpenCL	99	6	0xfc	1	1000	1024	1	0	tg64	7.46 ± 0.10
qwen35 4B Q4_0	2.44 GiB	4.21 B	OpenCL	99	6	0xfc	1	1000	1024	1	0	tg128	7.31 ± 0.03

build: 1ec7ba0 (9113)

This PR

Details

model	size	params	backend	ngl	threads	cpu_mask	cpu_strict	poll	n_ubatch	fa	mmap	test	t/s
qwen35 4B Q4_0	2.44 GiB	4.21 B	OpenCL	99	6	0xfc	1	1000	1024	1	0	pp64	95.43 ± 0.26
qwen35 4B Q4_0	2.44 GiB	4.21 B	OpenCL	99	6	0xfc	1	1000	1024	1	0	pp128	107.83 ± 0.31
qwen35 4B Q4_0	2.44 GiB	4.21 B	OpenCL	99	6	0xfc	1	1000	1024	1	0	tg64	9.05 ± 0.13
qwen35 4B Q4_0	2.44 GiB	4.21 B	OpenCL	99	6	0xfc	1	1000	1024	1	0	tg128	8.94 ± 0.02

build: 1ec7ba0 (9113)

Correctness: LD_LIBRARY_PATH=./lib:/vendor/lib64 ./bin/test-backend-ops test -o GATED_DELTA_NET

Details

Testing 2 devices

Backend 1/2: GPUOpenCL
Device description: QUALCOMM Adreno(TM) 750
Device memory: 11584 MB (10560 MB free)

GATED_DELTA_NET(type=f32,head_count=32,head_size=128,n_seq_tokens=1,n_seqs=1,v_repeat=1,permuted=0,kda=0): OK
GATED_DELTA_NET(type=f32,head_count=32,head_size=16,n_seq_tokens=1,n_seqs=1,v_repeat=1,permuted=0,kda=0): OK
GATED_DELTA_NET(type=f32,head_count=32,head_size=16,n_seq_tokens=1,n_seqs=1,v_repeat=1,permuted=1,kda=1): OK
GATED_DELTA_NET(type=f32,head_count=32,head_size=16,n_seq_tokens=1,n_seqs=1,v_repeat=1,permuted=0,kda=1): OK
GATED_DELTA_NET(type=f32,head_count=16,head_size=64,n_seq_tokens=1,n_seqs=2,v_repeat=1,permuted=0,kda=0): OK
GATED_DELTA_NET(type=f32,head_count=4,head_size=64,n_seq_tokens=4,n_seqs=1,v_repeat=1,permuted=0,kda=0): OK
GATED_DELTA_NET(type=f32,head_count=4,head_size=64,n_seq_tokens=4,n_seqs=2,v_repeat=1,permuted=0,kda=0): OK
GATED_DELTA_NET(type=f32,head_count=8,head_size=32,n_seq_tokens=4,n_seqs=2,v_repeat=2,permuted=0,kda=0): OK
GATED_DELTA_NET(type=f32,head_count=4,head_size=64,n_seq_tokens=4,n_seqs=2,v_repeat=1,permuted=1,kda=0): OK
GATED_DELTA_NET(type=f32,head_count=4,head_size=64,n_seq_tokens=4,n_seqs=1,v_repeat=1,permuted=1,kda=0): OK
GATED_DELTA_NET(type=f32,head_count=4,head_size=64,n_seq_tokens=1,n_seqs=1,v_repeat=1,permuted=0,kda=1): OK
GATED_DELTA_NET(type=f32,head_count=4,head_size=64,n_seq_tokens=1,n_seqs=2,v_repeat=1,permuted=0,kda=1): OK
GATED_DELTA_NET(type=f32,head_count=4,head_size=16,n_seq_tokens=1,n_seqs=2,v_repeat=1,permuted=0,kda=1): OK
GATED_DELTA_NET(type=f32,head_count=4,head_size=32,n_seq_tokens=4,n_seqs=1,v_repeat=1,permuted=0,kda=1): OK
GATED_DELTA_NET(type=f32,head_count=4,head_size=64,n_seq_tokens=4,n_seqs=2,v_repeat=1,permuted=0,kda=1): OK
GATED_DELTA_NET(type=f32,head_count=8,head_size=32,n_seq_tokens=4,n_seqs=2,v_repeat=2,permuted=0,kda=1): OK
GATED_DELTA_NET(type=f32,head_count=4,head_size=64,n_seq_tokens=4,n_seqs=2,v_repeat=1,permuted=1,kda=1): OK
GATED_DELTA_NET(type=f32,head_count=4,head_size=16,n_seq_tokens=4,n_seqs=2,v_repeat=1,permuted=1,kda=1): OK
GATED_DELTA_NET(type=f32,head_count=4,head_size=64,n_seq_tokens=64,n_seqs=1,v_repeat=1,permuted=0,kda=0): OK
GATED_DELTA_NET(type=f32,head_count=4,head_size=64,n_seq_tokens=127,n_seqs=1,v_repeat=1,permuted=0,kda=0): OK
GATED_DELTA_NET(type=f32,head_count=4,head_size=64,n_seq_tokens=256,n_seqs=1,v_repeat=1,permuted=0,kda=0): OK
GATED_DELTA_NET(type=f32,head_count=4,head_size=64,n_seq_tokens=65,n_seqs=1,v_repeat=1,permuted=0,kda=0): OK
GATED_DELTA_NET(type=f32,head_count=4,head_size=64,n_seq_tokens=100,n_seqs=1,v_repeat=1,permuted=0,kda=0): OK
GATED_DELTA_NET(type=f32,head_count=4,head_size=64,n_seq_tokens=200,n_seqs=1,v_repeat=1,permuted=0,kda=0): OK
GATED_DELTA_NET(type=f32,head_count=4,head_size=64,n_seq_tokens=127,n_seqs=2,v_repeat=1,permuted=0,kda=0): OK
GATED_DELTA_NET(type=f32,head_count=4,head_size=64,n_seq_tokens=64,n_seqs=1,v_repeat=1,permuted=0,kda=1): OK
GATED_DELTA_NET(type=f32,head_count=4,head_size=64,n_seq_tokens=33,n_seqs=1,v_repeat=1,permuted=0,kda=1): OK
GATED_DELTA_NET(type=f32,head_count=4,head_size=64,n_seq_tokens=100,n_seqs=1,v_repeat=1,permuted=0,kda=1): OK
28/28 tests passed
Backend OpenCL: OK
Backend 2/2: CPU
Skipping CPU backend
2/2 backends passed
OK

• I have read and agree with the contributing guidelines

• AI usage disclosure: YES

AI is used to first evaluate the current implementation in vulkan, hexagon and CPU. AI found vulkan and hexagon has the best implementation. Due to proximity between Vulkan and OpenCL, so Vulkan implementation was used as a reference. Since Adreno GPU doesn't support all the features of OpenCL, direct translation is not possible. Therefore, I manually inspected what is supported by the Adreno GPU and modify the code manually. I also tuned the COLUMNS_PER_LANE_GROUP (cpl) and SUBGROUP_PER_WG (spw) parameters manually.

[lhez]

@ymcki Thank you for adding GDN. There are whitespace errors - please fix them. I updated the backend initialization so you will need to resolve; I think you can move the check for qcom subgroup shuffle to ggml_opencl_is_device_supported.

[ymcki]

@ymcki Thank you for adding GDN. There are whitespace errors - please fix them. I updated the backend initialization so you will need to resolve; I think you can move the check for qcom subgroup shuffle to ggml_opencl_is_device_supported.

Fixed trailing spaces.

Moved check for qcom subgroup shuffle to ggml_cl_init instead of ggml_opencl_is_device_supported because I think the flag better stay in struct ggml_backend_opencl_context than struct ggml_backend_opencl_device_context.

[lhez]

@max-krasnyansky Could you take a look and ack when you get a chance?

[max-krasnyansky]

Nice!