CI optimizations - #1216
Open
quic-rishinr wants to merge 14 commits into
Open
Conversation
quic-rishinr
force-pushed
the
CI_optimization_fork
branch
from
July 26, 2026 13:45
991e341 to
9c91780
Compare
quic-rishinr
marked this pull request as draft
July 28, 2026 06:47
quic-rishinr
marked this pull request as ready for review
July 29, 2026 19:11
quic-rishinr
force-pushed
the
CI_optimization_fork
branch
2 times, most recently
from
August 3, 2026 10:30
da60fa0 to
ec5c89e
Compare
abukhoy
reviewed
Aug 4, 2026
| "vllm: marks vLLM tests", | ||
| "dummy_layers: marks tests running on reduced-layer dummy configs (per-PR lane)", | ||
| "full_layers: marks tests running on full-layer configs (nightly lane)" | ||
| ] No newline at end of file |
Contributor
There was a problem hiding this comment.
You have created a new marker for all other embedding, audio models. embedding_audio_model supposed to be added here.
Contributor
There was a problem hiding this comment.
I think, we should parallelize the stages in which we are not running with pytest multiple workers with assigned QAIC devices to each stage just like LLM stage.
Contributor
Author
There was a problem hiding this comment.
Agree, but when I enabled it, the time taken was higher. Need to check and assign right amount of threads for it.
Contributor
There was a problem hiding this comment.
We can remove test_causal_tlm_models.py and test_causal_lm_blocking_hqkv.py in this folder. And add all the blocking configs in one test.
ochougul
marked this pull request as draft
August 4, 2026 16:39
quic-rishinr
force-pushed
the
CI_optimization_fork
branch
from
August 5, 2026 08:15
ec5c89e to
89130c8
Compare
quic-rishinr
marked this pull request as ready for review
August 9, 2026 06:08
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
… added ci report generation module Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
…rmat Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
…dendunt tests, removed unnecessory tags Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
The per-PR causal-LM entries for qwen3_5_dense_text and qwen3_5_moe_text
carried a known_runtime_parity_issue xfail claiming QAIC decode tokens
diverge from HF after prefill in the Gated-DeltaNet retained-state
handoff. The xfail was over-broad: the hybrid modeling path is correct.
QEff-PyTorch fp32 matches HF fp32 greedy at every prefill and decode
step, and real Qwen3.5-0.8B (24 layers, 18 linear_attention + 6
full_attention) matches HF greedy 20/20 tokens on QAIC fp16.
The actual cause was the shape of the tiny-random checkpoints, not the
runtime. tiny-random/qwen3.5 has hidden_size=8 against vocab_size=248320,
and at that width the fp16 error accumulated through the Gated-DeltaNet
recurrence is enough to flip the greedy argmax. Widening the dummy config
makes the argmax fp16-stable on device.
Because the per-PR harness already random-re-initialises from the dummy
config (AutoModelForCausalLM.from_config in load_hf_causal_lm_model) and
_per_pr_dummy_config already applies a config_overrides dict, this needs
no code change - only config_overrides on the two entries, matching the
existing qwen2_5_dense_text idiom. Both entries move from compile-only
xfail to the full 7-variant runtime matrix, and supports_blocking is
turned on.
Three overrides are load-bearing and must not be trimmed:
- layer_types on the dense entry. Upstream tiny-random/qwen3.5 is four
linear_attention layers, so without the explicit
[linear, linear, linear, full_attention] the dense entry would stop
covering the hybrid linear-to-full handoff while still passing.
tiny-random/qwen3.5-moe already has a full_attention layer upstream
and so needs no override.
- head_dim stays at the upstream 32. qeff_apply_interleaved_mrope
slices the rotary tensor at offsets driven by mrope_section=[1,1,2];
the W section needs head_dim * partial_rotary_factor(0.25) > 4.
head_dim=16 yields 4 and aborts qaic-compile with
"Slice ... Start should be lesser than End".
- vocab_size stays at the upstream 248320. The entries use the real
tiny-random/qwen3.5 tokenizer, which emits ids up to 4888.
The duplicate qwen3_6_dense_text and qwen3_6_moe_text entries are
dropped. They route through the identical Qwen3_5ForCausalLM /
Qwen3_5MoeForCausalLM wrappers, so they added two more export, compile
and device runs per PR for byte-identical code.
hidden_size is part of the golden fingerprint, so goldens for both
entries are regenerated under their hub-id keys.
Validated on QAIC (Cloud AI 100):
- 14/14 with QEFF_REGENERATE_GOLDEN=1: dense 7/7 (501s), MoE 7/7 (508s)
- 28/28 steady state with regeneration off, which also re-runs
qwen3_dense_text and qwen2_5_dense_text as a no-regression check
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
… infer,refactored LORA to use smaller models and general cleanup Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
quic-rishinr
force-pushed
the
CI_optimization_fork
branch
from
August 9, 2026 06:08
65168a7 to
c99cab4
Compare
Contributor
Author
|
ci_ready |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test cases are following:
Fp16 end to end run subfunction True by default. (end to end run+ output verification)
Fp16 + Subfunction + CB (end to end run+ output verification)
Fp16 + Subfunction + speculation + CB (end to end run+ output verification)
Fp16 + Subfunction + CB with prefix caching (end to end run+ output verification)
Fp16 + Subfunction + CB + CCL (end to end run+ output verification)
FP32 export + FP16 compilation + Subfunction + CB + CCL (end to end run+ output verification)
BF16 export and BF16 compilation + Subfunction + CB + CCL (only till compilation)
Fp16 + Subfunction + CB + Blocking enabled
FP16 + Disagg model + Subfunction + CB + CCL for Moe models (end to end run+ output verification)