Layerwise int4 kimi

[abhishek-singh591]

Setup and Run Instructions

Follow the steps below to set up and run Kimi K2.5 layerwise export/compile using run.py.

Step 1: Download the Model

Download Kimi K2.5 from Hugging Face:

• https://huggingface.co/moonshotai/Kimi-K2.5

Step 2: Set Model Path

run.py now requires --model_path (no hardcoded default).

Example:

--model_path /home/huggingface_hub/models--moonshotai--Kimi-K2.5/snapshots/54383e83fa343a1331754112fb9e3410c55efa2f

Step 3: Command-Line Arguments

Argument	Description	Default
--model_path	Path to downloaded Kimi model (required)	None (required)
--aic_hw_version	Accelerator HW version passed to final compile	ai100
--window_size	Number of layers per export window	1
--layerwise_mode	single_qpc or multiple_qpc	single_qpc
--total_layers	Total text layers; auto-resolved from model config if not set	None
--num-devices	Number of devices for compile stages	1
--batch_size	Batch size for specialization config	1
--seq_len	Prefill/compile sequence length	1
--ctx_len	Context length	128
--num_cores	Number of accelerator cores	16
--mxfp6 / --no-mxfp6	Enable/disable MXFP6 matmul compile flag	Enabled
--mxint8_kv_cache / --no-mxint8_kv_cache	Enable/disable MXINT8 KV cache	Enabled
--enable_blocking / --no-enable_blocking	Enable/disable blocking in QAIC config	Disabled
--blocking_mode	Blocking mode	kv
--num_kv_heads_repeat	KV heads repeat count	1
--num_kv_blocks	Number of KV blocks	4
--head_block_size	Head block size	4
--absorption	Enable MLA absorption	Disabled
--online	Enable MLA online mode	Disabled
--prefill_only	Compile in prefill-only mode	Disabled

Step 4: Layerwise Strategy

• single_qpc:
  • Builds layerwise ONNX windows
  • Merges via QEfficient.utils.layerwise_pipeline(...)
  • Runs final full-model compile via compile_full_model.py flow
• multiple_qpc:
  • Compiles layerwise outputs directly with QEfficient.utils.compile_layerwise(...)
  • Runs QEfficient.utils.inference_pipeline(...)

Step 5: Run Examples

Minimal:

  python run.py \
    --model_path /path/to/Kimi-K2.5/snapshot

Your requested test shape (total_layers=3):

  --model_path /home/huggingface_hub/models--moonshotai--Kimi-K2.5/snapshots/54383e83fa343a1331754112fb9e3410c55efa2f \
  --total_layers 3 \
  --window_size 1 \
  --layerwise_mode single_qpc \
  --aic_hw_version ai100

Multiple-QPC mode:

  python run.py \
    --model_path /path/to/Kimi-K2.5/snapshot \
    --layerwise_mode multiple_qpc \
    --window_size 1

Note on Layer Windows

Windowing uses half-open intervals:

[start, end)

• start is inclusive
• end is exclusive

Windows are generated from the top layer range down to 0 (for example, with total_layers=3, window_size=1: (2,3), (1,2),
(0,1)).