Skip to content

Emruur/DeltaAttention

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

328 Commits
.vscode
.vscode
 
 
Paper
Paper
 
 
__pycache__
__pycache__
 
 
experiments/longbench
experiments/longbench
 
 
key_similarity_plots
key_similarity_plots
 
 
proxyExperiment/llama
proxyExperiment/llama
 
 
snellius_experiments/llama
snellius_experiments/llama
 
 
speedup_experiments_15
speedup_experiments_15
 
 
speedup_experiments_17
speedup_experiments_17
 
 
.DS_Store
.DS_Store
 
 
.gitignore
.gitignore
 
 
README.md
README.md
 
 
compute_key_similarity.py
compute_key_similarity.py
 
 
deltaDecoding.py
deltaDecoding.py
 
 
deltaFlashAttention.py
deltaFlashAttention.py
 
 
eval_all.py
eval_all.py
 
 
eval_speedup.py
eval_speedup.py
 
 
flashAttention.py
flashAttention.py
 
 
globVR.py
globVR.py
 
 
glob_set.py
glob_set.py
 
 
index.html
index.html
 
 
key_similarity_data.npz
key_similarity_data.npz
 
 
modeling_llama.py
modeling_llama.py
 
 
plot_attention_dominance.py
plot_attention_dominance.py
 
 
plot_key_similarity.py
plot_key_similarity.py
 
 
plot_speedup_comparison.py
plot_speedup_comparison.py
 
 
plotter.py
plotter.py
 
 
query_sim_23778769.err
query_sim_23778769.err
 
 
query_sim_23778769.out
query_sim_23778769.out
 
 
query_similarity.png
query_similarity.png
 
 
requirements.txt
requirements.txt
 
 
run.slurm
run.slurm
 
 
run_key.slurm
run_key.slurm
 
 
run_query_sim.slurm
run_query_sim.slurm
 
 
run_ruler.slurm
run_ruler.slurm
 
 
run_speedup.slurm
run_speedup.slurm
 
 
tritonModules.py
tritonModules.py
 
 
visualize_key_similarity.py
visualize_key_similarity.py
 
 
visualize_query_similarity.py
visualize_query_similarity.py
 
 

Repository files navigation

DeltaAttention

Read the draft paper

A prefill acceleration method for LLMs that compresses the key matrix $K$ along the sequence dimension before the $QK$ multiplication, reducing FLOPs proportionally to the compression ratio. Custom Triton kernels realize the speedup in practice, demonstrating wall-clock acceleration on long contexts.

About

DeltaAttention: accelerating LLM prefill by compressing the key matrix before QK multiplication — custom Triton kernels, wall-clock speedup on long contexts

emruur.github.io/DeltaAttention/

Resources

Readme
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages