Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Misc] Add attention sinks #3515

Draft
wants to merge 90 commits into
base: main
Choose a base branch
from
Draft

[Misc] Add attention sinks #3515

wants to merge 90 commits into from

Commits on Mar 15, 2024

  1. temp

    @felixzhu555
    felixzhu555 committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    7914879 View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2024

  1. wip

    @felixzhu555
    felixzhu555 committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    5b672d9 View commit details
    Browse the repository at this point in the history

  2. wip

    @felixzhu555
    felixzhu555 committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    b35d7ba View commit details
    Browse the repository at this point in the history

Commits on Mar 19, 2024

  1. wip

    @felixzhu555
    felixzhu555 committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    e90cb58 View commit details
    Browse the repository at this point in the history

  2. wip

    @felixzhu555
    felixzhu555 committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    831f18b View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2024

  1. change q pos

    @felixzhu555
    felixzhu555 committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    c8d86e6 View commit details
    Browse the repository at this point in the history

  2. evict

    @felixzhu555
    felixzhu555 committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    0bd7566 View commit details
    Browse the repository at this point in the history

Commits on Mar 31, 2024

  1. edit xformers

    @felixzhu555
    felixzhu555 committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    f0263a4 View commit details
    Browse the repository at this point in the history

  2. wip

    Felix Zhu committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    15b68ca View commit details
    Browse the repository at this point in the history

Commits on Apr 1, 2024

  1. wip

    Felix Zhu committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    9fe1895 View commit details
    Browse the repository at this point in the history

  2. wip

    @felixzhu555
    felixzhu555 committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    595638d View commit details
    Browse the repository at this point in the history

Commits on Apr 4, 2024

  1. wip

    @felixzhu555
    felixzhu555 committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    217743d View commit details
    Browse the repository at this point in the history

Commits on Apr 10, 2024

  1. wip

    @felixzhu555
    felixzhu555 committed Apr 10, 2024
    Configuration menu
    Copy the full SHA
    fd83c78 View commit details
    Browse the repository at this point in the history

Commits on Apr 13, 2024

  1. pull from main

    @felixzhu555
    felixzhu555 committed Apr 13, 2024
    Configuration menu
    Copy the full SHA
    12e0e97 View commit details
    Browse the repository at this point in the history

Commits on Apr 14, 2024

  1. wip

    @felixzhu555
    felixzhu555 committed Apr 14, 2024
    Configuration menu
    Copy the full SHA
    a9b094c View commit details
    Browse the repository at this point in the history

  2. cuda illegal memory access

    @felixzhu555
    felixzhu555 committed Apr 14, 2024
    Configuration menu
    Copy the full SHA
    25e599d View commit details
    Browse the repository at this point in the history

Commits on Apr 17, 2024

  1. wip

    Felix Zhu committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    d14b94e View commit details
    Browse the repository at this point in the history

Commits on Apr 18, 2024

  1. cache current prerope key inside llama instead of xformers

    @felixzhu555
    felixzhu555 committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    8bb1840 View commit details
    Browse the repository at this point in the history

  2. early eos

    @felixzhu555
    felixzhu555 committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    339305b View commit details
    Browse the repository at this point in the history

  3. fix small bugs

    @felixzhu555
    felixzhu555 committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    1157cf3 View commit details
    Browse the repository at this point in the history

Commits on Apr 21, 2024

  1. wip

    @felixzhu555
    felixzhu555 committed Apr 21, 2024
    Configuration menu
    Copy the full SHA
    0f0a414 View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2024

  1. fix prefill

    @felixzhu555
    felixzhu555 committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    6f01606 View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2024

  1. wip

    @felixzhu555
    felixzhu555 committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    740cbdb View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2024

  1. starting to work!

    @felixzhu555
    felixzhu555 committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    15d586a View commit details
    Browse the repository at this point in the history

  2. blockwise speedup

    @felixzhu555
    felixzhu555 committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    c4a50b4 View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2024

  1. wip

    @felixzhu555
    felixzhu555 committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    455c814 View commit details
    Browse the repository at this point in the history

Commits on May 15, 2024

  1. start removing loop over blocks

    @felixzhu555
    felixzhu555 committed May 15, 2024
    Configuration menu
    Copy the full SHA
    ee12294 View commit details
    Browse the repository at this point in the history

  2. wip

    @felixzhu555
    felixzhu555 committed May 15, 2024
    Configuration menu
    Copy the full SHA
    d29b559 View commit details
    Browse the repository at this point in the history

Commits on May 16, 2024

  1. wip

    @felixzhu555
    felixzhu555 committed May 16, 2024
    Configuration menu
    Copy the full SHA
    94ebe4d View commit details
    Browse the repository at this point in the history

Commits on May 17, 2024

  1. wip: after refactor, generation abruptly ends

    @felixzhu555
    felixzhu555 committed May 17, 2024
    Configuration menu
    Copy the full SHA
    899a7b3 View commit details
    Browse the repository at this point in the history

Commits on May 18, 2024

  1. speedup to 4 tok/s done

    @felixzhu555
    felixzhu555 committed May 18, 2024
    Configuration menu
    Copy the full SHA
    016a6c6 View commit details
    Browse the repository at this point in the history

  2. cache phys_bnums, 3x speedup

    @felixzhu555
    felixzhu555 committed May 18, 2024
    Configuration menu
    Copy the full SHA
    2186c13 View commit details
    Browse the repository at this point in the history

Commits on May 19, 2024

  1. move logic out of xformers.py

    @felixzhu555
    felixzhu555 committed May 19, 2024
    Configuration menu
    Copy the full SHA
    e7acfbe View commit details
    Browse the repository at this point in the history

  2. refactor into new layer

    @felixzhu555
    felixzhu555 committed May 19, 2024
    Configuration menu
    Copy the full SHA
    18042c6 View commit details
    Browse the repository at this point in the history

Commits on May 21, 2024

  1. pull from main

    @felixzhu555
    felixzhu555 committed May 21, 2024
    Configuration menu
    Copy the full SHA
    d2af329 View commit details
    Browse the repository at this point in the history

  2. wip

    @felixzhu555
    felixzhu555 committed May 21, 2024
    Configuration menu
    Copy the full SHA
    8fe15d4 View commit details
    Browse the repository at this point in the history

  3. add use_attention_sinks args

    @felixzhu555
    felixzhu555 committed May 21, 2024
    Configuration menu
    Copy the full SHA
    3ae06f5 View commit details
    Browse the repository at this point in the history

Commits on May 23, 2024

  1. investigating eviction issue

    @felixzhu555
    felixzhu555 committed May 23, 2024
    Configuration menu
    Copy the full SHA
    67c3bdf View commit details
    Browse the repository at this point in the history

Commits on May 24, 2024

  1. flash attn works

    @felixzhu555
    felixzhu555 committed May 24, 2024
    Configuration menu
    Copy the full SHA
    e09296b View commit details
    Browse the repository at this point in the history

Commits on May 30, 2024

  1. remove eviction & some refactoring

    @felixzhu555
    felixzhu555 committed May 30, 2024
    Configuration menu
    Copy the full SHA
    8f152d5 View commit details
    Browse the repository at this point in the history

Commits on Jun 1, 2024

  1. start mixtral

    @felixzhu555
    felixzhu555 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    a766775 View commit details
    Browse the repository at this point in the history

Commits on Jun 2, 2024

  1. refactor, start alibi, try falcon/bloom

    @felixzhu555
    felixzhu555 committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    1e44278 View commit details
    Browse the repository at this point in the history

  2. tiny

    @felixzhu555
    felixzhu555 committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    7413279 View commit details
    Browse the repository at this point in the history

Commits on Jun 3, 2024

  1. add mpt

    @felixzhu555
    felixzhu555 committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    05d7aa9 View commit details
    Browse the repository at this point in the history

  2. alibi not working

    @felixzhu555
    felixzhu555 committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    19a90f6 View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2024

  1. fix seq len bug -> eviction and alibi work

    @felixzhu555
    felixzhu555 committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    34df763 View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2024

  1. eviction moved to block manager -> rope works, alibi not yet

    @felixzhu555
    felixzhu555 committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    afb754c View commit details
    Browse the repository at this point in the history

  2. fix alibi bug

    @felixzhu555
    felixzhu555 committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    d7db6e1 View commit details
    Browse the repository at this point in the history

  3. beam search not supported

    @felixzhu555
    felixzhu555 committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    3d0929c View commit details
    Browse the repository at this point in the history

  4. small fix

    @felixzhu555
    felixzhu555 committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    13b48c4 View commit details
    Browse the repository at this point in the history

  5. pull main

    @felixzhu555
    felixzhu555 committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    9475536 View commit details
    Browse the repository at this point in the history

  6. refactor models

    @felixzhu555
    felixzhu555 committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    88a77d3 View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2024

  1. pull main

    @felixzhu555
    felixzhu555 committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    b3cfffb View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2024

  1. small

    @felixzhu555
    felixzhu555 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    3e229a0 View commit details
    Browse the repository at this point in the history

Commits on Jun 8, 2024

  1. tests wip

    @felixzhu555
    felixzhu555 committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    b834de8 View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2024

  1. tests failing

    @felixzhu555
    felixzhu555 committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    56b448a View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2024

  1. wip

    @felixzhu555
    felixzhu555 committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    7d9723c View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2024

  1. test correctness done

    @felixzhu555
    felixzhu555 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    2f92168 View commit details
    Browse the repository at this point in the history

  2. add attn backend to tests, add eviction test

    @felixzhu555
    felixzhu555 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    c8416a0 View commit details
    Browse the repository at this point in the history

  3. small

    @felixzhu555
    felixzhu555 committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    b31ae95 View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2024

  1. start refactor

    @felixzhu555
    felixzhu555 committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    f241532 View commit details
    Browse the repository at this point in the history

  2. add wrapper method

    @felixzhu555
    felixzhu555 committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    5c7f802 View commit details
    Browse the repository at this point in the history

  3. wip

    @felixzhu555
    felixzhu555 committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    143db31 View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2024

  1. pull main

    @felixzhu555
    felixzhu555 committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    0722ff0 View commit details
    Browse the repository at this point in the history

  2. refactor wip

    @felixzhu555
    felixzhu555 committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    e0848e3 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2024

  1. fix test

    @felixzhu555
    felixzhu555 committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    7abb285 View commit details
    Browse the repository at this point in the history

  2. small

    @felixzhu555
    felixzhu555 committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    5bf0d5c View commit details
    Browse the repository at this point in the history

  3. chunked prefill wip

    @felixzhu555
    felixzhu555 committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    ae31b1d View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2024

  1. wip

    @felixzhu555
    felixzhu555 committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    779b2a3 View commit details
    Browse the repository at this point in the history

  2. cuda mem error

    @felixzhu555
    felixzhu555 committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    d527920 View commit details
    Browse the repository at this point in the history

Commits on Jun 23, 2024

  1. chunked prefill working

    @felixzhu555
    felixzhu555 committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    87bd485 View commit details
    Browse the repository at this point in the history

  2. wip

    @felixzhu555
    felixzhu555 committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    0a1abf8 View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2024

  1. fix paxos paper

    @felixzhu555
    felixzhu555 committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    08fd48f View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2024

  1. wip

    @felixzhu555
    felixzhu555 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    65f5f6d View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' of https://github.com/vllm-project/vllm into add_… 

    …attention_sinks
    @felixzhu555
    felixzhu555 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    cb12d5f View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2024

  1. chunked prefill for alibi

    @felixzhu555
    felixzhu555 committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    fdc1365 View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2024

  1. add some docstrings

    @felixzhu555
    felixzhu555 committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    da75ff6 View commit details
    Browse the repository at this point in the history

  2. fix test

    @felixzhu555
    felixzhu555 committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    fa8a253 View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2024

  1. pull main

    @felixzhu555
    felixzhu555 committed Jun 29, 2024
    Configuration menu
    Copy the full SHA
    1763a44 View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2024

  1. fix after removal of logical block table

    @felixzhu555
    felixzhu555 committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    ef65724 View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2024

  1. change pos arange

    @felixzhu555
    felixzhu555 committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    38bd15f View commit details
    Browse the repository at this point in the history

  2. pull main

    @felixzhu555
    felixzhu555 committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    b0b8d0b View commit details
    Browse the repository at this point in the history

Commits on Aug 4, 2024

  1. small

    @felixzhu555
    felixzhu555 committed Aug 4, 2024
    Configuration menu
    Copy the full SHA
    7de1a21 View commit details
    Browse the repository at this point in the history

  2. small

    @felixzhu555
    felixzhu555 committed Aug 4, 2024
    Configuration menu
    Copy the full SHA
    1ecec38 View commit details
    Browse the repository at this point in the history

  3. pull main, breaking changes to be fixed

    @felixzhu555
    felixzhu555 committed Aug 4, 2024
    Configuration menu
    Copy the full SHA
    5f03373 View commit details
    Browse the repository at this point in the history

  4. fix updates from pull main

    @felixzhu555
    felixzhu555 committed Aug 4, 2024
    Configuration menu
    Copy the full SHA
    2da86a8 View commit details
    Browse the repository at this point in the history

  5. refactor forward: remove rem logic, move torch ops out of loop

    @felixzhu555
    felixzhu555 committed Aug 4, 2024
    Configuration menu
    Copy the full SHA
    71ca701 View commit details
    Browse the repository at this point in the history

  6. fix flash_attn.py

    @felixzhu555
    felixzhu555 committed Aug 4, 2024
    Configuration menu
    Copy the full SHA
    bce7902 View commit details
    Browse the repository at this point in the history

  7. fix tests

    @felixzhu555
    felixzhu555 committed Aug 4, 2024
    Configuration menu
    Copy the full SHA
    be779fb View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2024

  1. pull main

    @felixzhu555
    felixzhu555 committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    9d97b8d View commit details
    Browse the repository at this point in the history