Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Experimental] Prefix Caching Support #1669

Merged
merged 44 commits into from
Jan 18, 2024
Merged

[Experimental] Prefix Caching Support #1669

merged 44 commits into from
Jan 18, 2024

Commits on Nov 15, 2023

  1. add the prefix prefill triton kernel

    @caoshiyi
    caoshiyi committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    8d8ebb1 View commit details
    Browse the repository at this point in the history

  2. add the prefix class

    @caoshiyi
    caoshiyi committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    49aaf42 View commit details
    Browse the repository at this point in the history

  3. modify block manager & scheduler

    @caoshiyi
    caoshiyi committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    ac96e7f View commit details
    Browse the repository at this point in the history

  4. prepare inputs to the kernel

    @caoshiyi
    caoshiyi committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    bf3faa2 View commit details
    Browse the repository at this point in the history

Commits on Nov 16, 2023

  1. fix

    @caoshiyi
    caoshiyi committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    074a90f View commit details
    Browse the repository at this point in the history

  2. add prefix_pos

    @caoshiyi
    caoshiyi committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    de69ca4 View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2023

  1. fix prefix state transition

    @caoshiyi
    caoshiyi committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    f5bf25a View commit details
    Browse the repository at this point in the history

  2. tested on single request

    @caoshiyi
    caoshiyi committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    49b19bb View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2023

  1. add prefix_pos for offline inference

    @caoshiyi
    caoshiyi committed Nov 18, 2023
    Configuration menu
    Copy the full SHA
    1619d33 View commit details
    Browse the repository at this point in the history

  2. minor

    @caoshiyi
    caoshiyi committed Nov 18, 2023
    Configuration menu
    Copy the full SHA
    2309330 View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2023

  1. fix blocktable padding

    @caoshiyi
    caoshiyi committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    d8b2809 View commit details
    Browse the repository at this point in the history

Commits on Jan 1, 2024

  1. fix multi-gpu state transition

    @caoshiyi
    caoshiyi committed Jan 1, 2024
    Configuration menu
    Copy the full SHA
    50e7497 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into prefix

    @caoshiyi
    caoshiyi committed Jan 1, 2024
    Configuration menu
    Copy the full SHA
    9b4a63d View commit details
    Browse the repository at this point in the history

Commits on Jan 2, 2024

  1. clean

    @caoshiyi
    caoshiyi committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    33bfcff View commit details
    Browse the repository at this point in the history

  2. format

    @caoshiyi
    caoshiyi committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    1cef69f View commit details
    Browse the repository at this point in the history

  3. add support for alibi bias 

    Co-authored-by: DouHappy <2278958187@qq.com>
    @caoshiyi @DouHappy
    caoshiyi and DouHappy committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    98ee509 View commit details
    Browse the repository at this point in the history

Commits on Jan 5, 2024

  1. Merge branch 'main' into prefix

    @caoshiyi
    caoshiyi committed Jan 5, 2024
    Configuration menu
    Copy the full SHA
    6e07602 View commit details
    Browse the repository at this point in the history

  2. format

    @caoshiyi
    caoshiyi committed Jan 5, 2024
    Configuration menu
    Copy the full SHA
    a948cd3 View commit details
    Browse the repository at this point in the history

Commits on Jan 6, 2024

  1. clean

    @caoshiyi
    caoshiyi committed Jan 6, 2024
    Configuration menu
    Copy the full SHA
    ead42a2 View commit details
    Browse the repository at this point in the history

  2. clean

    @caoshiyi
    caoshiyi committed Jan 6, 2024
    Configuration menu
    Copy the full SHA
    7bcb509 View commit details
    Browse the repository at this point in the history

Commits on Jan 7, 2024

  1. clean & minor

    @caoshiyi
    caoshiyi committed Jan 7, 2024
    Configuration menu
    Copy the full SHA
    8bc52ca View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into prefix

    @caoshiyi
    caoshiyi committed Jan 7, 2024
    Configuration menu
    Copy the full SHA
    a534d6c View commit details
    Browse the repository at this point in the history

  3. format

    @caoshiyi
    caoshiyi committed Jan 7, 2024
    Configuration menu
    Copy the full SHA
    abb843b View commit details
    Browse the repository at this point in the history

  4. move prefix prefill kernel test to a separate file

    @caoshiyi
    caoshiyi committed Jan 7, 2024
    Configuration menu
    Copy the full SHA
    dc08e14 View commit details
    Browse the repository at this point in the history

  5. format

    @caoshiyi
    caoshiyi committed Jan 7, 2024
    Configuration menu
    Copy the full SHA
    f0f8f66 View commit details
    Browse the repository at this point in the history

Commits on Jan 13, 2024

  1. Merge branch 'main' into prefix

    @zhuohan123
    zhuohan123 committed Jan 13, 2024
    Configuration menu
    Copy the full SHA
    2389fdb View commit details
    Browse the repository at this point in the history

  2. fix test

    @zhuohan123
    zhuohan123 committed Jan 13, 2024
    Configuration menu
    Copy the full SHA
    3678af6 View commit details
    Browse the repository at this point in the history

  3. format kernel

    @zhuohan123
    zhuohan123 committed Jan 13, 2024
    Configuration menu
    Copy the full SHA
    037950b View commit details
    Browse the repository at this point in the history

  4. fix format

    @zhuohan123
    zhuohan123 committed Jan 13, 2024
    Configuration menu
    Copy the full SHA
    b414c77 View commit details
    Browse the repository at this point in the history

Commits on Jan 15, 2024

  1. [WIP] Refactor

    @zhuohan123
    zhuohan123 committed Jan 15, 2024
    2 Configuration menu
    Copy the full SHA
    bb4ca73 View commit details
    Browse the repository at this point in the history

  2. fix comments

    @zhuohan123
    zhuohan123 committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    37cd3fc View commit details
    Browse the repository at this point in the history

Commits on Jan 16, 2024

  1. Merge branch 'prefix' of https://github.com/caoshiyi/vllm into prefix

    @zhuohan123
    zhuohan123 committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    b09bbaa View commit details
    Browse the repository at this point in the history

  2. add example and test

    @zhuohan123
    zhuohan123 committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    58d2839 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into prefix

    @zhuohan123
    zhuohan123 committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    8370e7b View commit details
    Browse the repository at this point in the history

  4. add prefix caching test to ci

    @zhuohan123
    zhuohan123 committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    58c5cff View commit details
    Browse the repository at this point in the history

  5. fix ci

    @zhuohan123
    zhuohan123 committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    dc6e959 View commit details
    Browse the repository at this point in the history

Commits on Jan 17, 2024

  1. add comment

    @zhuohan123
    zhuohan123 committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    7dc2e87 View commit details
    Browse the repository at this point in the history

  2. add TODO

    @zhuohan123
    zhuohan123 committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    49b2684 View commit details
    Browse the repository at this point in the history

  3. fix

    @zhuohan123
    zhuohan123 committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    c9050d3 View commit details
    Browse the repository at this point in the history

  4. fix swapping logic

    @zhuohan123
    zhuohan123 committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    cfe1444 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into prefix

    @zhuohan123
    zhuohan123 committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    3bb9802 View commit details
    Browse the repository at this point in the history

  6. fix bug

    @zhuohan123
    zhuohan123 committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    29f4f96 View commit details
    Browse the repository at this point in the history

  7. fix correctness

    @zhuohan123
    zhuohan123 committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    bd56a69 View commit details
    Browse the repository at this point in the history

Commits on Jan 18, 2024

  1. add notes and small fix

    @zhuohan123
    zhuohan123 committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    6b00283 View commit details
    Browse the repository at this point in the history