Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TP llama with continuous batching #2709

Merged
merged 21 commits into from
Dec 14, 2023
Merged

TP llama with continuous batching #2709

merged 21 commits into from
Dec 14, 2023

Commits on Oct 12, 2023

  1. Enabled left side padding in llama2 model

    @mreso
    mreso committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    cdc01ed View commit details
    Browse the repository at this point in the history

  2. Add unit test for tp_llama

    @mreso
    mreso committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    5d27b13 View commit details
    Browse the repository at this point in the history

  3. Use no_grad context manager

    @mreso
    mreso committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    8022495 View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2023

  1. [WIP] Converting llama_tp into cont batching handler

    @mreso
    mreso committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    bc4375c View commit details
    Browse the repository at this point in the history

  2. [WIP]Implement prefill and decode for tp_llama

    @mreso
    mreso committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    ece426f View commit details
    Browse the repository at this point in the history

Commits on Oct 19, 2023

  1. WIP fixing decode

    @mreso
    mreso committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    a3f4b38 View commit details
    Browse the repository at this point in the history

  2. Fix return prefill vs decode format in tp_llama handler

    @mreso
    mreso committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    c405e0c View commit details
    Browse the repository at this point in the history

Commits on Oct 20, 2023

  1. Fix current_position index in llama_handler

    @mreso
    mreso committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    430fe2e View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2023

  1. Fix kv caching issue with different results for different padding len… 

    …gths
    @mreso
    mreso committed Oct 24, 2023
    Configuration menu
    Copy the full SHA
    85966dc View commit details
    Browse the repository at this point in the history

  2. Fix liniting error

    @mreso
    mreso committed Oct 24, 2023
    Configuration menu
    Copy the full SHA
    246bb94 View commit details
    Browse the repository at this point in the history

  3. Remove cuda dependency for tp_llama test

    @mreso
    mreso committed Oct 24, 2023
    Configuration menu
    Copy the full SHA
    9174b55 View commit details
    Browse the repository at this point in the history

  4. fix handler mock

    @mreso
    mreso committed Oct 24, 2023
    Configuration menu
    Copy the full SHA
    c377b81 View commit details
    Browse the repository at this point in the history

  5. Adjust expected result for 13b model in tp llama test

    @mreso
    mreso committed Oct 24, 2023
    Configuration menu
    Copy the full SHA
    5bce1ee View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2023

  1. Make continuous batching work with tp llama

    @mreso
    mreso committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    17fd172 View commit details
    Browse the repository at this point in the history

  2. Adjust sample txt

    @mreso
    mreso committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    6e57063 View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2023

  1. Add missing requirements

    @mreso
    mreso committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    92590db View commit details
    Browse the repository at this point in the history

  2. Add support for chat dialogs

    @mreso
    mreso committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    be59945 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into feature/tp_llama_with_continuous_batching

    @mreso
    mreso committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    eb934a6 View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2023

  1. Merge branch 'master' into feature/tp_llama_with_continuous_batching

    @mreso
    mreso committed Oct 30, 2023
    Configuration menu
    Copy the full SHA
    b7230ac View commit details
    Browse the repository at this point in the history

Commits on Dec 11, 2023

  1. Merge branch 'master' into feature/tp_llama_with_continuous_batching

    @HamidShojanazeri
    HamidShojanazeri committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    a22a37f View commit details
    Browse the repository at this point in the history

Commits on Dec 13, 2023

  1. Use model archiver config in tp_llama test

    @mreso
    mreso committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    df362b7 View commit details
    Browse the repository at this point in the history