Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Continous batching for single GPU LLM inference #2628

Merged
merged 49 commits into from Oct 4, 2023
Merged

Continous batching for single GPU LLM inference #2628

merged 49 commits into from Oct 4, 2023

Commits on Sep 6, 2023

  1. Make test/pytest/test_handler.py run stand-alone

    @mreso
    mreso committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    83b88aa View commit details
    Browse the repository at this point in the history

  2. Refactor if else statement

    @mreso
    mreso committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    5138de5 View commit details
    Browse the repository at this point in the history

  3. First working poc for streaming inference with continuous batching

    @mreso
    mreso committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    afd183a View commit details
    Browse the repository at this point in the history

  4. WIP: stopping criteria + caching

    @mreso
    mreso committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    490f676 View commit details
    Browse the repository at this point in the history

  5. FE add continuousbatching

    @lxning @mreso
    lxning authored and mreso committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    ae3d64e View commit details
    Browse the repository at this point in the history

  6. fmt

    @lxning @mreso
    lxning authored and mreso committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    5395cc5 View commit details
    Browse the repository at this point in the history

  7. Fmt

    @lxning @mreso
    lxning authored and mreso committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    c73c46c View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2023

  1. Fix continuous batching PoC; remove batchDelay if jobs are being proc… 

    …essed
    @mreso
    mreso committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    fd7b93a View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2023

  1. Add model_config.yaml

    @mreso
    mreso committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    e94ffc4 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2023

  1. Added ipynb for generate_next_token

    @mreso
    mreso committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    e81b79d View commit details
    Browse the repository at this point in the history

  2. Update notebook and move to right subfolder

    @mreso
    mreso committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    7eef111 View commit details
    Browse the repository at this point in the history

  3. Fix buffer underruns; wait until enough bytes are in for reading

    @mreso
    mreso committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    8f19e00 View commit details
    Browse the repository at this point in the history

  4. Add bandaid for bug in our otf

    @mreso
    mreso committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    cef6e33 View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2023

  1. Added test for otf protocol with context

    @mreso
    mreso committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    c6aece6 View commit details
    Browse the repository at this point in the history

  2. Fix buffer underrun; handle batch quota of zero correctly

    @mreso
    mreso committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    2883a30 View commit details
    Browse the repository at this point in the history

  3. Initial implementation of prefill + decode without kv caching for now

    @mreso
    mreso committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    e1f000f View commit details
    Browse the repository at this point in the history

  4. adds missing __init__py files

    @mreso
    mreso committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    e4f9b56 View commit details
    Browse the repository at this point in the history

  5. WIP kv caching

    @mreso
    mreso committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    2f4ef20 View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2023

  1. Fixed kv cache; missing tuple;

    @mreso
    mreso committed Sep 22, 2023
    Configuration menu
    Copy the full SHA
    6db970c View commit details
    Browse the repository at this point in the history

  2. Cleaned up streaming handler code

    @mreso
    mreso committed Sep 22, 2023
    Configuration menu
    Copy the full SHA
    8c3a890 View commit details
    Browse the repository at this point in the history

  3. Added cache cleaning

    @mreso
    mreso committed Sep 22, 2023
    Configuration menu
    Copy the full SHA
    481ce10 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2023

  1. clean up aggregator jobs forcontrol cmd

    @lxning
    lxning committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    6beb42c View commit details
    Browse the repository at this point in the history

  2. fmt

    @lxning
    lxning committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    eb396a5 View commit details
    Browse the repository at this point in the history

  3. fix streaming handler test

    @mreso
    mreso committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    be28f29 View commit details
    Browse the repository at this point in the history

  4. Rename streaming test into continuous batching test

    @mreso
    mreso committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    1bc2154 View commit details
    Browse the repository at this point in the history

  5. fmt

    @lxning
    lxning committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    b67942e View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2023

  1. Enable gpu usage in continuous batching unit test

    @mreso
    mreso committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    1ec6982 View commit details
    Browse the repository at this point in the history

  2. Add llama to stream notebook

    @mreso
    mreso committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    ae205be View commit details
    Browse the repository at this point in the history

  3. skip pull mgmt job if jobs is not empty

    @lxning
    lxning committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    48694a8 View commit details
    Browse the repository at this point in the history

  4. set pollMgmtJobStatus init value as false

    @lxning
    lxning committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    67ec104 View commit details
    Browse the repository at this point in the history

  5. fmt

    @lxning
    lxning committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    b2b33f1 View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2023

  1. only take describe request if jobsrepo is empty

    @lxning
    lxning committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    267bf07 View commit details
    Browse the repository at this point in the history

  2. init job

    @lxning
    lxning committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    387f548 View commit details
    Browse the repository at this point in the history

Commits on Sep 29, 2023

  1. Remove cont batching job if connection to client gets closed

    @mreso
    mreso committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    5f9b8fe View commit details
    Browse the repository at this point in the history

  2. Fix and reenable cached request logic

    @mreso
    mreso committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    e6d1df0 View commit details
    Browse the repository at this point in the history

  3. Fix linter error

    @mreso
    mreso committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    68829ba View commit details
    Browse the repository at this point in the history

  4. merege origin/feature/continous_batching_for_streaming commit 68829ba

    @lxning
    lxning committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    2825ae0 View commit details
    Browse the repository at this point in the history

  5. fmt

    @lxning
    lxning committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    289c702 View commit details
    Browse the repository at this point in the history

  6. remove llama2-13b stream_handler.py

    @lxning
    lxning committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    c5e7f7e View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2023

  1. revert otf

    @lxning
    lxning committed Sep 30, 2023
    Configuration menu
    Copy the full SHA
    088f330 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2023

  1. update maxDelay logic

    @lxning
    lxning committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    c8b4604 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2023

  1. replace size checking with isEmpty

    @lxning
    lxning committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    d78ce15 View commit details
    Browse the repository at this point in the history

  2. Use handler section

    @mreso
    mreso committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    835e17d View commit details
    Browse the repository at this point in the history

  3. Fix linter errors

    @mreso
    mreso committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    0b6e309 View commit details
    Browse the repository at this point in the history

  4. Fix linter error in oft mesg handler

    @mreso
    mreso committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    b774002 View commit details
    Browse the repository at this point in the history

  5. Fix linter error in test_otf_codec_protocol.py

    @mreso
    mreso committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    8ac8626 View commit details
    Browse the repository at this point in the history

  6. Merge remote-tracking branch 'origin/master' into feature/continous_b… 

    …atching_for_streaming
    @mreso
    mreso committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    7af5f2a View commit details
    Browse the repository at this point in the history

  7. Merge branch 'master' into feature/continous_batching_for_streaming

    @mreso
    mreso committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    4e802f0 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'master' into feature/continous_batching_for_streaming

    @mreso
    mreso committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    7855a9c View commit details
    Browse the repository at this point in the history