Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Skip computation of much of last layer & unused logits during prompt eval / large N #2700

Closed
wants to merge 15 commits into from
Closed

Skip computation of much of last layer & unused logits during prompt eval / large N #2700

wants to merge 15 commits into from

Commits on Aug 24, 2023

  1. Skip computation of unused logits during batch prompt eval (drop othe… 

    …r batch positions after writing their kv to cache)
    @ochafik
    ochafik committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    2cf4f62 View commit details
    Browse the repository at this point in the history

  2. skip-unused: disable skipping on ROCm / when LLAMA_USE_HIPBLAS

    @ochafik
    ochafik committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    7ec7ef9 View commit details
    Browse the repository at this point in the history

Commits on Aug 25, 2023

  1. Allow disabling unused logit skipping code w/ cmake / make options 

    cmake -DLLAMA_SKIP_UNUSED_LOGITS=OFF ...
LLAMA_NO_SKIP_UNUSED_LOGITS=1 make ...
    @ochafik
    ochafik committed Aug 25, 2023
    Configuration menu
    Copy the full SHA
    5553820 View commit details
    Browse the repository at this point in the history

  2. Tweak GPU offload when skipping unused logits computations

    @ochafik
    ochafik committed Aug 25, 2023
    Configuration menu
    Copy the full SHA
    3be6e8d View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into skip-unused-2

    @ochafik
    ochafik committed Aug 25, 2023
    Configuration menu
    Copy the full SHA
    e23fa92 View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2023

  1. fix offloading logic

    @JohannesGaessler
    JohannesGaessler committed Aug 27, 2023
    Configuration menu
    Copy the full SHA
    21df40d View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2023

  1. skip-unused: fix brackets & tabs

    @ochafik
    ochafik committed Aug 28, 2023
    Configuration menu
    Copy the full SHA
    2eaeb7e View commit details
    Browse the repository at this point in the history

  2. skip-unused: revert extra spaces

    @ochafik
    ochafik committed Aug 28, 2023
    Configuration menu
    Copy the full SHA
    f6a446e View commit details
    Browse the repository at this point in the history

  3. skip-unused: fix -ngl=1 case by ensure input & of view are offloaded … 

    …consistently
    @ochafik
    ochafik committed Aug 28, 2023
    Configuration menu
    Copy the full SHA
    9f5b781 View commit details
    Browse the repository at this point in the history

  4. Fix multiple offloading

    @JohannesGaessler
    JohannesGaessler committed Aug 28, 2023
    Configuration menu
    Copy the full SHA
    e9e8ac4 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2023

  1. Merge remote-tracking branch 'origin/master' into skip-unused-2

    @ochafik
    ochafik committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    d547e05 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'JohannesGaessler/skip-unused-2' into sk… 

    …ip-unused-2
    @ochafik
    ochafik committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    4974f37 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2023

  1. Merge remote-tracking branch 'origin/master' into skip-unused-2

    @ochafik
    ochafik committed Sep 11, 2023
    Configuration menu
    Copy the full SHA
    646caf9 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2023

  1. Merge remote-tracking branch 'origin/master' into skip-unused-2

    @ochafik
    ochafik committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    55d49b2 View commit details
    Browse the repository at this point in the history

  2. Makefile: move unused logits flags where they don't interfere w/ targets 

    (and also fix bad merge)
    @ochafik
    ochafik committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    58bb7d5 View commit details
    Browse the repository at this point in the history