Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cache attention keys + values to speed up inference #216

Merged
merged 7 commits into from
Jun 20, 2023
Merged

Cache attention keys + values to speed up inference #216

merged 7 commits into from
Jun 20, 2023

Commits on Jun 19, 2023

  1. cache attention keys/values during inference

    @epwalsh
    epwalsh committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    d605355 View commit details
    Browse the repository at this point in the history

  2. ssh, mypy

    @epwalsh
    epwalsh committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    8dcde40 View commit details
    Browse the repository at this point in the history

  3. clean up

    @epwalsh
    epwalsh committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    85b8fa1 View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2023

  1. testing

    @epwalsh
    epwalsh committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    c9f2b57 View commit details
    Browse the repository at this point in the history

  2. rename

    @epwalsh
    epwalsh committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    e9d4f52 View commit details
    Browse the repository at this point in the history

  3. fix

    @epwalsh
    epwalsh committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    c068d8b View commit details
    Browse the repository at this point in the history

  4. add option to only compute last logits

    @epwalsh
    epwalsh committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    3ab48df View commit details
    Browse the repository at this point in the history