Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add inference cache (cleaner version) #4

Open
wants to merge 19 commits into
base: weight-sharing
Choose a base branch
from
Open

Add inference cache (cleaner version) #4

wants to merge 19 commits into from

Commits on Dec 20, 2021

  1. Add FFN caching

    @borzunov
    borzunov committed Dec 20, 2021
    Configuration menu
    Copy the full SHA
    1cd8e20 View commit details
    Browse the repository at this point in the history

  2. Cache to_qkv and to_out in sparse attn, add debug prints

    @borzunov
    borzunov committed Dec 20, 2021
    Configuration menu
    Copy the full SHA
    4d431ac View commit details
    Browse the repository at this point in the history

  3. Cache full Attention

    @borzunov
    borzunov committed Dec 20, 2021
    Configuration menu
    Copy the full SHA
    2603776 View commit details
    Browse the repository at this point in the history

  4. Remove debug outputs

    @borzunov
    borzunov committed Dec 20, 2021
    Configuration menu
    Copy the full SHA
    112ea05 View commit details
    Browse the repository at this point in the history

  5. Cache pre-logits MLP

    @borzunov
    borzunov committed Dec 20, 2021
    Configuration menu
    Copy the full SHA
    6ba4cb6 View commit details
    Browse the repository at this point in the history

  6. Further optimize attention caching

    @borzunov
    borzunov committed Dec 20, 2021
    Configuration menu
    Copy the full SHA
    c333ea7 View commit details
    Browse the repository at this point in the history

Commits on Dec 21, 2021

  1. Fix mask in attention

    @borzunov
    borzunov committed Dec 21, 2021
    Configuration menu
    Copy the full SHA
    1fd45ca View commit details
    Browse the repository at this point in the history

  2. Don't cache MLPs since we can just pass only last item

    @borzunov
    borzunov committed Dec 21, 2021
    Configuration menu
    Copy the full SHA
    2b77018 View commit details
    Browse the repository at this point in the history

  3. Revert excess changes in attentions

    @borzunov
    borzunov committed Dec 21, 2021
    Configuration menu
    Copy the full SHA
    8e8dea8 View commit details
    Browse the repository at this point in the history

  4. Rename FixCacheKey -> CachedAs

    @borzunov
    borzunov committed Dec 21, 2021
    Configuration menu
    Copy the full SHA
    df89951 View commit details
    Browse the repository at this point in the history

  5. Save the current offset in cache

    @borzunov
    borzunov committed Dec 21, 2021
    Configuration menu
    Copy the full SHA
    059fe1b View commit details
    Browse the repository at this point in the history

  6. Use static masks to simulate axial attn

    @borzunov
    borzunov committed Dec 21, 2021
    Configuration menu
    Copy the full SHA
    b76b78e View commit details
    Browse the repository at this point in the history

  7. Add option to disable caching

    @borzunov
    borzunov committed Dec 21, 2021
    Configuration menu
    Copy the full SHA
    4c833a2 View commit details
    Browse the repository at this point in the history

  8. Make the cached version work

    @borzunov
    borzunov committed Dec 21, 2021
    Configuration menu
    Copy the full SHA
    1ff47c6 View commit details
    Browse the repository at this point in the history

  9. Speed up PreShiftToken

    @borzunov
    borzunov committed Dec 21, 2021
    Configuration menu
    Copy the full SHA
    94fda36 View commit details
    Browse the repository at this point in the history

Commits on Jan 10, 2022

  1. Remove excess changes

    @borzunov
    borzunov committed Jan 10, 2022
    Configuration menu
    Copy the full SHA
    adfce34 View commit details
    Browse the repository at this point in the history

  2. Add NonCached wrapper

    @borzunov
    borzunov committed Jan 10, 2022
    Configuration menu
    Copy the full SHA
    59cfc49 View commit details
    Browse the repository at this point in the history

  3. Rename use_static_masks -> optimize_for_inference

    @borzunov
    borzunov committed Jan 10, 2022
    Configuration menu
    Copy the full SHA
    4f496a4 View commit details
    Browse the repository at this point in the history

  4. Improve names and comments

    @borzunov
    borzunov committed Jan 10, 2022
    Configuration menu
    Copy the full SHA
    732226d View commit details
    Browse the repository at this point in the history