Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Core][Hash][Automatic Prefix caching] Accelerating the hashing function by avoiding deep copies #4696

Merged
merged 12 commits into from
May 14, 2024
Merged

[Core][Hash][Automatic Prefix caching] Accelerating the hashing function by avoiding deep copies #4696

merged 12 commits into from
May 14, 2024
+78 −3

Commits on May 8, 2024

  1. Kuntai: improve hashing to avoid multiple deep copies on prompt_token… 

    …_ids
    @KuntaiDu
    KuntaiDu committed May 8, 2024
    Configuration menu
    Copy the full SHA
    16fd11a View commit details
    Browse the repository at this point in the history

  2. Kuntai: update benchmark_hashing.py so that no huggingface token is r… 

    …equired
    @KuntaiDu
    KuntaiDu committed May 8, 2024
    Configuration menu
    Copy the full SHA
    e9dc917 View commit details
    Browse the repository at this point in the history

  3. Kuntai: fix formatting issues

    @KuntaiDu
    KuntaiDu committed May 8, 2024
    Configuration menu
    Copy the full SHA
    8112e0d View commit details
    Browse the repository at this point in the history

  4. Kuntai: fix issues after formatting

    @KuntaiDu
    KuntaiDu committed May 8, 2024
    Configuration menu
    Copy the full SHA
    57fd939 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2024

  1. Kuntai: adjust prompt_token_ids to tuple, and fix various type annota… 

    …tion conflict
    @KuntaiDu
    KuntaiDu committed May 9, 2024
    Configuration menu
    Copy the full SHA
    803572e View commit details
    Browse the repository at this point in the history

  2. Kuntai: let prompt_token_ids in RequestOutputs be List[int] so that i… 

    …t aligns with the expectation of test cases, plus moving benchmark_hashing.py to a new folder
    @KuntaiDu
    KuntaiDu committed May 9, 2024
    Configuration menu
    Copy the full SHA
    b9b38c0 View commit details
    Browse the repository at this point in the history

Commits on May 10, 2024

  1. Kuntai: update the test case of speculative decoding: explicitly conv… 

    …ert to list
    @KuntaiDu
    KuntaiDu committed May 10, 2024
    Configuration menu
    Copy the full SHA
    e0691a6 View commit details
    Browse the repository at this point in the history

  2. Kuntai: keep the prompt_ids as Tuple and change the test cases instead

    @KuntaiDu
    KuntaiDu committed May 10, 2024
    Configuration menu
    Copy the full SHA
    bf498b0 View commit details
    Browse the repository at this point in the history

  3. Kuntai: revert RequestOutput.prompt_token_ids back to List[int], in o… 

    …rder to be OpenAI-compatible
    @KuntaiDu
    KuntaiDu committed May 10, 2024
    Configuration menu
    Copy the full SHA
    b352246 View commit details
    Browse the repository at this point in the history

  4. Kuntai: make prompt_token_ids list, and add new private attribute _pr… 

    …ompt_token_ids_tuple to speed up hashing
    @KuntaiDu
    KuntaiDu committed May 10, 2024
    Configuration menu
    Copy the full SHA
    a868884 View commit details
    Browse the repository at this point in the history

  5. Kuntai: remove a list conversion

    @KuntaiDu
    KuntaiDu committed May 10, 2024
    Configuration menu
    Copy the full SHA
    49d994d View commit details
    Browse the repository at this point in the history

Commits on May 14, 2024

  1. Kuntai: remove list() conversion in get_token_ids

    @KuntaiDu
    KuntaiDu committed May 14, 2024
    Configuration menu
    Copy the full SHA
    e95d660 View commit details
    Browse the repository at this point in the history