Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tokenizer : special token handling #3538

Merged
merged 11 commits into from
Oct 17, 2023
Merged

tokenizer : special token handling #3538

merged 11 commits into from
Oct 17, 2023

Commits on Oct 8, 2023

  1. Rewrite special token handling from ggerganov#1931

    @staviq
    staviq committed Oct 8, 2023
    Configuration menu
    Copy the full SHA
    b592c70 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2023

  1. shorten param name, add st verification by type

    @staviq
    staviq committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    fc634d8 View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2023

  1. use offsets instead of copy by substr

    @staviq
    staviq committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    29e6b46 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into specialtokens

    @staviq
    staviq committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    eac5f54 View commit details
    Browse the repository at this point in the history

  3. formatting, remove copying iterator on delete

    @staviq
    staviq committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    f7b1205 View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2023

  1. Merge branch 'master' into HEAD

    @ggerganov
    ggerganov committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    04ac055 View commit details
    Browse the repository at this point in the history

  2. llama : normalize code-style

    @ggerganov
    ggerganov committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    5c6b2be View commit details
    Browse the repository at this point in the history

  3. swift fix

    @staviq
    staviq committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    0f1c569 View commit details
    Browse the repository at this point in the history

  4. print pfx/sfx if verb, main: split pfx input sfx

    @staviq
    staviq committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    5974d61 View commit details
    Browse the repository at this point in the history

  5. dont add space when using special tokens

    @staviq
    staviq committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    1c28116 View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2023

  1. minor : comment + spacing

    @ggerganov
    ggerganov committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    fc82541 View commit details
    Browse the repository at this point in the history