Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Lllama] Update tokenization code to ensure parsing of the special tokens [core] #24042

Merged
merged 4 commits into from
Jun 9, 2023
Merged

[Lllama] Update tokenization code to ensure parsing of the special tokens [core] #24042

merged 4 commits into from
Jun 9, 2023

Commits on Jun 6, 2023

  1. preventllama fast from returning token type ids

    @ArthurZucker
    ArthurZucker committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    c0c9672 View commit details
    Browse the repository at this point in the history

  2. remove type hints

    @ArthurZucker
    ArthurZucker committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    c8dee03 View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2023

  1. normalised False

    @ArthurZucker
    ArthurZucker committed Jun 7, 2023
    Configuration menu
    Copy the full SHA
    8622c17 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' of https://github.com/huggingface/transformers in… 

    …to fix-llama-fast
    @ArthurZucker
    ArthurZucker committed Jun 7, 2023
    Configuration menu
    Copy the full SHA
    8c8c782 View commit details
    Browse the repository at this point in the history