Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve additionnal vocabulary management #309

Merged
merged 9 commits into from Jun 19, 2020
Merged

Improve additionnal vocabulary management #309

merged 9 commits into from Jun 19, 2020

Commits on Jun 16, 2020

  1. Rust - Extract AddedVocabulary management from Tokenizer

    @n1t0
    n1t0 committed Jun 16, 2020
    Configuration menu
    Copy the full SHA
    66be62b View commit details
    Browse the repository at this point in the history

  2. Rust - Add slice and slice_bytes to NormalizedString

    @n1t0
    n1t0 committed Jun 16, 2020
    Configuration menu
    Copy the full SHA
    7dff86b View commit details
    Browse the repository at this point in the history

  3. Rust - Add AddedVocabulary + normalized option on AddedToken

    @n1t0
    n1t0 committed Jun 16, 2020
    Configuration menu
    Copy the full SHA
    397cc53 View commit details
    Browse the repository at this point in the history

  4. Rust - Fix/Tweak AddedVocabulary + Fix python tests

    @n1t0
    n1t0 committed Jun 16, 2020
    Configuration menu
    Copy the full SHA
    c6f633e View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2020

  1. AddedVocabulary - Add tests, update bindings + various tweaks

    @n1t0
    n1t0 committed Jun 18, 2020
    Configuration menu
    Copy the full SHA
    fc63d56 View commit details
    Browse the repository at this point in the history

  2. Rust - Fix byte-level decoding for added tokens

    @n1t0
    n1t0 committed Jun 18, 2020
    Configuration menu
    Copy the full SHA
    b92d739 View commit details
    Browse the repository at this point in the history

  3. Update CHANGELOGs

    @n1t0
    n1t0 committed Jun 18, 2020
    Configuration menu
    Copy the full SHA
    4c7a0ff View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2020

  1. Python - Update AddedToken repr

    @n1t0
    n1t0 committed Jun 19, 2020
    Configuration menu
    Copy the full SHA
    63edb95 View commit details
    Browse the repository at this point in the history

  2. Python - Make AddedToken pickable

    @n1t0
    n1t0 committed Jun 19, 2020
    Configuration menu
    Copy the full SHA
    898a4a8 View commit details
    Browse the repository at this point in the history