Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add split special tokens #30772

Merged
merged 21 commits into from
May 24, 2024
Merged

Add split special tokens #30772

merged 21 commits into from
May 24, 2024

Commits on May 24, 2024

  1. seems like split_special_tokens is used here

    @ArthurZucker
    ArthurZucker committed May 24, 2024
    Configuration menu
    Copy the full SHA
    56fd608 View commit details
    Browse the repository at this point in the history

  2. split special token

    @ArthurZucker
    Ita Zaporozhets authored and ArthurZucker committed May 24, 2024
    Configuration menu
    Copy the full SHA
    fbb144c View commit details
    Browse the repository at this point in the history

  3. add new line at end of file

    @ArthurZucker
    Ita Zaporozhets authored and ArthurZucker committed May 24, 2024
    Configuration menu
    Copy the full SHA
    abeaeb4 View commit details
    Browse the repository at this point in the history

  4. moving split special token test to common tests

    @ArthurZucker
    Ita Zaporozhets authored and ArthurZucker committed May 24, 2024
    Configuration menu
    Copy the full SHA
    99580a9 View commit details
    Browse the repository at this point in the history

  5. added assertions

    @ArthurZucker
    Ita Zaporozhets authored and ArthurZucker committed May 24, 2024
    Configuration menu
    Copy the full SHA
    a7ebe7d View commit details
    Browse the repository at this point in the history

  6. test

    @ArthurZucker
    ArthurZucker committed May 24, 2024
    Configuration menu
    Copy the full SHA
    224aae4 View commit details
    Browse the repository at this point in the history

  7. fixup

    @ArthurZucker
    ArthurZucker committed May 24, 2024
    Configuration menu
    Copy the full SHA
    80adc63 View commit details
    Browse the repository at this point in the history

  8. add co-author

    @itazap @ArthurZucker
    itazap authored and ArthurZucker committed May 24, 2024
    Configuration menu
    Copy the full SHA
    6dbe878 View commit details
    Browse the repository at this point in the history

  9. passing rest of args to gptsan_japanese, fixing tests

    @ArthurZucker
    Ita Zaporozhets authored and ArthurZucker committed May 24, 2024
    Configuration menu
    Copy the full SHA
    4000f63 View commit details
    Browse the repository at this point in the history

  10. removing direct comparison of fast and slow models

    @ArthurZucker
    Ita Zaporozhets authored and ArthurZucker committed May 24, 2024
    Configuration menu
    Copy the full SHA
    aec5f71 View commit details
    Browse the repository at this point in the history

  11. adding test support for UDOP and LayoutXLM

    @ArthurZucker
    Ita Zaporozhets authored and ArthurZucker committed May 24, 2024
    Configuration menu
    Copy the full SHA
    88b12aa View commit details
    Browse the repository at this point in the history

  12. ruff fix

    @ArthurZucker
    Ita Zaporozhets authored and ArthurZucker committed May 24, 2024
    Configuration menu
    Copy the full SHA
    fd5d1fe View commit details
    Browse the repository at this point in the history

  13. readd check if slow tokenizer

    @ArthurZucker
    Ita Zaporozhets authored and ArthurZucker committed May 24, 2024
    Configuration menu
    Copy the full SHA
    bb64b8d View commit details
    Browse the repository at this point in the history

  14. modify test to handle bos tokens

    @ArthurZucker
    Ita Zaporozhets authored and ArthurZucker committed May 24, 2024
    Configuration menu
    Copy the full SHA
    24473ac View commit details
    Browse the repository at this point in the history

  15. removing commented function

    @ArthurZucker
    Ita Zaporozhets authored and ArthurZucker committed May 24, 2024
    Configuration menu
    Copy the full SHA
    fb321f6 View commit details
    Browse the repository at this point in the history

  16. trigger build

    @ArthurZucker
    Ita Zaporozhets authored and ArthurZucker committed May 24, 2024
    Configuration menu
    Copy the full SHA
    fd357bd View commit details
    Browse the repository at this point in the history

  17. applying review feedback - updated docstrings, var names, and simplif… 

    …ied tests
    @ArthurZucker
    Ita Zaporozhets authored and ArthurZucker committed May 24, 2024
    Configuration menu
    Copy the full SHA
    34cdba6 View commit details
    Browse the repository at this point in the history

  18. ruff fixes

    @ArthurZucker
    Ita Zaporozhets authored and ArthurZucker committed May 24, 2024
    Configuration menu
    Copy the full SHA
    29af720 View commit details
    Browse the repository at this point in the history

  19. Update tests/test_tokenization_common.py 

    Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
    @itazap @ArthurZucker
    itazap and ArthurZucker committed May 24, 2024
    Configuration menu
    Copy the full SHA
    80b4e77 View commit details
    Browse the repository at this point in the history

  20. applying feedback, comments

    @ArthurZucker
    Ita Zaporozhets authored and ArthurZucker committed May 24, 2024
    Configuration menu
    Copy the full SHA
    f5bf109 View commit details
    Browse the repository at this point in the history

  21. shutil temp directory fix

    Ita Zaporozhets authored and Ita Zaporozhets committed May 24, 2024
    Configuration menu
    Copy the full SHA
    2ce7569 View commit details
    Browse the repository at this point in the history