Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[NeMo-UX] Fix tokenizer IO #9555

Merged
merged 4 commits into from
Jun 27, 2024
Merged

[NeMo-UX] Fix tokenizer IO #9555

merged 4 commits into from
Jun 27, 2024

Commits on Jun 27, 2024

  1. Adding tokenizer to io-test + making it pass

    @marcromeyn
    marcromeyn committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    43a4cc6 View commit details
    Browse the repository at this point in the history

  2. Handling tokenizer correctly inside dump_io

    @marcromeyn
    marcromeyn committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    6db6fd0 View commit details
    Browse the repository at this point in the history

  3. Apply isort and black reformatting 

    Signed-off-by: marcromeyn <marcromeyn@users.noreply.github.com>
    @marcromeyn
    marcromeyn committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    6056529 View commit details
    Browse the repository at this point in the history

  4. Removing not used import

    @marcromeyn
    marcromeyn committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    c694c5b View commit details
    Browse the repository at this point in the history