Skip to content

Add bytelevel normalizer to fix decode when adding tokens to BPE #2357

Add bytelevel normalizer to fix decode when adding tokens to BPE

Add bytelevel normalizer to fix decode when adding tokens to BPE #2357

Triggered via pull request June 18, 2024 07:00
Status Failure
Total duration 1m 59s
Artifacts

python.yml

on: pull_request
Matrix: Check everything builds & tests
Matrix: Check it builds for Windows 32-bit
Fit to window
Zoom out
Zoom in

Annotations

16 errors and 28 warnings
Check everything builds & tests (ubuntu-latest): tokenizers/src/tokenizer/mod.rs#L1320
this file contains an unclosed delimiter
Check everything builds & tests (ubuntu-latest)
could not compile `tokenizers` (lib) due to 1 previous error
Check everything builds & tests (ubuntu-latest)
The process '/home/runner/.cargo/bin/cargo' failed with exit code 101
Check everything builds & tests (macos-latest)
The job was canceled because "ubuntu-latest" failed.
Check everything builds & tests (macos-latest)
The operation was canceled.
Check it builds for Windows 32-bit (3.7): tokenizers/src/tokenizer/mod.rs#L1320
this file contains an unclosed delimiter
Check it builds for Windows 32-bit (3.7)
could not compile `tokenizers` (lib) due to 1 previous error
Check it builds for Windows 32-bit (3.7)
The process 'C:\Users\runneradmin\.cargo\bin\cargo.exe' failed with exit code 101
Check it builds for Windows 32-bit (3.9)
The job was canceled because "_3_7" failed.
Check it builds for Windows 32-bit (3.9): tokenizers/src/tokenizer/mod.rs#L1320
this file contains an unclosed delimiter
Check it builds for Windows 32-bit (3.9)
could not compile `tokenizers` (lib) due to 1 previous error
Check it builds for Windows 32-bit (3.9)
The process 'C:\Users\runneradmin\.cargo\bin\cargo.exe' failed with exit code 101
Check it builds for Windows 32-bit (3.10)
The job was canceled because "_3_7" failed.
Check it builds for Windows 32-bit (3.10)
The operation was canceled.
Check it builds for Windows 32-bit (3.8)
The job was canceled because "_3_7" failed.
Check it builds for Windows 32-bit (3.8)
The operation was canceled.
Check everything builds & tests (ubuntu-latest)
Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20: actions/checkout@v2, actions-rs/toolchain@v1, actions-rs/cargo@v1, actions/setup-python@v4, actions/cache@v1. For more information see: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.
Check everything builds & tests (ubuntu-latest)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/checkout@v2, actions-rs/toolchain@v1, actions-rs/cargo@v1, actions/cache@v1. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
Check everything builds & tests (ubuntu-latest)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Check everything builds & tests (ubuntu-latest)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Check everything builds & tests (ubuntu-latest)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Check everything builds & tests (ubuntu-latest)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Check everything builds & tests (macos-latest)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Check everything builds & tests (macos-latest)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Check everything builds & tests (macos-latest)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Check everything builds & tests (macos-latest)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Check it builds for Windows 32-bit (3.7)
Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20: actions/checkout@v3, actions-rs/toolchain@v1, actions/setup-python@v2, actions-rs/cargo@v1. For more information see: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.
Check it builds for Windows 32-bit (3.7)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions-rs/toolchain@v1, actions/setup-python@v2, actions-rs/cargo@v1. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
Check it builds for Windows 32-bit (3.7)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Check it builds for Windows 32-bit (3.7)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Check it builds for Windows 32-bit (3.7)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Check it builds for Windows 32-bit (3.7)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Check it builds for Windows 32-bit (3.9)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Check it builds for Windows 32-bit (3.9)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Check it builds for Windows 32-bit (3.9)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Check it builds for Windows 32-bit (3.9)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Check it builds for Windows 32-bit (3.10)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Check it builds for Windows 32-bit (3.10)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Check it builds for Windows 32-bit (3.10)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Check it builds for Windows 32-bit (3.10)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Check it builds for Windows 32-bit (3.8)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Check it builds for Windows 32-bit (3.8)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Check it builds for Windows 32-bit (3.8)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Check it builds for Windows 32-bit (3.8)
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/