Skip to content

Local DE Coach v0.4.4 — Fix transformers/tokenizers for Python 3.14

Choose a tag to compare

View all tags
@bif26 bif26 released this 13 Sep 01:48
· 18 commits to main since this release
v0.4.4
5066f09

🔧 Root cause found and fixed

The previous failures (python-Levenshtein, rapidfuzz 3.10.1, webrtcvad) were all symptoms of the same underlying issue: packages without Python 3.14 prebuilt wheels fall back to source build, which fails because their pyo3 dependency only supports Python up to 3.13.

This release fixes the root cause: transformers==4.46.3 pins tokenizers<0.21, and tokenizers 0.20.x only ships cp310 wheels (Python 3.10-specific). On Python 3.14, pip builds from source and hits the pyo3-ffi 0.22.5 error:

error: the configured Python interpreter version (3.14) is newer than
PyO3's maximum supported version (3.13)

✨ The fix

Bumped to versions that ship stable-ABI (cp39-abi3) wheels — these work on Python 3.9 through 3.14+ without recompilation:

Package Old New Why
transformers 4.46.3 4.49.0 Pulls tokenizers>=0.21 which has cp39-abi3 wheels
faster-whisper 1.1.0 1.1.1 Pulls ctranslate2>=4.8 which has cp314 wheels

The cp39-abi3 tag is Python's stable ABI — a single wheel that works on any Python ≥ 3.9. This is why tokenizers 0.21.4 installs cleanly on Python 3.14 while 0.20.3 does not.

🐛 Two scorer bugs also fixed

While testing with rapidfuzz, I found two latent bugs in the scorer that the old python-Levenshtein happened to mask:

Bug 1: find_suffix_mismatch off-by-one

  • When the last character of two words differed, the function returned an empty string instead of that character.
  • Example: 'guten' vs 'gute' returned '' instead of 'n'
  • Fix: use diff_idx + 1 to include the differing character

Bug 2: Layers 3 and 4 only checked 'equal' words

  • difflib.SequenceMatcher tags word pairs as equal, replace, insert, or delete.
  • The suffix and umlaut layers only processed equal pairs, missing substituted words.
  • Example: 'die Häuser sind groß' vs 'die hauser sind gross' — 'häuser' and 'hauser' are a replace pair, so the umlaut loss was never flagged.
  • Fix: process both equal and replace opcodes.

✅ End-to-end verified

I tested the complete setup flow in a clean venv on Python 3.12 (closest available to 3.14):

# Install torch CPU
pip install torch==2.13.0+cpu --index-url https://download.pytorch.org/whl/cpu
# Install requirements — ZERO source builds
pip install -r backend/requirements.txt --prefer-binary

Result:

Successfully installed transformers-4.49.0 tokenizers-0.21.4 ...
  ctranslate2-4.8.2 faster-whisper-1.1.1 ...
  # NO source builds triggered

Test suite:

tests/test_scorer.py::test_perfect_match_a1 PASSED
tests/test_scorer.py::test_extra_character_insertion PASSED
tests/test_scorer.py::test_missing_word_a1 PASSED
tests/test_scorer.py::test_suffix_mismatch_b1 PASSED
tests/test_scorer.py::test_umlaut_loss_b2 PASSED
tests/test_scorer.py::test_level_thresholds PASSED
tests/test_scorer.py::test_layer_activation_per_level PASSED
tests/test_scorer.py::test_word_errors_extracted_for_srs PASSED

============================== 8 passed in 0.10s ===============================

🚀 How to upgrade

cd /home/bif/Desktop/Lab/Local_DE_Coach
git pull origin main
./setup.sh

Since torch==2.13.0+cpu is already installed (and now rapidfuzz, psutil, Pillow too from v0.4.3), setup will only need to upgrade transformers (4.46.3 → 4.49.0) and faster-whisper (1.1.0 → 1.1.1). Both use prebuilt wheels — no source builds.

After setup:

./start.sh
# Open http://127.0.0.1:8766 → System page → Start backend

Full changelog: see git log v0.4.3..v0.4.4

Assets 2
Loading