Skip to content

v0.5.0 — spaCy, language auto-detect, mypy CI

Choose a tag to compare

@Mrrobi Mrrobi released this 25 May 00:44
b469afc

Highlights

  • Optional spaCy preprocessing — `PreprocessConfig(spacy_model="en_core_web_sm")` switches sentence splitting to spaCy when the model is installed; regex fallback runs otherwise (no error). `spacy_strip_pos=True` does POS-aware token removal that keeps named entities verbatim.
  • Language auto-detection — pass `source_lang="auto"` and the detected ISO code lands in `result.stats.resolved_lang`. Uses `langdetect` when installed (new `lang` extra), otherwise a stopword-overlap heuristic over the 12 bundled languages.
  • mypy strict-ish + CI job — `[tool.mypy]` enforces typed defs across the public surface; SDK-bridge modules are excluded via override. New `typecheck` job in `.github/workflows/ci.yml` runs mypy on every push/PR.
  • New docs pages — Language detection, spaCy integration. MkDocs nav and API reference updated.

Install

```bash
pip install narratoflow==0.5.0

optional extras:

pip install "narratoflow[nlp]" # spaCy
pip install "narratoflow[lang]" # langdetect
```

Example

```python
from narrato import Compressor

c = Compressor.from_profile("rag-en", provider="anthropic")
c.source_lang = "auto" # detect at runtime

result = c.compress(any_language_document)
print(result.stats["resolved_lang"]) # 'en' / 'de' / 'fi' / ...
```

Tests

63 passing + 1 skipped (spaCy-only). Was 55. New `tests/test_language.py`, `tests/test_spacy.py`.

Deferred

Learned-encoder R&D — genuine research, not worth half-shipping. Moved to v0.6.

Full notes: see CHANGELOG.md.