Skip to content

v1.14.0 - Security fixes and Lemmatizer efficiency updates

Latest
Latest

Choose a tag to compare

View all tags
@AngledLuffa AngledLuffa released this 15 Jul 01:53
v1.14.0
1f4bfdd

Stanza v1.14.0 Release Notes

Security Fixes

  • Fix a potential zip slip vulnerability when extracting downloaded model archives. While low-risk given that Stanza controls the resources being downloaded, extraction now validates that no file paths escape the target directory. See GHSA-2fwf-f686-7p34. #1621

  • Restrict the unpickler used when deserializing annotated Documents, and add a deprecation warning: in a future release, Document serialization will move to JSON entirely, removing the pickle dependency. See GHSA-487q-m798-cp85. #1626

  • Remove shell subprocess calls from make_lm_data.py, addressing GHSA-c9h2-qmqw-qf6h. As a side benefit, the charlm data preparation script is now fully portable to Windows. #1623

Bugfixes

  • Fix a bug where coreference chain annotations were written under the ner= MISC key instead of coref_chains= when serializing a Document to CoNLL-U, causing collision with real NER labels on the same token. Thank you @devteamaegis! #1628

New / Updated Models

  • Substantially updated Slovenian models: the new default sl_combined package mixes the SSJ and SST treebanks (reported by Kaja Dobrovoljc to be highly compatible), augments lemma and POS training with SUK 1.1 data, builds a lemma dictionary from Sloleks 3.1, and adds contextual lemma classifiers for the ambiguous pairs del/delo and rok/roka. #1625

Lemmatizer Improvements

  • Reorganize the lemmatizer dictionary to use a pos → word → lemma layout and store it gzip-compressed. This dramatically reduces load time for large models — Slovenian drops from 30+ seconds to under 5 seconds — and shrinks model sizes considerably. A conversion script for updating locally trained 1.13.0 models is included.

    Note: lemmatizer models from v1.13.0 are not compatible with v1.14.0. Please re-download or convert existing models. #1627

  • Reduce the hidden dimension of the contextual lemma classifier, making models smaller and faster without hurting accuracy. #1629

Dependency Parser

  • Post-process dependency parses to enforce uniqueness constraints on nsubj/csubj and obj relations: if the graph parser produces a node with multiple subjects or direct objects, the parser now reruns Chu-Liu-Edmonds iteratively (reusing the original neural scores) to find the best-scoring repair. This is on by default in the Pipeline. Addresses #1340. #1638

Tokenizer

  • Move comma-transposition augmentations from the data preparation script into the DataLoader, so that augmentation is applied on-the-fly during training rather than being baked in once at preprocessing time. This produces more balanced training and avoids accidentally affecting other annotators' data files. #1624

  • Add new structural feature functions to the tokenizer to help distinguish address-line formatting from normal running text, laying groundwork for fixing sentence-splitting errors on non-prose inputs. Addresses #1640. #1642

Interface Improvements

  • Add a stanza.utils.list_installed script that lists all locally cached Stanza models and their versions, without modifying anything on disk. Addresses #1542. #1632

  • Add a tokenize_with_speakers() convenience function for processing transcript-style text where each line begins with a speaker label, automatically assigning speaker metadata to sentences before passing them to the coref annotator. #1631

CharLM Training Infrastructure

For researchers building character language models for new languages, this release includes updated tooling for collecting and deduplicating training data from OSCAR. The previous OSCAR 2023 source is no longer accessible to new users and is broken with datasets >= 4.0; the new scripts target the OSCAR Community Crawl instead. Addresses #1622.

  • Add a download script for the OSCAR Community Crawl that bypasses load_dataset (which has a known bug with OSCAR), along with an inventory script to inspect the language breakdown of downloaded chunks. Also adds OSCAR language codes to constant.py and fixes a bug where extra language name aliases were being silently clobbered. #1633

  • Switch the near-deduplication strategy from TLSH to MinHash LSH. MinHash is faster, retains more content, and still achieves satisfactory deduplication rates as verified by the diagnostic script included in this PR. #1639

Contributors

Assets 2
Loading