1.0.6-sentences — sentence-level view

Superseded by 1.0.7-sentences, published the same day: it keeps names such as "Estados Unidos" whole, where this release splits them. The assets below are unchanged.

A sentence-level view of release 1.0.6. 9,378,448 sentences.

It is 1.0.5-sentences with one file changed: Colombia is now segmented. In 1.0.5 its rows were unpunctuated YouTube captions and passed through whole; 1.0.6 restores their punctuation, and its 2,018 rows become 173,215 sentences, median 120 characters.

Every Colombia sentence is marked segmenter = "pysbd/es on restored punctuation", because the boundaries rest on a model's punctuation rather than a transcriber's. How that punctuation scores against human punctuation is in the 1.0.6 notes. A few passages where the model chose commas throughout run on: 45 sentences are over 1,000 characters, 0.3% of the text.

The other 41 country files are byte-identical to 1.0.5-sentences. Total non-whitespace character mass in Colombia is identical to 1.0.6, and all 42 files share one schema.

load_ecd(country = 'Colombia', ecd_version = '1.0.6', unit = 'sentence')

Built with ecd_sentences.py.