1.0.7-sentences — sentence-level view

A sentence-level view of release 1.0.7. 9,377,931 sentences.

It is 1.0.5-sentences with one file changed: Colombia is now segmented. In 1.0.5 its rows were unpunctuated YouTube captions and passed through whole; 1.0.7 restores their punctuation, and its 2,018 rows become 172,698 sentences, median 121 characters. It supersedes 1.0.6-sentences, whose punctuation split names such as "Estados, Unidos".

Every Colombia sentence is marked segmenter = "pysbd/es on restored punctuation", because the boundaries rest on a model's punctuation rather than a transcriber's. How that punctuation scores against human punctuation is in the 1.0.7 notes. A few passages where the model chose commas throughout run on: 45 sentences are over 1,000 characters, 0.3% of the text.

The other 41 country files are byte-identical to 1.0.5-sentences. Total non-whitespace character mass in Colombia is identical to 1.0.7, and all 42 files share one schema.

load_ecd(country = 'Colombia', unit = 'sentence')   # 1.0.7 is the default from ecdata 1.5.0

Built with ecd_sentences.py.