1.0.6 — Colombia punctuation

Superseded by 1.0.7, published the same day: it keeps names such as "Estados Unidos" whole, where this release splits them. The assets below are unchanged.

Colombia has punctuation. Its rows are auto-generated YouTube captions — each a whole speech as one run of words, with no punctuation at all — and 1.0.6 restores it, with capitals. Nothing else differs from 1.0.5.

What changed since 1.0.5

Colombia's text is punctuated and cased

In 1.0.5, the opening of Duque's first row:

alcalde yo quiero hoy aprovechar porque hace hace unos días estábamos cumpliendo un año de gobierno para hacer un repaso de cosas trascendentales que se han hecho en tan solo un año y también hacer referencia a las que hemos hecho en bogotá en tan solo un año y voy a empezar por lo siguiente sin duda el bien público más preciado por una sociedad es la seguridad la seguridad es fundamental

In 1.0.6:

Alcalde, yo quiero hoy aprovechar, porque hace hace unos días estábamos cumpliendo un año de Gobierno, para hacer un repaso de cosas trascendentales que se han hecho en tan solo un año y también hacer referencia a las que hemos hecho en Bogotá en tan solo un año. Y voy a empezar por lo siguiente, sin duda, el bien público más preciado por una sociedad es la seguridad. La seguridad es fundamental

The repeated "hace hace" is the caption's, and stays.

Across Colombia's 2,018 rows and 4.42 million words, ecd_punctuate.py added:

mark	added
. full stop	171,408
, comma	373,394
¿ … ? question	2,226

The words are the captions' own. Punctuation is restored with token classifiers, which label the words they are given and have no way to add, drop, reorder or respell one. Every row was then checked against 1.0.5, independently of the code that produced it: same whitespace, same words up to case, at most a ¿ before a word and a . , or ? after it. All 2,018 pass. Recognition errors in the captions are still there; they are the source's, and correcting them is not an edit a dataset can make. [Aplausos] and [Risas] are untouched, and every column other than text is identical to 1.0.5.

The punctuation is a model's, not a transcriber's. Scored against human punctuation from this release — Mexico's stenographic press-conference transcripts and Argentina's speeches, stripped and restored:

F1	Mexico	Argentina
comma	0.68	0.58
full stop	0.65	0.57
sentence boundary	0.65	0.58
capital letter	0.80	0.73
question	0.30	0.28

Those are clean transcripts, so for captions these are an upper bound. About nine in ten of its sentence breaks fall where the transcriber put a full stop or a comma; sentences run somewhat shorter than a stenographer would write them. Few questions are marked — about one in five, three in four of them correctly — because marking more would turn statements into questions.

Two models, both pinned: kredor/punctuate-all places . , ?, and 1-800-BAD-CODE/punctuation_fullstop_truecase_romance cases the words and opens questions with ¿, conditioned on kredor's punctuation so the capitals follow the marks that are written. The data-validation README has the reasoning and the full evaluation.

Validation

ecd_validate.py reports the same 125 findings as for 1.0.5, finding for finding: 0 CRITICAL, 0 ERROR, 96 WARN, 29 INFO.

full_ecd.parquet is rebuilt from the country assets and reconciles row for row: 2,891,622 rows, 42 countries. The 41 other country files are byte-identical to 1.0.5.

Using it

install.packages('ecdata', repos = 'https://executive-communications-dataset.r-universe.dev')
load_ecd(country = 'Colombia', ecd_version = '1.0.6')
load_ecd(country = 'Colombia', ecd_version = '1.0.6', unit = 'sentence')

The sentence view of this release is 1.0.6-sentences, where Colombia is now segmented. 1.0.0 through 1.0.5 remain published and unchanged; 1.0.5 is the unpunctuated captions.