1.0.7 — Colombia punctuation, refined

Colombia has punctuation. Its rows are auto-generated YouTube captions — each a whole speech as one run of words, with no punctuation at all — and this restores it, with capitals. Nothing else differs from 1.0.5.

1.0.7 supersedes 1.0.6, published the same day with the same method. The difference is which phrases a mark may not split. 1.0.6 kept 55 connectives whole and wrote "Estados, Unidos" 385 times, "Naciones, Unidas" 182 and "a, veces" 73; 1.0.7 keeps 66 fixed phrases whole, and each one is checked against the release's own human punctuation (below). Use 1.0.7.

What changed since 1.0.5

Colombia's text is punctuated and cased

In 1.0.5, the opening of Duque's first row:

alcalde yo quiero hoy aprovechar porque hace hace unos días estábamos cumpliendo un año de gobierno para hacer un repaso de cosas trascendentales que se han hecho en tan solo un año y también hacer referencia a las que hemos hecho en bogotá en tan solo un año y voy a empezar por lo siguiente sin duda el bien público más preciado por una sociedad es la seguridad la seguridad es fundamental

In 1.0.7:

Alcalde, yo quiero hoy aprovechar, porque hace hace unos días estábamos cumpliendo un año de Gobierno, para hacer un repaso de cosas trascendentales que se han hecho en tan solo un año y también hacer referencia a las que hemos hecho en Bogotá en tan solo un año. Y voy a empezar por lo siguiente, sin duda, el bien público más preciado por una sociedad es la seguridad. La seguridad es fundamental

The repeated "hace hace" is the caption's, and stays.

Across Colombia's 2,018 rows and 4.42 million words, ecd_punctuate.py added:

mark	added
. full stop	170,971
, comma	372,343
¿ … ? question	2,119

The words are the captions' own. Punctuation is restored with token classifiers, which label the words they are given and have no way to add, drop, reorder or respell one. Every row was then checked against 1.0.5, independently of the code that produced it: same whitespace, same words up to case, at most a ¿ before a word and a . , or ? after it. All 2,018 pass. Recognition errors in the captions are still there; they are the source's, and correcting them is not an edit a dataset can make. [Aplausos] and [Risas] are untouched, and every column other than text is identical to 1.0.5.

The punctuation is a model's, not a transcriber's. Scored against human punctuation from this release — Mexico's stenographic press-conference transcripts and Argentina's speeches, stripped and restored:

F1	Mexico	Argentina
F1	Mexico	Argentina
---	---:	---:
comma	0.68	0.58
full stop	0.65	0.57
sentence boundary	0.66	0.58
capital letter	0.80	0.73
question	0.30	0.28

Those are clean transcripts, so for captions these are an upper bound. About nine in ten of its sentence breaks fall where the transcriber put a full stop or a comma; sentences run somewhat shorter than a stenographer would write them. Few questions are marked — about one in five, more than three in four of them correctly — because marking more would turn statements into questions.

Which phrases stay whole is checked, not assumed. kredor splits connectives — it marks a parenthetical by putting a comma after every word of it, "y, por, lo, tanto," — and names whose words are also ordinary words. No mark is written inside 66 fixed phrases, and ecd_punctuate.py --review-phrases holds each to the release: at least 100 uses in its human-punctuated Spanish, and split by transcribers in fewer than 1 in 100. Across 274,828 uses in ten countries, they split one 49 times. eso sí, más aún and por lo demás failed and are not kept whole.

Two models, both pinned: kredor/punctuate-all places . , ?, and 1-800-BAD-CODE/punctuation_fullstop_truecase_romance cases the words and opens questions with ¿, conditioned on kredor's punctuation so the capitals follow the marks that are written. The data-validation README has the reasoning and the full evaluation.

Validation

ecd_validate.py reports the same 125 findings as for 1.0.5, finding for finding: 0 CRITICAL, 0 ERROR, 96 WARN, 29 INFO.

full_ecd.parquet is rebuilt from the country assets and reconciles row for row: 2,891,622 rows, 42 countries. The 41 other country files are byte-identical to 1.0.5.

Using it

install.packages('ecdata', repos = 'https://executive-communications-dataset.r-universe.dev')
load_ecd(country = 'Colombia')                     # 1.0.7 is the default from ecdata 1.5.0
load_ecd(country = 'Colombia', unit = 'sentence')

ec.load_ecd(country='Colombia')                    # 1.0.7 is the default from ecdata 1.5.0

The sentence view of this release is 1.0.7-sentences, where Colombia is now segmented. 1.0.0 through 1.0.6 remain published and unchanged; 1.0.5 is the unpunctuated captions.