1.0.7-sentences — sentence-level view

A sentence-level view of release 1.0.7. 9,377,931 sentences.

It is 1.0.5-sentences with one file changed: Colombia is now segmented. In 1.0.5 its rows were unpunctuated YouTube captions and passed through whole; 1.0.7 restores their punctuation, and its 2,018 rows become 172,698 sentences, median 121 characters. It supersedes 1.0.6-sentences, whose punctuation split names such as "Estados, Unidos".

Every Colombia sentence is marked segmenter = "pysbd/es on restored punctuation", because the boundaries rest on a model's punctuation rather than a transcriber's. How that punctuation scores against human punctuation is in the 1.0.7 notes. A few passages where the model chose commas throughout run on: 45 sentences are over 1,000 characters, 0.3% of the text.

The other 41 country files are byte-identical to 1.0.5-sentences. Total non-whitespace character mass in Colombia is identical to 1.0.7, and all 42 files share one schema.

load_ecd(country = 'Colombia', unit = 'sentence')   # 1.0.7 is the default from ecdata 1.5.0

Built with ecd_sentences.py.

1.0.7 — Colombia punctuation, refined

Colombia has punctuation. Its rows are auto-generated YouTube captions — each a whole speech as one run of words, with no punctuation at all — and this restores it, with capitals. Nothing else differs from 1.0.5.

1.0.7 supersedes 1.0.6, published the same day with the same method. The difference is which phrases a mark may not split. 1.0.6 kept 55 connectives whole and wrote "Estados, Unidos" 385 times, "Naciones, Unidas" 182 and "a, veces" 73; 1.0.7 keeps 66 fixed phrases whole, and each one is checked against the release's own human punctuation (below). Use 1.0.7.

What changed since 1.0.5

Colombia's text is punctuated and cased

In 1.0.5, the opening of Duque's first row:

alcalde yo quiero hoy aprovechar porque hace hace unos días estábamos cumpliendo un año de gobierno para hacer un repaso de cosas trascendentales que se han hecho en tan solo un año y también hacer referencia a las que hemos hecho en bogotá en tan solo un año y voy a empezar por lo siguiente sin duda el bien público más preciado por una sociedad es la seguridad la seguridad es fundamental

In 1.0.7:

Alcalde, yo quiero hoy aprovechar, porque hace hace unos días estábamos cumpliendo un año de Gobierno, para hacer un repaso de cosas trascendentales que se han hecho en tan solo un año y también hacer referencia a las que hemos hecho en Bogotá en tan solo un año. Y voy a empezar por lo siguiente, sin duda, el bien público más preciado por una sociedad es la seguridad. La seguridad es fundamental

The repeated "hace hace" is the caption's, and stays.

Across Colombia's 2,018 rows and 4.42 million words, ecd_punctuate.py added:

mark	added
. full stop	170,971
, comma	372,343
¿ … ? question	2,119

The words are the captions' own. Punctuation is restored with token classifiers, which label the words they are given and have no way to add, drop, reorder or respell one. Every row was then checked against 1.0.5, independently of the code that produced it: same whitespace, same words up to case, at most a ¿ before a word and a . , or ? after it. All 2,018 pass. Recognition errors in the captions are still there; they are the source's, and correcting them is not an edit a dataset can make. [Aplausos] and [Risas] are untouched, and every column other than text is identical to 1.0.5.

The punctuation is a model's, not a transcriber's. Scored against human punctuation from this release — Mexico's stenographic press-conference transcripts and Argentina's speeches, stripped and restored:

F1	Mexico	Argentina
F1	Mexico	Argentina
---	---:	---:
comma	0.68	0.58
full stop	0.65	0.57
sentence boundary	0.66	0.58
capital letter	0.80	0.73
question	0.30	0.28

Those are clean transcripts, so for captions these are an upper bound. About nine in ten of its sentence breaks fall where the transcriber put a full stop or a comma; sentences run somewhat shorter than a stenographer would write them. Few questions are marked — about one in five, more than three in four of them correctly — because marking more would turn statements into questions.

Which phrases stay whole is checked, not assumed. kredor splits connectives — it marks a parenthetical by putting a comma after every word of it, "y, por, lo, tanto," — and names whose words are also ordinary words. No mark is written inside 66 fixed phrases, and ecd_punctuate.py --review-phrases holds each to the release: at least 100 uses in its human-punctuated Spanish, and split by transcribers in fewer than 1 in 100. Across 274,828 uses in ten countries, they split one 49 times. eso sí, más aún and por lo demás failed and are not kept whole.

Two models, both pinned: kredor/punctuate-all places . , ?, and 1-800-BAD-CODE/punctuation_fullstop_truecase_romance cases the words and opens questions with ¿, conditioned on kredor's punctuation so the capitals follow the marks that are written. The data-validation README has the reasoning and the full evaluation.

Validation

ecd_validate.py reports the same 125 findings as for 1.0.5, finding for finding: 0 CRITICAL, 0 ERROR, 96 WARN, 29 INFO.

full_ecd.parquet is rebuilt from the country assets and reconciles row for row: 2,891,622 rows, 42 countries. The 41 other country files are byte-identical to 1.0.5.

Using it

install.packages('ecdata', repos = 'https://executive-communications-dataset.r-universe.dev')
load_ecd(country = 'Colombia')                     # 1.0.7 is the default from ecdata 1.5.0
load_ecd(country = 'Colombia', unit = 'sentence')

ec.load_ecd(country='Colombia')                    # 1.0.7 is the default from ecdata 1.5.0

The sentence view of this release is 1.0.7-sentences, where Colombia is now segmented. 1.0.0 through 1.0.6 remain published and unchanged; 1.0.5 is the unpunctuated captions.

1.0.6-sentences — sentence-level view

Superseded by 1.0.7-sentences, published the same day: it keeps names such as "Estados Unidos" whole, where this release splits them. The assets below are unchanged.

A sentence-level view of release 1.0.6. 9,378,448 sentences.

It is 1.0.5-sentences with one file changed: Colombia is now segmented. In 1.0.5 its rows were unpunctuated YouTube captions and passed through whole; 1.0.6 restores their punctuation, and its 2,018 rows become 173,215 sentences, median 120 characters.

Every Colombia sentence is marked segmenter = "pysbd/es on restored punctuation", because the boundaries rest on a model's punctuation rather than a transcriber's. How that punctuation scores against human punctuation is in the 1.0.6 notes. A few passages where the model chose commas throughout run on: 45 sentences are over 1,000 characters, 0.3% of the text.

The other 41 country files are byte-identical to 1.0.5-sentences. Total non-whitespace character mass in Colombia is identical to 1.0.6, and all 42 files share one schema.

load_ecd(country = 'Colombia', ecd_version = '1.0.6', unit = 'sentence')

Built with ecd_sentences.py.

1.0.6 — Colombia punctuation

Superseded by 1.0.7, published the same day: it keeps names such as "Estados Unidos" whole, where this release splits them. The assets below are unchanged.

Colombia has punctuation. Its rows are auto-generated YouTube captions — each a whole speech as one run of words, with no punctuation at all — and 1.0.6 restores it, with capitals. Nothing else differs from 1.0.5.

What changed since 1.0.5

Colombia's text is punctuated and cased

In 1.0.5, the opening of Duque's first row:

alcalde yo quiero hoy aprovechar porque hace hace unos días estábamos cumpliendo un año de gobierno para hacer un repaso de cosas trascendentales que se han hecho en tan solo un año y también hacer referencia a las que hemos hecho en bogotá en tan solo un año y voy a empezar por lo siguiente sin duda el bien público más preciado por una sociedad es la seguridad la seguridad es fundamental

In 1.0.6:

Alcalde, yo quiero hoy aprovechar, porque hace hace unos días estábamos cumpliendo un año de Gobierno, para hacer un repaso de cosas trascendentales que se han hecho en tan solo un año y también hacer referencia a las que hemos hecho en Bogotá en tan solo un año. Y voy a empezar por lo siguiente, sin duda, el bien público más preciado por una sociedad es la seguridad. La seguridad es fundamental

The repeated "hace hace" is the caption's, and stays.

Across Colombia's 2,018 rows and 4.42 million words, ecd_punctuate.py added:

mark	added
. full stop	171,408
, comma	373,394
¿ … ? question	2,226

The words are the captions' own. Punctuation is restored with token classifiers, which label the words they are given and have no way to add, drop, reorder or respell one. Every row was then checked against 1.0.5, independently of the code that produced it: same whitespace, same words up to case, at most a ¿ before a word and a . , or ? after it. All 2,018 pass. Recognition errors in the captions are still there; they are the source's, and correcting them is not an edit a dataset can make. [Aplausos] and [Risas] are untouched, and every column other than text is identical to 1.0.5.

The punctuation is a model's, not a transcriber's. Scored against human punctuation from this release — Mexico's stenographic press-conference transcripts and Argentina's speeches, stripped and restored:

F1	Mexico	Argentina
comma	0.68	0.58
full stop	0.65	0.57
sentence boundary	0.65	0.58
capital letter	0.80	0.73
question	0.30	0.28

Those are clean transcripts, so for captions these are an upper bound. About nine in ten of its sentence breaks fall where the transcriber put a full stop or a comma; sentences run somewhat shorter than a stenographer would write them. Few questions are marked — about one in five, three in four of them correctly — because marking more would turn statements into questions.

Two models, both pinned: kredor/punctuate-all places . , ?, and 1-800-BAD-CODE/punctuation_fullstop_truecase_romance cases the words and opens questions with ¿, conditioned on kredor's punctuation so the capitals follow the marks that are written. The data-validation README has the reasoning and the full evaluation.

Validation

ecd_validate.py reports the same 125 findings as for 1.0.5, finding for finding: 0 CRITICAL, 0 ERROR, 96 WARN, 29 INFO.

full_ecd.parquet is rebuilt from the country assets and reconciles row for row: 2,891,622 rows, 42 countries. The 41 other country files are byte-identical to 1.0.5.

Using it

install.packages('ecdata', repos = 'https://executive-communications-dataset.r-universe.dev')
load_ecd(country = 'Colombia', ecd_version = '1.0.6')
load_ecd(country = 'Colombia', ecd_version = '1.0.6', unit = 'sentence')

The sentence view of this release is 1.0.6-sentences, where Colombia is now segmented. 1.0.0 through 1.0.5 remain published and unchanged; 1.0.5 is the unpunctuated captions.

1.0.5-sentences — sentence-level view

A sentence-level view of release 1.0.5. One row per sentence: 2,891,622 rows become 9,207,251 sentences, 3.2 per row.

This does not replace 1.0.5. That release is unchanged and remains the document-level data; this is a parallel view for work that needs one unit meaning the same thing in every country.

Why it exists

The unit of observation differs by country (#15). A row is a whole document in Brazil, a paragraph in Spain, a sentence in Canada, an HTML block in Czechia. Any cross-country count of rows therefore measures the scraper as much as the executive. Sentences give a unit that is comparable across all 42.

Schema

Every column from 1.0.5, plus four:

column	meaning
document_id	stable id for the source document, derived from its url
block_index	which original row within that document the sentence came from — the release's own grain, preserved rather than discarded
sentence_index	position of the sentence within the document
segmenter	which rules split it

text holds the sentence. Joining back to 1.0.5 is document_id, or url directly.

Segmentation

pysbd has rules for 11 of the 22 languages in the release: Spanish, German, French, Italian, Greek, Danish, Polish, Japanese, Chinese, Hindi, English. The other 11 use the English rules.

That fallback was tested against real rows rather than assumed. In Czech, Portuguese, Turkish, Korean, Hungarian, Hebrew, Indonesian, Icelandic, Norwegian and Georgian, fewer than 4% of long passages collapse to a single sentence. What it loses is language-specific abbreviation handling, so an abbreviation followed by a capital can split early. The segmenter column records which case each row was, so this is visible in the data rather than only in these notes.

Where the output is poor

Sentence splitting cannot improve on what it is given.

• Colombia is not segmented at all. Its rows are auto-generated YouTube captions with no punctuation and no capitalisation — 198 of 198 long rows yield a single "sentence". It passes through whole, marked segmenter = "none (unpunctuated source)", rather than being split wrongly.
• Czechia (47% of sentences under 20 characters), Republic of Korea (25%), Japan (20%) and Hong Kong (18%) are extracted block by block, so headings, captions and signature lines become "sentences". That is the source grain showing through.
• Italy is OCR'd PDF text with a running page header, so some sentences are page furniture and some carry OCR noise.
• Brazil: 56% of its rows carry no url, so 227,464 sentences have a null document_id. That is a property of 1.0.5, not of this view.

Verification

Total non-whitespace character mass is identical before and after for all 42 countries — nothing lost, duplicated or reordered.

73 documents across Portugal (52), Ecuador (12), the Dominican Republic (8) and Israel (1) do not appear, because every row of those documents has empty text.

Using it

Built with ecd_sentences.py. Until the packages expose it directly, download an asset and read it:

sentences <- arrow::read_parquet("chile.parquet")

import polars as pl
sentences = pl.read_parquet("chile.parquet")

1.0.5 — clean validation

The first release that validates clean: 0 CRITICAL, 0 ERROR. Release 1.0.0 had 19 and 55.

If you are coming from 1.0.0, read the 1.0.1 notes first — everything they describe applies here, in particular that row counts are 82.8% lower because that is how many rows were exact duplicates.

What changed since 1.0.4

type is usable in the US file

type held a president's name rather than a document category on 486,000 rows — about a third of the file, two fields crossed at the source. The American Presidency Project encodes the category in its url slug, so the correct value was recoverable from the file itself: slug prefix to type, learned from the documents that were correctly typed.

	rows
rebuilt from the source url	468,321
no reliable prefix, set to null	17,243
president names remaining	0

Measured on a 70/30 split of the correctly-typed documents, the mapping is 99.75% accurate on 10,013 held-out documents. Documents whose prefix did not clear that bar are null rather than guessed at.

type is still not a controlled vocabulary shared across countries (#20 stays open for that), but within the US file it now means what it says.

Colombia and Russia are documented rather than flagged

Neither was ever a fault, and both now carry the reason in the validation report and in COVERAGE.md:

• Colombia — the corpus is transcripts of YouTube videos published by the presidency. The url points at the video the transcript came from. That is the source, not a scraping fault.
• Russia — the rows come from a pre-existing dataset rather than a scrape of kremlin.ru, so the fields a scrape would have filled are absent: url and type are empty for every row, and the text is the English-language edition rather than the Russian original.

Nothing else differs from 1.0.4: 2,891,622 rows, 42 countries, full_ecd.parquet reconciling exactly.

What is left

The validator reports no errors, which means every defect that could be derived from the published files has been. What remains needs the source data back, or is a property of the corpus rather than a bug:

• #15 — the unit of observation is a paragraph in some countries and a whole document in others. Compare documents, not rows; COVERAGE.md gives per-country counts of both.
• #21 — coverage is very uneven. Ten countries hold fewer than 500 documents and three fewer than 100, against a median of 1,648 and a United States holding 60,404.
• #19 — file and office are empty everywhere. They need a decision about what they are for.
• #16 — 100 Venezuela rows could not be re-decoded; a byte was lost at ingest.
• #20 — type has no shared vocabulary across countries.

The scrapers that produced the duplication were fixed at source in #30 and #31, so a future build should not reintroduce it. Those fixes have not been re-run, so they are unverified until someone scrapes again.

Using it

install.packages('ecdata', repos = 'https://executive-communications-dataset.r-universe.dev')
load_ecd(country = 'United States of America', ecd_version = '1.0.5')

ec.load_ecd(country='United States of America', ecd_version='1.0.5')

Built with ecd_repair.py from the 1.0.0 assets. 1.0.0 through 1.0.4 remain published and unchanged.

1.0.4 — date corrections

1.0.3 plus three corrected dates. This is the first release with no remaining executive findings.

If you are coming from 1.0.0, read the 1.0.1 notes first — everything they describe applies here, in particular that row counts are 82.8% lower because that is how many rows were exact duplicates.

What changed since 1.0.3

Three documents carried the wrong year. Each was credited to an executive who had not yet taken office, and in each case checking the document showed the executive was right and the date was wrong:

country	was	now	what the document says
Chile	2022-01-07	2023-01-07	Marcela Ríos' resignation as Justice Minister and Luis Cordero's appointment — January 2023
Brazil	2016-02-07	2017-02-07	Lunch for President Macri at Itamaraty; his first official visit to Brasília was February 2017
Brazil	2016-01-24	2017-01-24	MCTI advisory council with minister Gilberto Kassab; that ministry was created in May 2016

year_of_statement moves with the date. Michel Temer's rows now begin 2016-05-12 and Gabriel Boric's 2022-03-11 — their actual inaugurations.

Nothing else differs from 1.0.3: 2,891,622 rows, 42 countries, full_ecd.parquet reconciling exactly.

Where executive now stands

Every finding against it is resolved:

• the Obama/Trump handover, corrected in 1.0.3 — 23,255 rows returned to Obama
• Italy's composite values, split in 1.0.3 — no executive anywhere packs two people
• nine misspelled names, corrected in 1.0.2
• the three dates above

The fifteen "overlapping terms" reported against 1.0.0 are also accounted for: twelve were an artefact of the validator comparing date spans rather than counting documents, and Gerald R. Ford's rows running to 1996 are 14 convention speeches, correctly attributed. See #13.

Still broken

ecd_validate.py reports 0 CRITICAL, 3 ERROR, against 19 CRITICAL and 55 ERROR in 1.0.0. All three need the source data back or a decision about what a column means, and none can be repaired from the published files:

• #18 — Colombia is sourced entirely from YouTube.
• #20 — type holds president names in the US file.
• #19/#22 — url is null for Russia, whose text is the English kremlin.ru edition rather than the Russian original. file and office remain empty everywhere.

Also unchanged: the paragraph-vs-document grain (#15), uneven coverage (#21, see COVERAGE.md), and 100 Venezuela rows that could not be re-decoded because a byte was lost at ingest (#16).

The scrapers that produced the duplication have now been fixed at source (#30, #31), so a future build should not reintroduce it.

Using it

install.packages('ecdata', repos = 'https://executive-communications-dataset.r-universe.dev')
load_ecd(country = 'United States of America', ecd_version = '1.0.4')

ec.load_ecd(country='United States of America', ecd_version='1.0.4')

Built with ecd_repair.py from the 1.0.0 assets. 1.0.0 through 1.0.3 remain published and unchanged.

1.0.3 — executive corrections

1.0.2 plus two corrections to executive, the column the audit called least trustworthy.

If you are coming from 1.0.0, read the 1.0.1 notes first — everything they describe applies here. In particular row counts are 82.8% lower than 1.0.0, because that is how many rows were exact duplicates.

What changed since 1.0.2

The Obama/Trump handover was off by a year

1.0.2   Barack Obama      156,496   2009-01-20 .. 2016-01-19
        Donald J. Trump   200,688   2016-01-20 .. 2021-01-20

1.0.3   Barack Obama      179,751   2009-01-20 .. 2017-01-19
        Donald J. Trump   177,433   2017-01-20 .. 2021-01-20

23,255 rows dated inside Obama's final year were credited to Trump — 11.6% of Trump's rows. Anything grouping US documents by executive over 2016 was wrong, in both directions. Every other US handover in the file was already correct to the day, which is what made this a single-boundary fix.

Italy's composite executives are split

Romano Prodi/Massimo D'Alema (1,956 rows) and Massimo D'Alema/Giuliano Amato (536) packed two people into one string, so they grouped and joined as neither. A further 593 rows credited to Giuliano Amato predated his term.

Romano Prodi         2,058   1996-05-18 .. 2008-04-24
Massimo D'Alema        930   1998-10-23 .. 2000-04-19
Giuliano Amato       1,738   2000-04-26 .. 2001-06-08

No composite executive values remain anywhere in the release.

Nothing else differs from 1.0.2: 2,891,622 rows, 42 countries, full_ecd.parquet reconciling exactly.

A correction to what we said about 1.0.2

The 1.0.2 notes said executive terms overlapped in 15 places and that Gerald R. Ford's rows ran to 1996. Both came from the original audit and both were wrong or overstated.

Twelve of the fifteen were a bug in the validator, which compared each executive's first and last date rather than counting documents. Any leader with two non-contiguous terms — Lars Løkke Rasmussen, Netanyahu, Putin — appeared to overlap a successor they never overlapped. In Denmark, Israel, Russia, Austria, Greece and Italy the first executive holds zero documents in the window supposedly shared.

Ford's rows running to 1996 is not a defect. It is 14 rows, one URL, type = 'Convention Speeches': Ford addressing the 1996 Republican National Convention. A further 33 Ford rows inside Carter's term are all dated 1977-01-20, Carter's inauguration, when both men legitimately published.

The validator now counts documents inside the window, and distinguishes alternating terms and handover days from real mis-attribution. Full correction in #13.

Still broken

ecd_validate.py reports 0 CRITICAL, 5 ERROR, against 19 CRITICAL and 55 ERROR in 1.0.0.

• #18 — Colombia is sourced entirely from YouTube.
• #19 — file and office are empty everywhere; url is null for Russia.
• #20 — type holds president names in the US file.
• #13 — six rows across Brazil and Chile are dated before their executive took office.
• #15 — the unit of observation is still a paragraph in some countries and a whole document in others.
• #21 — coverage is very uneven; ten countries hold fewer than 500 documents. See COVERAGE.md.
• #22 — Russia is the English kremlin.ru edition, not the Russian originals.
• #16 — 100 Venezuela rows could not be re-decoded; a byte was lost at ingest.
• #30, #31 — the duplication and the malformed Jamaica urls are repaired in these assets but not in the pipeline that builds them, so both will recur on the next build.

Using it

Until the packages are re-published, ask for this version explicitly:

load_ecd(country = 'United States of America', ecd_version = '1.0.3')

ec.load_ecd(country='United States of America', ecd_version='1.0.3')

Built with ecd_repair.py from the 1.0.0 assets. 1.0.0, 1.0.1 and 1.0.2 remain published and unchanged.

1.0.2 — repair release

1.0.1 plus one fix: language is now populated everywhere.

If you are on 1.0.0, read the 1.0.1 notes as well — everything they describe applies here. In particular row counts are 82.8% lower than 1.0.0, because that is how many rows were exact duplicates, and Ecuador and the Dominican Republic are far smaller than 1.0.0 implied.

What changed since 1.0.1

language was null for every row of portugal.parquet and united_states_of_america.parquet — 1,423,119 rows, 49% of the release. Both are now filled from the country dictionary the packages already ship, which has always said Portuguese and English. load_ecd(language = 'English') no longer silently omits the United States.

language is populated on all 2,891,622 rows. Nothing else differs from 1.0.1 — same rows, same repairs, byte-identical apart from that column.

India is untouched and still correctly bilingual: 51,559 Hindi rows and 31,123 English, labelled per row.

Still broken

ecd_validate.py reports 0 CRITICAL, 18 ERROR, down from 19 CRITICAL and 55 ERROR in 1.0.0 and 20 ERROR in 1.0.1. What remains needs the source data back or a decision about what a column means:

• #13 — the Obama/Trump handover is dated 2016-01-20 rather than 2017-01-20, so 23,255 rows dated inside Obama's final year are credited to Trump. Italy carries composite values (Massimo D'Alema/Giuliano Amato) that will not group or join, and six rows across Brazil, Chile and Italy are dated before their executive took office.

  Corrected after publication: these notes originally said executive terms overlap in 15 places and that Ford's rows run to 1996. Twelve of those fifteen were an artefact of the validator comparing date spans rather than counting documents — no document is double-attributed in Denmark, Israel, Russia, Austria, Greece or Italy/Berlusconi (fixed in #28). Ford's post-tenure rows are 14 convention speeches from 1996, correctly attributed. See #13 for the full correction.

• #15 — the unit of observation is still a paragraph in some countries and a whole document in others.

• #18 — Colombia is still sourced entirely from YouTube.

• #19 — file and office are still empty everywhere, and url is still null for Russia.

• #20 — type still holds president names in the US file.

• #21 — coverage is still very uneven: ten countries hold fewer than 500 documents, three fewer than 100.

• #22 — Russia is still the English kremlin.ru edition, not the Russian originals.

• #16 — 100 Venezuela rows could not be repaired: a byte was lost at ingest, so the text does not round-trip.

Using it

Until the packages are re-published, ask for this version explicitly:

load_ecd(country = 'India', ecd_version = '1.0.2')

ec.load_ecd(country='India', ecd_version='1.0.2')

Built with data-validation/ecd_repair.py from the 1.0.0 assets. 1.0.0 and 1.0.1 remain published and unchanged.

1.0.1 — repair release

A repair release. The corpus is unchanged in substance — no new documents, no re-scrape.
Every fix here was derived from the 1.0.0 assets themselves, using
data-validation/ecd_repair.py.

1.0.0 is left exactly as it was, so anything that cites it still reproduces.

Read this before you upgrade

Row counts drop by 82.8%, from 16,845,134 to 2,891,622. That is not data loss:
those rows were exact duplicates produced by a many-to-many join in the build. India
goes from 7,970,491 rows to 82,682; Denmark from 4,801,705 to 49,595. If your analysis
counted rows, its numbers will change, and the 1.0.1 numbers are the correct ones.

Ecuador and the Dominican Republic are much smaller than 1.0.0 implied. The two
files shared one pooled corpus under swapped labels. Split by source domain, Ecuador is
20,619 rows / 2,236 documents covering 2023-11-23 to 2024-03-19 (Daniel Noboa only), and
the Dominican Republic is 187,932 rows ending 2020-08-16 (Danilo Medina, plus the
handover day). The apparent 2012–2024 span for both was an artefact of the pooling.

Fixed

#10	Ecuador and the Dominican Republic no longer share a corpus; country comes from the source domain and executive from the date
#11	full_ecd.parquet rebuilt from the country assets: Portugal is 64,068 rows rather than one empty row, Ecuador appears once rather than twice, and every country reconciles exactly
#12	Exact duplicates on (country, url, text, date) removed
#13	Nine misspelled executive names corrected (Jooseph R. Biden, Jr., Dmitry Medvedec, Kyriakos Misotakis, Ioannis Samaras, Poul Nyrup Rasumussen, Christina Fernandez de Kirchner, Iralki Garibashvili, Jose Luis Rodriquez Zapatero, Andrew Manuel Lopez Obrador), and Brazil's language value (Portugese → Portuguese)
#14	One schema across all 42 assets: the 17 documented columns, date as Datetime(us, UTC). Portugal's urls and the Dominican Republic's subject are restored to url and title, so both are populated rather than null
#16	Venezuela's mis-decoded text repaired in 17,703 of 17,803 affected rows
—	611 Brazil rows that carried no country, isonumber, gwc, cowcodes, polity_v, polity_iv, vdem or language at all are now labelled

Still broken

ecd_validate.py reports 0 CRITICAL, 20 ERROR against these assets, down from
19 CRITICAL and 55 ERROR. What remains needs the source data back, or a decision about
what a column means, and none of it can be repaired from the published files:

• #13 — executive terms still overlap in 15 places. Gerald R. Ford's rows still run
  to 1996 and the Obama/Trump handover is still dated 2016-01-20. Correcting these needs
  a reviewed term table per country, not a rename map. executive is still the least
  trustworthy column in the dataset.
• #15 — the unit of observation is still a paragraph in some countries and a whole
  document in others.
• #18 — Colombia is still sourced entirely from YouTube.
• #19 — file and office are still empty everywhere; language is still null for
  the United States; url is still null for Russia.
• #20 — type still holds president names in the US file.
• #21 — per-country coverage is still very uneven: ten countries hold fewer than 500
  documents, three fewer than 100.
• #22 — Russia is still the English-language kremlin.ru edition, not the Russian
  originals.
• #16 — 100 Venezuela rows could not be repaired: a byte was lost at ingest, so the
  text does not round-trip cleanly.

Using it

The ecdata packages still default to 1.0.0. Until that default is bumped and
released, ask for this version explicitly:

load_ecd(country = 'India', ecd_version = '1.0.1')

ec.load_ecd(country='India', ecd_version='1.0.1')