1.0.5 — clean validation

The first release that validates clean: 0 CRITICAL, 0 ERROR. Release 1.0.0 had 19 and 55.

If you are coming from 1.0.0, read the 1.0.1 notes first — everything they describe applies here, in particular that row counts are 82.8% lower because that is how many rows were exact duplicates.

What changed since 1.0.4

type is usable in the US file

type held a president's name rather than a document category on 486,000 rows — about a third of the file, two fields crossed at the source. The American Presidency Project encodes the category in its url slug, so the correct value was recoverable from the file itself: slug prefix to type, learned from the documents that were correctly typed.

	rows
rebuilt from the source url	468,321
no reliable prefix, set to null	17,243
president names remaining	0

Measured on a 70/30 split of the correctly-typed documents, the mapping is 99.75% accurate on 10,013 held-out documents. Documents whose prefix did not clear that bar are null rather than guessed at.

type is still not a controlled vocabulary shared across countries (#20 stays open for that), but within the US file it now means what it says.

Colombia and Russia are documented rather than flagged

Neither was ever a fault, and both now carry the reason in the validation report and in COVERAGE.md:

• Colombia — the corpus is transcripts of YouTube videos published by the presidency. The url points at the video the transcript came from. That is the source, not a scraping fault.
• Russia — the rows come from a pre-existing dataset rather than a scrape of kremlin.ru, so the fields a scrape would have filled are absent: url and type are empty for every row, and the text is the English-language edition rather than the Russian original.

Nothing else differs from 1.0.4: 2,891,622 rows, 42 countries, full_ecd.parquet reconciling exactly.

What is left

The validator reports no errors, which means every defect that could be derived from the published files has been. What remains needs the source data back, or is a property of the corpus rather than a bug:

• #15 — the unit of observation is a paragraph in some countries and a whole document in others. Compare documents, not rows; COVERAGE.md gives per-country counts of both.
• #21 — coverage is very uneven. Ten countries hold fewer than 500 documents and three fewer than 100, against a median of 1,648 and a United States holding 60,404.
• #19 — file and office are empty everywhere. They need a decision about what they are for.
• #16 — 100 Venezuela rows could not be re-decoded; a byte was lost at ingest.
• #20 — type has no shared vocabulary across countries.

The scrapers that produced the duplication were fixed at source in #30 and #31, so a future build should not reintroduce it. Those fixes have not been re-run, so they are unverified until someone scrapes again.

Using it

install.packages('ecdata', repos = 'https://executive-communications-dataset.r-universe.dev')
load_ecd(country = 'United States of America', ecd_version = '1.0.5')

ec.load_ecd(country='United States of America', ecd_version='1.0.5')

Built with ecd_repair.py from the 1.0.0 assets. 1.0.0 through 1.0.4 remain published and unchanged.