Skip to content

1.0.1 — repair release

Choose a tag to compare

@juangomezcruces juangomezcruces released this 05 Aug 22:38
· 35 commits to main since this release
dd133de

A repair release. The corpus is unchanged in substance — no new documents, no re-scrape.
Every fix here was derived from the 1.0.0 assets themselves, using
data-validation/ecd_repair.py.

1.0.0 is left exactly as it was, so anything that cites it still reproduces.

Read this before you upgrade

Row counts drop by 82.8%, from 16,845,134 to 2,891,622. That is not data loss:
those rows were exact duplicates produced by a many-to-many join in the build. India
goes from 7,970,491 rows to 82,682; Denmark from 4,801,705 to 49,595. If your analysis
counted rows, its numbers will change, and the 1.0.1 numbers are the correct ones.

Ecuador and the Dominican Republic are much smaller than 1.0.0 implied. The two
files shared one pooled corpus under swapped labels. Split by source domain, Ecuador is
20,619 rows / 2,236 documents covering 2023-11-23 to 2024-03-19 (Daniel Noboa only), and
the Dominican Republic is 187,932 rows ending 2020-08-16 (Danilo Medina, plus the
handover day). The apparent 2012–2024 span for both was an artefact of the pooling.

Fixed

#10 Ecuador and the Dominican Republic no longer share a corpus; country comes from the source domain and executive from the date
#11 full_ecd.parquet rebuilt from the country assets: Portugal is 64,068 rows rather than one empty row, Ecuador appears once rather than twice, and every country reconciles exactly
#12 Exact duplicates on (country, url, text, date) removed
#13 Nine misspelled executive names corrected (Jooseph R. Biden, Jr., Dmitry Medvedec, Kyriakos Misotakis, Ioannis Samaras, Poul Nyrup Rasumussen, Christina Fernandez de Kirchner, Iralki Garibashvili, Jose Luis Rodriquez Zapatero, Andrew Manuel Lopez Obrador), and Brazil's language value (Portugese → Portuguese)
#14 One schema across all 42 assets: the 17 documented columns, date as Datetime(us, UTC). Portugal's urls and the Dominican Republic's subject are restored to url and title, so both are populated rather than null
#16 Venezuela's mis-decoded text repaired in 17,703 of 17,803 affected rows
— 611 Brazil rows that carried no country, isonumber, gwc, cowcodes, polity_v, polity_iv, vdem or language at all are now labelled

Still broken

ecd_validate.py reports 0 CRITICAL, 20 ERROR against these assets, down from
19 CRITICAL and 55 ERROR. What remains needs the source data back, or a decision about
what a column means, and none of it can be repaired from the published files:

  • #13 — executive terms still overlap in 15 places. Gerald R. Ford's rows still run
    to 1996 and the Obama/Trump handover is still dated 2016-01-20. Correcting these needs
    a reviewed term table per country, not a rename map. executive is still the least
    trustworthy column in the dataset.
  • #15 — the unit of observation is still a paragraph in some countries and a whole
    document in others.
  • #18 — Colombia is still sourced entirely from YouTube.
  • #19 — file and office are still empty everywhere; language is still null for
    the United States; url is still null for Russia.
  • #20 — type still holds president names in the US file.
  • #21 — per-country coverage is still very uneven: ten countries hold fewer than 500
    documents, three fewer than 100.
  • #22 — Russia is still the English-language kremlin.ru edition, not the Russian
    originals.
  • #16 — 100 Venezuela rows could not be repaired: a byte was lost at ingest, so the
    text does not round-trip cleanly.

Using it

The ecdata packages still default to 1.0.0. Until that default is bumped and
released, ask for this version explicitly:

load_ecd(country = 'India', ecd_version = '1.0.1')
ec.load_ecd(country='India', ecd_version='1.0.1')