Skip to content

1.0.2 — repair release

Choose a tag to compare

View all tags
@juangomezcruces juangomezcruces released this 06 Aug 08:11
· 31 commits to main since this release
1.0.2
c6d23f1
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

1.0.1 plus one fix: language is now populated everywhere.

If you are on 1.0.0, read the 1.0.1 notes as well — everything they describe applies here. In particular row counts are 82.8% lower than 1.0.0, because that is how many rows were exact duplicates, and Ecuador and the Dominican Republic are far smaller than 1.0.0 implied.

What changed since 1.0.1

language was null for every row of portugal.parquet and united_states_of_america.parquet — 1,423,119 rows, 49% of the release. Both are now filled from the country dictionary the packages already ship, which has always said Portuguese and English. load_ecd(language = 'English') no longer silently omits the United States.

language is populated on all 2,891,622 rows. Nothing else differs from 1.0.1 — same rows, same repairs, byte-identical apart from that column.

India is untouched and still correctly bilingual: 51,559 Hindi rows and 31,123 English, labelled per row.

Still broken

ecd_validate.py reports 0 CRITICAL, 18 ERROR, down from 19 CRITICAL and 55 ERROR in 1.0.0 and 20 ERROR in 1.0.1. What remains needs the source data back or a decision about what a column means:

  • #13 — the Obama/Trump handover is dated 2016-01-20 rather than 2017-01-20, so 23,255 rows dated inside Obama's final year are credited to Trump. Italy carries composite values (Massimo D'Alema/Giuliano Amato) that will not group or join, and six rows across Brazil, Chile and Italy are dated before their executive took office.

    Corrected after publication: these notes originally said executive terms overlap in 15 places and that Ford's rows run to 1996. Twelve of those fifteen were an artefact of the validator comparing date spans rather than counting documents — no document is double-attributed in Denmark, Israel, Russia, Austria, Greece or Italy/Berlusconi (fixed in #28). Ford's post-tenure rows are 14 convention speeches from 1996, correctly attributed. See #13 for the full correction.

  • #15 — the unit of observation is still a paragraph in some countries and a whole document in others.

  • #18 — Colombia is still sourced entirely from YouTube.

  • #19 — file and office are still empty everywhere, and url is still null for Russia.

  • #20 — type still holds president names in the US file.

  • #21 — coverage is still very uneven: ten countries hold fewer than 500 documents, three fewer than 100.

  • #22 — Russia is still the English kremlin.ru edition, not the Russian originals.

  • #16 — 100 Venezuela rows could not be repaired: a byte was lost at ingest, so the text does not round-trip.

Using it

Until the packages are re-published, ask for this version explicitly:

load_ecd(country = 'India', ecd_version = '1.0.2')
ec.load_ecd(country='India', ecd_version='1.0.2')

Built with data-validation/ecd_repair.py from the 1.0.0 assets. 1.0.0 and 1.0.1 remain published and unchanged.

Assets 45
Loading