1.0.5-sentences — sentence-level view

A sentence-level view of release 1.0.5. One row per sentence: 2,891,622 rows become 9,207,251 sentences, 3.2 per row.

This does not replace 1.0.5. That release is unchanged and remains the document-level data; this is a parallel view for work that needs one unit meaning the same thing in every country.

Why it exists

The unit of observation differs by country (#15). A row is a whole document in Brazil, a paragraph in Spain, a sentence in Canada, an HTML block in Czechia. Any cross-country count of rows therefore measures the scraper as much as the executive. Sentences give a unit that is comparable across all 42.

Schema

Every column from 1.0.5, plus four:

column	meaning
document_id	stable id for the source document, derived from its url
block_index	which original row within that document the sentence came from — the release's own grain, preserved rather than discarded
sentence_index	position of the sentence within the document
segmenter	which rules split it

text holds the sentence. Joining back to 1.0.5 is document_id, or url directly.

Segmentation

pysbd has rules for 11 of the 22 languages in the release: Spanish, German, French, Italian, Greek, Danish, Polish, Japanese, Chinese, Hindi, English. The other 11 use the English rules.

That fallback was tested against real rows rather than assumed. In Czech, Portuguese, Turkish, Korean, Hungarian, Hebrew, Indonesian, Icelandic, Norwegian and Georgian, fewer than 4% of long passages collapse to a single sentence. What it loses is language-specific abbreviation handling, so an abbreviation followed by a capital can split early. The segmenter column records which case each row was, so this is visible in the data rather than only in these notes.

Where the output is poor

Sentence splitting cannot improve on what it is given.

• Colombia is not segmented at all. Its rows are auto-generated YouTube captions with no punctuation and no capitalisation — 198 of 198 long rows yield a single "sentence". It passes through whole, marked segmenter = "none (unpunctuated source)", rather than being split wrongly.
• Czechia (47% of sentences under 20 characters), Republic of Korea (25%), Japan (20%) and Hong Kong (18%) are extracted block by block, so headings, captions and signature lines become "sentences". That is the source grain showing through.
• Italy is OCR'd PDF text with a running page header, so some sentences are page furniture and some carry OCR noise.
• Brazil: 56% of its rows carry no url, so 227,464 sentences have a null document_id. That is a property of 1.0.5, not of this view.

Verification

Total non-whitespace character mass is identical before and after for all 42 countries — nothing lost, duplicated or reordered.

73 documents across Portugal (52), Ecuador (12), the Dominican Republic (8) and Israel (1) do not appear, because every row of those documents has empty text.

Using it

Built with ecd_sentences.py. Until the packages expose it directly, download an asset and read it:

sentences <- arrow::read_parquet("chile.parquet")

import polars as pl
sentences = pl.read_parquet("chile.parquet")