Historical disease database version 1.2.0

This is version 1.2.0 of the historical disease database. It is created by performing regex searches for locations (municipalities) and diseases in historical newspaper articles from 1830-1940. There is not much post-processing, so note that there are likely still several issues. Please be careful when using this dataset, perform your own quality checks. When in doubt, contact us via GitHub.

The main improvement from this version is the more efficient and portable storage format as a single .parquet file. There are also fewer columns included. Note that the column cbscode indicates locations corresponding to the cbscode column from NLGIS's 1869 map of the Netherlands, available here and attached as a .geojson for archival purposes.

To join names with the database and get a polars dataframe, you can do something like this:

import polars as pl
import geopandas as gp

dd = pl.read_parquet("disease_database_v1.2.parquet")
nlgis = gp.read_file("nlgis_1869.geojson")
nl = pl.DataFrame(nlgis.iloc[:,1:3])
dd.join(nl, how="left", on="cbscode")

┌─────────┬──────┬───────┬─────────┬──────────────┬────────────┬────────┬─────────────────────────────────┐
│ disease ┆ year ┆ month ┆ cbscode ┆ mention_rate ┆ n_location ┆ n_both ┆ name                            │
│ ---     ┆ ---  ┆ ---   ┆ ---     ┆ ---          ┆ ---        ┆ ---    ┆ ---                             │
│ cat     ┆ i32  ┆ i8    ┆ i32     ┆ f64          ┆ u32        ┆ u32    ┆ str                             │
╞═════════╪══════╪═══════╪═════════╪══════════════╪════════════╪════════╪═════════════════════════════════╡
│ cholera ┆ 1830 ┆ 1     ┆ 1       ┆ 0.0          ┆ 5          ┆ 0      ┆ Adorp                           │
│ cholera ┆ 1830 ┆ 1     ┆ 2       ┆ 0.0          ┆ 3          ┆ 0      ┆ Aduard                          │
│ cholera ┆ 1830 ┆ 1     ┆ 3       ┆ 0.0          ┆ 3          ┆ 0      ┆ Appingedam                      │
│ cholera ┆ 1830 ┆ 1     ┆ 4       ┆ 0.0          ┆ 0          ┆ 0      ┆ Baflo                           │
│ cholera ┆ 1830 ┆ 1     ┆ 5       ┆ 0.0          ┆ 4          ┆ 0      ┆ Bedum                           │
│ …       ┆ …    ┆ …     ┆ …       ┆ …            ┆ …          ┆ …      ┆ …                               │
└─────────┴──────┴───────┴─────────┴──────────────┴────────────┴────────┴─────────────────────────────────┘