v0.5.0

Added

• converter.output_columns: restricts pandas to materializing only the configured columns while parsing CSV, via read_csv's usecols, instead of parsing every column and discarding the unwanted ones afterward. converter.max_workers_by_dataset overrides the scalar converter.max_workers for a single dataset, since a safe worker count depends on peak per-worker memory and that varies a lot by dataset
• filter.output_columns: per-dataset column projection on the filtered output, independent of columns_to_check's row-filtering. filter.compression also becomes a per-dataset configurable write codec (pyarrow already ships zstd/gzip/brotli/lz4, no new dependency needed)
• filter.float32_columns: opt-in, per-dataset, per-column narrowing of float64 output columns to float32. Off by default: real GDELT float data, AvgTone especially, has been observed with up to 15 significant figures, well past float32's ~7, and a round-trip test against 6.5M real rows changed the value on 31 to 100% of rows depending on the column. Narrowing low-cardinality integer columns was evaluated too and not implemented, since Parquet already dictionary-encodes those regardless of declared type and the measured gain was under 1%
• converter.compression: configurable Parquet codec for convert's own writes, same per-dataset shape filter.compression already had, defaulting to zstd for the same measured reason that default was chosen there
• crossref warns before a configured output_columns strips a dataset's required join key (GlobalEventID for gdelt_event, EventIds for both GKG 1.0 datasets, V2DOCUMENTIDENTIFIER for gdelt_gkg_v2, GLOBALEVENTID/MentionIdentifier for gdelt_mentions), checked by both run_filter and run_converter at their own configure time against a single REQUIRED_JOIN_COLUMNS mapping. crossref already raised a clear error for this, but only at join time, possibly after an unrelated sample run had already completed on the pruned output
• crossref_events_gkg_v1/_v2 warn when some or all sampled events predate that generation's real coverage start (GKG_V1_COVERAGE_START 2013-04-01, GKG_V2_COVERAGE_START 2015-02-18, both confirmed against GDELT's real file listings), the other way a join can legitimately find nothing besides a missing required column. Checked against DATEADDED, not Day (which reflects when an event is reported to have occurred and can be far in the past for retrospective reporting). Diagnostic only: events within coverage in the same sample still join normally
• crossref_events_gkg_auto (--gkg-version auto): attempts every eligible event against both crossref_events_gkg_v1 and crossref_events_gkg_v2, for a sample spanning both eras, e.g. the 2013-2015 window where only GKG 1.0 exists. Output concatenates both paths' results with a CrossrefSource column (v1/v2); an event matching both contributes one row per source. Events before GKG_V1_COVERAGE_START are skipped and logged, since neither generation has any data for them. --columns isn't supported in this mode, since GKG 1.0's 11 fields and GKG 2.1's 27 share no common name
• convert gains a .done marker for flat/daily output (Events daily archives, GKG 1.0, GKG 2.1, Mentions), matching the resumability the historical (Hive-partitioned) path already had and the skip-already-downloaded behavior scrape already had. Found necessary against a real 30,137-file Mentions batch: two independent runs each died to an OS-level kill around the same ~51% mark, having made no net progress relaunch to relaunch, because every attempt reprocessed every zip from file 1
• filter gains equivalent resumability, for the first time; its correctness surface is wider than convert's, since columns_to_check, output_columns, float32_columns, and compression all change what the filtered output contains

Changed

• filter.compression now defaults to zstd instead of snappy. Measured on real Events data (5.8M rows, all 58 columns): roughly 30% smaller than snappy, at comparable or faster write speed. Lossless, so no accuracy tradeoff in taking it as the default; per-dataset override remains available
• README, docs/index.md, and docs/getting-started.md now lead with pip install gdeltforge instead of "install from a clone," now that 0.4.0 is actually on PyPI; the clone/uv sync path is kept as a clearly-labeled "installing from source" option for contributors. Checked off the PyPI roadmap item, and fixed a leftover ↔ in docs/cli-reference.md (should have been <->, missed in the arrow-notation pass this session). pyproject.toml's description was also still the old Events-only tagline; fixed for the next release, though it won't retroactively change what's already published for 0.4.0

Fixed

• _detect_file_type never recognized real 15-minute YYYYMMDDHHMMSS GKG 2.1/Mentions filenames as any known cadence, since the file-type patterns in place at the time GKG 2.1/Mentions support shipped all assumed 8-digit daily or longer. Nothing surfaced the gap in practice, only because partitioning is never enabled for those two datasets. Added a real quarter_hourly file type; the non-detecting partitioning-off shortcut, previously mislabeled "daily" regardless of a file's real cadence, is renamed "flat", and routing to the historical (Hive-partitioned) path is now decided by whether partitioning.rules defines an entry for the detected file type rather than by a hardcoded "daily" string comparison
• The .done marker convert's historical (Hive-partitioned) path already had only ever recorded that a file had been processed, not under what configuration. Rerunning convert after changing output_columns would be silently skipped by a marker left from the old configuration, serving output shaped by settings that no longer match the current run. config_fingerprint/is_marked_done/mark_done now write the relevant config into the marker itself and compare its content, not just presence; a mismatch, including a pre-fingerprint empty marker from before this existed, is treated as not done
• crossref_events_gkg_v2 hardcoded MentionTimeDate and Confidence as required when reading Mentions, even though neither participates in the join itself (only GLOBALEVENTID and MentionIdentifier do). A Mentions dataset missing either one failed outright instead of joining successfully minus that payload field; both are now read only if present
• crossref_events_gkg_auto's routing excluded an event from v2 whenever its own DATEADDED predated GKG_V2_COVERAGE_START, on the reasoning that such an event could only match through GKG 1.0. Confirmed for real that reasoning doesn't hold: a Mentions row is timestamped by when it was created, not by its event's DATEADDED (a 2019-origin event was found referenced by a real Mentions row dated 2020), and a direct test against the complete 2015-02-18 through 2015-12-31 Mentions history (211.8M rows) plus a further scattered sample found zero matches for any pre-2015-02-18 event, not because none could exist but because the old routing never gave them the chance. The same asymmetry ran the other direction too, since GKG 1.0 remains live and daily-published today. Every eligible event is now attempted against both generations rather than routed to exactly one
• crossref_events_gkg_v2's dedup (drop_duplicates(keep="last"), keeping the most recently reprocessed article) silently depended on Path.glob's return order matching each file's real chronological position, true on NTFS by coincidence but not on Linux/ext4, where GitHub Actions CI failed by keeping the stale record instead of the reprocessed one. _dataset() now sorts its file list explicitly