GdeltForge 0.9.0

[0.9.0] - 2026-09-09

Fixed

• A hand-written settings.yaml that omitted a whole top-level section (columns, columns_numeric) or a nested one (filter.columns_to_check) crashed deep inside converter.py/filter.py/cli.py with a bare Error: 'columns' or Error: 'columns_to_check', naming the missing key with no indication of which file or section was at fault, since every one of those reads its section with a direct config["..."][...] access on the assumption the section is always present the way the bundled default always has it. New _deep_merge_defaults fills in whatever a real, user-supplied config doesn't mention from the bundled default, recursing into nested dicts, so a partial config now behaves like the bundled default for anything it doesn't set; a key already present in the user's config is never touched regardless of type, and only dict values recurse, so a user's own (possibly empty) list for one dataset is never padded with the default's entries for that same dataset. Found via a live comprehensive QA pass
• sample --mode filtered --stratify, with no --filter, failed with Error: --filter is required when mode == 'filtered', even though --stratify's own --help text only documented --n-per-group as a requirement. --stratify targets rows by group membership on its own and never needed a separate row-level --filter condition the way get_random_sample does; an omitted --filter now defaults to no filtering when --stratify is set, and only still raises for a plain --mode filtered call with no --stratify. Found via a live comprehensive QA pass
• Converting the real events-reduced dump failed with a raw WinError, the system cannot find the path specified, from a moderately nested project directory: the generated .tmp partition path landed at 265 characters, past Windows' default 260-character MAX_PATH, confirmed directly by reproducing success from a short path and failure only from a deep one. Every part filename repeated the full source zip stem (GDELT.MASTERREDUCEDV2.1979-2013, 32 characters) for no functional reason, since there is only ever one source file for this dataset; part filenames are now just part{chunk_idx}.parquet, still deterministic from the chunk index alone. Found via a live comprehensive QA pass
• crossref --gkg-version v1-counts could crash with a Rust-level memory allocation failure, or hang indefinitely with no progress, joining against real gkg-v1-counts data: one real row's EventIds field held 13,051 comma-separated ids against a same-file mean of ~37, and exploding an entire 64,000-row batch in one step let that single row dominate the whole batch's peak memory. New _iter_row_slices_bounded_by_explosion caps each explode step by its own output size (200,000 rows), not the outer batch's row count. A pathological row is processed alone when needed, never left to hold up or balloon the batch around it; verified a real join forced through several bounded steps produces byte-identical results to the unbounded case. Found via a live comprehensive QA pass
• Every gdeltforge command, including --help, crashed with an unhandled PermissionError in a working directory the process can't write to: cli.py builds its own logger with log_to_file=True unconditionally at import time, before argparse or any command dispatch ever runs, and get_logger() created the log directory with no error handling at all. get_logger() now falls back to console-only logging with a warning when the log directory or file can't be created. Found via a live comprehensive QA pass
• SIGKILL of a running scrape/convert/filter process left its already-dispatched ProcessPoolExecutor workers running as orphans, still pulling from the shared task queue, for minutes after the parent process was gone, since SIGKILL can't be caught by anything. gdeltforge now makes itself its own process group leader on POSIX at startup, so killing the whole group (kill -KILL -$(pgid)) reliably stops every worker too. A real SIGTERM handler also now cancels queued work immediately, the same way Ctrl+C already does, before a process manager's escalation to SIGKILL ever becomes necessary. Found via a live comprehensive QA pass
• Two concurrent filter or convert invocations writing to the same output path (a retry launched before realizing the first was still going, an overlapping cron run) could both fail with a raw FileNotFoundError, one process's os.replace() finding its own temp file already renamed away by the other: filter_single_file's and _write_partition_file's own hand-rolled temp-file naming never adopted the PID-suffix fix write_parquet_atomic/write_dataframe_atomic already carry for the identical race. Both now use a PID-suffixed name too, _write_partition_file by delegating to write_parquet_atomic directly, not duplicating the pattern. Found via a live comprehensive QA pass
• converter.partitioning.enabled, documented as an Events-only opt-in, broke convert for every other dataset (gkg-v2, mentions, gkg-v1, gkg-v1-counts) the moment it was turned on for Events' own yearly/monthly split, demanding a *_parquet_historical_directory path none of them could ever actually write to, since none of them can produce a yearly- or monthly-typed file. The historical-directory requirement is now scoped to whether a configured rule's file type could actually apply to the dataset being converted. Found via a live comprehensive QA pass
• Invalid YAML in a config file surfaced PyYAML's own raw parser traceback, not a clear, crafted message naming the config file, unlike every other malformed-config case load_config already handles (a missing file, an empty file, a directory, not a file). Found via a live comprehensive QA pass
• A columns_numeric value that couldn't be parsed as numeric (an out-of-range integer, unparseable garbage, or a whitespace-padded numeric string) was silently coerced to null during convert, indistinguishable in both the output and the log from a field that was genuinely blank to begin with. _read_csv and process_reduced_file now warn, naming the column and how many new nulls appeared, whenever the cast actually introduces one. Found via a live comprehensive QA pass
• A temp file orphaned by a killed or crashed write_parquet_atomic/write_dataframe_atomic call (a sample/crossref write, in practice) was never detected or cleaned up on a later run: the leftover-detection warning only ever checked the exact current-process PID-suffixed path, which a genuinely different, now-dead process's own leftover essentially never is. It now globs for any PID's leftover at the same destination and removes the ones whose process is confirmed no longer running, leaving a still-live PID's file (a genuinely concurrent writer) untouched. Found via a live comprehensive QA pass
• crossref listed and date-filtered each configured GKG/Mentions directory twice per run (crossref_events_gkg_auto, which calls straight into both single-version joins, up to six times), doubling the real I/O cost cli-reference.md's own capacity-planning numbers document: warn_if_directory_is_large and the actual scan each redid the identical listing pass independently, not sharing one. Both now consume one already-listed file set per directory, computed exactly once per run. Found via a live comprehensive QA pass
• sample (all three modes) and crossref crashed reading a real multi-file archive whose files agree on a column's name but disagree on its dtype, e.g. events' own Actor2Geo_Type (Float64 through 2007-10, Int64 from 2007-11 onward) or GKG 2.1's V2.1DATE (Float64 in 441 files scattered across six years, Int64 everywhere else). The existing schema union only reconciled a column missing entirely from some files; a column present everywhere but typed differently raised data type mismatch ... incoming: X != target: Y the moment a read crossed the boundary. sample's calendar/filtered scans, IndexedSampler.get_random_sample, and crossref's GKG/Mentions scans now detect a genuine dtype conflict across files and widen it (Int64 to Float64, mixed integer widths to Int64) before reading, falling back to a plain error naming the column and both dtypes for a non-numeric conflict that can't be widened safely.
• The Int64-to-Float64 widening above had no check against Float64's own precision limit: it represents an integer exactly only up to 2**53. A genuine ID/count column hit by the identical dtype drift, holding a value past that bound, would have been silently corrupted: a wrong value, not a crash. Every real GDELT column currently affected (small geo-type enums, a 14-digit datetime-as-integer, a 1-2 range) stays far under the limit, but the fix's correctness rested entirely on that fact holding, with nothing in the code checking it. Widening now reads the affected file's own min/max first (a cheap, parquet-statistics-only check, not a full column scan); a value past the safe bound raises a clear error naming the column, file, and value.
• A Ctrl+C/SIGINT arriving during any of sample's tqdm-wrapped loops (indexed loading, calendar/filtered/stratified sampling) or crossref's GKG 1.0 cross-referencing loop leaked a raw "Exception ignored in: <generator object tqdm.iter ...>" traceback fragment to stderr before the documented clean Interrupted. message. Each loop now drives its own tqdm object manually inside an explicit with block, not by iterating it directly: the bare for x in tqdm(iterable): form creates a second, separate generator internally whose own implicit close, on interrupt, has no legitimate way to propagate a further exception raised during it.
• A Ctrl+C arriving during scrape, convert, or filter's executor loop (downloading, converting, or filtering files in parallel) leaked the same kind of traceback fragment. Each of the three now drives its own `tq...

v0.8.0

GdeltForge 0.8.0: events-15min dataset, required --dataset (breaking)…

GdeltForge 0.7.0

[0.7.0] - 2026-08-19

Added

• convert/filter gain --delete-source: convert deletes each source zip once its parquet output is written and confirmed done, filter deletes each source (unfiltered, converted) parquet once its filtered output is written and confirmed done. Off by default, never deletes on a failed conversion/filter, and never runs ahead of the existing .done marker. sample deliberately doesn't get an equivalent option: a sample is many-to-one against its source, so deleting the source after one sample would discard the ability to draw another. Combined with any setting that narrows the output (output_columns for convert; columns_to_check/output_columns/float32_columns for filter), whatever that dropped or changed has nothing left to recover it from except redoing an earlier stage, so a warning fires once at the start of a run configured that way, shared between both commands (warn_if_delete_source_drops_recoverable_data)
• scrape/convert/filter gain --verbose. convert/filter used to log unconditional per-file lines (convert: "Processing ZIP"/"Skipping already converted"; filter: rows-kept summary/"Skipping already filtered") at INFO, which at gkg-v2/mentions scale (hundreds of thousands of 15-minute files) meant hundreds of thousands of terminal lines fighting the tqdm progress bar for the screen; scrape never had this problem, since its own per-attempt detail was already DEBUG-only. Those lines are now DEBUG too, so all three commands default to the same shape (setup line, progress bar, end-of-run summary); --verbose raises the relevant module's logger back to DEBUG for whoever actually wants per-file detail. convert/filter both run their per-file work inside a ProcessPoolExecutor worker, a genuinely separate process that re-imports the module fresh, so a level change made in the main process alone never reaches it; verbose is threaded through as a real instance attribute and re-applied independently inside each worker, not just set once where the flag is parsed
• scrape/convert/filter gain --quiet/-q, mutually exclusive with --verbose: raises the relevant module's logger to WARNING, suppressing the setup and end-of-run summary lines each command otherwise always prints, for scripted or cron use that only cares about problems. cli.py keeps its own logger separate from the stage modules (its own "Starting X stage..."/"X completed." lines are logged from there, not from scraper.py/converter.py/filter.py), so --quiet applies to it too via a small _apply_verbosity helper shared by all three commands
• scrape/convert/filter gain --force: bypasses the existing resumability check (scrape's already-downloaded-file check, convert/filter's .done marker) for files that would otherwise be skipped, reprocessing and overwriting them. Off by default, unrelated to --delete-source, which still only fires from a successful run's own success branch
• scrape/convert/filter gain --dry-run: reports how many files would be processed and how many would be skipped, without downloading, converting, or filtering anything. Runs after --force's own skip-list logic, so --force --dry-run together preview what a real forced run would actually do, not what an unforced one would. scrape's preview mirrors its own per-file existing-file check without making any network request; convert/filter short-circuit after building their to-process list, before any worker is submitted
• sample/crossref gain --export-format {parquet,csv} (default parquet): writes the finished, already-in-memory sample or join result as CSV instead of Parquet, for handing it to a tool that doesn't read Parquet. csv rewrites --out's extension to .csv regardless of what was passed, so the flag is authoritative over the file's own name. convert/filter/sample --mode filtered/crossref themselves stay Parquet-only: their streaming reads and pyarrow.dataset predicate-pushdown scanning of the input side have no equivalent in CSV, so only the single final-result write this flag touches is affected, not how a sample or join is computed. New write_dataframe_atomic in utils/io.py adds the CSV write path alongside the existing write_parquet_atomic, same atomic tmp-then-rename guarantee

GdeltForge 0.6.1

Fixed

• filter silently wrote nothing whenever columns_to_check was empty, exactly the bundled default config's own value for every dataset. filter_single_file treated an empty columns_to_check the same as "columns were configured but none exist in this file's schema", logging an error and returning before ever writing the output file, while the batch summary still reported every file processed successfully with 100% retention. sample and crossref then correctly found nothing in the (empty) output directory, surfacing as confusing downstream errors with no indication the real failure was upstream and silent. The early-return now only fires when columns_to_check is actually non-empty; an empty list falls through to the existing, already-correct no-op path (dropna(subset=[]) is a documented pandas no-op)
• crossref --events crashed with a confusing ArrowInvalid: ... magic bytes not found error when pointed at a directory instead of a single file, if that directory contained a convert/filter .done resumability marker (<name>.parquet.done, a real sibling of the data by design in every one of their output directories). pd.read_parquet was handing the raw path straight to pandas with no awareness of that convention. --events now accepts a directory properly: every *.parquet file in it is read and concatenated, .done markers and any other non-parquet sibling are never handed to the parquet reader

GdeltForge 0.6.0

Added

• load_config() gains a fourth fallback tier: GdeltForge's own built-in default, bundled inside the installed package via importlib.resources, used only when neither --config nor GDELTFORGE_CONFIG was given at all and ./config/settings.yaml doesn't exist either. Previously this situation was a hard FileNotFoundError on every single run, painful specifically for a pip install gdeltforge in an ephemeral environment like a Colab session, where nothing drops config/settings.example.yaml into the working directory the way a git clone does, so the file had to be reconstructed by hand every session reset. The bundled default is written out to the resolved config path on first use (best-effort; a read-only working directory still gets a working, in-memory-only config rather than an error), so it becomes a normal editable file for the rest of that session. An explicit --config/GDELTFORGE_CONFIG pointing at a missing path still raises, unchanged: that's almost always a typo, not a request for the built-in default
• The bundled default (src/gdeltforge/config/default_settings.yaml) is deliberately more conservative than settings.example.yaml, not the same content with the paths changed: real ./data/... paths (not settings.example.yaml's ./path_example/... placeholders) but no filter.columns_to_check values, output_columns, or float32_columns for any dataset, so a first run's output is never silently shaped by row-dropping or column-pruning choices the user never made. Confirmed dropna(subset=[]) is a documented pandas no-op before relying on it, not assumed
• crossref_events_gkg_v1/_v2 (and, by extension, crossref_events_gkg_auto, which calls both) now warn, once --events crosses 1,000,000 rows, that the module is designed for a bounded sample, not the full Events archive. Never blocks: an archive-scale join is a legitimate thing to ask for, just an expensive one that nothing previously flagged. The threshold and the numbers in the warning are measured, not guessed: building the join key set alone (set() -> list() -> a pyarrow isin() filter, the exact sequence both join paths run) was measured via tracemalloc at ~100 MB/1s per million events, ~800 MB/5s at 10M, ~5.2 GB/13s at 50M, before the archive scan itself even starts, since predicate pushdown prunes rows within a file, not which files get opened
• The same two functions now separately warn when the configured Mentions/GKG directory itself has more than 50,000 files, independent of --events size: crossref lists and opens every file in that directory on each run (Mentions/GKG 2.1/GKG 1.0 checked independently, so either side gets named specifically). Measured directly against real local data (3,127 real GKG 2.1 files: 0.20s; 33,303 real Mentions files: 2.47s), a roughly linear ~75 microseconds/file; extrapolated to the full historical GKG 2.1/Mentions archive (~385,728 files, see docs/configuration.md's "Capacity planning" section), that's ~29s just to list and open every file before a single row is read
• crossref_events_gkg_v2/crossref_events_gkg_auto gain on_duplicate_document ("all"/"latest"/"earliest", default "all"): GKG 2.1 can carry more than one record for the same document URL, confirmed against real data to sometimes hold genuinely different content across visits (a tag/listing page recrawled years apart), not always just a stale reprocessing of the same article, so picking a single winner is a real editorial choice rather than noise removal, and defaults to keeping every record rather than silently discarding one. "latest"/"earliest" remain available for a caller who specifically wants a single row per URL. CLI: crossref --on-duplicate-document {all,latest,earliest}
• crossref_events_gkg_v2/crossref_events_gkg_auto gain dedupe_mentions (default False). Mentions records one row per sentence that references an event, so an event quoted in several sentences of the same article produces several near-identical raw rows for one (event, article) relationship; by default every one of those rows is kept, so row count still reflects real mention frequency. Set to True (CLI: --collapse-duplicate-mentions) to instead collapse them into one row per (event, article), keeping the highest-Confidence version when Confidence is available, with a new Mention_Count column recording how many raw rows collapsed into it. Confirmed against real data: about 1.9% of (event, article) pairs have more than one raw Mentions row, and where they do, Confidence differs across them about 23% of the time, so collapsing picks a representative row rather than removing true duplicates
• crossref_events_gkg_v1/_v2/_auto gain start_date/end_date (CLI: --start-date/--end-date), narrowing which files in the configured Mentions/GKG directories get listed and opened at all, the same lever scrape/convert/filter already had and previously the one thing crossref's own large-directory warning said didn't exist. Reuses scrape's own filter_paths_by_date and filename date parsers rather than a separate mechanism. Only narrows the Mentions/GKG corpus, not --events: a Mentions row is timestamped by when it was recorded, not by its event's DATEADDED, so this is a real scope decision that can exclude a legitimate late mention of an in-range event, not a risk-free filter

v0.5.0

Added

• converter.output_columns: restricts pandas to materializing only the configured columns while parsing CSV, via read_csv's usecols, instead of parsing every column and discarding the unwanted ones afterward. converter.max_workers_by_dataset overrides the scalar converter.max_workers for a single dataset, since a safe worker count depends on peak per-worker memory and that varies a lot by dataset
• filter.output_columns: per-dataset column projection on the filtered output, independent of columns_to_check's row-filtering. filter.compression also becomes a per-dataset configurable write codec (pyarrow already ships zstd/gzip/brotli/lz4, no new dependency needed)
• filter.float32_columns: opt-in, per-dataset, per-column narrowing of float64 output columns to float32. Off by default: real GDELT float data, AvgTone especially, has been observed with up to 15 significant figures, well past float32's ~7, and a round-trip test against 6.5M real rows changed the value on 31 to 100% of rows depending on the column. Narrowing low-cardinality integer columns was evaluated too and not implemented, since Parquet already dictionary-encodes those regardless of declared type and the measured gain was under 1%
• converter.compression: configurable Parquet codec for convert's own writes, same per-dataset shape filter.compression already had, defaulting to zstd for the same measured reason that default was chosen there
• crossref warns before a configured output_columns strips a dataset's required join key (GlobalEventID for gdelt_event, EventIds for both GKG 1.0 datasets, V2DOCUMENTIDENTIFIER for gdelt_gkg_v2, GLOBALEVENTID/MentionIdentifier for gdelt_mentions), checked by both run_filter and run_converter at their own configure time against a single REQUIRED_JOIN_COLUMNS mapping. crossref already raised a clear error for this, but only at join time, possibly after an unrelated sample run had already completed on the pruned output
• crossref_events_gkg_v1/_v2 warn when some or all sampled events predate that generation's real coverage start (GKG_V1_COVERAGE_START 2013-04-01, GKG_V2_COVERAGE_START 2015-02-18, both confirmed against GDELT's real file listings), the other way a join can legitimately find nothing besides a missing required column. Checked against DATEADDED, not Day (which reflects when an event is reported to have occurred and can be far in the past for retrospective reporting). Diagnostic only: events within coverage in the same sample still join normally
• crossref_events_gkg_auto (--gkg-version auto): attempts every eligible event against both crossref_events_gkg_v1 and crossref_events_gkg_v2, for a sample spanning both eras, e.g. the 2013-2015 window where only GKG 1.0 exists. Output concatenates both paths' results with a CrossrefSource column (v1/v2); an event matching both contributes one row per source. Events before GKG_V1_COVERAGE_START are skipped and logged, since neither generation has any data for them. --columns isn't supported in this mode, since GKG 1.0's 11 fields and GKG 2.1's 27 share no common name
• convert gains a .done marker for flat/daily output (Events daily archives, GKG 1.0, GKG 2.1, Mentions), matching the resumability the historical (Hive-partitioned) path already had and the skip-already-downloaded behavior scrape already had. Found necessary against a real 30,137-file Mentions batch: two independent runs each died to an OS-level kill around the same ~51% mark, having made no net progress relaunch to relaunch, because every attempt reprocessed every zip from file 1
• filter gains equivalent resumability, for the first time; its correctness surface is wider than convert's, since columns_to_check, output_columns, float32_columns, and compression all change what the filtered output contains

Changed

• filter.compression now defaults to zstd instead of snappy. Measured on real Events data (5.8M rows, all 58 columns): roughly 30% smaller than snappy, at comparable or faster write speed. Lossless, so no accuracy tradeoff in taking it as the default; per-dataset override remains available
• README, docs/index.md, and docs/getting-started.md now lead with pip install gdeltforge instead of "install from a clone," now that 0.4.0 is actually on PyPI; the clone/uv sync path is kept as a clearly-labeled "installing from source" option for contributors. Checked off the PyPI roadmap item, and fixed a leftover ↔ in docs/cli-reference.md (should have been <->, missed in the arrow-notation pass this session). pyproject.toml's description was also still the old Events-only tagline; fixed for the next release, though it won't retroactively change what's already published for 0.4.0

Fixed

• _detect_file_type never recognized real 15-minute YYYYMMDDHHMMSS GKG 2.1/Mentions filenames as any known cadence, since the file-type patterns in place at the time GKG 2.1/Mentions support shipped all assumed 8-digit daily or longer. Nothing surfaced the gap in practice, only because partitioning is never enabled for those two datasets. Added a real quarter_hourly file type; the non-detecting partitioning-off shortcut, previously mislabeled "daily" regardless of a file's real cadence, is renamed "flat", and routing to the historical (Hive-partitioned) path is now decided by whether partitioning.rules defines an entry for the detected file type rather than by a hardcoded "daily" string comparison
• The .done marker convert's historical (Hive-partitioned) path already had only ever recorded that a file had been processed, not under what configuration. Rerunning convert after changing output_columns would be silently skipped by a marker left from the old configuration, serving output shaped by settings that no longer match the current run. config_fingerprint/is_marked_done/mark_done now write the relevant config into the marker itself and compare its content, not just presence; a mismatch, including a pre-fingerprint empty marker from before this existed, is treated as not done
• crossref_events_gkg_v2 hardcoded MentionTimeDate and Confidence as required when reading Mentions, even though neither participates in the join itself (only GLOBALEVENTID and MentionIdentifier do). A Mentions dataset missing either one failed outright instead of joining successfully minus that payload field; both are now read only if present
• crossref_events_gkg_auto's routing excluded an event from v2 whenever its own DATEADDED predated GKG_V2_COVERAGE_START, on the reasoning that such an event could only match through GKG 1.0. Confirmed for real that reasoning doesn't hold: a Mentions row is timestamped by when it was created, not by its event's DATEADDED (a 2019-origin event was found referenced by a real Mentions row dated 2020), and a direct test against the complete 2015-02-18 through 2015-12-31 Mentions history (211.8M rows) plus a further scattered sample found zero matches for any pre-2015-02-18 event, not because none could exist but because the old routing never gave them the chance. The same asymmetry ran the other direction too, since GKG 1.0 remains live and daily-published today. Every eligible event is now attempted against both generations rather than routed to exactly one
• crossref_events_gkg_v2's dedup (drop_duplicates(keep="last"), keeping the most recently reprocessed article) silently depended on Path.glob's return order matching each file's real chronological position, true on NTFS by coincidence but not on Linux/ext4, where GitHub Actions CI failed by keeping the stale record instead of the reprocessed one. _dataset() now sorts its file list explicitly

v0.4.0

Added

• gdeltforge codes command: looks up valid codes across seven CAMEO/FIPS-coded column families (CAMEO actor-country, FIPS geo-country, CAMEO ethnic, CAMEO known-group, CAMEO religion, CAMEO actor-type, and CAMEO event), with a --search filter. Needs no config file
• FilteredSampler now warns (not raises) when a filter value on a CAMEO/FIPS-coded column isn't recognized for that column's code family, e.g. a 3-letter CAMEO code used against a 2-letter FIPS column. Covers all seven families above, comparing case-insensitively since GDELT stores ethnic codes lowercase in real data while the source codebook uses uppercase
• Bundled CAMEO/FIPS reference data verified against every distinct value across a full archive scan (~542M rows, all seven families); added 7 previously-missing FIPS Pacific-island codes, 28 previously-missing CAMEO known-group codes for organizations absent from the public CAMEO manual, and 2 previously-missing CAMEO event codes (1213/1214, undocumented in the public manual's "reject material cooperation" branch but confirmed via GDELT's own TABARI/PETRARCH verb-pattern dictionary), each confirmed against real data rather than guessed. A handful of real event-code values ("X", "--", "---") are GDELT's own markers for unclassifiable rows, not CAMEO codes, and are deliberately excluded so a filter using one still warns
• gdeltforge sample --source {filtered,converted}: sampling can now read from the raw converted Parquet directory instead of always requiring the filter stage first
• --columns now applies to indexed and daily sampling too, not just filtered; both previously always read every column of every file they touched
• --dataset {gkg-v2,mentions} support for scrape/convert/filter/sample: GKG 2.1 (the current, actively-produced Global Knowledge Graph: themes, tone, GCAM, people, organizations) and Mentions (every re-report of an Event by a different article over time) alongside Events. Discovery uses GDELT's v2 master file list rather than Events' HTML directory listing, since GKG 2.1/Mentions publish every 15 minutes, not daily; a warning logs before a scrape that would download an unusually large number of files given that granularity. Column schemas and field order confirmed against the real production parser in aamend/spark-gdelt, not just the codebook
• --dataset {gkg-v1,gkg-v1-counts} support for scrape/convert/filter/sample: the legacy GKG 1.0 format (the primary GKG feed April 2013 through February 2015, still published daily since for backwards compatibility) and its separate, narrower Counts file (one row per count mention rather than per document). Unlike GKG 2.1, GKG 1.0 rows carry EventIds directly, so joining to Events skips the two-hop trip through Mentions that GKG 2.1 needs. Discovery reuses Events' HTML-directory-listing approach, not GKG 2.1/Mentions' master-file-list mechanism, since GKG 1.0 publishes daily like Events rather than every 15 minutes; the two paths are inferred (not yet directly confirmed) to share the same markup, based on both hitting an identical TLS certificate mismatch, i.e. the same underlying GCS bucket. Column schemas confirmed against the same real production parser used for GKG 2.1/Mentions
• gdeltforge crossref command: enriches a sampled Events output with GKG (themes, tone, people, organizations). --gkg-version {v1,v1-counts,v2} picks the join strategy, not just the data source, since GKG 1.0 carries EventIds directly (a direct join) while GKG 2.1 carries no event id at all and needs a two-hop join through Mentions on the source article's URL. Both pyarrow filter-pushdown scan only the rows relevant to the input sample, never materializing the full Mentions/GKG archive; both preserve the real many-to-many structure (one event can produce several output rows, one article covering several events contributes one row per event) rather than silently collapsing it, and dedupe GKG 2.1 rows on document URL so an article reprocessed across batches contributes exactly one row. GKG-side output columns are prefixed GKG_/Mention_ to avoid colliding with an identically-named Events column (NumArticles exists on both Events and GKG 1.0)
• --start-date/--end-date for convert and filter, matching the flags scrape already had. Both narrow which files get touched by filename, reusing the same date parsers scrape uses (now also matching the equivalent converted-Parquet names, e.g. 20200315.export.parquet alongside the raw 20200315.export.CSV.zip) rather than a new date-parsing path; a file whose date can't be determined is still processed rather than silently skipped, since a local directory can hold pre-daily yearly/monthly archives with no single day to compare against a range, unlike a remote listing where an unparseable entry is usually just noise. filter's version restricts which input files get read, not a row-level date filter, since filtering itself is row-wise NaN-dropping unrelated to dates

Changed

• README hero section: real CI/license/release badges, a tighter pitch, and a terminal-demo screenshot of codes and sample running against the live dataset
• The CLI now catches failures at the top level: any error prints Error: <message> and exits 1, and Ctrl+C prints Interrupted. and exits 130, instead of a raw Python traceback either way
• Breaking config change, foundational work for multi-dataset support (GKG, Mentions): columns_numeric and filter.columns_to_check are now nested under the dataset name (gdelt_event), matching how columns was already structured, instead of being flat lists assuming a single dataset. Update settings.yaml: wrap your existing columns_numeric: list as columns_numeric: {gdelt_event: [...]}, and likewise for filter.columns_to_check. scrape/convert/filter/sample also gain a --dataset {events,gkg-v1,gkg-v1-counts,gkg-v2,mentions} flag (default events, matching current behavior exactly)
• filter now runs across a worker pool (config: filter.max_workers, same shape and default as converter.max_workers) instead of processing files one at a time. Found running a real full historical Events archive through filter: 4,748 files, 866M rows, over an hour single-threaded, despite scrape and convert already being parallel
• README, docs/index.md, and docs/comparison.md no longer describe GdeltForge as Events-only: both explicitly said GKG/Mentions support was "on the roadmap, not built yet," which stopped being true once crossref shipped. Updated the tagline, the pipeline-stage diagrams and tables (both were also missing crossref and codes entirely), and the "reach for something else" comparison table and bullet list, which used to point GKG/Mentions/join needs elsewhere. Added a GKG-enriched-Events recipe to docs/recipes.md and a matching example to the README
• Breaking config change, no code changes required: the example paths.* values in settings.example.yaml now nest every dataset's four stages under one directory (data/events/raw, data/events/parquet, ..., data/gkg_v2/raw, ...) instead of data/raw, data/parquet, data/gkg_v2_raw, and so on sitting flat side by side. Events previously sat unprefixed at the top level while every other dataset already got its own directory; this was the one dataset left out of that pattern rather than a new one. Config keys are unchanged (dataset_path_key only ever resolves a key name, never a path value), so existing installs keep working with their current layout unless you choose to adopt the new one: update your settings.yaml path values and move your existing data/ subdirectories to match

Fixed

• GKG 1.0 and its Counts file both ship with a literal header row (DATE\tNUMARTS\t...); the converter read every dataset as headerless unconditionally, so that line was misread as a garbage data row on every converted file. Found by converting a real downloaded file rather than only a synthetic fixture
• crossref_events_gkg_v1's EventIds split assumed a semicolon-delimited list; real data uses commas. Any GKG 1.0 row naming more than one event had its whole EventIds string treated as a single token that could never match a real event, silently contributing zero rows for that row instead of one per event
• columns.gdelt_mentions was missing two real trailing columns (MentionDocTranslationInfo, Extras), confirmed live against the official GDELT Event Codebook V2.0. Every column in every converted Mentions file was silently reading the wrong value as a result (pandas absorbs the shortfall as an unnamed leading index rather than raising), including GLOBALEVENTID and MentionIdentifier, which crossref_events_gkg_v2's join depends on entirely
• A single file's download failure could crash an entire scrape instead of just failing that file: found running a real ~4,850-file full-history scrape, where a transient Windows file lock made the retry loop's own cleanup step raise, and that second exception went unhandled all the way up through the batch. Both the cleanup step and the per-file result handling in download_gdelt_files are now guarded, matching the resilience convert already had
• Converting a large batch could leave a truncated, corrupt parquet file at its final path if the process was interrupted mid-write, since both _save_parquet and _save_historical_parquet wrote straight to that path instead of through a temp file. Found for real converting a ~3,000-file GKG 2.1 batch. Both now write atomically, matching the pattern already used for scrape downloads and sample output
• The initial fix for the above (bulk-assigning all accepted rows in one indexed write) let pandas resolve same-batch slot collisions independently per column block, silently desyncing string columns from numeric ones when both were written together
• A later fix for that (per-column assignment via DataFrame.iloc) still r...

v0.3.0: Hardening pass

Added

• Documentation site (MkDocs Material), deployed to GitHub Pages
• docs/filtered-sampling.md and docs/recipes.md, replacing the old standalone guide and example scripts
• GitHub Actions CI: Ruff + Pyright, tests on Python 3.10 and 3.12, and a package-build check on every push/PR to main
• Pre-commit hooks: Ruff on every commit, Pyright and pytest on every push
• Unit test coverage for filter.py, samplers.py, indexer.py, and rng.py, including the reservoir-sampling and filter AND/OR DSL logic, which previously had none
• Community health files: Code of Conduct, Contributing guide, Security policy (private vulnerability reporting), issue templates, pull request template
• World-on-anvil emblem as the project's brand mark
• PyPI-readiness metadata in pyproject.toml (authors, keywords, classifiers, project URLs)
• hatch-vcs-based versioning

Changed

• Relicensed from MIT to Apache License 2.0
• README's "Known Limitations" and "Roadmap" sections consolidated into a single docs-site page instead of two independently-drifting copies
• scrape, convert, and filter now exit non-zero and report failed files instead of silently discarding partial failures

Removed

• filtered_sampling_guide.md, sample.example.sh, sample.example.cmd (content moved into the docs site)

Fixed

• CI silently installed the wrong Python version, since setup-uv has no python-version input (now passed via uv sync --python)
• Various Ruff and Pyright findings across the codebase
• _is_gdelt_dataset_file matched same-length non-ZIP files (e.g. 2020.csv) as monthly/yearly GDELT archives, since it only checked the digit-prefix length and never verified the .zip suffix
• Sample output (gdeltforge sample) was written directly to its destination path, so a process killed mid-write could leave a corrupt or empty file there with no indication anything was wrong; sample output and scraped downloads now write atomically via a temp file, and warn instead of silently overwriting a leftover incomplete file from a previous interrupted run

Full Changelog: v0.2.0...v0.3.0