GdeltForge 0.6.1
·
212 commits
to main
since this release
Fixed
filtersilently wrote nothing whenevercolumns_to_checkwas empty, exactly the bundled default config's own value for every dataset.filter_single_filetreated an emptycolumns_to_checkthe same as "columns were configured but none exist in this file's schema", logging an error and returning before ever writing the output file, while the batch summary still reported every file processed successfully with 100% retention.sampleandcrossrefthen correctly found nothing in the (empty) output directory, surfacing as confusing downstream errors with no indication the real failure was upstream and silent. The early-return now only fires whencolumns_to_checkis actually non-empty; an empty list falls through to the existing, already-correct no-op path (dropna(subset=[])is a documented pandas no-op)crossref --eventscrashed with a confusingArrowInvalid: ... magic bytes not founderror when pointed at a directory instead of a single file, if that directory contained aconvert/filter.doneresumability marker (<name>.parquet.done, a real sibling of the data by design in every one of their output directories).pd.read_parquetwas handing the raw path straight to pandas with no awareness of that convention.--eventsnow accepts a directory properly: every*.parquetfile in it is read and concatenated,.donemarkers and any other non-parquet sibling are never handed to the parquet reader