ingest: NotImplementedError when files are removed from a dataset directory

[lewisjared]

Summary

ref datasets ingest --source-type cmip6 raises NotImplementedError("Removing files is not yet supported") from register_dataset (climate_ref/datasets/base.py:335) whenever any dataset in the input tree has fewer files on disk than the DB has tracked for it. This aborts the whole ingest run; no progress on the rest of the catalog is committed.

Reproduction

1. Ingest a CMIP6 tree once: ref datasets ingest --source-type cmip6 /data/cmip6 (succeeds, registers N files for some dataset).
2. Remove or rename one of the .nc files belonging to that dataset on disk.
3. Re-run the same command.

Expected: the dataset is reconciled to match disk (file dropped from the registry), or at least the run continues for the rest of the catalog and the failing dataset is reported and skipped.

Actual: the run aborts on the first dataset where files_to_remove is non-empty.

Fixes

I'm not actually sure about the right behaviour here? Does this dirty any execution groups that uses this dataset?

[lewisjared]

Update: this fires on a fresh, empty database too, not just stale state.

Reproduced by running the weekly-reset workflow that:

1. Wipes the REF state dir (DB + caches)
2. ref db migrate
3. ref providers setup --skip-data --skip-validate
4. ref -v datasets ingest --source-type cmip6 --chunk-size 1000 /data/cmip6

Stats right before the crash (clean DB, no prior data):

IngestionStats(datasets_created=2302, datasets_updated=0, datasets_unchanged=0,
               files_added=62021, files_updated=0, files_removed=...)

Traceback identical: register_dataset → NotImplementedError("Removing files is not yet supported") after a WARNING climate_ref.datasets.base - Files to remove: [...].

Hypothesis: --chunk-size aligns to directories but a CMIP6 dataset (instance_id) can span chunk boundaries. Chunk A registers the dataset with some files, chunk B later contains the rest → register_dataset diffs new-chunk files against the already-persisted set and treats the absent-from-this-chunk files as removals.

If that's right, --chunk-size is unsafe whenever a CMIP6 dataset's files don't all land in the same chunk — i.e. fairly often, since per-dataset file counts vary and chunks are size-bounded. The dataset that fails on my data has been CMIP6.DAMIP.IPSL.IPSL-CM6A-LR.hist-GHG.r1i1p1f1.* both times.

Suggested fixes (any one works):

1. register_dataset should diff against the input chunk's known dataset members, not the full DB row, when the dataset is mid-stream. Or persist a dataset.streaming marker so removals from chunks aren't treated as authoritative.
2. Group rows by instance_id (or whichever column defines a CMIP6 dataset) when building chunks, so a single dataset never spans two chunks.
3. As a safety net, when chunk_size > 0, only treat removals as removals during the final chunk's pass, or require an explicit ref datasets prune step.