fix: implement soft-delete for ZipStore (issue #828)

[mohammadZuherJaserAsad]

Summary

Fixes #828 — implements soft-delete for ZipStore so that delete() and delete_dir() no longer raise NotImplementedError.

The ZIP format has no native entry-removal API (a known CPython limitation: python/cpython#51067). This PR uses the soft-delete approach suggested by @dcherian in the issue: overwrite the entry with an empty byte sentinel (b"") and filter it out in all read, exists, and list paths.

Changes

src/zarr/storage/_zip.py

• supports_deletes = True (was False)
• Added module-level _SOFT_DELETE_SENTINEL = b"" constant
• delete(key): writes sentinel to zip entry; no-op if key is absent
• delete_dir(prefix): collects live keys under prefix, calls delete() on each
• _get(): reads all bytes first; returns None if content equals sentinel
• exists(): returns False for entries whose content equals sentinel
• list(): deduplicates entries (duplicate names can appear after overwrites) and skips sentinel entries
• list_dir(): consumes filtered list() output for consistency

tests/test_store/test_zip.py

• 7 new targeted tests covering: basic delete, no-op cases, sibling-key safety, double-delete, delete_dir, and list() filtering
• test_api_integration updated: removed two pytest.raises(NotImplementedError) blocks that are no longer correct

tests/test_store/test_stateful.py

• Removed two pytest.skip(reason="ZipStore does not support delete") blocks

Design notes

• Why soft-delete? The only alternative that preserves persistence is rebuilding the entire zip on every delete, which is O(n) in archive size. The maintainer's own suggestion in the issue was the sentinel approach; prior PRs zipstore delitems via override #1184 and Implement soft delete for zip store #2838 validated it.
• Duplicate zip entries: When an entry is overwritten (including soft-deleted), namelist() returns the name twice. NameToInfo tracks the last-written entry, so reading always returns the sentinel. list() uses a seen set to yield each name at most once.
• Lock re-entrancy: delete_dir() holds the RLock while calling list_prefix() and then delete() (which also acquires it). This works because threading.RLock is re-entrant for the same thread.
• set_if_not_exists edge case: After a soft-delete the name is still in namelist(), so set_if_not_exists will not re-write it. This matches the documented semantics and existing zarr usage patterns.

Testing

pytest tests/test_store/test_zip.py -v
pytest tests/test_store/test_stateful.py -v -k "not slow_hypothesis"

[codecov]

Codecov Report

❌ Patch coverage is 44.18605% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.30%. Comparing base (b9d3964) to head (2b784fe).
⚠️ Report is 52 commits behind head on main.

Files with missing lines	Patch %	Lines
src/zarr/storage/_zip.py	44.18%	24 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4107      +/-   ##
==========================================
- Coverage   93.53%   84.30%   -9.24%     
==========================================
  Files          88       90       +2     
  Lines       11894    15415    +3521     
==========================================
+ Hits        11125    12995    +1870     
- Misses        769     2420    +1651     

Files with missing lines	Coverage Δ
src/zarr/storage/_zip.py	82.27% <44.18%> (-15.42%)	⬇️

... and 45 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[d-v-b]

@mkitti

[mkitti]

Soft delete is the best we can here I think. I have not looked closely at the implementation yet.

[mohammadZuherJaserAsad]

Hi @mkitti — following up since it's been about a month since this PR was opened. Since your last comment, the soft-delete approach has been redesigned: Slow Hypothesis CI now passes cleanly on the ZipInfo.comment-based marker (the earlier sentinel-value approach had a real collision bug that Hypothesis caught twice). The remaining failures are on the py3.12/py3.14 minimal-deps matrix in test_array.py, which looks unrelated to the files this PR touches. Would you or another maintainer have a chance to take a look when you get a moment? Happy to make any changes needed. Thanks for your time!

[mkitti]

Currently, the soft-delete mechanism is to add a new central directory entry_ with the sentinel marker in the zipinfo comment (the per-entry comment, not the archive comment). If a new entry is added of the same name is subsequently added after a soft delete, it becomes yet another entry in the file. Detecting the existence of a file thus requires the entire central directory to be read. Finding the first soft delete entry is not sufficient because there may be a subsequent entry indicating a new file has been added.

While my preference for a soft delete would be just to remove the entry from the central directory completely that would more of a hack. It is possible because infolist() returns a direct reference to the internal list structure (zipfile.filelist)
. One could modify that list in place and figure out how to rewrite the central directory.

There was an issue in the past where some zip clients might only look at the first entry of a zip file.

A recent development is that python/cpython#134627 has been merged on June 20th, 2026, a week before this pull request was created. The cpython pull request adds remove and repack methods.

The new zipfile.remove() implements my preference above by removing the entry from the central directory exactly.

Currently, the upstream pull request is not part of a tagged release. It would be part of c-python 3.16 at the earliest. For c-python versions that contain that feature we should probably just use zipfile.remove().

The question then is do we want to do something before that upstream release?

[mkitti]

https://github.com/danny0838/zipremove might be used to enable this feature before a Python release which contains zipfile.remove().