fix: replace diskcache with atomic file-per-key cache#130
Merged
Conversation
…butions diskcache (SQLite-backed) doesn't scale under heavy concurrent access. With 200+ parallel Snakemake jobs in the JUMP pipeline, three failures: 1. sqlite3.OperationalError on Cache.__init__ — opening the cache runs PRAGMA and INSERT (writes), so 200 processes deadlock on init 2. diskcache.Timeout on cache.set() in multiprocessing workers 3. Increasing timeout / adding retries just shifted the bottleneck Root cause: SQLite is single-writer. WAL mode helps concurrent reads but every Cache() constructor is a writer. No configuration fixes this at 200+ processes — it's an architectural mismatch. File-per-key with atomic writes (tempfile + os.replace) eliminates the shared lock entirely. Reads are just np.load (zero coordination). Writes go through a temp file on the same filesystem and os.replace, which is atomic on POSIX — this prevents the corruption that cytomining#128 originally fixed without introducing a shared bottleneck. Tested: 124 parallel copairs jobs (cold cache), 68/68 unit tests pass. Removes diskcache dependency.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows up on #128, which introduced diskcache to fix write corruption from parallel workers. That fix was correct for the write-race problem, but diskcache uses SQLite internally, and SQLite doesn't scale under heavy concurrent access.
Problem
With 200+ parallel Snakemake jobs (JUMP Cell Painting pipeline), the shared SQLite database becomes a bottleneck. We hit three failure modes, each uncovered after fixing the previous one:
sqlite3.OperationalError: database is lockedduringCache.__init__- the constructor runsPRAGMAandINSERT INTO Settings, which are writes. 200 processes deadlock on init.diskcache.core.Timeoutoncache.set()inside multiprocessing workers - steady-state write contention.get/setin retries too: it worked, but fragile. Every cache operation needed error handling for a problem that shouldn't exist.SQLite is single-writer. WAL mode (which diskcache enables) allows concurrent readers, but every
Cache()constructor is a writer. No timeout or retry configuration fixes 200 concurrent writer-on-open sessions - it's architectural.Fix
Replace diskcache with file-per-key caching using atomic writes:
np.load(path)- no lock, no coordination, scales to any number of processestempfile.mkstemp+np.save(fd)+os.replace(tmp, path)-os.replaceis atomic on POSIX (same filesystem), so readers see either the complete old file or the complete new file, never a partial writeThis addresses the original corruption from #128 (atomic writes prevent truncation/partial reads) without introducing a shared bottleneck (no SQLite, no locks).
What changed
_cache_read/_cache_write: two small helpers, stdlib onlynull_dist_cachedandget_null_dists: use file helpers instead ofdiskcache.Cachediskcachefrom dependenciestest_null_dist_cached_corrupt(verifies graceful recovery from corrupted cache files)Test plan