Hdf5 Cache #221
Hdf5 Cache #221
Conversation
…py-backed cache. We should probably unify the implementations into a single class.
It would delete the first element of cache when hdf5CacheGrp connected. Changing input can invalidate the cache, changing the cache does not.
…elism again New locks are lighter weight (only one real lock; list of blocked writes doesn't grow beyond size of in-flight requests) Some TODOs remain to test and fix, but they work for now
…xport tweaked typo in assert in rtype to test stop (not just start twice)
chaubold
force-pushed
the
master
branch
2 times, most recently
from
ed24eb1
to
9b5a9f0
Compare
|
in general I can see a |
This PR provides support for ilastik PR 1365 Blockwise Carving [https://github.com/ilastik/ilastik/pull/1365] and ilstiktools PR 4 Blockwise Carving [https://github.com/ilastik/ilastiktools/pull/4]. There were expensive calculations, done in parallel, that wouldn't all fit in memory; hence the file cache. Changes were made to original opBlockedHdf5Cache by Stuart to more simply track which blocks were already calculated (as I recall). If there is interest in reviving these three PRs, I can update them to the latest versions of ilastik / lazyflow / ilastiktools, and provide rational for the changes (it's from over two years ago, so I'll have to review them myself too). Hope that helps. |
This patch is based on h5-cache, but updated to sync with master. The primary changes were made to opUnblockedHdf5Cache, making the locking simpler, and making propagate dirty work correctly. It works with the ilastk/blockwise_carving patch.