Fs file caching (#10535) #1058
Fs file caching (#10535) #1058
Conversation
Reduce the number of locations that OMEROWrapper is created, passing directory when it is clearly server-side. Note: this currently does not compile because the OMEROWrapper lives in blitz while everything else lives in romio.
To simplify cleanup, having all caches written into a single
directory was added to the `loci.formats.Memoizer` class. Here,
hard-coded as `/tmp/memo`.
The various import related classes, howerver, are a tangle of
dependencies making it very difficult to not copy the same block of
code, e.g. the ReaderWrapper sequence and setting the proper
directory, throughout the code base.
By splitting OMEROWrapper into a CachingWrapper part in romio with
no dependencies on blitz we can mostly do this.
A few issues:
* now, client-side caching is disabled
* requires an insight dependency on romio
* enforces use of a wrapper through ome.io.bioformats
* allowed for preliminary, needed import class cleanup:
- use of `wrap()` is encouraged
- overriding `ReaderWrapper` method is discouraged
- discourage holding on to wrapper instances
|
Conflicting PR. Removed from build OMERO-merge-develop#233. See the console output for more details. |
Two branches both tried to refactor the copy-n-paste issues with reader creation. The combined solution means it's possible to override createBfReader, but by default a simple CachingWrapper will be returned. A rebase would have been unduly arduous since in the early commits of the fs-file-caching branch the CachingWrapper which would be needed wasn't present, making bissecting later impossible. Conflicts: components/romio/src/ome/io/nio/PixelsService.java
This reverts commit 6f69e38. Conflicts: components/romio/src/ome/io/bioformats/CachingWrapper.java components/romio/src/ome/io/nio/PixelsService.java
This reverts commit d37c4c7.
There is almost certainly more cleanup that can be done in terms of where the reader stack is constructed and how it is stored. This is at least a start, and is hopefully safe and minimally invasive. See #10750.
Memoizer refactoring
|
Likely last commits pushed to this branch and kryo-memoizer in bioformats, and testing locally. A quick validation should be possible by @pwalczysko tomorrow morning (or earlier if we re-launch gretzky). |
|
@joshmoore : just checked it, leica-scn, flex plate, svs - works fine - caching definitely in place and does what it should |
|
@pwalczysko: did you test locally? These changes haven't been deployed to gretzky yet. |
|
@joshmoore no I did not - will do |
|
@joshmoore my build with this PR fs-file-caching failed |
|
Yes, sorry. You'll need my |
This removes the last location with /tmp/memo hard-coded. Along with a change to Bio-Formats, caching will only occur when a directory is explicitly set. With the current setup, this means that caching does *not* occur during: * client-side import * server-side import * any where in Bio-Formats natively The only time memoization should occur is in the ome.io (i.e. romio) package during rendering and pixel reading.
|
@joshmoore : merged the kayo-memoizer branch into my components/bioformats and then merged fs-file caching branch into my test branch in my openmicroscopy folder and ran For the last couple of lines before the end see below: |
|
Did edit of etc/local.properties.example to include:
Works fine. |
|
Note: I updated my |
|
gh-1108 bumps the submodule pointer to include kryo-memoizer. |
Makes use of the new
loci.formats.Memoizerclass to cache all calls toIFormatReader.setId()on the server-side. The intent is that quick calls to the server from web (panning over a large image, etc) should incur lower cost per call.Memo files are created under
/tmp/memo(or similar), e.g.Loading the memo file is usually under 100ms, while calls to
setId()can be several seconds. (Memoizerskips caching whensetId()is faster than 100ms).Critical for testing is a breadth of file formats because the contents of each
Readermay or may not cause serialization issues. A single (unused) field in theOBFReader, for example, was able to crash the JVM.See:
--no-rebase FS only