Improve TIFF performance

[hinerm]

The current TIFFFormat is a direct port of the loci.formats TIFFFormat. Part of the goal for SCIFIO is to improve this performance.

As a part of this, the MinimalTIFFFormat should be restructured into just base components, and not be a format by itself.

[dscho]

Now that scijava-common has a Timing class, I suggest this strategy when trying to figure out what code needs the most improvement:

final Timing timing = Timing.start(true);
    ...
Timing.tick(timing);
    ...
Timing.tick(timing);
    ...
Timing.stop(timing);

This will accumulate nano-second timings (as precise as the platform allows, anyway) with only minimal performance impact. The stop method will sort the individual steps by time and offer the file name and line number to you (very convenient in Eclipse's console...) The reason for the true is that you can easily switch off the timing in the method by prefixing the true with an exclamation mark, both to unclutter the console and to avoid impacting performance by the benchmarking itself.

[hinerm]

Possible place to improve performance:

• tiff service for determining tiff type
• create a sub-plugin "TiffFormat"
• read the IFD once, pass it around to different checkers..

[ctrueden]

@hinerm: I agree, we need a TIFFService which can cache IFDs. See also issue #39.

[dietzc]

@gab1one is taking a look at the performance of TIFF reader / writer and is posting his results here for further discussion.

[hinerm]

This gist can be loaded into a reader, e.g. Eclipse Java Monitor, to view profiling results that clearly indicate TIFF is terrible (e.g. spending 1/3 of its time determining the format)

[hinerm]

Top offenders that profiling identified:

• TiffFormat.Parser.initMetadata > Location.list 41%
• AbstractChecker.isFormat.RandomAccessInputStream.<init> 13%
• AbstractFormat.createChecker 8%
• MicroManagerFormat.isFormat 8%

Lesser offenders but kind of suspicious:

• MasterFilterHelper.<init> 2%
• JPEGFormat.isFormat 2%
• DICOMFormat.isFormat 2%

[hinerm]

OK so I made a mistake in that I was only profiling io.scif.* packages. After adding net.imagej.* and org.scijava.* It now seems like an overwhelming amount of time is spent on context injection:

17KB TIFF

Typed parsing

CreateChecker

IsFormat

Reading (less than 3%)

This doesn't 100% tell us that the API is being used correctly (e.g. calling Context.inject when it is actually needed).

But my next steps are to profile with a naive caching mechanism, e.g. mapping classes to their field annotations.

If that proves to be successful then my thoughts for future steps would be incremental Context initialization (recursive pre-parsing and caching of fields and anything else..) that runs on a dedicated thread after a Context is created. This would also require a general caching service, e.g. a LRU cache.

[hinerm]

When testing with a larger image, of course, the Context initialization impact is negligible:

200MB OME-TIFF

[hinerm]

So there are several scenarios to consider with regard to performance here:

• Rapidly opening small files. See Cache discoverable items that do not change scijava/scijava-common#141
• TIFF IFD parsing
• MinMax computation. See Improve MinMaxFilter performance #249

[hinerm]

@dietzc ea52f8b is awesome for repeated reads of small files

[hinerm]

• File grouping settings. See Should files be grouped by default? #253

[hinerm]

Note that #203 is also relevant here as having a higher priority ImageJ (and/or SCIFIO) TIFF Format could also improve performance in many cases

[hinerm]

See also #342

[hinerm]

See also #310