Skip to content
This repository has been archived by the owner on Jan 8, 2021. It is now read-only.

Tiling performance #44

Merged
merged 8 commits into from
Jun 7, 2017
Merged

Tiling performance #44

merged 8 commits into from
Jun 7, 2017

Conversation

dgault
Copy link
Member

@dgault dgault commented Jun 5, 2017

Whats included in this PR:

  • Java code for tiling tests
  • Maven pom updates
  • Benchmark scripts updates
  • Jenkins scripts updates

What isn't included:

  • analysis scripts

Tests:
Large Image
Series 1 - Bazla-14-100-brain - 2015-06-19 23.34.11.ndpi
Starting with a tile size of 30720x25856, then dividing the tile x and y by 2 until tile size reaches 16x16.

Medium Image
Series 2 - Bazla-14-100-brain - 2015-06-19 23.34.11.ndpi
Starting with a tile size of 7680x6464, then subtracting 64 from the tile x and y until tile size reaches 16x16.

Manual Tiling
tubhiswt_C0_TP0.ome.tif
Starting with a tile of size 512x512, then dividing the tile x and y by 2 until tile size reaches 16x16. Writes each tile individually.

Auto Tiling
tubhiswt_C0_TP0.ome.tif
Starting with a tile of size 512x512, then dividing the tile x and y by 2 until tile size reaches 16x16. Writes a full plane with the auto tiling.

Available parameters:
iterations
input
output
results
tileXStart
tileYStart
tileXEnd
tileYEnd
tileOperator (either / or -)
tileIncrement (how much to divide or subtract tile size by)
series
autoTile

@@ -132,3 +132,48 @@ for %%T in (bbbc mitocheck tubhiswt) do (
install\bin\pixels-performance 1 !input! %outdir%\!test!-cpp.ome.tiff %resultsdir%\!test!-pixeldata-win-cpp-%%I.tsv
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned on the trello card I think here it would be useful to put the new logic into a separate batch file (e.g. tiling.bat), move the other logic into metadata.bat and pixeldata.bat, and then have jenkins-build.bat call tiling.bat. The common logic at the top of jenkins-build.bat can stay as it is--this is just splitting out the test invocation so we can select the tests we want to run for each benchmarking run; for tiling we don't need to run the other tests.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same applies equally to the Unix shell script.


FormatWriter writer = new OMETiffWriter();
OMEXMLMetadata writerMeta = new OMEXMLMetadataImpl();
writerMeta.setImageID("Image:0", 0);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section could all be replaced by conversion from CoreMetadata using MetadataTools. Not strictly necessary, but the helper will ensure the metadata contains the minimal necessary information.

System.out.print("pass " + i + ": init...");
System.out.flush();
OMEXMLMetadata meta = new OMEXMLMetadataImpl();
List<List<byte[]>> pixels = new ArrayList<List<byte[]>>();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this be better named planes, since it's a list of planes containing a list of tiles?

System.out.flush();
reader.setSeries(series);

List<byte[]> planes = new ArrayList<byte[]>();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this be better named tiles, since it's a list of tiles within a plane?

@sbesson
Copy link
Member

sbesson commented Jun 7, 2017

After pulling the latest Docker image including this PR (from http://hub.docker.com/r/snoopycrimecop/ome-files-performance) and executing, the benchmarking went through except for the tiling execution because of the file name

[sbesson@ome-c6220-1-4 ~]$ docker run --rm -it -v /scratch/tmp/data:/data snoopycrimecop/ome-files-performance:master_merge_trigger 
+ datapath=/data
+ binpath=/install/bin
+ resultpath=/data/results
+ outpath=/data/out
+ iterations=1
+ mkdir -p /data/results
...
+ for test in neff-histopathology tubhiswt
+ input=unknown
+ case "$test" in
+ input=/data/ndpi/neff-histopathology/Bazla-14-100-brain
+ - 2015-06-19 23.34.11.ndpi
/git/ome-files-performance/scripts/run_benchmarking: line 63: -: command not found
[sbesson@ome-c6220-1-4 ~]$
```

I am merging this PR as it contains the minimal code for the new tiling benchmark, builds correctly and unlocks future work. Next incremental step will be to fix the execution of the code and the execution scripts semantics as suggested  in https://github.com/openmicroscopy/ome-files-performance/pull/44#discussion_r120314320 and captured in https://trello.com/c/v0jZwWJv/13-scripts-and-naming. Minor comments about the variable names can also be fixed in a follow-up PR.

@sbesson sbesson merged commit c2bf0cc into ome:master Jun 7, 2017
@sbesson sbesson modified the milestone: 0.2.0 Jun 7, 2017
This was referenced Jun 13, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants