Projectors

[MichaelZinsmaier]

KNIP Projetor and ScreenImage classes

see Issue #19

[tpietzsch]

The old XYProjector and XYRandomAccessibleProjector should be unified with the new projector hierarchy.
I think we should even get rid of the XY stuff altogether and replace everything with selectable dimensions like in Projector2D and DimProjector2D.
Could you create a benchmark to find out wether replacing the hardcoded 0, 1 by final members dimX, dimY has any performance impact? If not, we get rid of all XY (we can add additional convenience constructors to use dimX=0, dimY=1).

CompositeProjector and DimProjector should be unified.

It would be really nice if we could use this as an opportunity to clean up the display package instead of convoluting it even further.

[MichaelZinsmaier]

I tested Projector2D X=0 Y=1 vs XYProjector on some random images. On my machine they are equally fast

[dietzc]

any further comments? what needs to be done before merging?

[tpietzsch]

As I said before, it would be nice if the display package could be cleaned up. This would involve making XYProjector and XYRandomAccessibleProjector extend Projector2D and RandomAccessibleProjector2D (doesn't exist yet). Find a unifying solution for CompositeXYProjector, DimProjector2D maybe involving Saalfelds new Composite stuff (I had hoped that Stephan would comment here, but I guess thats not going to happen). Move the more general Converters to the net.imglib.converter package (or a subpackage) because they are not display specific. Maybe get rid of the unused generic parameters in the Projector interface.

I cannot provide detailed step-by-step instructions on what I would like to see done exactly, sorry. In the time that would require me to figure out and describe , I could also do it myself. I'm not using the projectors at the moment and anyway I think the display package is a bit of a mess right now. So actually you could also just merge now and put this additional stuff in, it will not get worse by that.

[ctrueden]

Thanks for your comments, @tpietzsch. I agree that what is needed here is just another iteration of development according to the suggestions you have given, followed by another code review. It shouldn't be necessary to offer exact step-by-step instructions as these things sort of emerge on their own as you are coding. So anyway I think it would be worth it for @MichaelZinsmaier to take another pass at accomplishing some of these goals, as long as it is not too time consuming.

Note that GitHub has some great code review features: you can comment on PRs like we are doing now, but also on specific commits of a PR (or any commit, really), or even specific lines of a commit. So @tpietzsch, if you see specific places in the PR that you want to point out, you can do this.

[axtimwalde]

Sorry. I have commented on that in a different context but not executed the intended performance tests because I ran out of time due to other issues. Currently, I cannot do anything either. Regarding composites, my draft is available in two interactive imglib2-examples. They demonstrate how you can collapse a dimension into a Composite (check Views.collapse), and then use the basic Projector concept with a Converter to convert from a Composite to an ARGBType. My thinking is that this is a more elegant solution than special purpose CompositeProjectors but I haven't had the time yet to investigate potential performance penalties, nor is the design set in stone. Tobias has pointed out rightly that carefully reading, commenting, and discussing someone else's code can become similar or higher effort than writing it from scratch (even with GitHub's great annotation features). And, sadly, I currently do not have the time to review the display package. To not stop progress, I suggest to merge and expect it to be revised and broken any time in the future. Sorry, I know that this isn't very satisfying.

[ctrueden]

@axtimwalde: I agree that detailed review, especially of large patches, can take quite some time. And while code review improves the quality of the design, I think there is a balance. In this case I think @tpietzsch struck it right: he recommended some focused directions without getting into nitpicks of specific lines of code, so that @MichaelZinsmaier and @dietzc can do another design iteration. Personally I believe that faster iteration is superior to slower "high-quality" iteration.

I suggest to merge and expect it to be revised and broken any time in the future. Sorry, I know that this isn't very satisfying.

I think it is OK as long as ImgLib2 is still in beta. There will come a day when we need to commit to the API, though. (And even then, adding new API and deprecating old API is fine; we just have to be more careful about removing things and changing behavior.)

[dietzc]

We started the clean-up. Some comments:

1. We moved all converters moved to the converters package.
2. Unified hierarchy. We removed XY Projectors and replaced them with Projectors2D/AbstractProjector2D.
3. We extended Projectors2D to work on RandomAccessible source and RandomAccessibleInterval target. Therefore we were able to remove XYRandomAccessbileProjector. We ran the benchmarks and the results were nearly equal (+-1 ms) to the results of XYRandomAccessibleInterval.
4. Volatile: We moved the VolatileTypes into types. Furthermore we moved all the related projectors to the package display.projectors.volatileprojectors.
5. CompositeProjectors: We moved the converters to converters. After short discussion with @axtimwalde we kept the CompositeProjectors in the package display.projectors.composite. Open points: Compare existing implementations (Views.collapse vs. CompositeProjectors) and decide which should be kept. We will open an issue, so that we won't forget to do this.
6. There are several XY Projectors which we simply modified, such that they fit into the Projector2D hierarchy. They still behave exactly as before. However, they could be re-factored to work on arbitrary dimensions (not only XY i.e. 0,1).

If you have any further comments, suggestions or anything else, we would be happy to further improve the package structure. Afterwards we would suggest to merge :-)

Cheers,
Christian & Michael

[ctrueden]

@tpietzsch: Thanks for the suggestions.

• Item (1) was complete by the time you commented. 😉 (Unfortunately we cannot call a package volatile because it is a Java reserved keyword, so we settled on the slightly lamer volatiles.)
• Item (2) is addressed in the last few commits above. Turns out UnsignedByteScreenImage came before the others (was written by @axtimwalde rather than @MichaelZinsmaier), and is essentially identical functionality to Michael's ByteScreenImage but using much more code. So we just removed it, then generalized the remaining ArrayImg-based AWTScreenImage implementations to support eight common types instead of just two.

[ctrueden]

@dietzc: Just a reminder that apart from the FlatIterationOrder improvements, there is one other thing we might want to add to this branch before merging: a utility method somewhere for selecting the correct ArrayImgAWTScreenImage implementation from a given type and/or ArrayImg.

[tpietzsch]

I just noticed that the new Projector2D is 25% slower on real-transformed images than the old XYRandomAccessibleProjector. Unfortunately XYRandomAccessibleProjector was removed by this PR without a proper replacement.

I think this PR was merged prematurely and maybe a bit carelessly (for example, how did this happen? : https://github.com/imglib/imglib-tests/blob/master/src/test/java/tests/XYRandomAccessibleProjector.java)

I would like to reinstate the old XYRandomAccessibleProjector until somebody has time to make a proper RandomAccessibleProjector2D? What do you think @dietzc, @MichaelZinsmaier, @ctrueden?

[dietzc]

Hi Tobias,
could you provide the benchmark you were using? is the Projector2D is just slower on real-transformed images? Anyway: of course you can add the XYRandomAccessibleProjector again. But it seems that the "map" logic of the XYRandomAccessibleProjector could realtively easy be integrated in the Projector2D? I can try this tomorrow.

[dietzc]

why to wait until tomorrow: I just created a new branch with a few commits (https://github.com/imglib/imglib/tree/p2d-improvements). I added a new RandomAccessibleIntervalProjector2D and renamed Projetor2D to IterableIntervalProjector2D.

These implemenations are not benchmarked and not tested yet. I can/will do this tomorrow. if you want me to benchmark certain cases, you can send me your benchmark.

[dscho]

Benchmarks that are made part of the unit tests, in particular when they mark performance regressions as failures, would have caught this regression.

Maybe it would be a good idea to prevent the same thing from happening by adding proper benchmark tests.

[tpietzsch]

@dietzc I will put a benchmark on the p2d-improvements branch and let you know. I played around a bit yesterday with the projector used in ImageJVirtualStack (I was using that and it seemed slow which is why I started looking into this). It turned out that also when just projecting from an ArrayImg XYRandomAccessibleProjector only took 2/3 of the time Projector2D took. This seems a bit unintuitive, so maybe it's caused by something else that confused the JIT.

@dscho True. I just saw your comment imagej/imagej-ops#6 on junit-benchmarks. I didn't know about junit-benchmarks, I'll have a look!

[tpietzsch]

@dietzc I put a benchmark in the p2d-improvements branch: https://github.com/imglib/imglib/blob/ea6992e0e51d52f323214e904715020472c21629/realtransform/src/test/java/net/imglib2/display/ProjectorBenchmark.java

Results for me are:
When each test is run individually, median / best times over 100 runs are

IterableIntervalProjector2D Img
36 / 35 ms
RandomAccessibleProjector2D Img
25 / 24 ms
XYRandomAccessibleProjector Img
25 / 24 ms
IterableIntervalProjector2D Affine
285 / 269 ms
RandomAccessibleProjector2D Affine
279 / 263 ms
XYRandomAccessibleProjector Affine
272 / 259 ms

When running all tests in sequence (causing JIT deoptimization), median / best times over 100 runs converge to

IterableIntervalProjector2D Img
103 / 98 ms
RandomAccessibleProjector2D Img
32 / 30 ms
XYRandomAccessibleProjector Img
30 / 29 ms
IterableIntervalProjector2D Affine
288 / 275 ms
RandomAccessibleProjector2D Affine
279 / 269 ms
XYRandomAccessibleProjector Affine
264 / 248 ms

Your RandomAccessibleProjector2D is almost as fast as XYRandomAccessibleProjector.
With your modifications (and my bugfix :-) IterableIntervalProjector2D is not so bad on the real-transformed image anymore. However, interestingly, on the ArrayImg it is still worse. Also it takes a bigger hit from JIT deoptimization if projector.map() calls are polymorphic, making it 3 times slower than the RandomAccesibleProjector. I don't know why this happens. Maybe because the map() method of IterableIntervalProjector2D is quite a bit bigger

[dietzc]

@tpietzsch I found one more thing to optimize in the IterableIntervalCursor.

We used localizingCursors even if we didn't need them. I will commit, run benchmark and post them here. Give me 30min!

and thanks for the bug-fix. didn't test it yet ;-)

[dietzc]

@tpietzsch i pushed the modifications on IterableIntervalProjector2D.

Here are my benchmarks when each test is run individually (100times):

IterableIntervalProjector2D Img
median: 21 ms
best: 20 ms

RandomAccessibleProjector2D Img
median: 24 ms
best: 23 ms

XYRandomAccessibleProjector Img
median: 23 ms
best: 22 ms

IterableIntervalProjector2D Affine
median: 243 ms
best: 242 ms

RandomAccessibleProjector2D Affine
median: 254 ms
best: 252 ms

XYRandomAccessibleProjector Affine
median: 255 ms
best: 252 ms

---

Sequence:
IterableIntervalProjector2D Img
median: 21 ms
best: 20 ms

IterableIntervalProjector2D Affine
median: 258 ms
best: 247 ms

RandomAccessibleProjector2D Img
median: 24 ms
best: 23 ms

RandomAccessibleProjector2D Affine
median: 272 ms
best: 269 ms

XYRandomAccessibleProjector Img
median: 22 ms
best: 22 ms

XYRandomAccessibleProjector Affine
median: 248 ms
best: 245 ms

can you confirm these results? what do you think?

@ctrueden while actually hacking on this, I realized, that this is also a very nice case where we could use scijava-commons: If there exists a very fast implementation to render this kind of image (e.g. Plane-Wise ArrayImg of type ByteType, see ArrayImgXYByteProjector) then use this projector. Pretty much reminds me of imagej-ops, doesn't it?

[tpietzsch]

@dietzc I can confirm your results. Very nice! Thanks a lot!
For me now SNAPSHOT versions imglib-tests and imglib-tutorials don't compile because they have the old Projector2D still. Could you please commit those? (Your eclipse probably refactored those and you just didn't commit). Then I would clean up and merge to master

@ctrueden @dietzc Wrt. to scijava-commons, it would be educational to see how well this would work in this case. If one of you finds time to give it a try, I would be very interested.

[dscho]

As pointed out multiple times over the past years: median is not the correct measure here, unless you really want to benchmark the run time of the algorithm together with delays caused by your mail client, your web browser, your disk cache, etc.

[dscho]

imglib-tests and imglib-tutorials live in their own repositories now, as agreed upon some month ago. They should depend on release versions of imglib so that we get closer to reproducible builds that @axtimwalde asked for all these years ago.

[dscho]

Okay, I was stupid and indeed dropped everything I really have to do in favor of fixing the imglib-tests and imglib-tutorials projects. But they are already fixed! They already depend on (beta) release versions of imglib!

So @tpietzsch what am I missing???

[tpietzsch]

@dscho eventually the imglib-tests and imglib-tutorials will depend on the next (beta) release versions which have these refactorings. So why not commit to a branch now what @dietzc's eclipse probably did automatically already to save some work later?

[dscho]

@tpietzsch that does not explain why you had a problem with imglib-tests and imglib-tutorials.

[dietzc]

@tpietzsch I didn't have tests in my eclipse workspace, so eclipse unfortunately didn't do the job. Anyway: I don't see any compilation errors (just freshly imported tests as maven project to do the refactoring). Something is strange ;-))

[tpietzsch]

@dscho @dietzc I have locally modified the imglib-tests and imglib-tutorials projects to use the 2.0.0-SNAPSHOT version of imglib. That way, refactorings in one project carry over to the others automatically. I think it makes a lot of sense to do this. Otherwise the next poor soul who wants to bring those projects "up to speed" will need to dig through the git history to see what happened. Why do that, if you can have Eclipse do it right away... Probably it would be good to establish some kind of best pratice to do this, e.g., identically named branches on the other projects. What do you think?

Sorry for not being clear about what I meant earlier. There are no compile errors with imglib-tests and imglib-tutorials depending on the imglib beta. There will be problems when they are bumped to the next beta version that will include @dietzc's refactorings, and it would make that transition smoother by having the corresponding refactorings in branches in imglib-tests and imglib-tutorials already. Sorry about the confusion.

[dscho]

@tpietzsch okay, you can do that. But most likely others did not because -SNAPSHOT bindings bring with them a whole world of pain, as @axtimwalde explained patiently at the EuroBioImaging meeting in Barcelona 2012. So I doubt that @dietzc has done that (because it would distract him from the real goal). Therefore I think you'll have to fix that breakage yourself, or go back to release bindings.

[tpietzsch]

Okay, I can go back to release bindings then. My intention was just to make the transition process for the next releases of imglib-tests and imglib-tutorials as smooth as possible. But of course we can also fix that later. We all have too much time on our hands anyway.

[dietzc]

I'm a little confused at the moment, about what we actually have or want.
(a) We have imglib2-tests which depends on beta-releases of imglib2? Then we should do the refactoring later.
(b) we have imglib2-tests releases which depend on beta-relesaes of imglib2. if the imglib2-tests snapshot itself depends on imglib2-snapshot, then we should do the refactoring now
(c) we have imglib2-tests which only depends on imglib2-snapshot and they are released together (which is doubt), then we should do the refactoring now.

It's not about what I want to do, I simply don't know which option is correct? Option (b) makes sense. @dscho, @tpietzsch, @ctrueden, @axtimwalde any suggestions? (keep it simple as the problem is simple ;-)).

[ctrueden]

My vote is to worry about imglib2-tests et. al later. No one consumes this artifact, and it doesn't ship with Fiji; to put it bluntly, it isn't really important. (To a lesser extent, this is also true of imglib2-scripting.)

We are moving away from multi-module releases versioned together, toward each component having its own version. This means that in the future, e.g. imglib2-tests may have a different number of releases than, and be at a different version number than, say, imglib2 core. This is OK, and means it becomes much simpler to make new releases of only a component that changed, rather than having to cut a release of the entire imglib2 stack when most of the components had no changes.

The rule of thumb (which we may not follow 100% in all cases, but it is a general guideline) is that each .git repository has one or more artifacts which are versioned together. It is possible that ultimately the imglib.git itself may not even be a multi-module build anymore, if we get to the point where every component lives in its own git repository. I guess it will depend on how intimately tied together some of the components are with one another.

The development pattern has been that during the early rapid development phase, we have more modules in a multi-module build a la imglib.git and imagej.git, and over time as things become more stable, pieces of those repositories are split out into separate git repositories which become versioned separately.

Regarding imglib2-tests specifically: note that it contains only manual tests and examples, as opposed to unit tests. My preference would be for any critical functionality being tested by imglib2-tests to be converted into unit tests, so it gets tested automatically by Jenkins with every build. And for any "demo" or "example" type code in there to be converted into imglib2-tutorials and documented clearly for the community. If we did that in every case, we could actually get rid of imglib2-tests completely. Failing that, the remaining imglib2-tests could become "integration tests" which are handled by either maven-failsafe-plugin or maven-invoker-plugin, and run automatically by Jenkins, perhaps on a daily basis. But still in that case we should consider moving them into the most relevant other imglib component, rather than leaving them separate as they are now.

If it helps I could write this stuff up on the SciJava wiki somewhere. The development process is still evolving as the SciJava stack becomes more mature, so it's not like we have all the answers in the bag. We're just trying to do what works and will continue to work as time goes on. If anyone disagrees with this direction, we can certainly discuss and hash out a better way.

[ctrueden]

@ctrueden while actually hacking on this, I realized, that this is also a very nice case where we could use scijava-commons: If there exists a very fast implementation to render this kind of image (e.g. Plane-Wise ArrayImg of type ByteType, see ArrayImgXYByteProjector) then use this projector. Pretty much reminds me of imagej-ops, doesn't it?

@dietzc Good idea. I filed #66 so we could pursue it later as time allows.

[ctrueden]

A couple other things about the release bindings between components at different versions:

1. I know it is a pain when you are developing two or more components in tandem, and agree with @tpietzsch that locally it helps to do SNAPSHOT couplings and let Eclipse refactor stuff for you when possible.
2. We want the pom-scijava parent to be the master resource documenting which versions of which components are recommended for use (and tested) together. Jenkins will do this for us, by actually ensuring all those components at all those versions build against one another (including all unit tests passing of course). I am working on a new script which will allow Jenkins to do this. With such a thing in place, and all our components declared in pluginManagement of pom-scijava with version properties, releases will become dirt simple and we'll have the peace of mind that everything still works together afterwards every time. I.e.: this will catch breakages like API skew between, say, imglib core and imglib2-tests.

Eventually we will definitely produce a more formal write-up of these details, once this "QA" process is fully in place.