Perspective warp

[dschneiderch]

Describe your changes
Add function to fuse images that have different size/scale/proportions, e.g. from two (different) cameras in different locations . The function accepts 2 images and 4 pairs of points in order to compute a perspective transformation matrix so the images overlay nicely. I found this works much better than simple scale/rotate/translate https://cougphenomics.github.io/camera_overlays.html

Type of update
Is this a:

• New feature or feature enhancement

Associated issues
discussed in various forms here #615 and #634

[codecov]

Codecov Report

Merging #648 (3de53e0) into master (4b885cd) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##            master      #648   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files          129       130    +1     
  Lines         6120      6176   +56     
=========================================
+ Hits          6120      6176   +56     

Flag	Coverage Δ
unittests	100.00% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
plantcv/plantcv/transform/__init__.py	100.00% <100.00%> (ø)
plantcv/plantcv/transform/warp.py	100.00% <100.00%> (ø)
plantcv/__init__.py	100.00% <0.00%> (ø)

[dschneiderch]

@nfahlgren @HaleySchuhl The tests are failing on colorspaces() function which I didn't touch !?! It looks like it isn't importing the rgb2gray_* functions properly but i have no idea why.

[nfahlgren]

@dschneiderch I think I got it, it was a tricky one. The transform submodule is imported before the visualize submodule because visualize.colorspaces uses transform.resize_factor. The new transform.warp function now uses visulize.overlay_two_imgs. That's actually fine, but visualize.colorspaces also uses rgb2gray_hsv and rgb2gray_lab, which are imported after transform. So since transform imports from visualize I just had to import rgb2gray functions before transform. Clear as mud right? :)

[dschneiderch]

wow, well thanks. I would not have guessed that import order matters. that means there could be circular dependencies(!)

[nfahlgren]

@dschneiderch this is great! I just tested it on one of our old use cases, fitting a mask derived from an RGB image onto an image from our NIR cabinet. One little issue I ran into that I need to think about more is that our NIR images are 16-bit and the mask is 8-bit, so when they get overlayed in the debug image it looks like weird noise. I'm not sure that it needs to be fixed, I just converted the NIR image to 8-bit to get around it.

In thinking about this function and our existing crop_position_mask I'm wondering if we should consider deprecating crop_position_mask in favor of warp? I think this new approach is easier to use and is more likely to produce a good fit because it takes into account more than just scaling and translation. I can imagine a scenario where it's not possible to pick landmark points and crop_position_mask could still be used, but not sure that is going to be a common situation. What do @dschneiderch and @HaleySchuhl think?

[dschneiderch]

Thanks!
My first thought is to internally convert refimg to 8bit for the display but that might be problematic if you have really low values - for example from a bioluminescence image. Maybe we should convert img to the type of refimg? It'd be a little more user friendly.
From a theoretical standpoint warp is just a more advanced geometric transformation than scale and translate used in crop_position_translate (which I think could be vastly simplified by switching to opencv functions). What scenario were you thiking about not picking landmark points? You should be able to get the same results with warp as with crop_position_mask. if you know your cropping parameters x and y than you can use them to warp too. simplest case, the refimg coords are simply the image limits. I'd suggest labeling crop_position_mask as soft deprecated and point people towards warp. Hard for me to imagine a case when crop_position_mask is more useful than warp - maybe just easier to wrap your brain around what's happening.

[nfahlgren]

Yeah, just imagining a case where there's nothing in the background to use as anchor points. With crop_position_mask you could guess and check until you got the overlap the way you want but I think that would be harder with landmark points. But like I said, not sure that's a likely scenario and I do think warp is not only a more complete computational approach, but also easier if you have easy to identify landmark points (i.e. no guess and check necessary).

[dschneiderch]

You can always use the edges and center of the plant as landmarks and adjust from there, no? From: Noah Fahlgren <notifications@github.com> Sent: Friday, December 4, 2020 10:51 To: danforthcenter/plantcv <plantcv@noreply.github.com> Cc: Schneider, Dominik <dominik.schneider@wsu.edu>; Mention <mention@noreply.github.com> Subject: Re: [danforthcenter/plantcv] Perspective warp (#648) Yeah, just imagining a case where there's nothing in the background to use as anchor points. With crop_position_mask you could guess and check until you got the overlap the way you want but I think that would be harder with landmark points. But like I said, not sure that's a likely scenario and I do think warp is not only a more complete computational approach, but also easier if you have easy to identify landmark points (i.e. no guess and check necessary). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/danforthcenter/plantcv/pull/648*issuecomment-738955442__;Iw!!JmPEgBY0HMszNaDT!8Et2JjhyhD6mnmf44BO4y5QTin0XUGBAtVY_h9-zsyfZq66gdv2soSXnxWirN3WAEzlkZA$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/ABY5SZN47M73AHZXXTRQF4TSTEVSTANCNFSM4T2RLNXA__;!!JmPEgBY0HMszNaDT!8Et2JjhyhD6mnmf44BO4y5QTin0XUGBAtVY_h9-zsyfZq66gdv2soSXnxWirN3XlzS2S3g$>.

[nfahlgren]

True, that makes sense. So maybe we can mark crop_position_mask with a deprecation warning for the remainder of v3 and remove it in v4, if folks agree with that plan.

[HaleySchuhl]

That sounds like a good plan to me!

[nfahlgren]

Thanks @dschneiderch, this is a nice function to have integrated in!