Dynamically Stitching images

[MichaelGofron]

Hello foundry,

Thank you very much for the image stitching code. It is informative and works on the provided images.

I am having difficulty, however, with dynamically taking photos and stitching them together. When I take two images using my iphone camera and pass those images as the imageArray in CVViewController, it tries to stitch the images but returns a null image:

2015-06-04 11:18:30.312 CVOpenStitch[2631:498516] stitchedImage (null)
2015-06-04 11:18:30.313 CVOpenStitch[2631:498516] scrollview contentSize {0, 0}

I'm not sure why it is incapable of stitching the images together.

1. Do the images have to have pixel perfect alignment?
2. Do I need to identify features before sending it into the stitching algorithm?
3. Having looked at the stitching pipeline I'm not sure where the identifying features part of the algorithm is in your source code. Could you point that out?

-Thank you very much for your time
Michael Gofron

[foundry]

Michael, i have looked at your demo project. There are at least two issues causing you problems

• the image size from the camera is too large, and this is causing memory issues in the stitching pipeline.
• there is an issue with writing the images to the filesystem and reading them back. I can't pinpoint the exact issue but this write/read is giving you back images that break openCV. (By the way you are writing as PNG not JPEG, although that is not the root cause of your problem).
  If you make a test by (a) reducing the image size (b) keeping a reduced size image array in memory and passing those into openCV - then you will find the stitching process works.

I found your project fairly confusing, you should try to clean it up and simplify it. Your storyboard view controller instantiation is not correct for example (get rid of those UINavigationControllers, you don't need them).

In answer to your questions...

1. Alignments do not have to be pixel perfect, the stitching pipeline includes registration

2. You do not have to identify features before passing in to the stitcher, that is also part of the stitcher's internal pipeline

3. I am using the highest-level stitch algorithm offered by openCV. The magic happens here:

   Stitcher stitcher = Stitcher::createDefault(try_use_gpu);
   Stitcher::Status status = stitcher.stitch(imgs, pano);
   

The pipeline is detailed here: http://docs.opencv.org/modules/stitching/doc/introduction.html

As I am using the highest-level stich method, I am relying totally on openCV's default pipeline to deliver the resulting stitch.

As the doc says:

It’s possible to use this class without being aware of the entire stitching pipeline. However, to be able to achieve higher stitching stability and quality of the final images at least being familiar with the theory is recommended

It is possible to treat parts of the pipeline separately to get a higher degree of control over the process. The methods are all documented here.

[MichaelGofron]

Thank you very much for your answers foundry. I appreciate the time and thought you put in.

I have just tried compressing the images to 720 by 960 in order to have the images be the same resolution as the ones in the original project. I tested the project on an Iphone 4 and Iphone 6. While this ensured that memory warnings did not show up in the Iphone 4, it appeared to have not stopped the issue of the images not stitching properly.

I believe your second point is of note: there must be some issue saving or loading the images.

In my CameraViewController I save the images to my camera roll and to a location in the file system. When I pass those images in the file system to the CVViewController as the array of UIImages in my file system, the stitching method does not work. However, when I take the images from my camera roll and download them to my computer and load them into my project through the copy bundle resources section, the stitching algorithm can easily stitch any of the images I put (even the ones at 2448 by 3264). The only major difference between the two images is when I log their contents appears to be the imageAsset section (the top one is dragged into the project and copy resource bundle, the bottom one is loaded from the stored files):

Perhaps the problem might be associated with the fact that I used imageWithContentsOfFile as opposed to imageNamed. imageNamed caches the image which might result in significant performance boost if the stitching algorithm references the image often?

I have several more questions as a result

1. Does the algorithm require the images to be added to the project before hand or should it work with references to the UIImages stored in the file structure (I would assume the later).
2. What do you think might be causing this inability to stitch images programmatically stored and loaded from the file system.

Again I would like to thank you very much for your time and effort.

[MichaelGofron]

I believe I've managed to resolve the issue!

It was a foolish error on my part (as these things tend to be).
I was saving all the images to the same location so that the I was essentially passing to the stitching algorithm the same image. I have now fixed it and it can now start stitching different images.

I want to thank you again sincerely for your time foundry. Your comments helped me to start looking in the right direction

[foundry]

Hi Michael. I'm glad you found a solution. By the way, I just noticed you posted a question on Stack Overflow which you self-answered. You should accept your reply so the question is closed. Good luck with openCV!

[foundry]

Michael, I have at last managed to diagnose the real cause of this issue. Images taken natively with the iOS cameras are not stored in the correct orientation (they are rotated during display only). For a fix, I have added a category on UIImage to check the imageOrientation flag and if necessary rotate the pixel matrix of the image correctly before passing to openCV. See also this issue.

[taidypanda]

@foundry

the image size from the camera is too large, and this is causing memory issues in the stitching pipeline.

I have test four images with 3200*2400 , and use
Stitcher::createDefault stitcher.stitch(imgs, pano)
to get the final image, and the ios device had crashed, because of memory warning. Memory usage reaches 180%, the 600M+ memory usage
I don't know how to reduce it, and the max memory usage