/
pano.txt
219 lines (155 loc) · 7.17 KB
/
pano.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
Panorama stitching
==================
This section gives a step-by-step outline of how to perform panorama stitching
using the primitives found in scikit-image. The full source code may be found
at https://github.com/scikit-image/scikit-image-demos.
0. Data loading
---------------
The ``ImageCollection`` class provides an easy way of representing multiple
images on disk. For efficiency, images are not read until accessed.
.. code-block:: python
from skimage import io
ic = io.ImageCollection('data/*')
.. figure:: pano_4_0.png
Credit: Photographs taken in Petra, Jordan by François Malan. License:
CC-BY. :label:`petra`
Figure |nbsp| :ref:`petra` shows the Petra dataset, which displays the same
facade from two different angles. For this demonstration, we will
estimate a projective transformation that relates the two images. Since the
outer parts of these photographs do not comform well to such a model, we select
only the central parts. To further speed up the demonstration, images are
downscaled to 25% of their original size.
.. code-block:: python
from skimage.color import rgb2gray
from skimage import transform
image0 = rgb2gray(ic[0][:, 500:500+1987, :])
image1 = rgb2gray(ic[1][:, 500:500+1987, :])
image0 = transform.rescale(image0, 0.25)
image1 = transform.rescale(image1, 0.25)
1. Feature detection and matching
---------------------------------
"Oriented FAST and rotated BRIEF" (ORB) features [ORB]_ are detected in both
images. Each feature yields a binary descriptor; those are used to find the
putative matches shown in Figure |nbsp| :ref:`putativematches`.
.. code-block:: python
from skimage.feature import ORB, match_descriptors
orb = ORB(n_keypoints=1000, fast_threshold=0.05)
orb.detect_and_extract(image0)
keypoints1 = orb.keypoints
descriptors1 = orb.descriptors
orb.detect_and_extract(image1)
keypoints2 = orb.keypoints
descriptors2 = orb.descriptors
matches12 = match_descriptors(descriptors1,
descriptors2,
cross_check=True)
.. figure:: pano_11_1.png
Putative matches computed from ORB binary features.
:label:`putativematches`
2. Transform estimation
-----------------------
To filter the matches, we apply RANdom SAmple Consensus (RANSAC) [ransac]_, a
common method for outlier rejection. This iterative process estimates
transformation models based on randomly chosen subsets of matches, finally
selecting the model which corresponds best with the majority of matches. The
new matches are shown in Figure |nbsp| :ref:`matchesfiltered`.
.. code-block:: python
from skimage.transform import ProjectiveTransform
from skimage.measure import ransac
# Select keypoints from the source (image to be
# registered) and target (reference image).
src = keypoints2[matches12[:, 1]][:, ::-1]
dst = keypoints1[matches12[:, 0]][:, ::-1]
model_robust, inliers = \
ransac((src, dst), ProjectiveTransform,
min_samples=4, residual_threshold=2)
.. figure:: pano_15_1.png
Matches filtered using RANSAC. :label:`matchesfiltered`
3. Warping
----------
Next, we produce the panorama itself. The first step is to find the shape of
the output image by considering the extents of all warped images.
.. code-block:: python
r, c = image1.shape[:2]
# Note that transformations take coordinates in
# (x, y) format, not (row, column), in order to be
# consistent with most literature.
corners = np.array([[0, 0],
[0, r],
[c, 0],
[c, r]])
# Warp the image corners to their new positions.
warped_corners = model_robust(corners)
# Find the extents of both the reference image and
# the warped target image.
all_corners = np.vstack((warped_corners, corners))
corner_min = np.min(all_corners, axis=0)
corner_max = np.max(all_corners, axis=0)
output_shape = (corner_max - corner_min)
output_shape += np.abs(corner_min)
output_shape = output_shape[::-1]
The images are now warped according to the estimated transformation
model. Values outside the input images are set to -1 to distinguish the
"background".
A shift is added to ensure that both images are visible in their
entirety. Note that ``warp`` takes the inverse mapping as an input.
.. code-block:: python
from skimage.color import gray2rgb
from skimage.exposure import rescale_intensity
from skimage.transform import warp
from skimage.transform import SimilarityTransform
offset = SimilarityTransform(translation=-corner_min)
image0_ = warp(image0, offset.inverse,
output_shape=output_shape, cval=-1)
image1_ = warp(image1, (offset + model_robust).inverse,
output_shape=output_shape, cval=-1)
An alpha channel is added to the warped images before merging them into a
single image:
.. code-block:: python
def add_alpha(image, background=-1):
"""Add an alpha layer to the image.
The alpha layer is set to 1 for foreground
and 0 for background.
"""
rgb = gray2rgb(image)
alpha = (image != background)
return np.dstack((rgb, alpha))
image0_alpha = add_alpha(image0_)
image1_alpha = add_alpha(image1_)
merged = (image0_alpha + image1_alpha)
alpha = merged[..., 3]
# The summed alpha layers give us an indication of
# how many images were combined to make up each
# pixel. Divide by the number of images to get
# an average.
merged /= np.maximum(alpha, 1)[..., np.newaxis]
The merged image is shown in Figure |nbsp| :ref:`merged`. Note that, while
the columns are well aligned, the color intensities at the boundaries are not
well matched.
.. figure:: pano_23_0.png
The second input frame (middle) is warped to align with the first input
frame (left). The averaged image is shown on the right. :label:`merged`.
4. Blending
-----------
To blend images smoothly we make use of the open source package Enblend
[Enblend]_, which in turn employs multi-resolution splines and Laplacian
pyramids [burt_adelson_0]_, [burt_adelson_1]_. The final panorama is shown in
Figure |nbsp| :ref:`pano`.
.. figure:: pano_28_0.png
The final panorama image, registered and warped using scikit-image, blended
with Enblend. :label:`pano`.
.. [ORB] Ethan Rublee, Vincent Rabaud, Kurt Konolige and Gary Bradski "ORB: An
efficient alternative to SIFT and SURF". Proceedings of the 2011
International Conference on Computer Vision (ICCV), pp.2564-2571.
doi: 10.1109/ICCV.2011.6126544
.. [ransac] Martin A. Fischler and Robert C. Bolles. "Random Sample Consensus:
A Paradigm for Model Fitting with Applications to Image Analysis and
Automated Cartography". Comm. of the ACM 24 (6): 381–395, June 1981.
doi:10.1145/358669.358692
.. [burt_adelson_0] P. Burt and E. Adelson. "A Multiresolution Spline With
Application to Image Mosaics". ACM Transactions on Graphics, Vol. 2, No. 4,
October 1983. Pg. 217-236.
.. [burt_adelson_1] "The Laplacian Pyramid as a Compact Image Code".
IEEE Transactions on Communications, April 1983.
.. [Enblend] Enblend 4.0 documentation, August 2010,
http://enblend.sourceforge.net.