forked from udacity/CarND-Advanced-Lane-Lines
/
writeup.html
1669 lines (1481 loc) · 92.9 KB
/
writeup.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Advanced Lane Lines</title>
<!-- 2017-02-09 Thu 16:29 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<meta name="author" content="David A. Ventimiglia" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<style>@import 'https://fonts.googleapis.com/css?family=Quattrocento';</style>
<link rel="stylesheet" type="text/css" href="base.css"/>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
<script type="text/javascript" src="http://orgmode.org/mathjax/MathJax.js"></script>
<script type="text/javascript">
<!--/*--><![CDATA[/*><!--*/
MathJax.Hub.Config({
// Only one of the two following lines, depending on user settings
// First allows browser-native MathML display, second forces HTML/CSS
// config: ["MMLorHTML.js"], jax: ["input/TeX"],
jax: ["input/TeX", "output/HTML-CSS"],
extensions: ["tex2jax.js","TeX/AMSmath.js","TeX/AMSsymbols.js",
"TeX/noUndefined.js"],
tex2jax: {
inlineMath: [ ["\\(","\\)"] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"], ["\\begin{displaymath}","\\end{displaymath}"] ],
skipTags: ["script","noscript","style","textarea","pre","code"],
ignoreClass: "tex2jax_ignore",
processEscapes: false,
processEnvironments: true,
preview: "TeX"
},
showProcessingMessages: true,
displayAlign: "center",
displayIndent: "2em",
"HTML-CSS": {
scale: 100,
availableFonts: ["STIX","TeX"],
preferredFont: "TeX",
webFont: "TeX",
imageFont: "TeX",
showMathMenu: true,
},
MMLorHTML: {
prefer: {
MSIE: "MML",
Firefox: "MML",
Opera: "HTML",
other: "HTML"
}
}
});
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<h1 class="title">Advanced Lane Lines</h1>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">Introduction</h2>
<div class="outline-text-2" id="text-1">
<p>
The goals / steps of this project are the following:
</p>
<ul class="org-ul">
<li>Compute the camera calibration matrix and distortion coefficients
given a set of chessboard images.
</li>
<li>Apply a distortion correction to raw images.
</li>
<li>Use color transforms, gradients, etc., to create a thresholded
binary image.
</li>
<li>Apply a perspective transform to rectify binary image ("birds-eye
view").
</li>
<li>Detect lane pixels and fit to find the lane boundary.
</li>
<li>Determine the curvature of the lane and vehicle position with
respect to center.
</li>
<li>Warp the detected lane boundaries back onto the original image.
</li>
<li>Output visual display of the lane boundaries and numerical
estimation of lane curvature and vehicle position.
</li>
</ul>
</div>
<div id="outline-container-sec-1-1" class="outline-3">
<h3 id="sec-1-1">Setup</h3>
<div class="outline-text-3" id="text-1-1">
<p>
The initial setup includes creating the <a href="https://www.python.org/">Python</a> environment with
the packages that the project needs and uses.
</p>
<p>
<b>NOTE I</b>: The code for this project can be found in the
repository's <a href="lanelines.py">lanelines.py</a> file. <i>However</i>, all of the code in
that file was generated directly from the code blocks that appear
in this file and as such, contain no new information. Reviewers
can consult either that Python file or this document.
</p>
<p>
<b>NOTE II</b>: This document is presented in a variety of formats.
There is this Emacs Org-Mode <a href="writeup.html">file</a>, a <a href="writeup.pdf">PDF</a> generated using <i>LaTeX</i>,
an <a href="writeup.html">HTML</a> file, and a <a href="writeup.md">Markdown</a> file. The Markdown file will be
rendered directly by GitHub when viewed on the web. The HTML
version can be rendered either by cloning the repository to your
own computer and opening the file in a browser locally. Or, you
can view the same file in GitHub Pages at <a href="https://dventimi.github.io/CarND-Advanced-Lane-Lines/writeup.html">this link</a>. It looks
quite a bit better than the GitHub-rendered Markdown version.
</p>
<dl class="org-dl">
<dt> <a href="http://matplotlib.org/">matplotlib</a> </dt><dd>plotting and image processing tools
</dd>
<dt> <a href="http://www.numpy.org/">NumPy</a> </dt><dd>foundational scientific computing library
</dd>
<dt> <a href="http://zulko.github.io/moviepy/">MoviePy</a> </dt><dd>video processing tools
</dd>
<dt> <a href="http://opencv.org/">OpenCV</a> </dt><dd>computer vision library
</dd>
</dl>
<p>
The <a href="https://github.com/">GitHub</a> <a href="https://github.com/dventimi/CarND-Advanced-Lane-Lines">repository</a> for this project contains an <a href="environment.yml">environment.yml</a>
file that can be used to create and activate a <a href="https://conda.io/docs/">Conda</a> environment
with these commands.
</p>
<div class="org-src-container">
<pre class="src src-sh">conda env create --file environment.yml --name CarND-Advanced-Lane-Lines
<span style="color: #e090d7;">source</span> activate CarND-Advanced-Lane-Lines
</pre>
</div>
<p>
Once activated this environment is used to launch Python in
whatever way one likes, such as a <a href="https://www.python.org/shell/">Python shell</a>, a <a href="https://ipython.org/">IPython shell</a>,
or a <a href="http://jupyter.org/">jupyter notebook</a>. Having done that, the usual first step is
to import the packages that are used.
</p>
<div class="org-src-container">
<pre class="src src-python"><span id="coderef-deque" class="coderef-off"><span style="color: #b4fa70;">from</span> collections <span style="color: #b4fa70;">import</span> deque</span>
<span id="coderef-itertools" class="coderef-off"><span style="color: #b4fa70;">from</span> itertools <span style="color: #b4fa70;">import</span> groupby, islice, zip_longest, cycle, filterfalse</span>
<span style="color: #b4fa70;">from</span> moviepy.editor <span style="color: #b4fa70;">import</span> VideoFileClip
<span style="color: #b4fa70;">from</span> mpl_toolkits.axes_grid1 <span style="color: #b4fa70;">import</span> ImageGrid
<span id="coderef-profiler" class="coderef-off"><span style="color: #b4fa70;">import</span> cProfile</span>
<span style="color: #b4fa70;">import</span> cv2
<span style="color: #b4fa70;">import</span> glob
<span style="color: #b4fa70;">import</span> matplotlib
<span style="color: #b4fa70;">import</span> matplotlib.image <span style="color: #b4fa70;">as</span> mpimg
<span style="color: #b4fa70;">import</span> matplotlib.pyplot <span style="color: #b4fa70;">as</span> plt
<span style="color: #b4fa70;">import</span> numpy <span style="color: #b4fa70;">as</span> np
<span id="coderef-debug" class="coderef-off"><span style="color: #b4fa70;">import</span> pdb</span>
</pre>
</div>
<p>
Besides the third-party packages listed above, the project also
makes use of these standard-library library packages.
</p>
<dl class="org-dl">
<dt> <a href="#coderef-deque"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-deque');" onmouseout="CodeHighlightOff(this, 'coderef-deque');">deque</a> </dt><dd><a href="https://en.wikipedia.org/wiki/Circular_buffer">ring buffers</a> for moving averages
</dd>
<dt> <a href="#coderef-itertools"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-itertools');" onmouseout="CodeHighlightOff(this, 'coderef-itertools');">itertools</a> </dt><dd>handy for <a href="http://davidaventimiglia.com/python_generators.html">Python generators</a>
</dd>
<dt> <a href="#coderef-profiler"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-profiler');" onmouseout="CodeHighlightOff(this, 'coderef-profiler');">cProfile</a> </dt><dd>run-time <a href="https://docs.python.org/2/library/profile.html">optimization</a>
</dd>
<dt> <a href="#coderef-debug"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-debug');" onmouseout="CodeHighlightOff(this, 'coderef-debug');">pdb</a> </dt><dd>Python <a href="https://docs.python.org/3/library/pdb.html">debugger</a>
</dd>
</dl>
</div>
</div>
<div id="outline-container-sec-1-2" class="outline-3">
<h3 id="sec-1-2">Processing Pipeline</h3>
<div class="outline-text-3" id="text-1-2">
<p>
In order to detect lane lines in a video of a car driving on a
road, and then generate an annotated video with the detected lane
overlaid, we need an image processor that performs these two
tasks–detection and annotation–on every frame of the video.
That image processor encompasses a "processing pipeline."
</p>
<p>
The pipeline depends on these preliminary tasks.
</p>
<ol class="org-ol">
<li>Camera Calibration
</li>
<li>Perspective Measurement
</li>
</ol>
<p>
Then, the pipeline applies these stages.
</p>
<ol class="org-ol">
<li>Distortion Correction
</li>
<li>Gradient and Color Thresholds
</li>
<li>Perspective Transform
</li>
<li>Lane-line Detection
</li>
</ol>
<p>
Let us examine these preliminary tasks and pipeline stages in
greater detail.
</p>
</div>
<div id="outline-container-sec-1-2-1" class="outline-4">
<h4 id="sec-1-2-1">Camera Calibration</h4>
<div class="outline-text-4" id="text-1-2-1">
<p>
<a href="http://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html">Camera calibration</a> measures the distortion inherent in cameras
that utilize lenses so that the images taken with the camera can
be corrected by removing the distortion. A standard way to do
this is to measure the distortion the camera imposes on standard
images of known geometry. Checkerboard patterns are useful for
this tasks because of their high contrast, known geometry, and
regular pattern.
</p>
<p>
The <a href="#coderef-measure_distortion"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-measure_distortion');" onmouseout="CodeHighlightOff(this, 'coderef-measure_distortion');"><code>measure_distortion</code></a> function takes a Python <a href="https://docs.python.org/2/library/stdtypes.html#sequence-types-str-unicode-list-tuple-bytearray-buffer-xrange">sequence</a> of
checkerboard image filenames taken at different distances,
center-offsets, and orientations and applies the OpenCV
functions <a href="http://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html#findchessboardcorners"><code>findChessboardCorners</code></a> and <a href="http://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html#drawchessboardcorners"><code>drawChessboardCorners</code></a> to
identify corners in the images and highlight the corners. Then,
the <a href="http://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html#calibratecamera"><code>calibrateCamera</code></a> function measures the distortion. This
function <a href="#coderef-measure_distortion_reval"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-measure_distortion_reval');" onmouseout="CodeHighlightOff(this, 'coderef-measure_distortion_reval');">returns</a> the distortion parameters and matrix, along
with a sequence of tuples with the original filenames and the
annotated images.
</p>
<div class="org-src-container">
<pre class="src src-python"><span id="coderef-measure_distortion" class="coderef-off"><span style="color: #b4fa70;">def</span> <span style="color: #fce94f;">measure_distortion</span>(calibration_files):</span>
<span style="color: #fcaf3e;">files</span> = calibration_files
<span style="color: #fcaf3e;">objp</span> = np.zeros((9*6,3), np.float32)
<span style="color: #fcaf3e;">objp</span>[:,:2] = np.mgrid[0:9,0:6].T.reshape(-1,2)
<span style="color: #fcaf3e;">stage1</span> = <span style="color: #e090d7;">map</span>(<span style="color: #b4fa70;">lambda</span> x: (x,), cycle(files))
<span style="color: #fcaf3e;">stage2</span> = <span style="color: #e090d7;">map</span>(<span style="color: #b4fa70;">lambda</span> x: x + (mpimg.imread(x[0]),), stage1)
<span style="color: #fcaf3e;">stage3</span> = <span style="color: #e090d7;">map</span>(<span style="color: #b4fa70;">lambda</span> x: x + (cv2.findChessboardCorners(cv2.cvtColor(x[1], cv2.COLOR_RGB2GRAY), (9,6)),), stage2)
<span style="color: #fcaf3e;">stage4</span> = <span style="color: #e090d7;">map</span>(<span style="color: #b4fa70;">lambda</span> x: x + (cv2.drawChessboardCorners(np.copy(x[1]), (9,6), *(x[2][::-1])),), stage3)
<span style="color: #fcaf3e;">filenames</span>,<span style="color: #fcaf3e;">images</span>,<span style="color: #fcaf3e;">corners</span>,<span style="color: #fcaf3e;">annotated_images</span> = <span style="color: #e090d7;">zip</span>(*<span style="color: #e090d7;">filter</span>(<span style="color: #b4fa70;">lambda</span> x: x[2][0], islice(stage4, <span style="color: #e090d7;">len</span>(files))))
<span style="color: #fcaf3e;">_</span>,<span style="color: #fcaf3e;">imgpoints</span> = <span style="color: #e090d7;">zip</span>(*corners)
<span style="color: #fcaf3e;">objpoints</span> = [objp <span style="color: #b4fa70;">for</span> i <span style="color: #b4fa70;">in</span> <span style="color: #e090d7;">range</span>(<span style="color: #e090d7;">len</span>(imgpoints))]
<span style="color: #fcaf3e;">ret</span>, <span style="color: #fcaf3e;">mtx</span>, <span style="color: #fcaf3e;">dist</span>, <span style="color: #fcaf3e;">rvecs</span>, <span style="color: #fcaf3e;">tvecs</span> = cv2.calibrateCamera(objpoints, imgpoints, <span style="color: #e090d7;">list</span>(islice(stage2,1))[0][1].shape[:2:][::-1], <span style="color: #e9b2e3;">None</span>, <span style="color: #e9b2e3;">None</span>)
<span id="coderef-measure_distortion_reval" class="coderef-off"> <span style="color: #b4fa70;">return</span> mtx, dist, <span style="color: #e090d7;">zip</span>(filenames, annotated_images)</span>
</pre>
</div>
<p>
This function is used in subsequent distortion corrections.
</p>
</div>
</div>
<div id="outline-container-sec-1-2-2" class="outline-4">
<h4 id="sec-1-2-2">Distortion Correction</h4>
<div class="outline-text-4" id="text-1-2-2">
<p>
The <a href="#coderef-get_undistorter"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-get_undistorter');" onmouseout="CodeHighlightOff(this, 'coderef-get_undistorter');"><code>get_undistorter</code></a> function takes a sequence of calibration
checkerboard image filenames, applies the <code>measure_distortion</code>
function, and <a href="#coderef-get_undistorter_retval"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-get_undistorter_retval');" onmouseout="CodeHighlightOff(this, 'coderef-get_undistorter_retval');">returns</a> a new function. The new function function
uses the OpenCV <a href="http://docs.opencv.org/2.4/modules/imgproc/doc/geometric_transformations.html#void%20undistort(InputArray%20src,%20OutputArray%20dst,%20InputArray%20cameraMatrix,%20InputArray%20distCoeffs,%20InputArray%20newCameraMatrix)"><code>undistort</code></a> function to remove distortion from
images taken with the same camera.
</p>
<div class="org-src-container">
<pre class="src src-python"><span id="coderef-get_undistorter" class="coderef-off"><span style="color: #b4fa70;">def</span> <span style="color: #fce94f;">get_undistorter</span>(calibration_files):</span>
<span style="color: #fcaf3e;">mtx</span>,<span style="color: #fcaf3e;">dist</span>,<span style="color: #fcaf3e;">annotated_images</span> = measure_distortion(calibration_files)
<span id="coderef-get_undistorter_retval" class="coderef-off"> <span style="color: #b4fa70;">return</span> <span style="color: #b4fa70;">lambda</span> x: cv2.undistort(x, mtx, dist, <span style="color: #e9b2e3;">None</span>, mtx), annotated_images</span>
</pre>
</div>
<p>
In the example shown below, we <a href="#coderef-get_fn"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-get_fn');" onmouseout="CodeHighlightOff(this, 'coderef-get_fn');">get</a> an "image undistorter"
function for a set of calibration images.
</p>
<div class="org-src-container">
<pre class="src src-python"><span id="coderef-get_fn" class="coderef-off"><span style="color: #fcaf3e;">undistort</span>,<span style="color: #fcaf3e;">annotated_images</span> = get_undistorter(glob.glob(<span style="color: #e9b96e;">"camera_cal/*.jpg"</span>))</span>
<span style="color: #fcaf3e;">fig</span> = plt.figure()
<span style="color: #fcaf3e;">grid</span> = ImageGrid(fig, 111, nrows_ncols=(4,4), axes_pad=0.0)
<span id="coderef-apply_fn" class="coderef-off"><span style="color: #b4fa70;">for</span> p <span style="color: #b4fa70;">in</span> <span style="color: #e090d7;">zip</span>(annotated_images, grid):</span>
p[1].imshow(p[0][1])
fig.savefig(<span style="color: #e9b96e;">"output_images/annotated_calibration_images.jpg"</span>)
</pre>
</div>
<p>
The annotated calibration images are shown in the figure below.
</p>
<div class="figure">
<p><img src="output_images/annotated_calibration_images.jpg" alt="annotated_calibration_images.jpg" width="800px" />
</p>
</div>
<p>
As discussed shortly, the effects of image distortion can be
subtle and difficult notice with the naked eye. It helps
therefore to apply it to examples where the effect will be more
vivid. The first of the camera calibration images that we
recently used to <i>measure</i> the camera distortion is a good
candidate for <i>correcting</i> distortion. The following figure has
the original, distorted image.
</p>
<div class="figure">
<p><img src="camera_cal/calibration1.jpg" alt="calibration1.jpg" width="800px" />
</p>
</div>
<p>
It should be evident at a minimum that there is radial
distortion as the horizontal and vertical lines—which should
be straight—are curved outward from the center.
</p>
<p>
Next we use the camera matrix and distortion coefficients
embedded with in the <code>undistort</code> function that we obtained in
order to correct for these effects.
</p>
<div class="org-src-container">
<pre class="src src-python"><span style="color: #fcaf3e;">fig</span> = plt.figure()
plt.imshow(undistort(mpimg.imread(<span style="color: #e9b96e;">"camera_cal/calibration1.jpg"</span>)))
fig.savefig(<span style="color: #e9b96e;">"output_images/undistorted_calibration1.jpg"</span>)
</pre>
</div>
<div class="figure">
<p><img src="output_images/undistorted_calibration1.jpg" alt="undistorted_calibration1.jpg" width="800px" />
</p>
</div>
<p>
Next, we show the effects of applying the image undistorter to a
sequence of 6 road images taken with this same camera. These 6
images are a test sequence that will reappear many times through
the remainder of this discussion as other image processing steps
are taken up.
</p>
<p>
The <a href="#coderef-visualize"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-visualize');" onmouseout="CodeHighlightOff(this, 'coderef-visualize');"><code>visualize</code></a> function helps us view a gallery of test images
in "ganged up" layout, and this is helpful as we develop the
processing pipeline stages.
</p>
<div class="org-src-container">
<pre class="src src-python"><span id="coderef-visualize" class="coderef-off"><span style="color: #b4fa70;">def</span> <span style="color: #fce94f;">visualize</span>(filename, a):</span>
<span style="color: #fcaf3e;">fig</span>, <span style="color: #fcaf3e;">axes</span> = plt.subplots(2,3,figsize=(24,12),subplot_kw={<span style="color: #e9b96e;">'xticks'</span>:[],<span style="color: #e9b96e;">'yticks'</span>:[]})
fig.subplots_adjust(hspace=0.03, wspace=0.05)
<span style="color: #b4fa70;">for</span> p <span style="color: #b4fa70;">in</span> <span style="color: #e090d7;">zip</span>(<span style="color: #e090d7;">sum</span>(axes.tolist(),[]), a):
p[0].imshow(p[1],cmap=<span style="color: #e9b96e;">'gray'</span>)
plt.tight_layout()
fig.savefig(filename)
plt.close()
</pre>
</div>
<p>
The 6 test images that we use repeatedly are shown in the figure
below, without any image processing at all.
</p>
<div class="org-src-container">
<pre class="src src-python">visualize(<span style="color: #e9b96e;">"output_images/test_images.jpg"</span>,
(mpimg.imread(f) <span style="color: #b4fa70;">for</span> f <span style="color: #b4fa70;">in</span> cycle(glob.glob(<span style="color: #e9b96e;">"test_images/test*.jpg"</span>))))
</pre>
</div>
<div class="figure">
<p><img src="output_images/test_images.jpg" alt="test_images.jpg" width="800px" />
</p>
</div>
<p>
These test images are shown again, only this time the image
undistorter that we acquired above now is used to remove
distortion introduced by the camera. The effect is subtle and
difficult to notice, but close inspection shows that at least a
small amount of radial distortion is removed by this process.
</p>
<div class="org-src-container">
<pre class="src src-python">visualize(<span style="color: #e9b96e;">"output_images/undistorted_test_images.jpg"</span>,
(undistort(mpimg.imread(f)) <span style="color: #b4fa70;">for</span> f <span style="color: #b4fa70;">in</span> cycle(glob.glob(<span style="color: #e9b96e;">"test_images/test*.jpg"</span>))))
</pre>
</div>
<div class="figure">
<p><img src="output_images/undistorted_test_images.jpg" alt="undistorted_test_images.jpg" width="800px" />
</p>
</div>
<p>
Next, we move on to perspective measurement.
</p>
</div>
</div>
<div id="outline-container-sec-1-2-3" class="outline-4">
<h4 id="sec-1-2-3">Perspective Measurement</h4>
<div class="outline-text-4" id="text-1-2-3">
<p>
Perspective measurement applies to two-dimensional images taken
of three-dimensional scenes wherein objects of
interest–typically planar objects like roads–are oriented such
that their <a href="http://mathworld.wolfram.com/NormalVector.html">normal vector</a> is not parallel with the camera's line
of site. Another way to put it is that the planar object is not
parallel with the <a href="https://en.wikipedia.org/wiki/Image_plane">image plane</a>. While there undoubtedly are more
sophisticated, perhaps automated or semi-automated ways of doing
this, a tried-and-true method is to identify a non-rectilinear
region in the image that corresponds to the planar object of
interest (the road) and then map those to a corresponding
rectilinear region on the <a href="https://en.wikipedia.org/wiki/Image_plane">image plane</a>.
</p>
<p>
The <a href="#coderef-measure_warp"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-measure_warp');" onmouseout="CodeHighlightOff(this, 'coderef-measure_warp');"><code>measure_warp</code></a> function helps measure perspective. It takes
an image as a <a href="https://docs.scipy.org/doc/numpy/reference/generated/numpy.array.html">NumPy array</a> and displays the image to the user in
an interactive window. The user only has to click four corners
in sequence for the source region and then close the interactive
window. The <a href="#coderef-dst_region"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-dst_region');" onmouseout="CodeHighlightOff(this, 'coderef-dst_region');">destination region</a> on the <a href="https://en.wikipedia.org/wiki/Image_plane">image plane</a> for now is
<a href="#coderef-set_dst"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-set_dst');" onmouseout="CodeHighlightOff(this, 'coderef-set_dst');">hard-code</a> to a bounding box between the top and bottom of the
image and 300 pixels from the left edge and 300 pixels from the
right edge. These values were obtained through experimentation,
and while they are not as sophisticated as giving the user
interactive control, they do have the virtue of being perfectly
rectilinear. This is something that is difficult to achieve
manually. Setting the src region coordinates, along with
drawing guidelines to aid the eye, is accomplished in an
<a href="#coderef-event_handler"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-event_handler');" onmouseout="CodeHighlightOff(this, 'coderef-event_handler');">event handler</a> function for mouse-click events. The function
<a href="#coderef-measure_warp_retval"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-measure_warp_retval');" onmouseout="CodeHighlightOff(this, 'coderef-measure_warp_retval');">returns</a> the transformation matrix \(M\) and the inverse
transformation matrix \(M_{inv}\).
</p>
<div class="org-src-container">
<pre class="src src-python"><span id="coderef-measure_warp" class="coderef-off"><span style="color: #b4fa70;">def</span> <span style="color: #fce94f;">measure_warp</span>(img):</span>
<span style="color: #fcaf3e;">top</span> = 0
<span style="color: #fcaf3e;">bottom</span> = img.shape[0]
<span id="coderef-event_handler" class="coderef-off"> <span style="color: #b4fa70;">def</span> <span style="color: #fce94f;">handler</span>(e):</span>
<span style="color: #b4fa70;">if</span> <span style="color: #e090d7;">len</span>(src)<4:
plt.axhline(<span style="color: #e090d7;">int</span>(e.ydata), linewidth=2, color=<span style="color: #e9b96e;">'r'</span>)
plt.axvline(<span style="color: #e090d7;">int</span>(e.xdata), linewidth=2, color=<span style="color: #e9b96e;">'r'</span>)
<span id="coderef-set_src" class="coderef-off"> src.append((<span style="color: #e090d7;">int</span>(e.xdata),<span style="color: #e090d7;">int</span>(e.ydata)))</span>
<span style="color: #b4fa70;">if</span> <span style="color: #e090d7;">len</span>(src)==4:
<span id="coderef-set_dst" class="coderef-off"> dst.extend([(300,bottom),(300,top),(980,top),(980,bottom)])</span>
<span style="color: #fcaf3e;">was_interactive</span> = matplotlib.is_interactive()
<span style="color: #b4fa70;">if</span> <span style="color: #b4fa70;">not</span> matplotlib.is_interactive():
plt.ion()
<span style="color: #fcaf3e;">fig</span> = plt.figure()
plt.imshow(img)
<span style="color: #b4fa70;">global</span> src
<span style="color: #b4fa70;">global</span> dst
<span id="coderef-src_region" class="coderef-off"> <span style="color: #fcaf3e;">src</span> = []</span>
<span id="coderef-dst_region" class="coderef-off"> <span style="color: #fcaf3e;">dst</span> = []</span>
<span style="color: #fcaf3e;">cid1</span> = fig.canvas.mpl_connect(<span style="color: #e9b96e;">'button_press_event'</span>, handler)
<span style="color: #fcaf3e;">cid2</span> = fig.canvas.mpl_connect(<span style="color: #e9b96e;">'close_event'</span>, <span style="color: #b4fa70;">lambda</span> e: e.canvas.stop_event_loop())
fig.canvas.start_event_loop(timeout=-1)
<span id="coderef-getperspectivetransform" class="coderef-off"> <span style="color: #fcaf3e;">M</span> = cv2.getPerspectiveTransform(np.asfarray(src, np.float32), np.asfarray(dst, np.float32))</span>
<span style="color: #fcaf3e;">Minv</span> = cv2.getPerspectiveTransform(np.asfarray(dst, np.float32), np.asfarray(src, np.float32))
matplotlib.interactive(was_interactive)
<span id="coderef-measure_warp_retval" class="coderef-off"> <span style="color: #b4fa70;">return</span> M, Minv</span>
</pre>
</div>
<p>
Like with the <code>get_undistorter</code> function described above, we use
<a href="https://www.programiz.com/python-programming/closure">Python closures</a> to create a function generator called
<a href="#coderef-get_warpers"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-get_warpers');" onmouseout="CodeHighlightOff(this, 'coderef-get_warpers');"><code>get_warpers</code></a>, which measures the perspective, remembers the
transformation matrices, and then generate a new function that
uses OpenCV <a href="http://docs.opencv.org/2.4/modules/imgproc/doc/geometric_transformations.html#warpperspective"><code>warpPerspective</code></a> to transform a target image. Note
that it actually <a href="#coderef-get_warpers_retval"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-get_warpers_retval');" onmouseout="CodeHighlightOff(this, 'coderef-get_warpers_retval');">generates</a> two functions, both to "warp" and
"unwarp" images.
</p>
<div class="org-src-container">
<pre class="src src-python"><span id="coderef-get_warpers" class="coderef-off"><span style="color: #b4fa70;">def</span> <span style="color: #fce94f;">get_warpers</span>(corrected_image):</span>
<span style="color: #fcaf3e;">M</span>, <span style="color: #fcaf3e;">Minv</span> = measure_warp(corrected_image)
<span style="color: #b4fa70;">return</span> <span style="color: #b4fa70;">lambda</span> x: cv2.warpPerspective(x,
M,
x.shape[:2][::-1],
flags=cv2.INTER_LINEAR), \
<span style="color: #b4fa70;">lambda</span> x: cv2.warpPerspective(x,
Minv,
x.shape[:2][::-1],
<span id="coderef-get_warpers_retval" class="coderef-off"> flags=cv2.INTER_LINEAR), M, Minv</span>
</pre>
</div>
<p>
The following code illustrates how this is put into practice.
We get an image with the matplotlib <a href="http://matplotlib.org/api/image_api.html#matplotlib.image.imread"><code>imread</code></a> function, correct
for camera distortion using the <code>undistort</code> function we
generated with the <code>undistorter</code> function created above (after
camera calibration on checkerboard images), then use
<code>get_warpers</code> to generate both the <code>warp</code> and <code>unwarp</code>
functions. It also returns the \(M\) and \(M_{inv}\) matrices as
<code>M</code> and <code>Minv</code> for good measure.
</p>
<div class="org-src-container">
<pre class="src src-python"><span style="color: #fcaf3e;">warp</span>,<span style="color: #fcaf3e;">unwarp</span>,<span style="color: #fcaf3e;">M</span>,<span style="color: #fcaf3e;">Minv</span> = get_warpers(undistort(mpimg.imread(<span style="color: #e9b96e;">"test_images/straight_lines2.jpg"</span>)))
</pre>
</div>
<p>
The next sequence of four figures illustrates the interactive
experience the user has in this operation, showing step-by-step
the orthogonal guidelines that appear. The trapezoidal area
formed bout the outside bottom two corners and the inside top
two corners of the last figure defines the source region that is
then mapped to the target region. Again, as discussed above the
target region is a rectangle running from the bottom of the
image to the top, 300 pixels in from the left edge and 300
pixels in from the right edge.
</p>
<div class="figure">
<p><img src="output_images/figure_3-1.png" alt="figure_3-1.png" width="800px" />
</p>
</div>
<div class="figure">
<p><img src="output_images/figure_3-2.png" alt="figure_3-2.png" width="800px" />
</p>
</div>
<div class="figure">
<p><img src="output_images/figure_3-3.png" alt="figure_3-3.png" width="800px" />
</p>
</div>
<div class="figure">
<p><img src="output_images/figure_3-4.png" alt="figure_3-4.png" width="800px" />
</p>
</div>
<p>
Equipped not just with an <code>undistort</code> function (obtained via
camera calibration) but also a <code>warp</code> (obtained via
perspective measurement) function, we can compose both functions
in the proper sequence (<code>undistort</code> then <code>warp</code>) and apply it to
our 6 test images.
</p>
<div class="org-src-container">
<pre class="src src-python">visualize(<span style="color: #e9b96e;">"output_images/warped_undistorted_test_images.jpg"</span>,
(warp(undistort(mpimg.imread(f))) <span style="color: #b4fa70;">for</span> f <span style="color: #b4fa70;">in</span> cycle(glob.glob(<span style="color: #e9b96e;">"test_images/test*.jpg"</span>))))
</pre>
</div>
<p>
As you can see in the following gallery we now have a
"birds-eye" (i.e. top-down) view of the road for these 6 test
images. Note also that the perspective transform has also had
the effect of shoving out of the frame much of the extraneous
details (sky, trees, guardrails, other cars). This is
serendipitous as it saves us from having to apply a mask just to
the lane region.
</p>
<div class="figure">
<p><img src="output_images/warped_undistorted_test_images.jpg" alt="warped_undistorted_test_images.jpg" width="800px" />
</p>
</div>
<p>
Camera calibration and perspective measurement are preliminary
steps that occur before applying the processing pipeline to
images taken from the video stream. However, they are essential
and they enable the distortion correction and perspective
transformation steps which <i>are</i> part of the processing
pipeline. Another set of essential pipeline steps involve
gradient ant color thresholds, discussed in the next sections.
</p>
</div>
</div>
<div id="outline-container-sec-1-2-4" class="outline-4">
<h4 id="sec-1-2-4">Gradient and Color Thresholds</h4>
<div class="outline-text-4" id="text-1-2-4">
<p>
Next we develop a set of useful utility functions for scaling
images, taking gradients across them, isolating different color
channels, and generating binary images.
</p>
<p>
The <a href="#coderef-scale"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-scale');" onmouseout="CodeHighlightOff(this, 'coderef-scale');"><code>scale</code></a> function scales the values of NumPy image arrays to
arbitrary ranges (e.g., [0,1] or [0,255]). The default range is
[0,255], and this is useful in order to give all images the same
scale. Different operations (e.g., taking gradients, producing
binary images) can introduce different scales and it eases
combining and comparing images when they have the same scale.
</p>
<div class="org-src-container">
<pre class="src src-python"><span id="coderef-scale" class="coderef-off"><span style="color: #b4fa70;">def</span> <span style="color: #fce94f;">scale</span>(img, factor=255.0):</span>
<span style="color: #fcaf3e;">scale_factor</span> = np.<span style="color: #e090d7;">max</span>(img)/factor
<span style="color: #b4fa70;">return</span> (img/scale_factor).astype(np.uint8)
</pre>
</div>
<p>
The <a href="#coderef-derivative"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-derivative');" onmouseout="CodeHighlightOff(this, 'coderef-derivative');"><code>derivative</code></a> function uses the OpenCV <a href="http://docs.opencv.org/2.4/modules/imgproc/doc/filtering.html#sobel"><code>sobel</code></a> function to
apply the <a href="https://en.wikipedia.org/wiki/Sobel_operator">Sobel operator</a> in order to estimate derivatives in the
\(x\) and \(y\) directions across the image. For good measure, it
also <a href="#coderef-derivative_retval"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-derivative_retval');" onmouseout="CodeHighlightOff(this, 'coderef-derivative_retval');">returns</a> both the <i>magnitude</i> and the <i>direction</i> of the
<a href="https://en.wikipedia.org/wiki/Gradient">gradient</a> computed from these derivative estimates.
</p>
<div class="org-src-container">
<pre class="src src-python"><span id="coderef-derivative" class="coderef-off"><span style="color: #b4fa70;">def</span> <span style="color: #fce94f;">derivative</span>(img, sobel_kernel=3):</span>
<span style="color: #fcaf3e;">derivx</span> = np.absolute(cv2.Sobel(img, cv2.CV_64F, 1, 0, ksize=sobel_kernel))
<span style="color: #fcaf3e;">derivy</span> = np.absolute(cv2.Sobel(img, cv2.CV_64F, 0, 1, ksize=sobel_kernel))
<span style="color: #fcaf3e;">gradmag</span> = np.sqrt(derivx**2 + derivy**2)
<span style="color: #fcaf3e;">absgraddir</span> = np.arctan2(derivy, derivx)
<span id="coderef-derivative_retval" class="coderef-off"> <span style="color: #b4fa70;">return</span> scale(derivx), scale(derivy), scale(gradmag), absgraddir</span>
</pre>
</div>
<p>
The <a href="#coderef-grad"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-grad');" onmouseout="CodeHighlightOff(this, 'coderef-grad');"><code>grad</code></a> function adapts the <code>derivative</code> function to return
both the gradient <i>magnitude</i> and <i>direction</i>. You might wonder
what this function adds to the <code>derivative</code> function, and that
is a valid consideration. Largely it exists because the lecture
notes seemed to suggest that it's worthwhile to use different
kernel sizes for the Sobel operator when computing the gradient
direction. In hindsight it's not clear this function really is
adding value and it may be removed in future versions.
</p>
<div class="org-src-container">
<pre class="src src-python"><span id="coderef-grad" class="coderef-off"><span style="color: #b4fa70;">def</span> <span style="color: #fce94f;">grad</span>(img, k1=3, k2=15):</span>
<span id="coderef-grad_m" class="coderef-off"> <span style="color: #fcaf3e;">_</span>,<span style="color: #fcaf3e;">_</span>,<span style="color: #fcaf3e;">g</span>,<span style="color: #fcaf3e;">_</span> = derivative(img, sobel_kernel=k1)</span>
<span id="coderef-grad_p" class="coderef-off"> <span style="color: #fcaf3e;">_</span>,<span style="color: #fcaf3e;">_</span>,<span style="color: #fcaf3e;">_</span>,<span style="color: #fcaf3e;">p</span> = derivative(img, sobel_kernel=k2)</span>
<span style="color: #b4fa70;">return</span> g,p
</pre>
</div>
<p>
The <a href="#coderef-hls_select"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-hls_select');" onmouseout="CodeHighlightOff(this, 'coderef-hls_select');"><code>hls_select</code></a> function is a convenience that fans out the
three channels of the <a href="https://en.wikipedia.org/wiki/HSL_and_HSV">HLS color-space</a> into separate NumPy
arrays.
</p>
<div class="org-src-container">
<pre class="src src-python"><span id="coderef-hls_select" class="coderef-off"><span style="color: #b4fa70;">def</span> <span style="color: #fce94f;">hls_select</span>(img):</span>
<span style="color: #fcaf3e;">hsv</span> = cv2.cvtColor(img, cv2.COLOR_RGB2HLS).astype(np.<span style="color: #e090d7;">float</span>)
<span style="color: #fcaf3e;">h</span> = hsv[:,:,0]
<span style="color: #fcaf3e;">l</span> = hsv[:,:,1]
<span style="color: #fcaf3e;">s</span> = hsv[:,:,2]
<span style="color: #b4fa70;">return</span> h,l,s
</pre>
</div>
<p>
The <a href="#coderef-rgb_select"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-rgb_select');" onmouseout="CodeHighlightOff(this, 'coderef-rgb_select');"><code>rgb_select</code></a> function is another convenience that returns
the three channels of the <a href="https://en.wikipedia.org/wiki/RGB_color_space">RGB color-space</a>.
</p>
<div class="org-src-container">
<pre class="src src-python"><span id="coderef-rgb_select" class="coderef-off"><span style="color: #b4fa70;">def</span> <span style="color: #fce94f;">rgb_select</span>(img):</span>
<span style="color: #fcaf3e;">rgb</span> = img
<span style="color: #fcaf3e;">r</span> = rgb[:,:,0]
<span style="color: #fcaf3e;">g</span> = rgb[:,:,1]
<span style="color: #fcaf3e;">b</span> = rgb[:,:,2]
<span style="color: #b4fa70;">return</span> r,g,b
</pre>
</div>
<p>
The <a href="#coderef-threshold"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-threshold');" onmouseout="CodeHighlightOff(this, 'coderef-threshold');"><code>threshold</code></a> function is a convenience that applies
<code>thresh_min</code> and <code>thresh_max</code> <i>min-max</i> values and logical
operations in order to obtain "binary" images. Binary images
have activated pixels (non-zero values) for desired features.
</p>
<div class="org-src-container">
<pre class="src src-python"><span id="coderef-threshold" class="coderef-off"><span style="color: #b4fa70;">def</span> <span style="color: #fce94f;">threshold</span>(img, thresh_min=0, thresh_max=255):</span>
<span style="color: #fcaf3e;">binary_output</span> = np.zeros_like(img)
<span style="color: #fcaf3e;">binary_output</span>[(img >= thresh_min) & (img <= thresh_max)] = 1
<span style="color: #b4fa70;">return</span> binary_output
</pre>
</div>
<p>
The <a href="#coderef-land_lor"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-land_lor');" onmouseout="CodeHighlightOff(this, 'coderef-land_lor');"><code>land</code></a> and <a href="#coderef-land_lor"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-land_lor');" onmouseout="CodeHighlightOff(this, 'coderef-land_lor');"><code>lor</code></a> functions are conveniences for combining
binary images, either with logical <a href="https://en.wikipedia.org/wiki/Logical_conjunction">conjunction</a> or <a href="https://en.wikipedia.org/wiki/Logical_disjunction">disjunction</a>,
respectively.
</p>
<div class="org-src-container">
<pre class="src src-python"><span id="coderef-land_lor" class="coderef-off"><span style="color: #fcaf3e;">land</span> = <span style="color: #b4fa70;">lambda</span> *x: np.logical_and.<span style="color: #e090d7;">reduce</span>(x)</span>
<span style="color: #fcaf3e;">lor</span> = <span style="color: #b4fa70;">lambda</span> *x: np.logical_or.<span style="color: #e090d7;">reduce</span>(x)
</pre>
</div>
<p>
There are various ways of doing this. Another way is to stack
binary image arrays using the NumPy <a href="https://docs.scipy.org/doc/numpy/reference/generated/numpy.stack.html"><code>stack</code></a> function and then
interleave various combinations of such interleavings along with
the NumPy <a href="https://docs.scipy.org/doc/numpy/reference/generated/numpy.any.html#numpy-any"><code>any</code></a> function and <a href="https://docs.scipy.org/doc/numpy/reference/generated/numpy.all.html#numpy-all"><code>all</code></a> function. It's a clever
approach, but I find that applying the NumPy <a href="https://docs.scipy.org/doc/numpy/reference/generated/numpy.logical_and.html#numpy-logical-and"><code>logical_and</code></a> and
<a href="https://docs.scipy.org/doc/numpy/reference/generated/numpy.logical_or.html#numpy-logical-or"><code>logical_or</code></a> functions as above leads to less typing.
</p>
<p>
The <a href="#coderef-highlight"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-highlight');" onmouseout="CodeHighlightOff(this, 'coderef-highlight');"><code>highlight</code></a> function composes the color channel selection,
gradient estimation, binary threshold, logical composition, and
scaling operations to an input image in order to "highlight" the
desired features, such as lane lines. Note that distortion
correction and perspective transformation are considered outside
the scope of this function. In a real pipeline, those two
operations almost certainly should be applied to an image before
presenting it to the <a href="#coderef-highlight"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-highlight');" onmouseout="CodeHighlightOff(this, 'coderef-highlight');"><code>highlight</code></a> function. In general, they
need not be, which can be useful during the exploratory phase of
pipeline development.
</p>
<div class="org-src-container">
<pre class="src src-python"><span id="coderef-highlight" class="coderef-off"><span style="color: #b4fa70;">def</span> <span style="color: #fce94f;">highlight</span>(img):</span>
<span style="color: #fcaf3e;">r</span>,<span style="color: #fcaf3e;">g</span>,<span style="color: #fcaf3e;">b</span> = rgb_select(img)
<span style="color: #fcaf3e;">h</span>,<span style="color: #fcaf3e;">l</span>,<span style="color: #fcaf3e;">s</span> = hls_select(img)
<span style="color: #fcaf3e;">o01</span> = threshold(r, 200, 255)
<span style="color: #fcaf3e;">o02</span> = threshold(g, 200, 255)
<span style="color: #fcaf3e;">o03</span> = threshold(s, 200, 255)
<span style="color: #b4fa70;">return</span> scale(lor(land(o01,o02),o03))
</pre>
</div>
<p>
In fact, the highlight and undistort operations are combined
<i>without</i> perspective transform in the next gallery of 6 test
images. This is an example of a common iteration pattern while
exploring pipeline options.
</p>
<div class="org-src-container">
<pre class="src src-python">visualize(<span style="color: #e9b96e;">"output_images/binary_undistorted_test_images.jpg"</span>,
(highlight(undistort(mpimg.imread(f))) <span style="color: #b4fa70;">for</span> f <span style="color: #b4fa70;">in</span> cycle(glob.glob(<span style="color: #e9b96e;">"test_images/test*.jpg"</span>))))
</pre>
</div>
<div class="figure">
<p><img src="output_images/binary_undistorted_test_images.jpg" alt="binary_undistorted_test_images.jpg" width="800px" />
</p>
</div>
</div>
</div>
<div id="outline-container-sec-1-2-5" class="outline-4">
<h4 id="sec-1-2-5">Perspective Transform</h4>
<div class="outline-text-4" id="text-1-2-5">
<p>
Armed with a pipeline which, based on the 6 test images, we
believe may be a good candidate for detecting lane lines, we
then see what the pipeline-processed test images look like after
transforming them to a "bird's-eye" view.
</p>
<div class="org-src-container">
<pre class="src src-python">visualize(<span style="color: #e9b96e;">"output_images/warped_binary_undistorted_images.jpg"</span>,
(warp(highlight(undistort(mpimg.imread(f)))) <span style="color: #b4fa70;">for</span> f <span style="color: #b4fa70;">in</span> cycle(glob.glob(<span style="color: #e9b96e;">"test_images/test*.jpg"</span>))))
</pre>
</div>
<div class="figure">
<p><img src="output_images/warped_binary_undistorted_images.jpg" alt="warped_binary_undistorted_images.jpg" width="800px" />
</p>
</div>
</div>
</div>
<div id="outline-container-sec-1-2-6" class="outline-4">
<h4 id="sec-1-2-6">Lane-Finding</h4>
<div class="outline-text-4" id="text-1-2-6">
<p>
Lane-line detection can be done somewhat laboriously–but
perhaps more accurately–using a "sliding window" technique.
Roughly, the algorithm implemented in
<a href="#coderef-detect_lines_sliding_window"class="coderef" onmouseover="CodeHighlightOn(this, 'coderef-detect_lines_sliding_window');" onmouseout="CodeHighlightOff(this, 'coderef-detect_lines_sliding_window');"><code>detect_lines_sliding_window</code></a> below has these steps, also
discussed in the code comments.
</p>
<ol class="org-ol">
<li>Take a histogram across the bottom of the image.
</li>
<li>Find the histogram peaks to identify the lane lines at the
bottom of the image.
</li>
<li>Divide the image into a vertical stack of narrow horizontal
slices.
</li>
<li>Select activated pixels (remember, the input is a binary
image) only in a "neighborhood" of our current estimate of
the lane position. This neighborhood is the "sliding
window." To bootstrap the process, our initial estimate of
the lane line location is taken from the histogram peak steps
listed above. Essentially, we are removing "outliers"
</li>
<li>Estimate the new lane-line location for this window from the
mean of the pixels falling within the sliding window.
</li>
<li>March vertically up through the stack, repeating this process.
</li>
<li>Select all activated pixels within all of our sliding windows.
</li>
<li>Fit a quadratic function to these selected pixels, obtaining
model parameters.
</li>
</ol>
<p>