Finding Lane Lines on the Road
In this project, I used Python and OpenCV to find lane lines in the road images.
The following techniques are used:
- Color Selection
- Canny Edge Detection
- Region of Interest Selection
- Hough Transform Line Detection
Finally, I applied all the techniques to process video clips to find lane lines in them.
Let's load and examine the test images.
Lines are in white or yellow. A white lane is a series of alternating dots and short lines, which we need to detect as one line.
RGB Color Space
The images are loaded in RGB color space. Let's try selecting only yellow and white colors in the images using the RGB channels.
Reference: RGB Color Code Chart
# image is expected be in RGB color space def select_rgb_white_yellow(image): # white color mask lower = np.uint8([200, 200, 200]) upper = np.uint8([255, 255, 255]) white_mask = cv2.inRange(image, lower, upper) # yellow color mask lower = np.uint8([190, 190, 0]) upper = np.uint8([255, 255, 255]) yellow_mask = cv2.inRange(image, lower, upper) # combine the mask mask = cv2.bitwise_or(white_mask, yellow_mask) masked = cv2.bitwise_and(image, image, mask = mask) return masked
It looks pretty good except the two in which the yellow lines are not clear due to the dark shade from the tree on the left.
HSL and HSV Color Space
cv2.cvtColor, we can convert RGB image into different color space. For example, HSL and HSV color space.
HSV Color Space
How does it look when RGB images are converted into HSV color space?
def convert_hsv(image): return cv2.cvtColor(image, cv2.COLOR_RGB2HSV)
The yellow lines are very clear including the ones under the shades but the white lines are less clear.
HSL Color Space
How does it look like when images are converted from RGB to HSL color space?
def convert_hls(image): return cv2.cvtColor(image, cv2.COLOR_RGB2HLS)
Both the white and yellow lines are clearly recognizable. Also, the yellow lines under the shades are clearly shown.
Let's build a filter to select those white and yellow lines. I want to select particular range of each channels (Hue, Saturation and Light).
cv2.inRangeto filter the white color and the yellow color seperately.
The function returns 255 when the filter conditon is satisfied. Otherwise, it returns 0.
cv2.bitwise_orto combine these two binary masks.
The combined mask returns 255 when either white or yellow color is detected.
cv2.bitwise_andto apply the combined mask onto the original RGB image
def select_white_yellow(image): converted = convert_hls(image) # white color mask lower = np.uint8([ 0, 200, 0]) upper = np.uint8([255, 255, 255]) white_mask = cv2.inRange(converted, lower, upper) # yellow color mask lower = np.uint8([ 10, 0, 100]) upper = np.uint8([ 40, 255, 255]) yellow_mask = cv2.inRange(converted, lower, upper) # combine the mask mask = cv2.bitwise_or(white_mask, yellow_mask) return cv2.bitwise_and(image, image, mask = mask) white_yellow_images = list(map(select_white_yellow, test_images))
For the white color,
- I chose high Light value.
- I did not filter Hue, Saturation values.
For the yellow color,
- I chose Hue around 30 to choose yellow color.
- I chose relatively high Saturation to exclude yellow hills
The combined mask filters the yellow and white lines very clearly.
Canny Edge Detection
The Canny edge detector was developed by John F. Canny in 1986.
We want to detect edges in order to find straight lines especially lane lines. For this,
cv2.cvtColorto convert images into gray scale
cv2.GaussianBlurto smooth out rough edges
cv2.Cannyto find edges
Let's take a look at each step in details.
Note: Canny Edge Detection Wikipedia has a good description in good details.
The images should be converted into gray scaled ones in order to detect shapes (edges) in the images. This is because the Canny edge detection measures the magnitude of pixel intensity changes or gradients (more on this later).
Here, I'm converting the white and yellow line images from the above into gray scale for edge detection.
def convert_gray_scale(image): return cv2.cvtColor(image, cv2.COLOR_RGB2GRAY) gray_images = list(map(convert_gray_scale, white_yellow_images))
Gaussian Smoothing (Gaussian Blur)
When there is an edge (i.e. a line), the pixel intensity changes rapidly (i.e. from 0 to 255) which we want to detect. But before doing so, we should make the edges smoother. As you can see, the above images have many rough edges which causes many noisy edges to be detected.
cv2.GaussianBlur to smooth out edges.
def apply_smoothing(image, kernel_size=15): """ kernel_size must be postivie and odd """ return cv2.GaussianBlur(image, (kernel_size, kernel_size), 0)
The GaussianBlur takes a
kernel_size parameter which you'll need to play with to find one that works best. I tried 3, 5, 9, 11, 15, 17 (they must be positive and odd) and check the edge detection (see the next section) result. The bigger the
kernel_size value is, the more blurry the image becomes.
kearnel_size value requires more time to process. It is not noticeable with the test images but we should keep that in mind (later we'll be processing video clips). So, we should prefer smaller values if the effect is similar.
blurred_images = list(map(lambda image: apply_smoothing(image), gray_images))
cv2.Canny takes two threshold values which requires some explanation.
it is essential to filter out the edge pixel with the weak gradient value and preserve the edge with the high gradient value. Thus two threshold values are set to clarify the different types of edge pixels, one is called high threshold value and the other is called the low threshold value. If the edge pixel’s gradient value is higher than the high threshold value, they are marked as strong edge pixels. If the edge pixel’s gradient value is smaller than the high threshold value and larger than the low threshold value, they are marked as weak edge pixels. If the pixel value is smaller than the low threshold value, they will be suppressed.
According to the OpenCV documentation, the double thresholds are used as follows:
- If a pixel gradient is higher than the upper threshold, the pixel is accepted as an edge
- If a pixel gradient value is below the lower threshold, then it is rejected.
- If the pixel gradient is between the two thresholds, then it will be accepted only if it is connected to a pixel that is above the upper threshold.
- Canny recommended a upper:lower ratio between 2:1 and 3:1.
These two threshold values are empirically determined. Basically, you will need to define them by trials and errors.
I first set the
low_threshold to zero and then adjust the
high_threshold is too high, you find no edges. If
high_threshold is too low, you find too many edges. Once you find a good
high_threshold, adjust the
low_threshold to discard the weak edges (noises) connected to the strong edges.
def detect_edges(image, low_threshold=50, high_threshold=150): return cv2.Canny(image, low_threshold, high_threshold) edge_images = list(map(lambda image: detect_edges(image), blurred_images))
Region of Interest Selection
When finding lane lines, we don't need to check the sky and the hills.
Roughly speaking, we are interested in the area surrounded by the red lines below:
So, we exclude outside the region of interest by apply a mask.
def filter_region(image, vertices): """ Create the mask using the vertices and apply it to the input image """ mask = np.zeros_like(image) if len(mask.shape)==2: cv2.fillPoly(mask, vertices, 255) else: cv2.fillPoly(mask, vertices, (255,)*mask.shape) # in case, the input image has a channel dimension return cv2.bitwise_and(image, mask) def select_region(image): """ It keeps the region surrounded by the `vertices` (i.e. polygon). Other area is set to 0 (black). """ # first, define the polygon by vertices rows, cols = image.shape[:2] bottom_left = [cols*0.1, rows*0.95] top_left = [cols*0.4, rows*0.6] bottom_right = [cols*0.9, rows*0.95] top_right = [cols*0.6, rows*0.6] # the vertices are an array of polygons (i.e array of arrays) and the data type must be integer vertices = np.array([[bottom_left, top_left, top_right, bottom_right]], dtype=np.int32) return filter_region(image, vertices) # images showing the region of interest only roi_images = list(map(select_region, edge_images))
Now we have lane lines but we need to recognize them as lines. Especially, two lines: the left lane and the right lane.
Hough Transform Line Detection
cv2.HoughLinesP to detect lines in the edge images.
There are several parameters you'll need to tweak and tune:
- rho: Distance resolution of the accumulator in pixels.
- theta: Angle resolution of the accumulator in radians.
- threshold: Accumulator threshold parameter. Only those lines are returned that get enough votes (>
- minLineLength: Minimum line length. Line segments shorter than that are rejected.
- maxLineGap: Maximum allowed gap between points on the same line to link them.
More details can be found:
def hough_lines(image): """ `image` should be the output of a Canny transform. Returns hough lines (not the image with lines) """ return cv2.HoughLinesP(image, rho=1, theta=np.pi/180, threshold=20, minLineLength=20, maxLineGap=300) list_of_lines = list(map(hough_lines, roi_images))
list_of_lines contains a list of lines detected. With the above parameters, approximately 5-15 lines are detected for each image.
Let's draw the lines onto the original images.
Averaging and Extrapolating Lines
There are multiple lines detected for a lane line. We should come up with an averaged line for that.
Also, some lane lines are only partially recognized. We should extrapolate the line to cover full lane line length.
We want two lane lines: one for the left and the other for the right. The left lane should have a positive slope, and the right lane should have a negative slope. Therefore, we'll collect positive slope lines and negative slope lines separately and take averages.
Note: in the image,
y coordinate is reversed. The higher
y value is actually lower in the image. Therefore, the slope is negative for the left lane, and the slope is positive for the right lane.
def average_slope_intercept(lines): left_lines =  # (slope, intercept) left_weights =  # (length,) right_lines =  # (slope, intercept) right_weights =  # (length,) for line in lines: for x1, y1, x2, y2 in line: if x2==x1: continue # ignore a vertical line slope = (y2-y1)/(x2-x1) intercept = y1 - slope*x1 length = np.sqrt((y2-y1)**2+(x2-x1)**2) if slope < 0: # y is reversed in image left_lines.append((slope, intercept)) left_weights.append((length)) else: right_lines.append((slope, intercept)) right_weights.append((length)) # add more weight to longer lines left_lane = np.dot(left_weights, left_lines) /np.sum(left_weights) if len(left_weights) >0 else None right_lane = np.dot(right_weights, right_lines)/np.sum(right_weights) if len(right_weights)>0 else None return left_lane, right_lane # (slope, intercept), (slope, intercept)
Using the above
average_lines function, we can calculate average slope and intercept for the left and right lanes of each image.
Let's draw the lanes. I need to convert the slope and intercept into pixel points.
def make_line_points(y1, y2, line): """ Convert a line represented in slope and intercept into pixel points """ if line is None: return None slope, intercept = line # make sure everything is integer as cv2.line requires it x1 = int((y1 - intercept)/slope) x2 = int((y2 - intercept)/slope) y1 = int(y1) y2 = int(y2) return ((x1, y1), (x2, y2))
draw_lines except a list of lines as the second parameter. Each line is a list of 4 values (x1, y1, x2, y2). The data type needs to be integer for
cv2.line to work without throwing an error.
def lane_lines(image, lines): left_lane, right_lane = average_slope_intercept(lines) y1 = image.shape # bottom of the image y2 = y1*0.6 # slightly lower than the middle left_line = make_line_points(y1, y2, left_lane) right_line = make_line_points(y1, y2, right_lane) return left_line, right_line def draw_lane_lines(image, lines, color=[255, 0, 0], thickness=20): # make a separate image to draw lines and combine with the orignal later line_image = np.zeros_like(image) for line in lines: if line is not None: cv2.line(line_image, *line, color, thickness) # image1 * α + image2 * β + λ # image1 and image2 must be the same shape. return cv2.addWeighted(image, 1.0, line_image, 0.95, 0.0) lane_images =  for image, lines in zip(test_images, list_of_lines): lane_images.append(draw_lane_lines(image, lane_lines(image, lines)))
I'm drawing lanes on video clips.
from collections import deque QUEUE_LENGTH=50 class LaneDetector: def __init__(self): self.left_lines = deque(maxlen=QUEUE_LENGTH) self.right_lines = deque(maxlen=QUEUE_LENGTH) def process(self, image): white_yellow = select_white_yellow(image) gray = convert_gray_scale(white_yellow) smooth_gray = apply_smoothing(gray) edges = detect_edges(smooth_gray) regions = select_region(edges) lines = hough_lines(regions) left_line, right_line = lane_lines(image, lines) def mean_line(line, lines): if line is not None: lines.append(line) if len(lines)>0: line = np.mean(lines, axis=0, dtype=np.int32) line = tuple(map(tuple, line)) # make sure it's tuples not numpy array for cv2.line to work return line left_line = mean_line(left_line, self.left_lines) right_line = mean_line(right_line, self.right_lines) return draw_lane_lines(image, (left_line, right_line))
Let's try the one with the solid white lane on the right first.
def process_video(video_input, video_output): detector = LaneDetector() clip = VideoFileClip(os.path.join('test_videos', video_input)) processed = clip.fl_image(detector.process) processed.write_videofile(os.path.join('output_videos', video_output), audio=False)
The video inputs are in test_videos folder. The video outputs are generated in output_videos folder.
%time process_video('solidWhiteRight.mp4', 'white.mp4') %time process_video('solidYellowLeft.mp4', 'yellow.mp4') %time process_video('challenge.mp4', 'extra.mp4')
The project was successful in that the video images clearly show the lane lines are detected properly and lines are very smoothly handled.
It only detects the straight lane lines. It is an advanced topic to handle curved lanes (or the curvature of lanes). We'll need to use perspective transformation and also poly fitting lane lines rather than fitting to straight lines.
Having said that, the lanes near the car are mostly straight in the images. The curvature appears at further distance unless it's a steep curve. So, this basic lane finding technique is still very useful.
Another thing is that it won't work for steep (up or down) roads because the region of interest mask is assumed from the center of the image.
For steep roads, we first need to detect the horizontal line (between the sky and the earth) so that we can tell up to where the lines should extend.