You can use these applications to create a license plate dataset and then use that dataset to train and test a license plate recogntion system.
The code is all Python3 and uses Keras, OpenCV3 and dlib libraries. Structure and content is influenced by PyImageSearch. The Performance when the model is trained with the training dataset is:
- 96.80% correct chars.
- 84.91% correct plates.
Using the pre-trained model and the verification dataset
- 98.7% characters correct.
- 93.9% plates correct.
Note that plate accuracy is dependent on the number of char errors per plate, and is also very sensitive to character accuracy. Every 1% increase in character detection accuracy above 86% can potentially add 7% to the plate accuracy. Also plate accuracy for a given character accuracy is variable and dependent on the number of character errors per plate. Thus character accuracy is a better metric for measuring performance.
The license plate recognizer is composed of two major blocks.
- A dlib FHOG+LSVM plate detector that crops the license plate from an image.
- A Keras based CNN that classifies the characters in the license plate. This is based on Multi-digit Number Recogniion from Street View and Number Plate Recogntion with Tensorflow
The provided dataset only contains California license plates with 7 characters per plate. Ideally the character detector would be able to classify plates with less than 7 characters. This paper explains how to use the ouput layer to identify the number of characters, but I encountered a number of issues in attempting to implement this scheme.
- I could not figure out how to configure the Keras output layer with the character count.
- Training back prop must be disabled for output characters that are not present. I could not figure how to do this in Keras.
- Not enough training data. Plates with less than 7 characters are not too common. This could maybe be solved by cropping 7 char plates to display less characters.
You can run the code against this dataset of gray, cropped, resized, and annotated license plate images.
You can run the code with this pre-trained Keras model
Play the video clips, using Shift+S to save the still images. This works best on a powerful desktop CPU. Note that the stills are saved to Pictures by default.
Create a file linking image filenames with plate text. Images are displayed in a window along with a text entry box. If no plate text is entered, then images are placed in the discard directory thus resulting in less images that need to be processed by imglab. Output stored in labels.txt.
Limitations; multiple plates per image are not supported. If this is something that you need, then you will have to hand edit the .xml file generated by imglab.
Create plate location data using dlib imglab. This will give you an xml file linking filenames with plate location data
eg create xml file list of image files at ./myImageDataSet
imglab -c ./plateAnnotations.xml ./myImageDataSet
eg Open the GUI to label and annotate the images listed in the xml file
imglab ./plateAnnotations.xml
Read plate text data from labels.txt and add to xml annotation file. Note that the modified .xml file can still be read by imglab, but if you save the .xml file from imglab then the text annotation will be deleted. In other words, ensure that you have completed all your imglab editing before you run this app.
Read plate text and location from xml file, crop the lplate from the associated image, convert to gray scale, resize to 128x64 and save to a new file with the plate text embedded in the filename.
Once your dataset is large enough to successfully train the plate detector and char classifier, then you no longer need to use VLC, imglab, etc. You can can use this app to generate more cropped annotated images. See below for description
The downloadable dataset is augmented with 1000 artificial plates. Test results indicated a useful increase in performance when augmenting real images with artificially generated images. Although the artificial plates are not a true representation of real plates with real backgrounds, they do help introduce more character variability. CA plates are predominantly car plates which follow the format DLLLDDD, where D = digit, and L=letter. Furthermore the first digit is age related so some leading digits are more common than others. However, truck plates and custom plates do not follow this format, but they are not as common, thus not well represented in the training data. The artificial data may help to offset this skewing of the plate text in the real images.
If you want to experiment with generating artificial plates, you can use gen_plates.py to generate your own artificial dataset.
Train dlib FHOG+LSVM plate detector
Read images from specified directory, and display each image with predicted bounding box.
Load images specified in xml file, and compare prediction bounding boxes against ground truth BB.
ANPR character detector loosely based on Matt Earl example. Assumes 7 characters per plate. Output layer is configured to classify the seven characters. Each character can be any alpha or any numeric for a total of 36 classes for each character. See Multi-digit Number Recogniion from Street View. Needs a fair bit of training data. If you have more than around 500 "real" images then you can train on real data, otherwise you will need to train on artificially generated images. I used Matt Earl modified code to generate the artificial datset. Real images work best. But the artificial images are still useful for data augmentation.
Image formats:
- The images contain just the plate, and the image filename contains the plate text in the form 00006667_6SGM738_1.png. --annFile is not defined and defaults to None
- The images contain more than the plate, and an augmented imglab .xml file provides plate location and text information for each image. -annFile defines the imglab .xml file.
Every 50 epochs, if the val_loss is the best yet, then the trained model is saved to file: "--modelPath" with filename: model-_<loss.hdf5>, eg model_2699-0.1537.hdf5
Load a previously trained model, and predict plate text in an image data set. Image data set format is the same as for anpr_char_det_train.py Prints prediction results, and then selects 5 images at random, displays the image and prints the ground truth plate text and the predicted text.
Load video files, detect plates using dlib FHOG+LSVM. Crop the plate from the image, and then use a convNet to classify the characters in the plate. Save the original image and the cropped image to file. Suffix the filename with the lplate text. This app can be used to generate training data for the char classifier convNet. The license plate text will be wrong for some images, and this will need to be manually corrected before using as training data. As the convNet performance improves, the number of character errors declines, and the hand editing of filenames becomes easier.
Similar to detectLPInVideoFiles.py, but detects plates in a live video feed.