Images Classification Recipe
Convolutional Neural Networks (CNN) and Transfer Learning for Heritage Images Classification
This recipe exposes an image classification scenario, aiming to deduce the technique or genre of images (picture, drawing, map...) using a Convolutional Neural Networks trained model (supervised approach with transfer learning).
This recipe may have various library use cases, particularly for cataloguing and information retrieval systems.
This recipe is based on BnF materials
This recipes includes a basic introduction to neural networks and deep learning (formal neuron model, neural networks, convolutional neural networks, transfer learning) and a hands-on session:
- Creation of an images dataset for training
- Training of a classification model with commercial APIs or open source IA frameworks
- Application of the model to the heritage images to be classified
The IIIF standard API is leveraged to extract images from digital repositories, but raw files may also be used.
For the theory, see the BnF github for a 45 mn introduction course (FR and EN versions, direct link).
Prerequisites: IBM Watson Studio account or Google Cloud AutoML account (see the setup documents, FR and EN versions)
A four classes scenario dataset (picture/drawing/map/noise) can be downloaded here, but it's up to you to build your own use case.
The dataset illustrates this scenario:
- filtering of "noisy" illustrations (blank pages, text pages)
- illustrations classification in 3 categories (picture, drawing, map)
IBM Watson Studio and Google Cloud AutoML have been tested and this howto documents the setup of a new user account and the creation of a visual recognition project for both platforms.
The following steps suppose you are using Watson Studio, but the Google AutoML case is very similar. This howto is also documented in a presentation (FR and EN versions).
Once Watson Studio web app is launched, choose the "Classify Images" custom model to create your new classification project.
In the platform
Outside the platform, using code
Prerequisites: basic scripting and command line skills (Python scripts are used)
An AI framework must be used: TensorFlow (Google), PyTorch (Facebook), CNTK (Microsoft), Caffe2, Keras, etc.
This implementation leverages the Inception-v3 model and applies a transfert learning method: the Inception-v3 model last layer is retrained on the images ground truth dataset.
First, TensorFlow must be installed.
Three Python scripts (within the Tensorflow framework) are used to train (and evaluate) a local model:
- split.py: the GT dataset is splitted in a training set (e.g. 2/3) and an evaluation set (1/3). The GT local dataset directory and the training/evaluation ratio must be defined in the script.
- retrain.py: the training set is used to train the last layer of the Inception-v3 model. The training dataset path and the generated model path must be defined. The Inception model is downloaded from the retrain.py script.
- label_image.py: the evaluation set is labeled by the model. The model path and the input images path must be defined.
To classify a set of images and output the results in a CSV file:
>python3 label_image.py > out.csv
Running the script outputs a line per classified image:
bd carte dessin filtrecouv filtretxt gravure photo foundClass realClass success imgTest
0.01 0.00 0.96 0.00 0.00 0.03 0.00 dessin OUT_img 0 btv1b10100491m-1-1.jpg
0.09 0.10 0.34 0.03 0.01 0.40 0.03 gravure OUT_img 0 btv1b10100495d-1-1.jpg
...
Each line describes the best classified class (according to its probability) and also the probability for all the other classes.
https://cloud.google.com/vision/automl/docs/beginners-guide?hl=en ...