Expert eyeglasses recommendation system with Generative Adversarial Networks written in Python, 2020.
Live Demo: Glasses:eyes:Expert
- Description
- Installation
- Reminder about API-services
- Additional features
- How to use ExpertGlasses
- Performance benchmarks
- Sources
The task of recommending eyeframes based on people's appearance is quite hard and there are no similar projects available at the moment. This system takes into account over 20 facial attributes, maps them into eyeglasses features with the help of expert module (which consists with over 40 branches). Expert module apply written beforehand rules to get necessary mappings. High interpretability of such approach guaranties user's understanding and loyality towards the system.
To get necessary attributes from the photo this system uses bunch of machine learning algorithms and existing free services, i.e BetaFaceAPI and Face++. Face++ allows to locate bounding boxes of faces in images, while BetaFace uses various classifiers to get most of the secondary-importance features. So to use this repository, you should have access to the internet.
To detect face shape probabilities, iris color, forehead size, jawtype type, skintone the system uses own pretrained convolutional neural networks (CNNs). These models run on your local machine on CPU.
To get all necessary eyeframes attributes, the large dataset (>8k records) of eyeframes was parsed and processed. Because in real life there are not so many eyeframe models in the local shop available, the generation of unique eyewear by given features was implemented. The system use a conditional GAN followed by Super Resolution GAN to create non-existing high-definition images of the eyeframes.
Requirements:
- Python 3.6 or greater
- requests
- numpy
- pandas
- opencv_python_headless
- scikit_image
- tensorflow
- Keras_Applications
- PySimpleGUI
For the full list of requirements consider to check requirements.txt
To install repository, use the following commands:
Windows:
$ git clone https://github.com/Defasium/expertglasses
$ cd expertglasses/
$ pip3 install -r requirements.txt
Linux:
$ git clone https://github.com/Defasium/expertglasses
$ cd expertglasses/
$ sudo pip3 install -r requirements.txt
Linux inside virtualenv:
$ git clone https://github.com/Defasium/expertglasses
$ cd expertglasses/
$ pip3 install -r requirements.txt
In current version without Face++ api key you can't use Expert Recommender System!
While BetaFace api has the same free api key for everyone with some limitations, allowing you to upload up to 500 images from your IP per day. Face++ utilizes a different strategy, requiring you to create an account on their site. In this account you can generate your own free api key and api secret in the Apps -> API key section:
After that consider changing values of API_KEY and API_SECRET in the faceplusplus.py module:
...
API_KEY = "api_key"
API_SECRET = "api_secret"
RETURN_ATTRIBUTES = "gender,beauty,age,smiling,headpose,ethnicity,skinstatus"
RETURN_LANDMARK = 1
...In the github repository there are no currently some functionality, e.g. pretrained GAN models and best selling eyeglasses. If you want to include this features in the system, you need to download necessary files. Links are listed bellow.
| Storage Service | Link |
|---|---|
| GDrive | |
| DropBox | |
| Mega |
Download 2 files, cgan.h5 and srgan.h5, and place them in utils/ directory.
| Storage Service | Link |
|---|---|
| GDrive | |
| DropBox | |
| Mega |
Download files, create a directory named abtest/ and extract man/ and woman/ directories inside of it.
At the current state there are only 2 ways of using expert eyeglasses recommender system: in your python code and via GUI.
To use this system, simply import class in your python code:
from expert_backend import ExpertEyeglassesRecommenderAfter that create an instance of this class with a specified path to the image:
ins = ExpertEyeglassesRecommender('test.jpg')
# by passing a `lang` parameter you can specify language, which will be used at explanation step
ins = ExpertEyeglassesRecommender('test.jpg', lang='en')Initialization of class may take quite a long time (from 30 second up to 2 minutes).
After initialization recomendations will be completed and to get the top 6 best images of eyeglasses use:
ins.plot_recommendations()By changing strategy value you can obtain different results:
# in standart strategy feature with the biggest value impacts the results the most
ins.plot_recommendations(strategy='standart')
# in factorized strategy (default value) features are divided into three main groups:
# shape of eyewear, its rim and other features, thus making results more various
ins.plot_recommendations(strategy='factorized')
# in factorized plus strategy every feature greatly influence the final result
# thus providing many varios nonsimilar eyeframes
ins.plot_recommendations(strategy='factorized_plus')
# in color only strategy the resulting images will have the same color as in color vector
ins.plot_recommendations(strategy='color_only')
# shape only strategy's results are similar to standart's one but doesn't take into account color
ins.plot_recommendations(strategy='shape_only')If you have downloaded abtest directory, you can also try recommending random most popular eyeglasses (so the system don't use any information about your appearance):
# will output 6 randomly chosen eyeglasses
ins.plot_recommendations(strategy='most_popular')To get explanation of the system try:
print(ins.description)To work with new image use:
ins.update_image('test2.jpg')
ins.expert_module()You can alse use url as an argument:
ins.update_image('https://github.com/Defasium/expertglasses/blob/master/assets/gui.png?raw=true')
ins.expert_module()To generate unique image with GANs (implying that you have downloaded pretrained models, look Download pretrained GAN models section) use:
image = ins.generate_unique(show=True)Will generate eyeglasses according to features, plot it with matplotlib if show is True and returns an numpy.ndarray
Graphical user interface is powered by PySimpleGui framework, which allows fast and simple prototyping for cross-platform systems.
To launch GUI, use the following command from terminal:
$ python gui.py
In the first popup window you should load an image to process, i.e. the fast start, module would automatically calculate all the necessary stuff in background. After that all machine learning models will be loaded in RAM (loading usually takes from 30 seconds up to 2 minutes). When progressbar will be filled, the main interface will appear.
In this section you can find information about current performance of the system, speed benchmarks and so on.
Speed benchmarks were tested on 2 different machines:
- Windows 10 with 4 cores 2.3 GHz Intel® Core™ i5-4200U and 4 GB of RAM (low spec)
- Linux Ubuntu 16.04 LTS with 4 cores 3.4 GHz Intel® Core™ i7-4770 and 20 GB of RAM (high spec)
The results are listed below:
| Action | Time, low spec | Time, high spec | Standart deviation, low spec | Standart deviation, high spec |
|---|---|---|---|---|
| Image alignment | 4.7s | 3.9s | 2.4s | 0.5s |
| Cached image alignment | 62.0ms | 39.1ms | 55.0ms | 5.7ms |
| BetaFace api request | 1.5s | 1.6s | 0.4s | 0.4s |
| Cached betaFace api request | 2.0μs | 0.7μs | 0.2μs | 0.3μs |
| Extraction of facial attributes | 2.7s | 2.3s | 1.6s | 1.2s |
| Cached extraction of facial attributes | 3.0μs | 2.4μs | 0.1μs | 1.8μs |
| Features translation | 0.78s | 0.11s | 0.62s | 0.54s |
| Cached features translation | 15.0μs | 3.2μs | 12.1μs | 1.6μs |
| Database search with cosine distances | 1.1s | 0.8s | 0.4s | 0.4s |
| Unique eyeglasses generation | 4.2s | 4.0s | 0.2s | 1.4s |
| Initialization of class instance | 44.7s | 27.2s | 1.9s | 2.7s |
To measure quality of recommendations A/B testing was made with ~50 participants. After that normalized discounted cumulative gain at top 5 items was calculated. The results are shown below:
As you can see, expert recommender system outperforms the results of randomly chosen best selling eyeframes.
For classifiers this system uses modified version of ShuffleNetV2, implemented by @opconty.
For upscaling generated images this system uses modified version of @krasserm Super Resolution GAN implementation.