When citing this work, please include:
- Armstrong, S; Cooper, J; Daniels-Koch, O; and Gorman, R, “The HappyFaces Benchmark”,” Aligned AI Limited published public benchmark, 2022.
- Arigbabu, OA, et al. "Smile detection using hybrid face representation." Journal of Ambient Intelligence and Humanized Computing (2016): 1-12.
- Sanderson, C and Lovell, BC, Multi-Region Probabilistic Histograms for Robust and Scalable Identity Inference. ICB 2009, LNCS 5558, pp. 199-208, 2009
- Huang, GB; Mattar, M; Berg, T and Learned-Miller, E (2007), Labeled faces in the wild: a database for studying face recognition in unconstrained environments. University of Massachusetts, Amherst, Technical Report
We want to thank those who have contributed to making and labeling the faces we used in our benchmark.
Why did a classifier that was trained to identify collapsed lungs end up detecting chest drains instead?
Because the training data was insufficient to distinguish actual collapsed lungs from chest drains – a treatment for collapsed lungs. Chest drains are visually far simpler than collapsed lungs and the two features were correlated, so the algorithm was able to perform well by learning to identify the simplest feature.
Classifiers will generally learn the simplest feature that predicts the label, whether it is what we humans had in mind, or not. Human oversight can sometimes catch this error, but human oversight is slow, expensive, and not fully reliable (as the humans may not even realise what the algorithm is actually doing before a potentially dangerous mistake is made).
Detecting the ‘wrong’ feature means that the classifier will fail to generalise as expected – when deployed on X-rays of real humans with real, untreated, collapsed lungs, it will classify them as healthy, since they don’t have a chest drain.
We can solve this problem by using algorithms which can detect multiple different features that explain the same label. One way of achieving this would be to train an ensemble of algorithms, such that each makes predictions based on different features in the input. When these algorithms disagree, clarification from a human can be requested.
This benchmark allows the design and testing of those sorts of algorithms. It uses a simple dataset of human faces with different expressions and different labels, which are fully correlated in a labeled set, but which differ in an unlabeled set that represents the data the algorithm may encounter upon deployment in the real world.
The aim of this benchmark is to encourage the design of classifiers that are capable of using multiple different features to classify the same image. The features themselves must be deduced by the classifiers without being specifically labeled, though they may use a large unlabeled dataset on which the features vary. We have constructed a benchmark where the features are very different: facial expressions versus written text.
The images in these datasets consist of smiling (Happy) or non-smiling (Sad) faces. The images themselves come from the Labeled Faces in the Wild dataset, a publicly available dataset of faces, annotated with names.). The sub-lists of smiling and non-smiling faces come from the Dataset for Smile Detection from Face Images.
The images are modified by printing “HAPPY” or “SAD” in red letters upon the image. Here are some examples of the labeled data:
|
|
|
And here are some unlabeled examples that normal classifiers will misclassify:
|
|
|
The image names consist of the name of the person depicted (derived from the Labeled Faces in the Wild name for the file) followed by one of four tags:
- FHWH (Face Happy, Writing HAPPY): Smiling faces with “HAPPY” written across them.
- FHWS (Face Happy, Writing SAD): Smiling faces with “SAD” written across them.
- FSWH (Face Sad, Writing HAPPY): Unsmiling faces with “HAPPY” written across them.
- FSWS (Face Sad, Writing SAD): Unsmiling faces with “SAD” written across them.
FSWH and FSWS images in which the face and the writing do not express the same emotion are referred to as cross types.
The images are inside a folder named with the same tag – so a smiling face with “SAD” written on it has the FHWS tag in its name and is located in a FHWS subfolder.
The benchmark sets consist of three datasets, each constructed by randomly sampling from among the relevant images:
- The labeled dataset. This contains only images with the ‘correct’ text on them – FHWH and FSWS. It has 100 of each type, for a total of 200 images.
- The unlabeled dataset. This contains images of all four types, including the cross types – FHWS and FSWH. It has 150 images of each type, for a total of 600 images of which 300 are sampled for use during training according to a mix rate (see below).
- The validation dataset. This also contains images of all four types. It has 50 images of each type, for a total of 200 images.
There are thus 200 images each of types FHWS and FSWH, and 300 images each of types FHWH and FSWS, for 1000 images total. A list of the names of each image and the dataset it belongs to is included as the file “images_list.txt”. We have also kept back a test dataset, of the same size and composition as the validation dataset.
A binary classifier trained on the labeled data would learn to classify images based on text only, as the text perfectly correlates with the labels,and is a much simpler feature to learn than the facial expression. If this classifier was then used to infer the label for an ambiguous image – a cross type – it would then classify that image according to the text rather than the facial expression:
|
Train classifier once... |
…classify or mis-classify ambiguous image |
If the feature that we humans want the classifier to base its predictions on is actually the facial expression, we’re in trouble.
The challenge is then to develop an algorithm which produces two binary outputs for each image, where the first identifies happy facial expressions, and the second identifies the written text ‘HAPPY’.. Only labels for the images in the “labeled” dataset may be used during training. This labeled set has perfect correlation between facial expression and written text, so it is deliberately difficult to distinguish these two features.
|
Train multiple classifiers that differ on ambiguous images |
Labels from the unlabeled set may not be used, either explicitly or implicitly (which means that, for example, there must be nothing informative about the order in which unlabeled images are fed to the classifiers, nor can the classifier make use of the names of the images, which do include feature information).
Lee et al. propose Diversify and Disambiguate, which works well when the unlabeled dataset is evenly balanced between all four image categories – but we cannot assume this to be true for arbitrary unlabelled datasets in the wild.Imbalanced datasets can, of course, be rebalanced – however this is akin to manual labeling and as such is prohibitively costly and difficult to scale.
Thus these methods must function at different mix rates. The mix rate is a real number between 0 and 1 that denotes the proportion of cross types (FHWS and FSWH) in the unlabeled dataset. A mix rate of 0 has only FHWH and FSWS (as per the labeled dataset), a mix rate of 0.5 has equal amounts in each category, while a mix rate of 1 has only cross types FHWS and FSWH.
An unlabeled dataset with a mix rate of r is constructed as follows:
⌊150r/2⌋ images are randomly selected from the unlabeled FHWS subset. An equal number of images are randomly selected from the unlabeled FSWH subset.
Similarly, ⌈150(1-r)/2⌉ images are randomly selected from FHWH, and the same number from FSWS. Thus the unlabeled dataset always contains exactly 300 images (of the possible 600 in the full unlabeled dataset), no matter the mix rate.
Performance is assessed on the validation set only, according to the accuracy of the worst of the two outputs.
The validation dataset includes images from all four categories in equal quantities (50 images of each type).
Of the two outputs produced by the algorithm for each image, one should be designated the ‘face’ classifier, and the other the ‘writing’ classifier, such that the ‘face’ classifier positively identifies all images that contain a smiling facial expression (FHWH and FHWS), and the ‘writing’ classifier positively identifies all images that contain the written text ‘HAPPY’ (FHWH and FSWH).
Thus, perfect predictions would look like this:
|
Image Class |
Output 1 (‘face’ classifier) |
Output 2 (‘writing’ classifier) |
|
FHWH (“Face Happy, Writing HAPPY”) |
1 |
1 |
|
FHWS (“Face Happy, Writing SAD”) |
1 |
0 |
|
FSWH (“Face Sad, Writing HAPPY”) |
0 |
1 |
|
FSWS (“Face Sad, Writing SAD”) |
0 |
0 |
Overall performance is the performance of the worst of the two classifiers.
A few performance benchmarks to bear in mind:
- 50%. If one or both of the classifiers classify randomly, the expected accuracy is around 50%. This is not good performance.
- 75%. If both classifiers are perfectly correct on the labeled images (FHWH and FSWS) and guess randomly on the other images, expected accuracy is 50%+50%/2=75%. Accuracy must therefore be above 75% if the classifiers are truly detecting the distinct underlying features.
In more realistic settings with large datasets from the wild, these ambiguous cross type images may rarely appear. For example, because chest drains are such a common intervention for treating collapsed lungs, images of collapsed lungs without chest drains may be poorly represented in the training set – and so it is all the more important to be able to learn from them effectively.
For this reason, our benchmark is particularly concerned with algorithms that perform well when very few ambiguous images are present in the unlabeled set – and so performance is calculated against two metrics:
- What is the lowest mix-rate where the algorithm achieves greater than 90% accuracy? (Lower is better!)
- What is the average accuracy on mix-rates between 0% and 30%? (As measured by the AUC within this range, normalised so that perfect performance is 1)
Below we show the performance of two different algorithms on this benchmark: Diversify and Disambiguate (DivDis), and two versions of our forthcoming method Extrapolate, the second of which (Extrapolate low data) is tuned to optimise performance on sparse cross types:
The current state of the art performance on the HappyFaces benchmark provided by Extrapolate low data is as follows:
Please help us beat these benchmarks ^_^