Update readme.

LICENSE

@@ -1,3 +1,5 @@
+For all files except those in "misc" directory:
+
 MIT License
 
 Copyright (c) 2021 kosuke1701

README.md

@@ -1,11 +1,11 @@
 # Danbooru 2020 Zero-shot Anime Character Identification Dataset (ZACI-20)
 
-The goal of this dataset is creating human-level character identification models which do not require retraining on novel characters. The dataset is derived from [Danbooru2020 dataset](https://www.gwern.net/Danbooru2020).
+The goal of this dataset is creating human-level character identification models which do not require retraining on novel characters. The dataset is derived from [Danbooru2020 dataset](https://www.gwern.net/Danbooru2020) [Anonymous+2021].
 
 ## Features
 
-* Large-scale
-  - 1.45M images of 39K characters (train dataset).
+* Large-scale character face image dataset.
+  - 1.45M face images of 39K characters (train dataset).
 * Designed for zero-shot setting.
   - Characters in the test dataset do not appear in the train dataset, allowing us to test model performance on novel characters.
 * Human annotated test dataset.
@@ -14,13 +14,54 @@ The goal of this dataset is creating human-level character identification models
 
 ## Benchmarks
 
+### Random negative pairs
+
+* Negative image pairs with different character labels are randomly sampled in this test set.
+  - **Limitation:**
+    - Since negative pairs are sampled in a completely random manner, most of them are *easy negative* which consists of images of clearly different characters.
+    - Thus, model performance tends to be over-estimated.
+
+![](misc/random.png)
+<div style="text-align: center;">
+Figure 1. Examples of random negative pairs (each column).
+</div>
+
+<br><br>
+
+<div style="text-align: center;">
+Table 1. Performance of benchmark models (random negative pairs).
+</div>
+
 | model name | FPR (%) | FNR (%) | EER (%) | note |
 |---|---|---|---|---|
 | Human | 1.59 | 13.9 | N/A | by kosuke1701 |
 | ResNet-152 | **2.40** | 13.9 | 8.89 | w/ RandAug, Contrastive loss. [0206_resnet152](https://github.com/kosuke1701/AnimeCV/releases/download/0111_best_randaug/0206_resnet152.zip) by kosuke1701 |
 | SE-ResNet-152 | 2.43 | 13.9 | **8.15** | w/ RandAug, Contrastive loss. [0206_seresnet152](https://github.com/kosuke1701/AnimeCV/releases/download/0111_best_randaug/0206_seresnet152.zip) by kosuke1701 |
 | ResNet-18 | 5.08 | 13.9 | 9.59 | w/ RandAug, Contrastive loss. [0206_resnet18](https://github.com/kosuke1701/AnimeCV/releases/download/0111_best_randaug/0206_resnet18.zip) by kosuke1701 |
 
+### Adversarially sampled negative pairs
+
+* Negative image pairs which are most confusing to a trained model are kept as test set.
+  - 0206_resnet152 is used as the trained model.
+  - Negative image pairs are sorted by their predicted scores, and pairs with largest scores are selected.
+
+![](misc/adversarial.png)
+<div style="text-align: center;">
+Figure 2. Examples of adversarial negative pairs (each column). Those pairs highlight common errors by benchmark models.
+</div>
+
+<br><br>
+
+<div style="text-align: center;">
+Table 2. Performance of benchmark models (adversarial negative pairs).
+</div>
+
+| model name | FPR (%) | FNR (%) | EER (%) | note |
+|---|---|---|---|---|
+In progress.
+
+### Participation
+
 * **Your participation is welcome!!** Please create an issue if you want to add your model to this list.
 * Please do not use test dataset to tune hyperparameters!!
 * You can use external resources. However, please do not use any data with character labels in test dataset to ensure fair comparison.
@@ -49,21 +90,42 @@ The goal of this dataset is creating human-level character identification models
   - `python evaluate.py --test-pairs dataset/zaci20_test_pairs.csv --test-dataset-dir zaci20_test`
   - If you want to evaluate my benchmarks, download and unzip compressed model files. [AnimeCV]() should be installed to run my benchmarks.
 
+## Todo
+
+- [ ] Create more difficult test dataset by adversarially sample negative image pairs.
+
 ## Notes
+### Preprocessing
 
-* Face annotations of [AnimeCV](https://github.com/kosuke1701/AnimeCV) is used to crop images.
+* Face annotations by pre-trained EfficientDet model [Tan+2020] is used to crop images. The annotations are publicly available at [AnimeCV](https://github.com/kosuke1701/AnimeCV).
   - See https://github.com/kosuke1701/AnimeCV/releases/tag/0.0 for more details.
+* All cropped face images are resized to the size of `224 x 224`. 
 * `/` in original character labels of Danbooru 2020 is replaced by `__`.
-* I follow the methodology of [a previous work](https://github.com/grapeot/Danbooru2018AnimeCharacterRecognitionDataset) to construct this dataset.
-* Benchmark models by me (kosuke1701) is trained with [this code](https://github.com/kosuke1701/optuna-metric-learning).
+
+### Data selection
+* The methodology of [Wang,2019] is used to select images in this dataset.
+  - More specifically, I only kept the images with only one character tag and only one face annotation.
+* Character tags with only one image are removed.
+* I split the set of characters into train/test set.
+  - Characters in the test set is randomly selected from the set of characters with only two face images.
+  - In this way, the number of images in the train set is maximized while keeping the diversity of characters in the test set.
+* Since evaluating and annotating all negative pairs in the test set is time-consuming task, a subset of negative pairs are included in the test set.
+
+### Benchmarks
+* Benchmark models (0206_resnet, 0206_seresnet, 0206_resnet18) are trained with [this code](https://github.com/kosuke1701/optuna-metric-learning).
   - `python -u -m optuna_metric_learning.train --conf <CONFIG_FN> --model-def-fn examples/image_folder_example.py  --max-epoch 60 --patience 3 --n-fold 100`
   - Corresponding `<CONFIG_FN>`:
-    - `tuned_configs/resnet18.json`
+    - `tuned_configs/resnet18.json` (for 0206_resnet18)
+    - `tuned_configs/resnet152.json` (for 0206_resnet152)
+    - `tuned_configs/seresnet152.json` (for 0206_seresnet152)
   - Results of conducted hyperparameter tuning on my private dataset is listed in [this Google spreadsheet](https://docs.google.com/spreadsheets/d/1kf4XnnEpWFugO--S1zD2lOv8jWyPAyYnL66POnNZKEM/edit?usp=sharing).
 
-## Todo
+### Statistics
 
-- [ ] Create more difficult test dataset by adversarially sample negative image pairs.
+| | size (MB) | #images | #characters |
+|---|---|---|---|
+| train | 16,644  | 1,451,527 | 39.038 |
+| test | 26 | 2006 | 1003 |
 
 ## Citation
 
@@ -91,4 +153,12 @@ If you found this dataset or my benchmark models useful, please consider citing
         type = {dataset,model},
         year = {2021},
         month = {February} }
-```
+```
+
+## References
+
+[Wang,2019] Yan Wang. "Danbooru2018 Anime Character Recognition Dataset," 2019, github.com/grapeot/Danbooru2018AnimeCharacterRecognitionDataset (accessed: 2021-02-11).
+
+[Tan+2020] Tan, Mingxing, Ruoming Pang, and Quoc V. Le. "Efficientdet: Scalable and efficient object detection." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.
+
+[Anonymous+2021] Anonymous, The Danbooru Community, and Gwern Branwen.“Danbooru2020: A Large-Scale Crowdsourced and Tagged Anime Illustration Dataset.” 2021, www.gwern.net/Danbooru2020 (accessed: 2021-02-11).