Initial commit

.gitignore

@@ -0,0 +1,16 @@
+.venv
+__pycache__
+.ipynb_checkpoints
+.pytest_cache
+
+build/
+dist/
+*.egg-info
+*.so
+*.c
+
+tmp
+personal_experiment
+.idea
+outputs
+scripts

README.md

@@ -0,0 +1,154 @@
+# Robust And Decomposable Average Precision for Image Retrieval (NeurIPS 2021)
+
+This repository contains the source code for our [ROADMAP paper (NeurIPS 2021)](https://arxiv.org/abs/2110.01445).
+
+![outline](https://github.com/elias-ramzi/ROADMAP/blob/main/picture/outline.png)
+
+## Use ROADMAP
+
+```
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -e .
+```
+
+
+## Datasets
+
+We use the following datasets for our submission
+
+- CUB-200-2011 (download link available on this website : http://www.vision.caltech.edu/visipedia/CUB-200.html)
+- Stanford Online Products (you can download it here : https://cvgl.stanford.edu/projects/lifted_struct/)
+- INaturalist-2018 (obtained from here https://github.com/visipedia/inat_comp/tree/master/2018#Data)
+
+
+## Run the code
+
+<details>
+  <summary><b>SOP</b></summary><br/>
+
+  The following command reproduce our results for Table 4.
+
+  ```
+  CUDA_VISIBLE_DEVICES=0 python roadmap/single_experiment_runner.py \
+  'experience.experiment_name=sop_ROADMAP_${dataset.sampler.kwargs.batch_size}_sota' \
+  experience.seed=333 \
+  experience.max_iter=100 \
+  'experience.log_dir=${env:HOME}/experiments/ROADMAP' \
+  optimizer=sop \
+  model=resnet \
+  transform=sop_big \
+  dataset=sop \
+  dataset.sampler.kwargs.batch_size=128 \
+  dataset.sampler.kwargs.batches_per_super_pair=10 \
+  loss=roadmap
+  ```
+
+  With the transformer backbone :
+
+  ```
+  CUDA_VISIBLE_DEVICES=0 python roadmap/single_experiment_runner.py \
+  'experience.experiment_name=sop_ROADMAP_${dataset.sampler.kwargs.batch_size}_DeiT' \
+  experience.seed=333 \
+  experience.max_iter=75 \
+  'experience.log_dir=${env:HOME}/experiments/ROADMAP' \
+  optimizer=sop_deit \
+  model=deit \
+  transform=sop \
+  dataset=sop \
+  dataset.sampler.kwargs.batch_size=128 \
+  dataset.sampler.kwargs.batches_per_super_pair=10 \
+  loss=roadmap
+  ```
+</details>
+
+
+<details>
+  <summary><b>INaturalist</b></summary><br/>
+
+  For ROADMAP sota results:
+
+  ```
+  CUDA_VISIBLE_DEVICES='0,1,2' python roadmap/single_experiment_runner.py \
+  --multirun \
+  'experience.experiment_name=inat_ROADMAP_${dataset.sampler.kwargs.batch_size}_sota' \
+  experience.seed=333 \
+  experience.max_iter=90 \
+  'experience.log_dir=${env:HOME}/experiments/ROADMAP' \
+  optimizer=inaturalist \
+  model=resnet \
+  transform=inaturalist \
+  dataset=inaturalist \
+  dataset.sampler.kwargs.batch_size=384 \
+  loss=roadmap_inat
+  ```
+</details>
+
+
+<details>
+  <summary><b>CUB-200-2011</b></summary><br/>
+
+  For ROADMAP sota results:
+
+  ```
+  CUDA_VISIBLE_DEVICES=0 python roadmap/single_experiment_runner.py \
+  'experience.experiment_name=cub_ROADMAP_${dataset.sampler.kwargs.batch_size}_sota' \
+  experience.seed=333 \
+  experience.max_iter=200 \
+  'experience.log_dir=${env:HOME}/experiments/ROADMAP' \
+  optimizer=cub \
+  model=resnet_max_ln \
+  transform=cub_big \
+  dataset=cub \
+  dataset.sampler.kwargs.batch_size=128 \
+  loss=roadmap
+  ```
+
+  ```
+  CUDA_VISIBLE_DEVICES=0 python roadmap/single_experiment_runner.py \
+  'experience.experiment_name=cub_ROADMAP_${dataset.sampler.kwargs.batch_size}_sota_DeiT' \
+  experience.seed=333 \
+  experience.max_iter=150 \
+  'experience.log_dir=${env:HOME}/experiments/ROADMAP' \
+  optimizer=cub_deit \
+  model=deit \
+  transform=cub \
+  dataset=cub \
+  dataset.sampler.kwargs.batch_size=128 \
+  loss=roadmap
+  ```
+
+</details>
+
+
+The results are not exactly the same as my code changed a bit (for instance the random seed are not the same).
+
+
+## Contacts
+
+If you have any questions don't hesitate to create an issue on this repository. Or send me an email at elias.ramzi@lecnam.net.
+
+Don't hesitate to cite our work:
+```
+@inproceedings{
+ramzi2021robust,
+title={Robust and Decomposable Average Precision for Image Retrieval},
+author={Elias Ramzi and Nicolas THOME and Cl{\'e}ment Rambour and Nicolas Audebert and Xavier Bitot},
+booktitle={Thirty-Fifth Conference on Neural Information Processing Systems},
+year={2021},
+url={https://openreview.net/forum?id=VjQw3v3FpJx}
+}
+```
+
+
+## Resources
+- Pytorch Metric Learning (PML): https://github.com/KevinMusgrave/pytorch-metric-learning
+- SmoothAP: https://github.com/Andrew-Brown1/Smooth_AP
+- Blackbox: https://github.com/martius-lab/blackbox-backprop
+- FastAP: https://github.com/kunhe/FastAP-metric-learning
+- SoftBinAP: https://github.com/naver/deep-image-retrieval
+- timm: https://github.com/rwightman/pytorch-image-models
+- PyTorch: https://github.com/pytorch/pytorch
+- Hydra: https://github.com/facebookresearch/hydra
+- Faiss: https://github.com/facebookresearch/faiss
+- Ray: https://github.com/ray-project/ray

dev_requirements.txt

@@ -0,0 +1,12 @@
+wheel
+six
+appdirs
+ordered_set
+ipython
+jupyter
+ipdb
+autopep8
+flake8
+pylint
+isort
+jedi==0.17.1

requirements.txt

@@ -0,0 +1,17 @@
+numpy
+torch==1.8.1
+torchvision==0.9.1
+faiss-gpu==1.6.5
+pillow
+pandas
+tqdm
+tensorboard
+matplotlib
+pytorch-metric-learning
+timm
+ray[tune]
+ray[default]
+hydra-core==1.0.6
+hydra_colorlog==1.0.1
+hydra-ax-sweeper==1.1.0
+hydra-ray-launcher==0.1.4

roadmap/config/dataset/cub.yaml

@@ -0,0 +1,10 @@
+# @package _group_
+name: Cub200Dataset
+kwargs:
+  data_dir: /local/DEEPLEARNING/image_retrieval/CUB_200_2011
+
+sampler:
+  name: MPerClassSampler
+  kwargs:
+    batch_size: 128
+    samples_per_class: 4

roadmap/config/dataset/inaturalist.yaml

@@ -0,0 +1,10 @@
+# @package _group_
+name: INaturalistDataset
+kwargs:
+  data_dir: /local/DEEPLEARNING/image_retrieval/Inaturalist
+
+sampler:
+  name: MPerClassSampler
+  kwargs:
+    batch_size: 128
+    samples_per_class: 4

roadmap/config/dataset/inshop.yaml

@@ -0,0 +1,13 @@
+# @package _group_
+name: InShopDataset
+kwargs:
+  data_dir: /local/DEEPLEARNING/image_retrieval/inshop
+  hierarchy_mode: 'all'
+
+sampler:
+  name: HierarchicalSampler
+  kwargs:
+    batch_size: 128
+    samples_per_class: 8
+    batches_per_super_pair: 4
+    nb_categories: 2

roadmap/config/dataset/sfm120k.yaml

@@ -0,0 +1,19 @@
+# @package _group_
+name: SfM120kDataset
+kwargs:
+  data_dir: /local/DEEPLEARNING/image_retrieval/landmarks/sfm120k
+
+sampler:
+  name: MPerClassSampler
+  kwargs:
+    batch_size: 128
+    samples_per_class: 4
+
+evaluation:
+  - name: RevisitedDataset
+    kwargs:
+      data_dir: /local/DEEPLEARNING/image_retrieval/landmarks/rparis6k
+
+  - name: RevisitedDataset
+    kwargs:
+      data_dir: /local/DEEPLEARNING/image_retrieval/landmarks/roxford5k

roadmap/config/dataset/sop.yaml

@@ -0,0 +1,11 @@
+# @package _group_
+name: SOPDataset
+kwargs:
+  data_dir: /local/DEEPLEARNING/image_retrieval/Stanford_Online_Products
+
+sampler:
+  name: HierarchicalSampler
+  kwargs:
+    batch_size: 128
+    samples_per_class: 4
+    batches_per_super_pair: 10

roadmap/config/default.yaml

@@ -0,0 +1,27 @@
+defaults:
+
+    - experience: default
+
+    - dataset: sop
+
+    - loss: roadmap
+
+    - memory: default
+
+    - model: resnet
+
+    - optimizer: sop
+
+    - transform: sop
+
+    - hydra/job_logging: colorlog
+
+    - hydra/hydra_logging: colorlog
+
+hydra:
+  run:
+    dir: ${experience.log_dir}/${experience.experiment_name}/outputs
+
+  sweep:
+    dir: ${experience.log_dir}
+    subdir: ${experience.experiment_name}/outputs

roadmap/config/experience/default.yaml

@@ -0,0 +1,34 @@
+# @package _group_
+experiment_name: ???
+log_dir: /share/DEEPLEARNING/datasets/image_retrieval/experiments/
+seed: ???
+resume: null
+maybe_resume: False
+force_lr: null
+
+warm_up: -1
+warm_up_key: fc
+
+max_iter: 50
+
+train_eval_freq: -1
+val_eval_freq: -1
+test_eval_freq: 5
+eval_bs: 96
+save_model: 50
+eval_split: test
+principal_metric: mean_average_precision_at_r_level0
+log_grad: False
+with_AP: False
+landmarks: False
+
+split: null
+kfold: null
+split_random_state: null
+with_super_labels: False
+
+num_workers: 16
+pin_memory: True
+
+sub_batch: 128
+update_type: base_update

roadmap/config/experience/landmarks.yaml

@@ -0,0 +1,34 @@
+# @package _group_
+experiment_name: ???
+log_dir: /share/DEEPLEARNING/datasets/image_retrieval/experiments/
+seed: ???
+resume: null
+maybe_resume: False
+force_lr: null
+
+warm_up: -1
+warm_up_key: fc
+
+max_iter: 50
+
+train_eval_freq: -1
+val_eval_freq: -1
+test_eval_freq: 5
+eval_bs: 96
+save_model: 50
+eval_split: rparis6k
+principal_metric: mapH
+log_grad: False
+with_AP: False
+landmarks: True
+
+split: null
+kfold: null
+split_random_state: null
+with_super_labels: False
+
+num_workers: 16
+pin_memory: True
+
+sub_batch: 128
+update_type: base_update

roadmap/config/hydra/launcher/ray_launcher.yaml

@@ -0,0 +1,10 @@
+# @package hydra.launcher
+_target_: hydra_plugins.hydra_ray_launcher.ray_launcher.RayLauncher
+ray:
+  init:
+    address: null
+    _temp_dir: ${env:HOME}/experiments/tmp
+
+  remote:
+    num_gpus: 1
+    num_cpus: 16

roadmap/config/loss/affineap.yaml

@@ -0,0 +1,7 @@
+# @package _group_
+- name:  AffineAP
+  weight: 1.0
+  kwargs:
+    theta: 0.5
+    mu_n: 0.025
+    mu_p: 0.025

roadmap/config/loss/blackboxap.yaml

@@ -0,0 +1,6 @@
+# @package _group_
+- name: BlackBoxAP
+  weight: 1.0
+  kwargs:
+    lambda_val: 4.0
+    margin: 0.02