notebooks/evaluate_model.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## In this notebook we will see how to evaluate on the following benchmarks:\n",
    "\n",
    "- Pittsburgh (pitts30k-val, pitts30k-test and pitts250k-test) [1]\n",
    "- MapillarySLS [2]\n",
    "- Cross Season [3]\n",
    "- ESSEX [3]\n",
    "- Inria Holidays [3]\n",
    "- Nordland [3]\n",
    "- SPED [3]\n",
    "\n",
    "[1] NetVLAD: CNN architecture for weakly supervised place recognition (https://github.com/Relja/netvlad)\n",
    "\n",
    "[2] Mapillary Street-Level Sequences: A Dataset for Lifelong Place Recognition (https://github.com/FrederikWarburg/mapillary_sls)\n",
    "\n",
    "[3] VPR-Bench: An Open-Source Visual Place Recognition Evaluation Framework with Quantifiable Viewpoint and Appearance Change (https://github.com/MubarizZaffar/VPR-Bench)\n",
    "\n",
    "You'll need to download Pittsburgh dataset from [1] (you need to email Relja for the dataset), MapillarySLS validation from [2]. For the other datasets, visit [3] for detail on their amazing benchmark, they also host those datasets on this link (https://surfdrive.surf.nl/files/index.php/s/sbZRXzYe3l0v67W), huge thanks.\n",
    "\n",
    "---\n",
    "\n",
    "**Note:** I rewrote the code for loading these datasets to ensure consistency in evaluation across all datasets and to improve its speed. The original code for these datasets was slow for valid reasons. For instance, VPR-Bench calculates multiple metrics, including latency, which requires individual image processing in the forward pass. MSLS offers various evaluation modes, such as Image_to_Image, Sequence_to_Sequence, Sequence_to_Image, among others. In this project, we focus solely on measuring recall@K and as a result, we can significantly speed up the validation process. Therefoe, you'll need to use the precomputed ground_truth that we provide in this repo (in the directory datasets).\n",
    "\n",
    "That being said, all you need to do is download the dataset and place it in a specific directory (we will need the dataset images). After that, you can hard-code the directory path into a global variable, as we will show in the following steps.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import sys\n",
    "sys.path.append('..') # append parent directory, we need it\n",
    "\n",
    "import torch\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "import torchvision.transforms as T\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from tqdm.notebook import tqdm\n",
    "from utils.validation import get_validation_recalls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "MEAN=[0.485, 0.456, 0.406]; STD=[0.229, 0.224, 0.225]\n",
    "\n",
    "IM_SIZE = (320, 320)\n",
    "\n",
    "def input_transform(image_size=IM_SIZE):\n",
    "    return T.Compose([\n",
    "        # T.Resize(image_size, interpolation=T.InterpolationMode.BICUBIC),\n",
    "\t\tT.Resize(image_size,  interpolation=T.InterpolationMode.BILINEAR),\n",
    "        \n",
    "        T.ToTensor(),\n",
    "        T.Normalize(mean=MEAN, std=STD)\n",
    "    ])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this project, we provide for each benchmark (or test dataset) a Dataset Class that encapsulates images sequentially as follows: \n",
    "\n",
    "$[R_1, R_2, ..., R_n, Q_1, Q_2, ..., Q_m]$ where $R_i$ are the reference images and $Q_i$ are the queries. We keep the number of references and queries as variables in the object so that we can split into references/queries later when evaluating. We also store a ground_truth matrix that indicates which references are prositives for each query.\n",
    "\n",
    "**Note:** make sure that for every [BenchmarkClass].py, the global variable DATASET_ROOT (where each dataset images are located) is well initialized, otherwise you won't be able to run the following steps. Also, GT_ROOT is the location of the precomputed ground_truth and filenames that WE PROVIDED (by default in ../datasets/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dataloaders.val.CrossSeasonDataset import CrossSeasonDataset\n",
    "from dataloaders.val.EssexDataset import EssexDataset\n",
    "from dataloaders.val.InriaDataset import InriaDataset\n",
    "from dataloaders.val.NordlandDataset import NordlandDataset\n",
    "from dataloaders.val.SPEDDataset import SPEDDataset\n",
    "from dataloaders.val.MapillaryDataset import MSLS\n",
    "from dataloaders.val.PittsburghDataset import PittsburghDataset\n",
    "\n",
    "\n",
    "\n",
    "def get_val_dataset(dataset_name, input_transform=input_transform()):\n",
    "    dataset_name = dataset_name.lower()\n",
    "    \n",
    "    if 'cross' in dataset_name:\n",
    "        ds = CrossSeasonDataset(input_transform = input_transform)\n",
    "    \n",
    "    elif 'essex' in dataset_name:\n",
    "        ds = EssexDataset(input_transform = input_transform)\n",
    "    \n",
    "    elif 'inria' in dataset_name:    \n",
    "        ds = InriaDataset(input_transform = input_transform)\n",
    "    \n",
    "    elif 'nordland' in dataset_name:    \n",
    "        ds = NordlandDataset(input_transform = input_transform)\n",
    "    \n",
    "    elif 'sped' in dataset_name:\n",
    "        ds = SPEDDataset(input_transform = input_transform)\n",
    "    \n",
    "    elif 'msls' in dataset_name:\n",
    "        ds = MSLS(input_transform = input_transform)\n",
    "\n",
    "    elif 'pitts' in dataset_name:\n",
    "        ds = PittsburghDataset(which_ds=dataset_name, input_transform = input_transform)\n",
    "    else:\n",
    "        raise ValueError\n",
    "    \n",
    "    num_references = ds.num_references\n",
    "    num_queries = ds.num_queries\n",
    "    ground_truth = ds.ground_truth\n",
    "    return ds, num_references, num_queries, ground_truth\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define a function to which we give a model, a dataloader and it returns the resulting representations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_descriptors(model, dataloader, device):\n",
    "    descriptors = []\n",
    "    with torch.no_grad():\n",
    "        for batch in tqdm(dataloader, 'Calculating descritptors...'):\n",
    "            imgs, labels = batch\n",
    "            output = model(imgs.to(device)).cpu()\n",
    "            descriptors.append(output)\n",
    "\n",
    "    return torch.cat(descriptors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now load a pre-trained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from main import VPRModel\n",
    "\n",
    "# define which device you'd like run experiments on (cuda:0 if you only have one gpu)\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "model = VPRModel(backbone_arch='resnet50', \n",
    "                 layers_to_crop=[],\n",
    "                 agg_arch='ConvAP',\n",
    "                 agg_config={'in_channels': 2048,\n",
    "                            'out_channels': 512,\n",
    "                            's1' : 2,\n",
    "                            's2' : 2},\n",
    "        )\n",
    "\n",
    "\n",
    "state_dict = torch.load('../LOGS/best_models/resnet50_ConvAP_512_2x2.ckpt') # link to the trained weights\n",
    "model.load_state_dict(state_dict)\n",
    "# model.load_state_dict(state_dict['state_dict'])\n",
    "model.eval()\n",
    "model = model.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running validation on one of the benchmarks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "89320d203ab14374b95eabdda39aed1d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Calculating descritptors...:   0%|          | 0/10 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Descriptor dimension 2048\n",
      "\n",
      "\n",
      "+----------------------------------------------------------------+\n",
      "|                   Performance on CrossSeason                   |\n",
      "+----------+--------+--------+--------+--------+--------+--------+\n",
      "|    K     |   1    |   5    |   10   |   15   |   20   |   25   |\n",
      "+----------+--------+--------+--------+--------+--------+--------+\n",
      "| Recall@K | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 |\n",
      "+----------+--------+--------+--------+--------+--------+--------+\n"
     ]
    }
   ],
   "source": [
    "# all_datasets = ['CrossSeason' ,'Essex' ,'Inria' ,'Nordland' ,'SPED' ,'MSLS']\n",
    "val_dataset_name = 'CrossSeason'\n",
    "batch_size = 40\n",
    "\n",
    "val_dataset, num_references, num_queries, ground_truth = get_val_dataset(val_dataset_name)\n",
    "val_loader = DataLoader(val_dataset, num_workers=4, batch_size=batch_size)\n",
    "\n",
    "descriptors = get_descriptors(model, val_loader, device)\n",
    "print(f'Descriptor dimension {descriptors.shape[1]}')\n",
    "\n",
    "# now we split into references and queries\n",
    "r_list = descriptors[ : num_references].cpu()\n",
    "q_list = descriptors[num_references : ].cpu()\n",
    "recalls_dict, preds = get_validation_recalls(r_list=r_list,\n",
    "                                    q_list=q_list,\n",
    "                                    k_values=[1, 5, 10, 15, 20, 25],\n",
    "                                    gt=ground_truth,\n",
    "                                    print_results=True,\n",
    "                                    dataset_name=val_dataset_name,\n",
    "                                    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating on all benchmarks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating on CrossSeason\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "946e290812a14b9f81cbafce6098d7c9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Calculating descritptors...:   0%|          | 0/10 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Descriptor dimension 2048\n",
      "\n",
      "\n",
      "+----------------------------------------------------------------+\n",
      "|                   Performance on CrossSeason                   |\n",
      "+----------+--------+--------+--------+--------+--------+--------+\n",
      "|    K     |   1    |   5    |   10   |   15   |   20   |   25   |\n",
      "+----------+--------+--------+--------+--------+--------+--------+\n",
      "| Recall@K | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 |\n",
      "+----------+--------+--------+--------+--------+--------+--------+\n",
      "========> DONE!\n",
      "\n",
      "\n",
      "Evaluating on Essex\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "37ea01e6d96247db874ed079cff48e29",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Calculating descritptors...:   0%|          | 0/11 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Descriptor dimension 2048\n",
      "\n",
      "\n",
      "+----------------------------------------------------------+\n",
      "|                   Performance on Essex                   |\n",
      "+----------+-------+-------+-------+-------+-------+-------+\n",
      "|    K     |   1   |   5   |   10  |   15  |   20  |   25  |\n",
      "+----------+-------+-------+-------+-------+-------+-------+\n",
      "| Recall@K | 74.29 | 93.33 | 97.14 | 98.57 | 99.05 | 99.52 |\n",
      "+----------+-------+-------+-------+-------+-------+-------+\n",
      "========> DONE!\n",
      "\n",
      "\n",
      "Evaluating on Inria\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dd6f9ca1a089406b933bd1bb00250d91",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Calculating descritptors...:   0%|          | 0/21 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Descriptor dimension 2048\n",
      "\n",
      "\n",
      "+----------------------------------------------------------+\n",
      "|                   Performance on Inria                   |\n",
      "+----------+-------+-------+-------+-------+-------+-------+\n",
      "|    K     |   1   |   5   |   10  |   15  |   20  |   25  |\n",
      "+----------+-------+-------+-------+-------+-------+-------+\n",
      "| Recall@K | 81.67 | 91.00 | 93.00 | 93.00 | 93.67 | 94.33 |\n",
      "+----------+-------+-------+-------+-------+-------+-------+\n",
      "========> DONE!\n",
      "\n",
      "\n",
      "Evaluating on MSLS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "875a746f0942447692191d76538e791c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Calculating descritptors...:   0%|          | 0/491 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Descriptor dimension 2048\n",
      "\n",
      "\n",
      "+----------------------------------------------------------+\n",
      "|                   Performance on MSLS                    |\n",
      "+----------+-------+-------+-------+-------+-------+-------+\n",
      "|    K     |   1   |   5   |   10  |   15  |   20  |   25  |\n",
      "+----------+-------+-------+-------+-------+-------+-------+\n",
      "| Recall@K | 83.38 | 90.27 | 92.16 | 93.65 | 93.78 | 94.05 |\n",
      "+----------+-------+-------+-------+-------+-------+-------+\n",
      "========> DONE!\n",
      "\n",
      "\n",
      "Evaluating on SPED\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f4dd57a18bce4b10a5a92a04a1ed2eef",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Calculating descritptors...:   0%|          | 0/31 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Descriptor dimension 2048\n",
      "\n",
      "\n",
      "+----------------------------------------------------------+\n",
      "|                   Performance on SPED                    |\n",
      "+----------+-------+-------+-------+-------+-------+-------+\n",
      "|    K     |   1   |   5   |   10  |   15  |   20  |   25  |\n",
      "+----------+-------+-------+-------+-------+-------+-------+\n",
      "| Recall@K | 81.55 | 90.77 | 93.41 | 94.73 | 96.05 | 96.38 |\n",
      "+----------+-------+-------+-------+-------+-------+-------+\n",
      "========> DONE!\n",
      "\n",
      "\n",
      "Evaluating on Nordland\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "494c82ebcae74815a371246b1ebcff60",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Calculating descritptors...:   0%|          | 0/759 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Descriptor dimension 2048\n",
      "\n",
      "\n",
      "+----------------------------------------------------------+\n",
      "|                 Performance on Nordland                  |\n",
      "+----------+-------+-------+-------+-------+-------+-------+\n",
      "|    K     |   1   |   5   |   10  |   15  |   20  |   25  |\n",
      "+----------+-------+-------+-------+-------+-------+-------+\n",
      "| Recall@K | 40.25 | 56.63 | 63.19 | 66.78 | 70.36 | 72.14 |\n",
      "+----------+-------+-------+-------+-------+-------+-------+\n",
      "========> DONE!\n",
      "\n",
      "\n",
      "Evaluating on pitts30k_test\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3096d874061344c6a4891e89161223f8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Calculating descritptors...:   0%|          | 0/421 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Descriptor dimension 2048\n",
      "\n",
      "\n",
      "+----------------------------------------------------------+\n",
      "|               Performance on pitts30k_test               |\n",
      "+----------+-------+-------+-------+-------+-------+-------+\n",
      "|    K     |   1   |   5   |   10  |   15  |   20  |   25  |\n",
      "+----------+-------+-------+-------+-------+-------+-------+\n",
      "| Recall@K | 90.57 | 95.13 | 96.21 | 96.83 | 97.20 | 97.52 |\n",
      "+----------+-------+-------+-------+-------+-------+-------+\n",
      "========> DONE!\n",
      "\n",
      "\n",
      "Evaluating on pitts250k_test\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fdcf1d1dded44bee80397fa70ba4f4e0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Calculating descritptors...:   0%|          | 0/2306 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Descriptor dimension 2048\n",
      "\n",
      "\n",
      "+----------------------------------------------------------+\n",
      "|              Performance on pitts250k_test               |\n",
      "+----------+-------+-------+-------+-------+-------+-------+\n",
      "|    K     |   1   |   5   |   10  |   15  |   20  |   25  |\n",
      "+----------+-------+-------+-------+-------+-------+-------+\n",
      "| Recall@K | 92.34 | 97.49 | 98.43 | 98.79 | 98.95 | 99.13 |\n",
      "+----------+-------+-------+-------+-------+-------+-------+\n",
      "========> DONE!\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "val_dataset_names = ['CrossSeason' ,'Essex' ,'Inria', 'MSLS', 'SPED', 'Nordland', 'pitts30k_test', 'pitts250k_test']\n",
    "batch_size = 40\n",
    "\n",
    "for val_name in val_dataset_names:\n",
    "    val_dataset, num_references, num_queries, ground_truth = get_val_dataset(val_name)\n",
    "    val_loader = DataLoader(val_dataset, num_workers=4, batch_size=batch_size)\n",
    "    print(f'Evaluating on {val_name}')\n",
    "    descriptors = get_descriptors(model, val_loader, device)\n",
    "    \n",
    "    print(f'Descriptor dimension {descriptors.shape[1]}')\n",
    "    r_list = descriptors[ : num_references]\n",
    "    q_list = descriptors[num_references : ]\n",
    "\n",
    "    recalls_dict, preds = get_validation_recalls(r_list=r_list,\n",
    "                                                q_list=q_list,\n",
    "                                                k_values=[1, 5, 10, 15, 20, 25],\n",
    "                                                gt=ground_truth,\n",
    "                                                print_results=True,\n",
    "                                                dataset_name=val_name,\n",
    "                                                faiss_gpu=False\n",
    "                                                )\n",
    "    del descriptors\n",
    "    print('========> DONE!\\n\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  },
  "vscode": {
   "interpreter": {
    "hash": "13e28b63dcb11f8cc8ca6da5fb3a89358e6ee1c494e903d975090f8ad11f1453"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}