notebooks/Demo 3 - Computing the ETAB score.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fc4ed6c0-a8d5-4127-b055-5cb7c2818216",
   "metadata": {},
   "source": [
    "<p align=\"center\">\n",
    "  <img src=\"assets/etab_logo.png\" width=\"320\">\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a46f6567-871d-48dc-a885-843fef6c520a",
   "metadata": {},
   "source": [
    "# Demo 3: Computing the ETAB score"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c616552a-f2e2-4959-abc0-537e4b4ddb98",
   "metadata": {},
   "source": [
    "This notebook provides a brief tutorial for computing the ETAB score. We start by importing the relevant modules as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "f2971bb9-1536-4755-a2c6-5a038ec3df32",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "current_dir = os.path.split(os.getcwd())[0]\n",
    "\n",
    "os.chdir(current_dir)\n",
    "\n",
    "from etab.scores import *\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "\n",
    "cuda_device      = 0\n",
    "device           = torch.device(\"cuda:%d\" % cuda_device if torch.cuda.is_available() else \"cpu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18dfdef6-aca3-46f0-b3c2-5bccb187a316",
   "metadata": {},
   "source": [
    "To compute the ETAB score of any pre-trained representation, we need to instantiate one of the backbone architectures supported in the ETAB zoo and then specify its pre-trained weights. In the example below, we instantiate a ResNet-50 backbone model pre-trained with ImageNet-1K weights. We can then pass the beackbone model and architecture type to the *ETABscore* function as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "6a8b96c3-78e2-4d0c-9109-7dcb55eecfbc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|| ETAB score >> Running benchmark task: a0-A4-E\n",
      "|| ETAB benchmark >>> Loading 700 echocardiograms with A4 views from the echonet dataset\n",
      "|| ETAB benchmark >>> Training a ResNet-50 backbone with a SegFormer head\n",
      "\u001b[35mEpoch: \u001b[36m 1/10 \u001b[35mTrain steps: \u001b[36m14 \u001b[35mVal steps: \u001b[36m3 \u001b[32m2.76s \u001b[35mloss:\u001b[94m 5.518127\u001b[35m acc:\u001b[94m 45.341902\u001b[35m fscore_macro:\u001b[94m 0.377684\u001b[35m jaccard_index:\u001b[94m 0.255285\u001b[35m val_loss:\u001b[94m 2.120828\u001b[35m val_acc:\u001b[94m 61.779820\u001b[35m val_fscore_macro:\u001b[94m 0.445834\u001b[35m val_jaccard_index:\u001b[94m 0.339720\u001b[0m\n",
      "Epoch 1: val_loss improved from inf to 2.12083, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 2/10 \u001b[35mTrain steps: \u001b[36m14 \u001b[35mVal steps: \u001b[36m3 \u001b[32m2.68s \u001b[35mloss:\u001b[94m 2.991097\u001b[35m acc:\u001b[94m 60.150308\u001b[35m fscore_macro:\u001b[94m 0.473989\u001b[35m jaccard_index:\u001b[94m 0.349475\u001b[35m val_loss:\u001b[94m 5.590778\u001b[35m val_acc:\u001b[94m 36.401440\u001b[35m val_fscore_macro:\u001b[94m 0.322638\u001b[35m val_jaccard_index:\u001b[94m 0.204345\u001b[0m\n",
      "\u001b[35mEpoch: \u001b[36m 3/10 \u001b[35mTrain steps: \u001b[36m14 \u001b[35mVal steps: \u001b[36m3 \u001b[32m2.70s \u001b[35mloss:\u001b[94m 1.479366\u001b[35m acc:\u001b[94m 75.153767\u001b[35m fscore_macro:\u001b[94m 0.565223\u001b[35m jaccard_index:\u001b[94m 0.451043\u001b[35m val_loss:\u001b[94m 1.060548\u001b[35m val_acc:\u001b[94m 80.051278\u001b[35m val_fscore_macro:\u001b[94m 0.562546\u001b[35m val_jaccard_index:\u001b[94m 0.465164\u001b[0m\n",
      "Epoch 3: val_loss improved from 2.12083 to 1.06055, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 4/10 \u001b[35mTrain steps: \u001b[36m14 \u001b[35mVal steps: \u001b[36m3 \u001b[32m2.72s \u001b[35mloss:\u001b[94m 0.791970\u001b[35m acc:\u001b[94m 84.597692\u001b[35m fscore_macro:\u001b[94m 0.639920\u001b[35m jaccard_index:\u001b[94m 0.531949\u001b[35m val_loss:\u001b[94m 0.642697\u001b[35m val_acc:\u001b[94m 85.693671\u001b[35m val_fscore_macro:\u001b[94m 0.643505\u001b[35m val_jaccard_index:\u001b[94m 0.537984\u001b[0m\n",
      "Epoch 4: val_loss improved from 1.06055 to 0.64270, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 5/10 \u001b[35mTrain steps: \u001b[36m14 \u001b[35mVal steps: \u001b[36m3 \u001b[32m2.67s \u001b[35mloss:\u001b[94m 0.503182\u001b[35m acc:\u001b[94m 89.870067\u001b[35m fscore_macro:\u001b[94m 0.698375\u001b[35m jaccard_index:\u001b[94m 0.593367\u001b[35m val_loss:\u001b[94m 0.727635\u001b[35m val_acc:\u001b[94m 85.871359\u001b[35m val_fscore_macro:\u001b[94m 0.657856\u001b[35m val_jaccard_index:\u001b[94m 0.549258\u001b[0m\n",
      "\u001b[35mEpoch: \u001b[36m 6/10 \u001b[35mTrain steps: \u001b[36m14 \u001b[35mVal steps: \u001b[36m3 \u001b[32m2.70s \u001b[35mloss:\u001b[94m 0.417174\u001b[35m acc:\u001b[94m 91.708634\u001b[35m fscore_macro:\u001b[94m 0.731224\u001b[35m jaccard_index:\u001b[94m 0.626806\u001b[35m val_loss:\u001b[94m 0.523509\u001b[35m val_acc:\u001b[94m 93.341041\u001b[35m val_fscore_macro:\u001b[94m 0.658345\u001b[35m val_jaccard_index:\u001b[94m 0.572811\u001b[0m\n",
      "Epoch 6: val_loss improved from 0.64270 to 0.52351, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 7/10 \u001b[35mTrain steps: \u001b[36m14 \u001b[35mVal steps: \u001b[36m3 \u001b[32m2.73s \u001b[35mloss:\u001b[94m 0.352086\u001b[35m acc:\u001b[94m 92.764749\u001b[35m fscore_macro:\u001b[94m 0.751119\u001b[35m jaccard_index:\u001b[94m 0.647851\u001b[35m val_loss:\u001b[94m 0.275833\u001b[35m val_acc:\u001b[94m 94.212572\u001b[35m val_fscore_macro:\u001b[94m 0.753632\u001b[35m val_jaccard_index:\u001b[94m 0.654111\u001b[0m\n",
      "Epoch 7: val_loss improved from 0.52351 to 0.27583, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 8/10 \u001b[35mTrain steps: \u001b[36m14 \u001b[35mVal steps: \u001b[36m3 \u001b[32m2.74s \u001b[35mloss:\u001b[94m 0.284964\u001b[35m acc:\u001b[94m 93.798573\u001b[35m fscore_macro:\u001b[94m 0.773825\u001b[35m jaccard_index:\u001b[94m 0.672388\u001b[35m val_loss:\u001b[94m 0.344805\u001b[35m val_acc:\u001b[94m 92.194308\u001b[35m val_fscore_macro:\u001b[94m 0.743430\u001b[35m val_jaccard_index:\u001b[94m 0.639182\u001b[0m\n",
      "\u001b[35mEpoch: \u001b[36m 9/10 \u001b[35mTrain steps: \u001b[36m14 \u001b[35mVal steps: \u001b[36m3 \u001b[32m2.72s \u001b[35mloss:\u001b[94m 0.242992\u001b[35m acc:\u001b[94m 94.551105\u001b[35m fscore_macro:\u001b[94m 0.796522\u001b[35m jaccard_index:\u001b[94m 0.697378\u001b[35m val_loss:\u001b[94m 0.223311\u001b[35m val_acc:\u001b[94m 95.272866\u001b[35m val_fscore_macro:\u001b[94m 0.793339\u001b[35m val_jaccard_index:\u001b[94m 0.695831\u001b[0m\n",
      "Epoch 9: val_loss improved from 0.27583 to 0.22331, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m10/10 \u001b[35mTrain steps: \u001b[36m14 \u001b[35mVal steps: \u001b[36m3 \u001b[32m2.80s \u001b[35mloss:\u001b[94m 0.232331\u001b[35m acc:\u001b[94m 94.782129\u001b[35m fscore_macro:\u001b[94m 0.803753\u001b[35m jaccard_index:\u001b[94m 0.705594\u001b[35m val_loss:\u001b[94m 0.318418\u001b[35m val_acc:\u001b[94m 94.672239\u001b[35m val_fscore_macro:\u001b[94m 0.735844\u001b[35m val_jaccard_index:\u001b[94m 0.639222\u001b[0m\n",
      "Restoring data from checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mTest steps: \u001b[36m7 \u001b[32m0.97s \u001b[35mtest_loss:\u001b[94m 0.235089\u001b[35m test_acc:\u001b[94m 95.064706\u001b[35m test_fscore_macro:\u001b[94m 0.793045\u001b[35m test_jaccard_index:\u001b[94m 0.695015\u001b[0m\n",
      "|| ETAB score >> Running benchmark task: a0-A4-C\n",
      "|| ETAB benchmark >>> Loading 700 echocardiograms with A4 views from the camus dataset\n",
      "|| ETAB benchmark >>> Training a ResNet-50 backbone with a SegFormer head\n",
      "\u001b[35mEpoch: \u001b[36m 1/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.28s \u001b[35mloss:\u001b[94m 2.016165\u001b[35m acc:\u001b[94m 76.745028\u001b[35m fscore_macro:\u001b[94m 0.603248\u001b[35m jaccard_index:\u001b[94m 0.481361\u001b[35m val_loss:\u001b[94m 1.296800\u001b[35m val_acc:\u001b[94m 80.004207\u001b[35m val_fscore_macro:\u001b[94m 0.546596\u001b[35m val_jaccard_index:\u001b[94m 0.455245\u001b[0m\n",
      "Epoch 1: val_loss improved from inf to 1.29680, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 2/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.21s \u001b[35mloss:\u001b[94m 0.991310\u001b[35m acc:\u001b[94m 85.457635\u001b[35m fscore_macro:\u001b[94m 0.713965\u001b[35m jaccard_index:\u001b[94m 0.593929\u001b[35m val_loss:\u001b[94m 0.983203\u001b[35m val_acc:\u001b[94m 88.283510\u001b[35m val_fscore_macro:\u001b[94m 0.628072\u001b[35m val_jaccard_index:\u001b[94m 0.535082\u001b[0m\n",
      "Epoch 2: val_loss improved from 1.29680 to 0.98320, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 3/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.24s \u001b[35mloss:\u001b[94m 0.803134\u001b[35m acc:\u001b[94m 87.389855\u001b[35m fscore_macro:\u001b[94m 0.743421\u001b[35m jaccard_index:\u001b[94m 0.626100\u001b[35m val_loss:\u001b[94m 0.941274\u001b[35m val_acc:\u001b[94m 89.276548\u001b[35m val_fscore_macro:\u001b[94m 0.677792\u001b[35m val_jaccard_index:\u001b[94m 0.575025\u001b[0m\n",
      "Epoch 3: val_loss improved from 0.98320 to 0.94127, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 4/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.92s \u001b[35mloss:\u001b[94m 0.561784\u001b[35m acc:\u001b[94m 90.378983\u001b[35m fscore_macro:\u001b[94m 0.791001\u001b[35m jaccard_index:\u001b[94m 0.681377\u001b[35m val_loss:\u001b[94m 0.747431\u001b[35m val_acc:\u001b[94m 88.590515\u001b[35m val_fscore_macro:\u001b[94m 0.720444\u001b[35m val_jaccard_index:\u001b[94m 0.608485\u001b[0m\n",
      "Epoch 4: val_loss improved from 0.94127 to 0.74743, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 5/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.26s \u001b[35mloss:\u001b[94m 0.413847\u001b[35m acc:\u001b[94m 92.130736\u001b[35m fscore_macro:\u001b[94m 0.820952\u001b[35m jaccard_index:\u001b[94m 0.718496\u001b[35m val_loss:\u001b[94m 0.650926\u001b[35m val_acc:\u001b[94m 89.299308\u001b[35m val_fscore_macro:\u001b[94m 0.730312\u001b[35m val_jaccard_index:\u001b[94m 0.619258\u001b[0m\n",
      "Epoch 5: val_loss improved from 0.74743 to 0.65093, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 6/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.23s \u001b[35mloss:\u001b[94m 0.383776\u001b[35m acc:\u001b[94m 92.837706\u001b[35m fscore_macro:\u001b[94m 0.835123\u001b[35m jaccard_index:\u001b[94m 0.736596\u001b[35m val_loss:\u001b[94m 0.938660\u001b[35m val_acc:\u001b[94m 90.220646\u001b[35m val_fscore_macro:\u001b[94m 0.689818\u001b[35m val_jaccard_index:\u001b[94m 0.587428\u001b[0m\n",
      "\u001b[35mEpoch: \u001b[36m 7/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.25s \u001b[35mloss:\u001b[94m 0.353041\u001b[35m acc:\u001b[94m 93.287176\u001b[35m fscore_macro:\u001b[94m 0.843346\u001b[35m jaccard_index:\u001b[94m 0.747444\u001b[35m val_loss:\u001b[94m 0.499454\u001b[35m val_acc:\u001b[94m 91.636993\u001b[35m val_fscore_macro:\u001b[94m 0.793349\u001b[35m val_jaccard_index:\u001b[94m 0.686935\u001b[0m\n",
      "Epoch 7: val_loss improved from 0.65093 to 0.49945, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 8/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.28s \u001b[35mloss:\u001b[94m 0.294357\u001b[35m acc:\u001b[94m 94.118820\u001b[35m fscore_macro:\u001b[94m 0.859494\u001b[35m jaccard_index:\u001b[94m 0.769220\u001b[35m val_loss:\u001b[94m 0.733796\u001b[35m val_acc:\u001b[94m 86.406074\u001b[35m val_fscore_macro:\u001b[94m 0.730805\u001b[35m val_jaccard_index:\u001b[94m 0.611692\u001b[0m\n",
      "\u001b[35mEpoch: \u001b[36m 9/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.26s \u001b[35mloss:\u001b[94m 0.302629\u001b[35m acc:\u001b[94m 94.176277\u001b[35m fscore_macro:\u001b[94m 0.863932\u001b[35m jaccard_index:\u001b[94m 0.775022\u001b[35m val_loss:\u001b[94m 0.483053\u001b[35m val_acc:\u001b[94m 91.640980\u001b[35m val_fscore_macro:\u001b[94m 0.786983\u001b[35m val_jaccard_index:\u001b[94m 0.680266\u001b[0m\n",
      "Epoch 9: val_loss improved from 0.49945 to 0.48305, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m10/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.26s \u001b[35mloss:\u001b[94m 0.268044\u001b[35m acc:\u001b[94m 94.637766\u001b[35m fscore_macro:\u001b[94m 0.870685\u001b[35m jaccard_index:\u001b[94m 0.784682\u001b[35m val_loss:\u001b[94m 0.471826\u001b[35m val_acc:\u001b[94m 92.105476\u001b[35m val_fscore_macro:\u001b[94m 0.803640\u001b[35m val_jaccard_index:\u001b[94m 0.699107\u001b[0m\n",
      "Epoch 10: val_loss improved from 0.48305 to 0.47183, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "Restoring data from checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mTest steps: \u001b[36m5 \u001b[32m0.88s \u001b[35mtest_loss:\u001b[94m 0.434163\u001b[35m test_acc:\u001b[94m 92.886520\u001b[35m test_fscore_macro:\u001b[94m 0.819357\u001b[35m test_jaccard_index:\u001b[94m 0.718379\u001b[0m\n",
      "|| ETAB score >> Running benchmark task: a0-A2-C\n",
      "|| ETAB benchmark >>> Loading 700 echocardiograms with A2 views from the camus dataset\n",
      "|| ETAB benchmark >>> Training a ResNet-50 backbone with a SegFormer head\n",
      "\u001b[35mEpoch: \u001b[36m 1/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.23s \u001b[35mloss:\u001b[94m 4.564139\u001b[35m acc:\u001b[94m 57.442389\u001b[35m fscore_macro:\u001b[94m 0.499453\u001b[35m jaccard_index:\u001b[94m 0.355438\u001b[35m val_loss:\u001b[94m 1.902234\u001b[35m val_acc:\u001b[94m 73.556589\u001b[35m val_fscore_macro:\u001b[94m 0.574509\u001b[35m val_jaccard_index:\u001b[94m 0.451992\u001b[0m\n",
      "Epoch 1: val_loss improved from inf to 1.90223, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 2/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.21s \u001b[35mloss:\u001b[94m 2.080506\u001b[35m acc:\u001b[94m 74.125486\u001b[35m fscore_macro:\u001b[94m 0.631547\u001b[35m jaccard_index:\u001b[94m 0.493747\u001b[35m val_loss:\u001b[94m 1.824893\u001b[35m val_acc:\u001b[94m 70.543731\u001b[35m val_fscore_macro:\u001b[94m 0.574348\u001b[35m val_jaccard_index:\u001b[94m 0.442468\u001b[0m\n",
      "Epoch 2: val_loss improved from 1.90223 to 1.82489, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 3/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.21s \u001b[35mloss:\u001b[94m 1.147256\u001b[35m acc:\u001b[94m 83.365945\u001b[35m fscore_macro:\u001b[94m 0.717845\u001b[35m jaccard_index:\u001b[94m 0.591509\u001b[35m val_loss:\u001b[94m 2.061541\u001b[35m val_acc:\u001b[94m 67.928669\u001b[35m val_fscore_macro:\u001b[94m 0.562897\u001b[35m val_jaccard_index:\u001b[94m 0.426836\u001b[0m\n",
      "\u001b[35mEpoch: \u001b[36m 4/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.22s \u001b[35mloss:\u001b[94m 0.834607\u001b[35m acc:\u001b[94m 86.938488\u001b[35m fscore_macro:\u001b[94m 0.757456\u001b[35m jaccard_index:\u001b[94m 0.638410\u001b[35m val_loss:\u001b[94m 1.544836\u001b[35m val_acc:\u001b[94m 80.434559\u001b[35m val_fscore_macro:\u001b[94m 0.679477\u001b[35m val_jaccard_index:\u001b[94m 0.550130\u001b[0m\n",
      "Epoch 4: val_loss improved from 1.82489 to 1.54484, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 5/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.22s \u001b[35mloss:\u001b[94m 0.614344\u001b[35m acc:\u001b[94m 89.965394\u001b[35m fscore_macro:\u001b[94m 0.796995\u001b[35m jaccard_index:\u001b[94m 0.686794\u001b[35m val_loss:\u001b[94m 0.791385\u001b[35m val_acc:\u001b[94m 85.079541\u001b[35m val_fscore_macro:\u001b[94m 0.721817\u001b[35m val_jaccard_index:\u001b[94m 0.599856\u001b[0m\n",
      "Epoch 5: val_loss improved from 1.54484 to 0.79138, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 6/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.24s \u001b[35mloss:\u001b[94m 0.513726\u001b[35m acc:\u001b[94m 91.345051\u001b[35m fscore_macro:\u001b[94m 0.817904\u001b[35m jaccard_index:\u001b[94m 0.713242\u001b[35m val_loss:\u001b[94m 0.601684\u001b[35m val_acc:\u001b[94m 89.159538\u001b[35m val_fscore_macro:\u001b[94m 0.767811\u001b[35m val_jaccard_index:\u001b[94m 0.654447\u001b[0m\n",
      "Epoch 6: val_loss improved from 0.79138 to 0.60168, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 7/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.19s \u001b[35mloss:\u001b[94m 0.438433\u001b[35m acc:\u001b[94m 92.734891\u001b[35m fscore_macro:\u001b[94m 0.840757\u001b[35m jaccard_index:\u001b[94m 0.743150\u001b[35m val_loss:\u001b[94m 0.629767\u001b[35m val_acc:\u001b[94m 91.322545\u001b[35m val_fscore_macro:\u001b[94m 0.809533\u001b[35m val_jaccard_index:\u001b[94m 0.703756\u001b[0m\n",
      "\u001b[35mEpoch: \u001b[36m 8/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.19s \u001b[35mloss:\u001b[94m 0.384203\u001b[35m acc:\u001b[94m 93.659019\u001b[35m fscore_macro:\u001b[94m 0.856821\u001b[35m jaccard_index:\u001b[94m 0.764932\u001b[35m val_loss:\u001b[94m 0.465367\u001b[35m val_acc:\u001b[94m 93.247146\u001b[35m val_fscore_macro:\u001b[94m 0.824423\u001b[35m val_jaccard_index:\u001b[94m 0.724971\u001b[0m\n",
      "Epoch 8: val_loss improved from 0.60168 to 0.46537, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 9/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.21s \u001b[35mloss:\u001b[94m 0.327370\u001b[35m acc:\u001b[94m 94.248727\u001b[35m fscore_macro:\u001b[94m 0.867213\u001b[35m jaccard_index:\u001b[94m 0.779417\u001b[35m val_loss:\u001b[94m 0.601120\u001b[35m val_acc:\u001b[94m 88.779142\u001b[35m val_fscore_macro:\u001b[94m 0.768360\u001b[35m val_jaccard_index:\u001b[94m 0.654017\u001b[0m\n",
      "\u001b[35mEpoch: \u001b[36m10/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.78s \u001b[35mloss:\u001b[94m 0.315899\u001b[35m acc:\u001b[94m 94.670463\u001b[35m fscore_macro:\u001b[94m 0.874618\u001b[35m jaccard_index:\u001b[94m 0.789948\u001b[35m val_loss:\u001b[94m 1.031990\u001b[35m val_acc:\u001b[94m 78.402336\u001b[35m val_fscore_macro:\u001b[94m 0.632872\u001b[35m val_jaccard_index:\u001b[94m 0.507684\u001b[0m\n",
      "Restoring data from checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mTest steps: \u001b[36m5 \u001b[32m0.87s \u001b[35mtest_loss:\u001b[94m 0.471768\u001b[35m test_acc:\u001b[94m 93.770609\u001b[35m test_fscore_macro:\u001b[94m 0.839656\u001b[35m test_jaccard_index:\u001b[94m 0.743991\u001b[0m\n",
      "|| ETAB score >> Running benchmark task: a1-A4-C\n",
      "|| ETAB benchmark >>> Loading 700 echocardiograms with A4 views from the camus dataset\n",
      "|| ETAB benchmark >>> Training a ResNet-50 backbone with a SegFormer head\n",
      "\u001b[35mEpoch: \u001b[36m 1/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.19s \u001b[35mloss:\u001b[94m 3.861156\u001b[35m acc:\u001b[94m 58.681603\u001b[35m fscore_macro:\u001b[94m 0.418960\u001b[35m jaccard_index:\u001b[94m 0.316351\u001b[35m val_loss:\u001b[94m 0.823491\u001b[35m val_acc:\u001b[94m 82.839516\u001b[35m val_fscore_macro:\u001b[94m 0.533667\u001b[35m val_jaccard_index:\u001b[94m 0.457087\u001b[0m\n",
      "Epoch 1: val_loss improved from inf to 0.82349, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 2/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.22s \u001b[35mloss:\u001b[94m 2.102879\u001b[35m acc:\u001b[94m 69.210933\u001b[35m fscore_macro:\u001b[94m 0.476248\u001b[35m jaccard_index:\u001b[94m 0.379734\u001b[35m val_loss:\u001b[94m 1.026111\u001b[35m val_acc:\u001b[94m 79.079377\u001b[35m val_fscore_macro:\u001b[94m 0.512577\u001b[35m val_jaccard_index:\u001b[94m 0.432406\u001b[0m\n",
      "\u001b[35mEpoch: \u001b[36m 3/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.25s \u001b[35mloss:\u001b[94m 1.159376\u001b[35m acc:\u001b[94m 79.293597\u001b[35m fscore_macro:\u001b[94m 0.532968\u001b[35m jaccard_index:\u001b[94m 0.444804\u001b[35m val_loss:\u001b[94m 1.038403\u001b[35m val_acc:\u001b[94m 81.595321\u001b[35m val_fscore_macro:\u001b[94m 0.531166\u001b[35m val_jaccard_index:\u001b[94m 0.451449\u001b[0m\n",
      "\u001b[35mEpoch: \u001b[36m 4/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.18s \u001b[35mloss:\u001b[94m 0.840889\u001b[35m acc:\u001b[94m 83.706825\u001b[35m fscore_macro:\u001b[94m 0.567536\u001b[35m jaccard_index:\u001b[94m 0.480284\u001b[35m val_loss:\u001b[94m 0.700434\u001b[35m val_acc:\u001b[94m 85.566536\u001b[35m val_fscore_macro:\u001b[94m 0.561035\u001b[35m val_jaccard_index:\u001b[94m 0.482485\u001b[0m\n",
      "Epoch 4: val_loss improved from 0.82349 to 0.70043, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 5/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.24s \u001b[35mloss:\u001b[94m 0.606028\u001b[35m acc:\u001b[94m 87.374626\u001b[35m fscore_macro:\u001b[94m 0.602769\u001b[35m jaccard_index:\u001b[94m 0.514930\u001b[35m val_loss:\u001b[94m 0.598294\u001b[35m val_acc:\u001b[94m 88.021678\u001b[35m val_fscore_macro:\u001b[94m 0.557317\u001b[35m val_jaccard_index:\u001b[94m 0.488536\u001b[0m\n",
      "Epoch 5: val_loss improved from 0.70043 to 0.59829, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 6/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.22s \u001b[35mloss:\u001b[94m 0.469880\u001b[35m acc:\u001b[94m 89.837312\u001b[35m fscore_macro:\u001b[94m 0.630321\u001b[35m jaccard_index:\u001b[94m 0.541626\u001b[35m val_loss:\u001b[94m 0.445879\u001b[35m val_acc:\u001b[94m 90.508031\u001b[35m val_fscore_macro:\u001b[94m 0.636230\u001b[35m val_jaccard_index:\u001b[94m 0.547919\u001b[0m\n",
      "Epoch 6: val_loss improved from 0.59829 to 0.44588, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 7/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.23s \u001b[35mloss:\u001b[94m 0.434318\u001b[35m acc:\u001b[94m 90.717431\u001b[35m fscore_macro:\u001b[94m 0.646427\u001b[35m jaccard_index:\u001b[94m 0.555861\u001b[35m val_loss:\u001b[94m 0.343352\u001b[35m val_acc:\u001b[94m 93.587861\u001b[35m val_fscore_macro:\u001b[94m 0.658944\u001b[35m val_jaccard_index:\u001b[94m 0.574027\u001b[0m\n",
      "Epoch 7: val_loss improved from 0.44588 to 0.34335, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 8/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.23s \u001b[35mloss:\u001b[94m 0.322319\u001b[35m acc:\u001b[94m 92.713192\u001b[35m fscore_macro:\u001b[94m 0.682821\u001b[35m jaccard_index:\u001b[94m 0.589385\u001b[35m val_loss:\u001b[94m 0.591458\u001b[35m val_acc:\u001b[94m 86.888865\u001b[35m val_fscore_macro:\u001b[94m 0.566362\u001b[35m val_jaccard_index:\u001b[94m 0.490167\u001b[0m\n",
      "\u001b[35mEpoch: \u001b[36m 9/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.22s \u001b[35mloss:\u001b[94m 0.303212\u001b[35m acc:\u001b[94m 93.056544\u001b[35m fscore_macro:\u001b[94m 0.689896\u001b[35m jaccard_index:\u001b[94m 0.595995\u001b[35m val_loss:\u001b[94m 0.330053\u001b[35m val_acc:\u001b[94m 92.349111\u001b[35m val_fscore_macro:\u001b[94m 0.644546\u001b[35m val_jaccard_index:\u001b[94m 0.559693\u001b[0m\n",
      "Epoch 9: val_loss improved from 0.34335 to 0.33005, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m10/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.21s \u001b[35mloss:\u001b[94m 0.298650\u001b[35m acc:\u001b[94m 92.998010\u001b[35m fscore_macro:\u001b[94m 0.695856\u001b[35m jaccard_index:\u001b[94m 0.600585\u001b[35m val_loss:\u001b[94m 0.272079\u001b[35m val_acc:\u001b[94m 94.454318\u001b[35m val_fscore_macro:\u001b[94m 0.669326\u001b[35m val_jaccard_index:\u001b[94m 0.584432\u001b[0m\n",
      "Epoch 10: val_loss improved from 0.33005 to 0.27208, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "Restoring data from checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mTest steps: \u001b[36m5 \u001b[32m0.89s \u001b[35mtest_loss:\u001b[94m 0.306084\u001b[35m test_acc:\u001b[94m 94.673726\u001b[35m test_fscore_macro:\u001b[94m 0.675263\u001b[35m test_jaccard_index:\u001b[94m 0.589586\u001b[0m\n",
      "|| ETAB score >> Running benchmark task: a1-A2-C\n",
      "|| ETAB benchmark >>> Loading 700 echocardiograms with A2 views from the camus dataset\n",
      "|| ETAB benchmark >>> Training a ResNet-50 backbone with a SegFormer head\n",
      "\u001b[35mEpoch: \u001b[36m 1/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.27s \u001b[35mloss:\u001b[94m 1.330988\u001b[35m acc:\u001b[94m 82.343493\u001b[35m fscore_macro:\u001b[94m 0.505277\u001b[35m jaccard_index:\u001b[94m 0.439450\u001b[35m val_loss:\u001b[94m 0.482524\u001b[35m val_acc:\u001b[94m 92.432504\u001b[35m val_fscore_macro:\u001b[94m 0.565890\u001b[35m val_jaccard_index:\u001b[94m 0.508739\u001b[0m\n",
      "Epoch 1: val_loss improved from inf to 0.48252, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 2/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.28s \u001b[35mloss:\u001b[94m 0.696768\u001b[35m acc:\u001b[94m 88.204461\u001b[35m fscore_macro:\u001b[94m 0.572249\u001b[35m jaccard_index:\u001b[94m 0.498198\u001b[35m val_loss:\u001b[94m 0.537446\u001b[35m val_acc:\u001b[94m 89.280668\u001b[35m val_fscore_macro:\u001b[94m 0.562197\u001b[35m val_jaccard_index:\u001b[94m 0.495736\u001b[0m\n",
      "\u001b[35mEpoch: \u001b[36m 3/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.25s \u001b[35mloss:\u001b[94m 0.509385\u001b[35m acc:\u001b[94m 90.003019\u001b[35m fscore_macro:\u001b[94m 0.595336\u001b[35m jaccard_index:\u001b[94m 0.518734\u001b[35m val_loss:\u001b[94m 0.385362\u001b[35m val_acc:\u001b[94m 92.623120\u001b[35m val_fscore_macro:\u001b[94m 0.598080\u001b[35m val_jaccard_index:\u001b[94m 0.529237\u001b[0m\n",
      "Epoch 3: val_loss improved from 0.48252 to 0.38536, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 4/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.25s \u001b[35mloss:\u001b[94m 0.386851\u001b[35m acc:\u001b[94m 91.970331\u001b[35m fscore_macro:\u001b[94m 0.638028\u001b[35m jaccard_index:\u001b[94m 0.553868\u001b[35m val_loss:\u001b[94m 0.421054\u001b[35m val_acc:\u001b[94m 91.458111\u001b[35m val_fscore_macro:\u001b[94m 0.608219\u001b[35m val_jaccard_index:\u001b[94m 0.531932\u001b[0m\n",
      "\u001b[35mEpoch: \u001b[36m 5/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.81s \u001b[35mloss:\u001b[94m 0.321803\u001b[35m acc:\u001b[94m 93.236009\u001b[35m fscore_macro:\u001b[94m 0.663673\u001b[35m jaccard_index:\u001b[94m 0.576412\u001b[35m val_loss:\u001b[94m 0.876540\u001b[35m val_acc:\u001b[94m 83.848145\u001b[35m val_fscore_macro:\u001b[94m 0.520867\u001b[35m val_jaccard_index:\u001b[94m 0.453226\u001b[0m\n",
      "\u001b[35mEpoch: \u001b[36m 6/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.31s \u001b[35mloss:\u001b[94m 0.315269\u001b[35m acc:\u001b[94m 93.155373\u001b[35m fscore_macro:\u001b[94m 0.667997\u001b[35m jaccard_index:\u001b[94m 0.579390\u001b[35m val_loss:\u001b[94m 0.336976\u001b[35m val_acc:\u001b[94m 92.701291\u001b[35m val_fscore_macro:\u001b[94m 0.638292\u001b[35m val_jaccard_index:\u001b[94m 0.556402\u001b[0m\n",
      "Epoch 6: val_loss improved from 0.38536 to 0.33698, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 7/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.26s \u001b[35mloss:\u001b[94m 0.253736\u001b[35m acc:\u001b[94m 94.173597\u001b[35m fscore_macro:\u001b[94m 0.696238\u001b[35m jaccard_index:\u001b[94m 0.604410\u001b[35m val_loss:\u001b[94m 0.303990\u001b[35m val_acc:\u001b[94m 94.618144\u001b[35m val_fscore_macro:\u001b[94m 0.662912\u001b[35m val_jaccard_index:\u001b[94m 0.580186\u001b[0m\n",
      "Epoch 7: val_loss improved from 0.33698 to 0.30399, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m 8/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.23s \u001b[35mloss:\u001b[94m 0.218045\u001b[35m acc:\u001b[94m 94.714664\u001b[35m fscore_macro:\u001b[94m 0.712062\u001b[35m jaccard_index:\u001b[94m 0.618933\u001b[35m val_loss:\u001b[94m 0.339068\u001b[35m val_acc:\u001b[94m 92.533793\u001b[35m val_fscore_macro:\u001b[94m 0.626953\u001b[35m val_jaccard_index:\u001b[94m 0.548014\u001b[0m\n",
      "\u001b[35mEpoch: \u001b[36m 9/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.28s \u001b[35mloss:\u001b[94m 0.196239\u001b[35m acc:\u001b[94m 95.278707\u001b[35m fscore_macro:\u001b[94m 0.727738\u001b[35m jaccard_index:\u001b[94m 0.633851\u001b[35m val_loss:\u001b[94m 0.282073\u001b[35m val_acc:\u001b[94m 94.826036\u001b[35m val_fscore_macro:\u001b[94m 0.671333\u001b[35m val_jaccard_index:\u001b[94m 0.587094\u001b[0m\n",
      "Epoch 9: val_loss improved from 0.30399 to 0.28207, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mEpoch: \u001b[36m10/10 \u001b[35mTrain steps: \u001b[36m9 \u001b[35mVal steps: \u001b[36m2 \u001b[32m2.35s \u001b[35mloss:\u001b[94m 0.206370\u001b[35m acc:\u001b[94m 95.142054\u001b[35m fscore_macro:\u001b[94m 0.726085\u001b[35m jaccard_index:\u001b[94m 0.632035\u001b[35m val_loss:\u001b[94m 0.261636\u001b[35m val_acc:\u001b[94m 94.808319\u001b[35m val_fscore_macro:\u001b[94m 0.679363\u001b[35m val_jaccard_index:\u001b[94m 0.593125\u001b[0m\n",
      "Epoch 10: val_loss improved from 0.28207 to 0.26164, saving file to checkpoints/latest_model/best_weight.ckpt\n",
      "Restoring data from checkpoints/latest_model/best_weight.ckpt\n",
      "\u001b[35mTest steps: \u001b[36m5 \u001b[32m0.89s \u001b[35mtest_loss:\u001b[94m 0.277636\u001b[35m test_acc:\u001b[94m 94.719564\u001b[35m test_fscore_macro:\u001b[94m 0.680977\u001b[35m test_jaccard_index:\u001b[94m 0.594089\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "from torchvision.models import resnet50\n",
    "\n",
    "input_backbone               = resnet50(pretrained=True)\n",
    "\n",
    "ETAB_score, task_scores_dict = ETABscore(backbone_architecture=\"ResNet-50\", \n",
    "                                         backbone_model=input_backbone, \n",
    "                                         task_weights=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19497a00-27b9-4393-9229-b6345639a950",
   "metadata": {},
   "source": [
    "The output of the *ETABScore* function is the score associated with the input model along with a breakdown of the performance for each benchmark task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "219e6db9-cda0-4979-a472-1955bf6ab602",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.6682118535041809,\n",
       " {'a0-A4-E': 0.6950145959854126,\n",
       "  'a0-A4-C': 0.7183789014816284,\n",
       "  'a0-A2-C': 0.7439906597137451,\n",
       "  'a1-A4-C': 0.5895856618881226,\n",
       "  'a1-A2-C': 0.5940894484519958})"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ETAB_score, task_scores_dict"
   ]
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "python3",
   "name": "pytorch-gpu.1-11.m91",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/pytorch-gpu.1-11:m91"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}