docs/tutorials/train_cnn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e5d6e344-3ed7-4b0a-aa36-2b83d4842bff",
   "metadata": {},
   "source": [
    "# Train a CNN\n",
    "\n",
    "Convolutional neural networks (CNNs) are popular tools for creating automated machine learning classifiers on images or image-like samples. By converting audio into a two-dimensional frequency vs. time representation such as a spectrogram, we can generate image-like samples that can be used to train CNNs. \n",
    "\n",
    "This tutorial demonstrates the basic use of OpenSoundscape's `preprocessors` and `cnn` modules for training CNNs and making predictions using CNNs.\n",
    "\n",
    "Under the hood, OpenSoundscape uses Pytorch for machine learning tasks. By using the class `opensoundscape.ml.cnn.CNN`, you can train and predict with PyTorch's powerful CNN architectures in just a few lines of code. \n",
    "\n",
    "## Run this tutorial\n",
    "\n",
    "This tutorial is more than a reference! It's a Jupyter Notebook which you can run and modify on Google Colab or your own computer.\n",
    "\n",
    "|Link to tutorial|How to run tutorial|\n",
    "| :- | :- |\n",
    "| [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/kitzeslab/opensoundscape/blob/master/docs/tutorials/train_cnn.ipynb) | The link opens the tutorial in Google Colab. Uncomment the \"installation\" line in the first cell to install OpenSoundscape. |\n",
    "| [![Download via DownGit](https://img.shields.io/badge/GitHub-Download-teal?logo=github)](https://minhaskamal.github.io/DownGit/#/home?url=https://github.com/kitzeslab/opensoundscape/blob/master/docs/tutorials/train_cnn.ipynb) | The link downloads the tutorial file to your computer. Follow the [Jupyter installation instructions](https://opensoundscape.org/en/latest/installation/jupyter.html), then open the tutorial file in Jupyter. |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b52ecca1-702b-4fa3-a48b-61025f55d8fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# if this is a Google Colab notebook, install opensoundscape in the runtime environment\n",
    "if 'google.colab' in str(get_ipython()):\n",
    "  %pip install opensoundscape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4d88b73-77d1-4c00-a83a-8466fd79e15e",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59c9eee8-c65c-4df1-95d0-15dda341ee0a",
   "metadata": {},
   "source": [
    "### Import needed packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "972e3e01-c85f-415d-95cc-9b695332f738",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/SML161/opensoundscape/opensoundscape/ml/cnn.py:18: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n",
      "  from tqdm.autonotebook import tqdm\n"
     ]
    }
   ],
   "source": [
    "# the cnn module provides classes for training/predicting with various types of CNNs\n",
    "from opensoundscape import CNN\n",
    "\n",
    "#other utilities and packages\n",
    "import torch\n",
    "import pandas as pd\n",
    "from pathlib import Path\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import random \n",
    "import subprocess\n",
    "from glob import glob\n",
    "import sklearn\n",
    "\n",
    "#set up plotting\n",
    "from matplotlib import pyplot as plt\n",
    "plt.rcParams['figure.figsize']=[15,5] #for large visuals\n",
    "%config InlineBackend.figure_format = 'retina'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22adf5d6-403d-4a06-bc85-477cdc60ec07",
   "metadata": {},
   "source": [
    "### Set random seeds\n",
    "\n",
    "Set manual seeds for Pytorch and Python. These essentially \"fix\" the results of any stochastic steps in model training, ensuring that training results are reproducible. You probably don't want to do this when you actually train your model, but it's useful for debugging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "68e09bd5-e86d-44e0-8ffa-0f8ee699c31f",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(0)\n",
    "random.seed(0)\n",
    "np.random.seed(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1c60bac-280a-4d72-80b6-2659f6ecd83d",
   "metadata": {},
   "source": [
    "### Download files\n",
    "\n",
    "Training a machine learning model requires some pre-labeled data. These data, in the form of audio recordings or spectrograms, are labeled with whether or not they contain the sound of the species of interest. \n",
    "\n",
    "These data can be obtained from online databases such as Xeno-Canto.org, or by labeling one's own ARU data using a program like Cornell's Raven sound analysis software. In this example we are using a set of annotated avian soundscape recordings that were annotated using the software Raven Pro 1.6.4 (Bioacoustics Research Program 2022):\n",
    "\n",
    "<blockquote><i>An annotated set of audio recordings of Eastern North American birds containing frequency, time, and species information. </i> Lauren M. Chronister,  Tessa A. Rhinehart,  Aidan Place,  Justin Kitzes.\n",
    "https://doi.org/10.1002/ecy.3329 \n",
    "</blockquote>\n",
    "\n",
    "These are the same data that are used by the annotation and preprocessing tutorials, so you can skip this step if you've already downloaded them there.\n",
    "\n",
    "Download the datasets to your current working directory and unzip them. You can do so by running the cell below OR\n",
    "\n",
    "- Download and unzip both `annotation_Files.zip` and `mp3_Files.zip` from the https://datadryad.org/stash/dataset/doi:10.5061/dryad.d2547d81z  \n",
    "- Move the unzipped contents into a subfolder of the current folder called `./annotated_data/`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7d8bf5cf-6c0b-43d6-a3bc-62657597fbec",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "# Note: the \"!\" preceding each line below allows us to run bash commands in a Jupyter notebook\n",
    "# If you are not running this code in a notebook, input these commands into your terminal instead\n",
    "!wget -O annotation_Files.zip https://datadryad.org/stash/downloads/file_stream/641805;\n",
    "!wget -O mp3_Files.zip https://datadryad.org/stash/downloads/file_stream/641807;\n",
    "!mkdir annotated_data;\n",
    "!unzip annotation_Files.zip -d ./annotated_data/Annotation_Files;\n",
    "!unzip mp3_Files.zip -d ./annotated_data/Recordings;"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82705d0a-f5f7-4104-8ea7-461ca7f72e4e",
   "metadata": {},
   "source": [
    "## Prepare audio data\n",
    "\n",
    "To prepare audio data for machine learning, we need to convert our annotated data into clip-level labels.\n",
    "\n",
    "These steps are covered in depth in other tutorials, so we'll just set our clip labels up quickly for this example.\n",
    "\n",
    "First, get exactly matched lists of audio files and their corresponding selection files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "61cbd28e-1e20-4709-95e7-dadf7f8b3f2c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Set the current directory to where the dataset is downloaded\n",
    "dataset_path = Path(\"./annotated_data/\")\n",
    "\n",
    "# Make a list of all of the selection table files\n",
    "selection_files = glob(f\"{dataset_path}/Annotation_Files/*/*.txt\")\n",
    "\n",
    "# Create a list of audio files, one corresponding to each Raven file\n",
    "# (Audio files have the same names as selection files with a different extension)\n",
    "audio_files = [f.replace('Annotation_Files','Recordings').replace('.Table.1.selections.txt','.mp3') for f in selection_files]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adc6709e-9508-4f08-b1ea-30d8662161b1",
   "metadata": {},
   "source": [
    "Next, convert the selection files and audio files to a `BoxedAnnotations` object, which contains the time, frequency, and label information for all annotations for every recording in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "77f3f7a5-e074-4313-a1bd-6b5a4c98612e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from opensoundscape.annotations import BoxedAnnotations\n",
    "# Create a dataframe of annotations\n",
    "annotations = BoxedAnnotations.from_raven_files(\n",
    "    selection_files,\n",
    "    audio_files)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0b8c74cb-3fbf-4f29-8ed5-d62f51b645a4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "# Parameters to use for label creation\n",
    "clip_duration = 3\n",
    "clip_overlap = 0\n",
    "min_label_overlap = 0.25\n",
    "species_of_interest = [\"NOCA\", \"EATO\", \"SCTA\", \"BAWW\", \"BCCH\", \"AMCR\", \"NOFL\"]\n",
    "\n",
    "# Create dataframe of one-hot labels\n",
    "clip_labels = annotations.one_hot_clip_labels(\n",
    "    clip_duration = clip_duration, \n",
    "    clip_overlap = clip_overlap,\n",
    "    min_label_overlap = min_label_overlap,\n",
    "    class_subset = species_of_interest # You can comment this line out if you want to include all species.\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "71d2b3ae-a37b-4e2a-a0c0-4bd41fce40ae",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>NOCA</th>\n",
       "      <th>EATO</th>\n",
       "      <th>SCTA</th>\n",
       "      <th>BAWW</th>\n",
       "      <th>BCCH</th>\n",
       "      <th>AMCR</th>\n",
       "      <th>NOFL</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>file</th>\n",
       "      <th>start_time</th>\n",
       "      <th>end_time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">annotated_data/Recordings/Recording_4/Recording_4_Segment_21.mp3</th>\n",
       "      <th>0.0</th>\n",
       "      <th>3.0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3.0</th>\n",
       "      <th>6.0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6.0</th>\n",
       "      <th>9.0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9.0</th>\n",
       "      <th>12.0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12.0</th>\n",
       "      <th>15.0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                        NOCA  \\\n",
       "file                                               start_time end_time         \n",
       "annotated_data/Recordings/Recording_4/Recording... 0.0        3.0        0.0   \n",
       "                                                   3.0        6.0        0.0   \n",
       "                                                   6.0        9.0        0.0   \n",
       "                                                   9.0        12.0       0.0   \n",
       "                                                   12.0       15.0       0.0   \n",
       "\n",
       "                                                                        EATO  \\\n",
       "file                                               start_time end_time         \n",
       "annotated_data/Recordings/Recording_4/Recording... 0.0        3.0        1.0   \n",
       "                                                   3.0        6.0        1.0   \n",
       "                                                   6.0        9.0        1.0   \n",
       "                                                   9.0        12.0       0.0   \n",
       "                                                   12.0       15.0       1.0   \n",
       "\n",
       "                                                                        SCTA  \\\n",
       "file                                               start_time end_time         \n",
       "annotated_data/Recordings/Recording_4/Recording... 0.0        3.0        0.0   \n",
       "                                                   3.0        6.0        0.0   \n",
       "                                                   6.0        9.0        0.0   \n",
       "                                                   9.0        12.0       1.0   \n",
       "                                                   12.0       15.0       0.0   \n",
       "\n",
       "                                                                        BAWW  \\\n",
       "file                                               start_time end_time         \n",
       "annotated_data/Recordings/Recording_4/Recording... 0.0        3.0        0.0   \n",
       "                                                   3.0        6.0        0.0   \n",
       "                                                   6.0        9.0        0.0   \n",
       "                                                   9.0        12.0       0.0   \n",
       "                                                   12.0       15.0       0.0   \n",
       "\n",
       "                                                                        BCCH  \\\n",
       "file                                               start_time end_time         \n",
       "annotated_data/Recordings/Recording_4/Recording... 0.0        3.0        0.0   \n",
       "                                                   3.0        6.0        0.0   \n",
       "                                                   6.0        9.0        0.0   \n",
       "                                                   9.0        12.0       0.0   \n",
       "                                                   12.0       15.0       0.0   \n",
       "\n",
       "                                                                        AMCR  \\\n",
       "file                                               start_time end_time         \n",
       "annotated_data/Recordings/Recording_4/Recording... 0.0        3.0        0.0   \n",
       "                                                   3.0        6.0        0.0   \n",
       "                                                   6.0        9.0        0.0   \n",
       "                                                   9.0        12.0       0.0   \n",
       "                                                   12.0       15.0       0.0   \n",
       "\n",
       "                                                                        NOFL  \n",
       "file                                               start_time end_time        \n",
       "annotated_data/Recordings/Recording_4/Recording... 0.0        3.0        0.0  \n",
       "                                                   3.0        6.0        0.0  \n",
       "                                                   6.0        9.0        0.0  \n",
       "                                                   9.0        12.0       0.0  \n",
       "                                                   12.0       15.0       0.0  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clip_labels.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7ec6fac-fb79-43dc-86c9-d66230189a94",
   "metadata": {},
   "source": [
    "## Create train, validation, and test datasets\n",
    "\n",
    "To train and test a model, we use three datasets:\n",
    "* The **training dataset** is used to fit your machine learning model to the audio data. \n",
    "* The **validation dataset** is a held-out dataset that is used to select hyperparameters (e.g. how many epochs to train for) during training\n",
    "* The **test dataset** is another held-out dataset that we use to check how the model performs on data that were not available at all during training.\n",
    "\n",
    "While both the training and validation datasets are used while trained the model, the test dataset is never touched until the model is fully trained and completed.\n",
    "\n",
    "The training and validation datasets may be gathered from the same source as each other. In contrast, the test dataset is often gathered from a different source to assess whether the model's performance generalizes to a real-world problem. For example, training and validation data might be drawn from an online database like Xeno-Canto, whereas the testing data is from your own field data. \n",
    "\n",
    "### Create a test dataset\n",
    "\n",
    "We'll separate the test dataset first. For a good assessment of the model's generalization, we want the test set to be independent of the training and validation datasets. For example, we don't want to use clips from the same source recording in the training dataset and the test dataset.\n",
    "\n",
    "For this example, we'll use the recordings in the folders `Recording_1`, `Recording_2` and `Recording_3` as our training and validation data, and use the recordings in folder `Recording_4` as our test data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d8190cbf-d9ad-400d-ad44-789eead2a656",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select all files from Recording_4 as a test set\n",
    "mask = clip_labels.reset_index()['file'].apply(lambda x: 'Recording_4' in x).values\n",
    "test_set = clip_labels[mask]\n",
    "\n",
    "# All other files will be used as a training set\n",
    "train_and_val_set = clip_labels.drop(test_set.index)\n",
    "\n",
    "# Save .csv tables of the training and validation sets to keep a record of them\n",
    "train_and_val_set.to_csv(\"./annotated_data/train_and_val_set.csv\")\n",
    "test_set.to_csv(\"./annotated_data/test_set.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b7fe29b-d6e7-4593-9b44-813c5aafb00b",
   "metadata": {},
   "source": [
    "If you wanted, you could load the training and testing set from these saved CSV files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "81f53802-c25f-4cbe-ab7f-531b80f38cec",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_and_val_set = pd.read_csv('./annotated_data/train_and_val_set.csv',index_col=[0,1,2])\n",
    "test_set = pd.read_csv('./annotated_data/test_set.csv',index_col=[0,1,2])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afb99584-33fc-4889-83b5-4c912e3c3188",
   "metadata": {},
   "source": [
    "### Split training and validation datasets\n",
    "\n",
    "Now, separate the remaining non-test data into training and validation datasets.\n",
    "\n",
    "The idea of keeping a separate validation dataset is that, throughout training, we can 'peek' at the performance on the validation set to choose hyperparameters. (This is in contrast to the test dataset, which we will not look at until we've finished training our model.)\n",
    "\n",
    "One important hyperparameter is the number of **epochs** to train to, in order to prevent overfitting. Each epoch includes one round of fitting on each training sample. \n",
    "\n",
    "If a model's performance on a training dataset continues to improve as it trains, but its performance on the validation dataset plateaus, this could incate the model is **overfitting** on the training dataset, learning information specific to those particular samples instead of gaining the ability to generalize to new data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "2f47db9c-bf65-46b9-b64b-040d13ea17e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split our training data into training and validation sets\n",
    "train_df, valid_df = sklearn.model_selection.train_test_split(train_and_val_set, test_size=0.1, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "74268296-4323-46c5-8a47-9f343f77844f",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df.to_csv(\"./annotated_data/train_set.csv\")\n",
    "valid_df.to_csv(\"./annotated_data/valid_set.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21d30e3e-eda1-4476-8ebf-db4b0844a1d0",
   "metadata": {},
   "source": [
    "### Resample data for even class representation\n",
    "\n",
    "Before training, we will balance the number of samples of each class in the training set. This helps the model learn all of the classes, rather than paying too much attention to the classes with the most labeled annotations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5a75f8ae-c81b-4a1b-b62e-87fe1b64eca0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from opensoundscape.data_selection import resample\n",
    "\n",
    "# upsample (repeat samples) so that all classes have 800 samples\n",
    "balanced_train_df = resample(train_df,n_samples_per_class=800,random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9730295-df2d-4fca-85d8-a7d756b1763f",
   "metadata": {},
   "source": [
    "## Set up model\n",
    "\n",
    "Now we create a model object. We have to select several parameters when creating this object: its `architecture`, `classes`, and `sample_duration`. \n",
    "\n",
    "Some additional parameters can also be changed at this step, such as the preprocessor used to create spectrograms and the shape of the spectrograms. \n",
    "\n",
    "For more detail on this step, see the [\"Customize CNN training\"](\"tutorials/CNN.html\") tutorial.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe66d592-fb5b-4e9d-a832-9ae123b9a442",
   "metadata": {},
   "source": [
    "### Create CNN object"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c5061ad-3fae-4b00-967e-f1101ff5165e",
   "metadata": {},
   "source": [
    "Now, create a CNN object with this architecture, the classes we put into the dataframe above, and the same sample duration as we selected above.\n",
    "\n",
    "The first time you run this script for a particular architecture, OpenSoundscape will download the desired architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "c61f98fb-0791-4e3d-ab51-ee36ae3e1dd5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a CNN object designed to recognize 3-second samples\n",
    "from opensoundscape import CNN\n",
    "\n",
    "# Use resnet34 architecture\n",
    "architecture = 'resnet34'\n",
    "\n",
    "# Can use this code to get your classes, if needed\n",
    "class_list = list(train_df.columns)\n",
    "\n",
    "model = CNN(\n",
    "    architecture = architecture,\n",
    "    classes = class_list,\n",
    "    sample_duration = clip_duration #3s, selected above\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f92a8de1-3d6b-4f03-bd61-dae8c17f1ddf",
   "metadata": {},
   "source": [
    "### Check model device\n",
    "\n",
    "If a GPU is available on your computer, the CNN object automatically selects it for accellerating performance. You can override `.device` to use a specific device such as `cpu` or `cuda:3`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "9de0c6df-d999-4791-b358-312a076f6888",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model.device is: mps\n"
     ]
    }
   ],
   "source": [
    "print(f'model.device is: {model.device}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c901111-323f-485d-bb45-f97a8abedafb",
   "metadata": {},
   "source": [
    "### Set up WandB model logging\n",
    "\n",
    "While this step is optional, it is very helpful for model training. In this step, we set up model logging on a service called **Weights & Biases** (AKA WandB). \n",
    "\n",
    "Weights & Biases is a free website you can use to monitor model training. It is integrated with OpenSoundscape to include helpful functions such as checking on your model's training progress in real time, visualizing the spectrograms created for training your model, comparing multiple tries at training the same model, and more. For more information, check out this [blog post](https://wandb.ai/wandb_fc/repo-spotlight/reports/Community-Spotlight-OpenSoundscape--Vmlldzo0MDcwMTI4). \n",
    "\n",
    "The instructions below will help you set up `wandb` logging:\n",
    "\n",
    "* Create an account on the [Weights and Biases website](https://wandb.ai/). \n",
    "* The first time you use `wandb`, you'll need to run `wandb.login()` in Python or `wandb login` on the command line, then enter the API key from your [settings](https://wandb.ai/settings) page\n",
    "* In a Python script where you want to log model training, use `wandb.init()` as demonstrated below. The \"Entity\" or team option allows runs and projects to be shared across members in a group, making it easy to collaborate and see progress of other team members' runs.\n",
    "\n",
    "\n",
    "As training progresses, performance metrics will be plotted to the wandb logging platform and visible on this run's web page. For example, this [wandb web page](https://wandb.ai/kitzeslab/opensoundscape%20training%20demo/runs/w1xyk7zr/workspace?workspace=user-samlapp) shows the content logged to wandb when this notebook was run by the Kitzes Lab. By default, OpenSoundscape + WandB integration creates several pages with information about the model:\n",
    "\n",
    "- Overview: hyperparameters, run description, and hardware available during the run\n",
    "- Charts: \"Samples\" panel with audio and images of preprocessed samples (useful for checking that your preprocessing performs as expected and your labels are correct)\n",
    "- Charts: graphs of each class's performance metrics over training time\n",
    "- Model: summary of model architecture\n",
    "- Logs: standard output of training script\n",
    "- System: computational performance metrics including memory, CPU use, etc\n",
    "\n",
    "When training several models and comparing performance, the \"Project\" page of WandB provides comparisons of metrics and hyperparameters across training runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "113a1a3c-1b0b-4159-83d7-43f7cc1a0d24",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Calling wandb.login() after wandb.init() has no effect.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Finishing last run (ID:j3o8kasj) before initializing another..."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "81dc01148d2a4be691fa148652276cb2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Label(value='0.001 MB of 0.001 MB uploaded (0.000 MB deduped)\\r'), FloatProgress(value=1.0, max…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">Train CNN</strong> at: <a href='https://wandb.ai/kitzeslab/OpenSoundscape%20tutorials/runs/j3o8kasj' target=\"_blank\">https://wandb.ai/kitzeslab/OpenSoundscape%20tutorials/runs/j3o8kasj</a><br/>Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>./wandb/run-20231016_150518-j3o8kasj/logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Successfully finished last run (ID:j3o8kasj). Initializing new run:<br/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f4a995c71a5540338a3cbc8760294e19",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Label(value='Waiting for wandb.init()...\\r'), FloatProgress(value=0.016753148600000028, max=1.0…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "wandb version 0.15.12 is available!  To upgrade, please run:\n",
       " $ pip install wandb --upgrade"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.13.11"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>/Users/SML161/opensoundscape/docs/tutorials/wandb/run-20231016_150544-1bqjhvna</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/kitzeslab/OpenSoundscape%20tutorials/runs/1bqjhvna' target=\"_blank\">Train CNN</a></strong> to <a href='https://wandb.ai/kitzeslab/OpenSoundscape%20tutorials' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/kitzeslab/OpenSoundscape%20tutorials' target=\"_blank\">https://wandb.ai/kitzeslab/OpenSoundscape%20tutorials</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/kitzeslab/OpenSoundscape%20tutorials/runs/1bqjhvna' target=\"_blank\">https://wandb.ai/kitzeslab/OpenSoundscape%20tutorials/runs/1bqjhvna</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import wandb\n",
    "try:\n",
    "    wandb.login()\n",
    "    wandb_session = wandb.init(\n",
    "        entity='kitzeslab', #replace with your entity/group name\n",
    "        project='OpenSoundscape tutorials',\n",
    "        name='Train CNN',\n",
    "    )\n",
    "except: #if wandb.init fails, don't use wandb logging\n",
    "    print('failed to create wandb session. wandb session will be None')\n",
    "    wandb_session = None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f865c2ff-441b-40eb-a6d9-7665452c5add",
   "metadata": {},
   "source": [
    "## Train the CNN\n",
    "\n",
    "Finally, train the CNN for two epoch. Typically, we would train the model for more than two epochs, but because training is slow and is much better done outside of a Jupyter Notebook, we just include this as a short demonstration of training.\n",
    "\n",
    "Each **epoch** is one pass-through of all of the samples in the training dataset, plus running predictions on the validation dataset. \n",
    "\n",
    "Each epoch is composed of smaller groups of samples called **batches**. The machine learning model predicts on every sample in the batch, then the model weights are updated based on those samples. Larger batches can increase training speed, but require more memory. If you get a memory error, try reducing the batch size.\n",
    "\n",
    "We use default training parameters, but many aspects of CNN training can be customized (see the \"Customize CNN training\" tutorial for examples)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "981bffa6-842e-4e76-bbf1-ad92a3a72dee",
   "metadata": {},
   "outputs": [],
   "source": [
    "checkpoint_folder = Path(\"model_training_checkpoints\")\n",
    "checkpoint_folder.mkdir(exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2b9afcf-fb51-45e8-9450-bcb2d17bac97",
   "metadata": {},
   "source": [
    "training on mps (Apple Silicon GPU) requires PyTorch >= 2.1.0. If we have an older "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "94dc7e68-b57c-4f8c-981c-d32d7c90d394",
   "metadata": {},
   "outputs": [],
   "source": [
    "# if model.device ==  torch.device('mps'):\n",
    "#     model.device=torch.device('cpu')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "2ea86e7f-5533-4815-bf34-31e141002dd2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training Epoch 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/site-packages/wandb/sdk/lib/import_hooks.py:243: DeprecationWarning: Deprecated since Python 3.4. Use importlib.util.find_spec() instead.\n",
      "  loader = importlib.find_loader(fullname, path)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e1c6d1fbaa27402eb8428af0ad5455e8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/88 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0 [batch 0/88, 0.00%] \n",
      "\tDistLoss: 0.727\n",
      "Metrics:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmp6jxpys8kwandb'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpac4qdf9gwandb'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmph6691dw2wandb'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmp4b_c1qyfwandb'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpt5aakdslwandb-artifacts'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpe9f877wswandb-artifacts'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpjc300kncwandb-artifacts'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpkxmcdxx5wandb-artifacts'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmp3tcagek9wandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpbj9hmspiwandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmphco9txwjwandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmprxqezoglwandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmp96z81r69wandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpo3mqj0yywandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpokv366frwandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpkg3tw4lywandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Metrics:\n",
      "\tMAP: 0.477\n",
      "\n",
      "Validation.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "038e574b2ec24a4a852378f057519998",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/8 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpvp05gl5wwandb'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpedbcjn25wandb'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpryrx0maewandb'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpa32pa72uwandb'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmp5wshcksjwandb-artifacts'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmprqzxphlmwandb-artifacts'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmp5lovd_21wandb-artifacts'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmp5ti4y2lkwandb-artifacts'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpyd3cln8rwandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmp7calzk13wandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmp4weete30wandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpjyxb9mobwandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpqbdk2k_wwandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpimi6bb3awandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmprb9ttemywandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmp8por1a1cwandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Metrics:\n",
      "\tMAP: 0.513\n",
      "\n",
      "Training Epoch 1\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d393a5094f954a359aaa6db436ae04fa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/88 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1 [batch 0/88, 0.00%] \n",
      "\tDistLoss: 0.391\n",
      "Metrics:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmp5vvkm7k7wandb'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmp6l6binkpwandb'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpnrtogtt4wandb'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmp7wtyaae4wandb'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpk59cslfywandb-artifacts'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpur9f_kznwandb-artifacts'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpbfzo72aqwandb-artifacts'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmp7n1z6q5zwandb-artifacts'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpz03a9iwjwandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpde3ups51wandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpm7468p11wandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpaio2dla_wandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpkbvv9nxlwandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmp07ldf49cwandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpumiww9fkwandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmps7uobq44wandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Metrics:\n",
      "\tMAP: 0.773\n",
      "\n",
      "Validation.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b82f25f75d0247f68f43fe4374b53a08",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/8 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpfndmgk36wandb'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmps0oxqyycwandb'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpy117imb9wandb'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmptv1n0sq9wandb'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmppg9bacviwandb-artifacts'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpnniababfwandb-artifacts'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpgvgcxj3uwandb-artifacts'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpazgxzpfmwandb-artifacts'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpnakqywpywandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpg5jub8qawandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpqro9quz8wandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpidenu7_lwandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpcb_7207kwandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpqezg4n7swandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmpyegs1litwandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/tmp5wyufb7pwandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Metrics:\n",
      "\tMAP: 0.681\n",
      "\n",
      "Best Model Appears at Epoch 1 with Validation score 0.681.\n"
     ]
    }
   ],
   "source": [
    "#%%capture --no-stdout --no-display\n",
    "# Uncomment the line above to silence outputs from this cell\n",
    "\n",
    "model.train(\n",
    "    balanced_train_df, \n",
    "    valid_df, \n",
    "    epochs = 2, \n",
    "    batch_size = 64, \n",
    "    log_interval = 100, #log progress every 100 batches\n",
    "    num_workers = 4, #4 parallelized cpu tasks for preprocessing\n",
    "    wandb_session = wandb_session,\n",
    "    save_interval = 10, #save checkpoint every 10 epochs\n",
    "    save_path = checkpoint_folder #location to save checkpoints\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b498f89-d856-45b5-bfe6-b9e94e603ada",
   "metadata": {},
   "source": [
    "Once this is finished running, you have trained the CNN. \n",
    "\n",
    "**Clean up:** Run the following cell to delete the files created in this tutorial. However, these files are used in other tutorials, so you may wish not to delete them just yet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "440ca518-abcd-4bac-94e8-12ff8b8e46b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil\n",
    "shutil.rmtree('./annotated_data')\n",
    "shutil.rmtree('./wandb')\n",
    "shutil.rmtree('./model_training_checkpoints')\n",
    "Path('annotation_Files.zip').unlink()\n",
    "Path('mp3_Files.zip').unlink()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "opso_dev",
   "language": "python",
   "name": "opso_dev"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}