identify_birds_using_their_voice.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "identify_birds_using_their_voice",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iRJQH_5dg7C3",
        "colab_type": "text"
      },
      "source": [
        "We will mainly use two libraries for audio acquisition and playback:\n",
        "1)Librosa\n",
        "It is a Python module to analyze audio signals in general but geared more towards music. It includes the nuts and bolts to build a MIR(Music information retrieval) system. It has been very well documented along with a lot of examples and tutorials.\n",
        "2)IPython.display.Audio\n",
        "IPython.display.Audio lets you play audio directly in a jupyter notebook."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BVeE3Q92gzV3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install librosa"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E7kRbw20hSb_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np # linear algebra\n",
        "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
        "import os"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XZVXCN1jhbBu",
        "colab_type": "text"
      },
      "source": [
        "Loading an audio file"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l8NrY5XehYTH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import librosa\n",
        "audio_data = 'birdsong-recognition/train_audio/nutwoo/XC462016.mp3'\n",
        "x , sr = librosa.load(audio_data)\n",
        "print(type(x), type(sr))\n",
        "print(x.shape, sr)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "69GgPB9Uhm9f",
        "colab_type": "text"
      },
      "source": [
        "This returns an audio time series as a numpy array with a default sampling rate(sr) of 22KHZ mono. We can change this behavior by resampling at 44.1KHz."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FkkY99lQhtwM",
        "colab_type": "text"
      },
      "source": [
        "Playing Audio:\n",
        "Using,IPython.display.Audio you can play the audio in your jupyter notebook"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zg-wpSx-hgZu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "librosa.load(audio_data, sr=44100)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PG6xVXUMhvju",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import IPython.display as ipd\n",
        "ipd.Audio(audio_data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SX4vk8QCh4kZ",
        "colab_type": "text"
      },
      "source": [
        "Visualizing Audio:\n",
        "We can plot the audio array using librosa.display.waveplot:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bxDBglduh0ci",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%matplotlib inline\n",
        "import matplotlib.pyplot as plt\n",
        "import librosa.display\n",
        "plt.figure(figsize=(14, 5))\n",
        "librosa.display.waveplot(x, sr=sr)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ez2pVLzqjYdW",
        "colab_type": "text"
      },
      "source": [
        "Spectrogram\n",
        "A spectrogram is a visual way of representing the signal strength, or “loudness”, of a signal over time at various frequencies present in a particular waveform. Not only can one see whether there is more or less energy at, for example, 2 Hz vs 10 Hz, but one can also see how energy levels vary over time.\n",
        "A spectrogram is usually depicted as a heat map, i.e., as an image with the intensity shown by varying the color or brightness.\n",
        "We can display a spectrogram using. librosa.display.specshow."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ageOjHxQh0vA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "X = librosa.stft(x)\n",
        "Xdb = librosa.amplitude_to_db(abs(X))\n",
        "plt.figure(figsize=(14, 5))\n",
        "librosa.display.specshow(Xdb, sr=sr, x_axis='time', y_axis='hz')\n",
        "plt.colorbar()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kAHB2y2AjlKu",
        "colab_type": "text"
      },
      "source": [
        ".stft() converts data into short term Fourier transform. STFT converts signals such that we can know the amplitude of the given frequency at a given time. Using STFT we can determine the amplitude of various frequencies playing at a given time of an audio signal. .specshow is used to display a spectrogram.\n",
        "The vertical axis shows frequencies (from 0 to 10kHz), and the horizontal axis shows the time of the clip. Since we see that all action is taking place at the bottom of the spectrum, we can convert the frequency axis to a logarithmic one."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VJeS0QT-jpXj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "librosa.display.specshow(Xdb, sr=sr, x_axis='time', y_axis='log')\n",
        "plt.colorbar()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I0pHsXCojuvR",
        "colab_type": "text"
      },
      "source": [
        "Create an Audio Signal:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "prdvFprfjuBm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sr = 22050 # sample rate\n",
        "T = 5.0    # seconds\n",
        "t = np.linspace(0, T, int(T*sr), endpoint=False) # time variable\n",
        "x = 0.5*np.sin(2*np.pi*220*t)# pure sine wave at 220 Hz\n",
        "#Playing the audio\n",
        "ipd.Audio(x, rate=sr) # load a NumPy array\n",
        "#Saving the audio\n",
        "librosa.output.write_wav('tone_220.wav', x, sr)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nPS3ELc3j0DQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import sklearn\n",
        "spectral_centroids = librosa.feature.spectral_centroid(x, sr=sr)[0]\n",
        "spectral_centroids.shape\n",
        "# Computing the time variable for visualization\n",
        "plt.figure(figsize=(12, 4))\n",
        "frames = range(len(spectral_centroids))\n",
        "t = librosa.frames_to_time(frames)\n",
        "# Normalising the spectral centroid for visualisation\n",
        "def normalize(x, axis=0):\n",
        "    return sklearn.preprocessing.minmax_scale(x, axis=axis)\n",
        "#Plotting the Spectral Centroid along the waveform\n",
        "librosa.display.waveplot(x, sr=sr, alpha=0.4)\n",
        "plt.plot(t, normalize(spectral_centroids), color='b')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3BzX1pTokBWg",
        "colab_type": "text"
      },
      "source": [
        ".spectral_centroid will return an array with columns equal to a number of frames present in your sample."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y2SBdiddkCe6",
        "colab_type": "text"
      },
      "source": [
        "Spectral Rolloff:\n",
        "It is a measure of the shape of the signal. It represents the frequency at which high frequencies decline to 0. To obtain it, we have to calculate the fraction of bins in the power spectrum where 85% of its power is at lower frequencies.\n",
        "librosa.feature.spectral_rolloff computes the rolloff frequency for each frame in a signal:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KnWIQTPdkGyr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "spectral_rolloff = librosa.feature.spectral_rolloff(x+0.01, sr=sr)[0]\n",
        "plt.figure(figsize=(12, 4))\n",
        "librosa.display.waveplot(x, sr=sr, alpha=0.4)\n",
        "plt.plot(t, normalize(spectral_rolloff), color='r')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y7zVW4XrkNv1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "spectral_bandwidth_2 = librosa.feature.spectral_bandwidth(x+0.01, sr=sr)[0]\n",
        "spectral_bandwidth_3 = librosa.feature.spectral_bandwidth(x+0.01, sr=sr, p=3)[0]\n",
        "spectral_bandwidth_4 = librosa.feature.spectral_bandwidth(x+0.01, sr=sr, p=4)[0]\n",
        "plt.figure(figsize=(15, 9))\n",
        "librosa.display.waveplot(x, sr=sr, alpha=0.4)\n",
        "plt.plot(t, normalize(spectral_bandwidth_2), color='r')\n",
        "plt.plot(t, normalize(spectral_bandwidth_3), color='g')\n",
        "plt.plot(t, normalize(spectral_bandwidth_4), color='y')\n",
        "plt.legend(('p = 2', 'p = 3', 'p = 4'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HPdRgv7AlxCQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x, sr = librosa.load(audio_data)\n",
        "#Plot the signal:\n",
        "plt.figure(figsize=(14, 5))\n",
        "librosa.display.waveplot(x, sr=sr)\n",
        "# Zooming in\n",
        "n0 = 9000\n",
        "n1 = 9100\n",
        "plt.figure(figsize=(14, 5))\n",
        "plt.plot(x[n0:n1])\n",
        "plt.grid()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5FYXXh-QnXVk",
        "colab_type": "text"
      },
      "source": [
        "A zero-crossing is a point where the sign of a mathematical function changes (e.g. from positive to negative)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vGjZXovrlxYz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "zero_crossings = librosa.zero_crossings(x[n0:n1], pad=False)\n",
        "print(sum(zero_crossings))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5dyfC3GcncF8",
        "colab_type": "text"
      },
      "source": [
        "Mel-Frequency Cepstral Coefficients(MFCCs)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AM3rFZLdl3Ad",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "fs=10\n",
        "mfccs = librosa.feature.mfcc(x, sr=fs)\n",
        "print(mfccs.shape)\n",
        "(20, 97)\n",
        "#Displaying  the MFCCs:\n",
        "plt.figure(figsize=(15, 7))\n",
        "librosa.display.specshow(mfccs, sr=sr, x_axis='time')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Odhl_yPYnhiH",
        "colab_type": "text"
      },
      "source": [
        "Chroma feature"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BIwfTW8Mncys",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "hop_length=12\n",
        "chromagram = librosa.feature.chroma_stft(x, sr=sr, hop_length=hop_length)\n",
        "plt.figure(figsize=(15, 5))\n",
        "librosa.display.specshow(chromagram, x_axis='time', y_axis='chroma', hop_length=hop_length, cmap='coolwarm')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Cxs7bK9Ln62q",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sub = pd.read_csv(\"birdsong-recognition/sample_submission.csv\")\n",
        "sub.to_csv(\"submission.csv\", index=False)  # this will be overwritten if everything goes well"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l8emcnOxnog1",
        "colab_type": "text"
      },
      "source": [
        "Resnet Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2AJ5gazGn8yI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class ResNet(nn.Module):\n",
        "    def __init__(self, base_model_name: str, pretrained=False,\n",
        "                 num_classes=264):\n",
        "        super().__init__()\n",
        "        base_model = models.__getattribute__(base_model_name)(\n",
        "            pretrained=pretrained)\n",
        "        layers = list(base_model.children())[:-2]\n",
        "        layers.append(nn.AdaptiveMaxPool2d(1))\n",
        "        self.encoder = nn.Sequential(*layers)\n",
        "\n",
        "        in_features = base_model.fc.in_features\n",
        "\n",
        "        self.classifier = nn.Sequential(\n",
        "            nn.Linear(in_features, 1024), nn.ReLU(), nn.Dropout(p=0.2),\n",
        "            nn.Linear(1024, 1024), nn.ReLU(), nn.Dropout(p=0.2),\n",
        "            nn.Linear(1024, num_classes))\n",
        "\n",
        "    def forward(self, x):\n",
        "        batch_size = x.size(0)\n",
        "        x = self.encoder(x).view(batch_size, -1)\n",
        "        x = self.classifier(x)\n",
        "        multiclass_proba = F.softmax(x, dim=1)\n",
        "        multilabel_proba = F.sigmoid(x)\n",
        "        return {\n",
        "            \"logits\": x,\n",
        "            \"multiclass_proba\": multiclass_proba,\n",
        "            \"multilabel_proba\": multilabel_proba\n",
        "        }\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GMXOKoPCpe6m",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model_config = {\n",
        "    \"base_model_name\": \"resnet50\",\n",
        "    \"pretrained\": False,\n",
        "    \"num_classes\": 264\n",
        "}\n",
        "\n",
        "melspectrogram_parameters = {\n",
        "    \"n_mels\": 128,\n",
        "    \"fmin\": 20,\n",
        "    \"fmax\": 16000\n",
        "}\n",
        "\n",
        "weights_path = \"birdcall-resnet50-init-weights/best.pth\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UiFySwROpoqx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "BIRD_CODE = {\n",
        "    'aldfly': 0, 'ameavo': 1, 'amebit': 2, 'amecro': 3, 'amegfi': 4,\n",
        "    'amekes': 5, 'amepip': 6, 'amered': 7, 'amerob': 8, 'amewig': 9,\n",
        "    'amewoo': 10, 'amtspa': 11, 'annhum': 12, 'astfly': 13, 'baisan': 14,\n",
        "    'baleag': 15, 'balori': 16, 'banswa': 17, 'barswa': 18, 'bawwar': 19,\n",
        "    'belkin1': 20, 'belspa2': 21, 'bewwre': 22, 'bkbcuc': 23, 'bkbmag1': 24,\n",
        "    'bkbwar': 25, 'bkcchi': 26, 'bkchum': 27, 'bkhgro': 28, 'bkpwar': 29,\n",
        "    'bktspa': 30, 'blkpho': 31, 'blugrb1': 32, 'blujay': 33, 'bnhcow': 34,\n",
        "    'boboli': 35, 'bongul': 36, 'brdowl': 37, 'brebla': 38, 'brespa': 39,\n",
        "    'brncre': 40, 'brnthr': 41, 'brthum': 42, 'brwhaw': 43, 'btbwar': 44,\n",
        "    'btnwar': 45, 'btywar': 46, 'buffle': 47, 'buggna': 48, 'buhvir': 49,\n",
        "    'bulori': 50, 'bushti': 51, 'buwtea': 52, 'buwwar': 53, 'cacwre': 54,\n",
        "    'calgul': 55, 'calqua': 56, 'camwar': 57, 'cangoo': 58, 'canwar': 59,\n",
        "    'canwre': 60, 'carwre': 61, 'casfin': 62, 'caster1': 63, 'casvir': 64,\n",
        "    'cedwax': 65, 'chispa': 66, 'chiswi': 67, 'chswar': 68, 'chukar': 69,\n",
        "    'clanut': 70, 'cliswa': 71, 'comgol': 72, 'comgra': 73, 'comloo': 74,\n",
        "    'commer': 75, 'comnig': 76, 'comrav': 77, 'comred': 78, 'comter': 79,\n",
        "    'comyel': 80, 'coohaw': 81, 'coshum': 82, 'cowscj1': 83, 'daejun': 84,\n",
        "    'doccor': 85, 'dowwoo': 86, 'dusfly': 87, 'eargre': 88, 'easblu': 89,\n",
        "    'easkin': 90, 'easmea': 91, 'easpho': 92, 'eastow': 93, 'eawpew': 94,\n",
        "    'eucdov': 95, 'eursta': 96, 'evegro': 97, 'fiespa': 98, 'fiscro': 99,\n",
        "    'foxspa': 100, 'gadwal': 101, 'gcrfin': 102, 'gnttow': 103, 'gnwtea': 104,\n",
        "    'gockin': 105, 'gocspa': 106, 'goleag': 107, 'grbher3': 108, 'grcfly': 109,\n",
        "    'greegr': 110, 'greroa': 111, 'greyel': 112, 'grhowl': 113, 'grnher': 114,\n",
        "    'grtgra': 115, 'grycat': 116, 'gryfly': 117, 'haiwoo': 118, 'hamfly': 119,\n",
        "    'hergul': 120, 'herthr': 121, 'hoomer': 122, 'hoowar': 123, 'horgre': 124,\n",
        "    'horlar': 125, 'houfin': 126, 'houspa': 127, 'houwre': 128, 'indbun': 129,\n",
        "    'juntit1': 130, 'killde': 131, 'labwoo': 132, 'larspa': 133, 'lazbun': 134,\n",
        "    'leabit': 135, 'leafly': 136, 'leasan': 137, 'lecthr': 138, 'lesgol': 139,\n",
        "    'lesnig': 140, 'lesyel': 141, 'lewwoo': 142, 'linspa': 143, 'lobcur': 144,\n",
        "    'lobdow': 145, 'logshr': 146, 'lotduc': 147, 'louwat': 148, 'macwar': 149,\n",
        "    'magwar': 150, 'mallar3': 151, 'marwre': 152, 'merlin': 153, 'moublu': 154,\n",
        "    'mouchi': 155, 'moudov': 156, 'norcar': 157, 'norfli': 158, 'norhar2': 159,\n",
        "    'normoc': 160, 'norpar': 161, 'norpin': 162, 'norsho': 163, 'norwat': 164,\n",
        "    'nrwswa': 165, 'nutwoo': 166, 'olsfly': 167, 'orcwar': 168, 'osprey': 169,\n",
        "    'ovenbi1': 170, 'palwar': 171, 'pasfly': 172, 'pecsan': 173, 'perfal': 174,\n",
        "    'phaino': 175, 'pibgre': 176, 'pilwoo': 177, 'pingro': 178, 'pinjay': 179,\n",
        "    'pinsis': 180, 'pinwar': 181, 'plsvir': 182, 'prawar': 183, 'purfin': 184,\n",
        "    'pygnut': 185, 'rebmer': 186, 'rebnut': 187, 'rebsap': 188, 'rebwoo': 189,\n",
        "    'redcro': 190, 'redhea': 191, 'reevir1': 192, 'renpha': 193, 'reshaw': 194,\n",
        "    'rethaw': 195, 'rewbla': 196, 'ribgul': 197, 'rinduc': 198, 'robgro': 199,\n",
        "    'rocpig': 200, 'rocwre': 201, 'rthhum': 202, 'ruckin': 203, 'rudduc': 204,\n",
        "    'rufgro': 205, 'rufhum': 206, 'rusbla': 207, 'sagspa1': 208, 'sagthr': 209,\n",
        "    'savspa': 210, 'saypho': 211, 'scatan': 212, 'scoori': 213, 'semplo': 214,\n",
        "    'semsan': 215, 'sheowl': 216, 'shshaw': 217, 'snobun': 218, 'snogoo': 219,\n",
        "    'solsan': 220, 'sonspa': 221, 'sora': 222, 'sposan': 223, 'spotow': 224,\n",
        "    'stejay': 225, 'swahaw': 226, 'swaspa': 227, 'swathr': 228, 'treswa': 229,\n",
        "    'truswa': 230, 'tuftit': 231, 'tunswa': 232, 'veery': 233, 'vesspa': 234,\n",
        "    'vigswa': 235, 'warvir': 236, 'wesblu': 237, 'wesgre': 238, 'weskin': 239,\n",
        "    'wesmea': 240, 'wessan': 241, 'westan': 242, 'wewpew': 243, 'whbnut': 244,\n",
        "    'whcspa': 245, 'whfibi': 246, 'whtspa': 247, 'whtswi': 248, 'wilfly': 249,\n",
        "    'wilsni1': 250, 'wiltur': 251, 'winwre3': 252, 'wlswar': 253, 'wooduc': 254,\n",
        "    'wooscj2': 255, 'woothr': 256, 'y00475': 257, 'yebfly': 258, 'yebsap': 259,\n",
        "    'yehbla': 260, 'yelwar': 261, 'yerwar': 262, 'yetvir': 263\n",
        "}\n",
        "\n",
        "INV_BIRD_CODE = {v: k for k, v in BIRD_CODE.items()}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U2FyD5GLpr02",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kk8rKABLpwGz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def mono_to_color(X: np.ndarray,\n",
        "                  mean=None,\n",
        "                  std=None,\n",
        "                  norm_max=None,\n",
        "                  norm_min=None,\n",
        "                  eps=1e-6):\n",
        "    \"\"\"\n",
        "    Code from https://www.kaggle.com/daisukelab/creating-fat2019-preprocessed-data\n",
        "    \"\"\"\n",
        "    # Stack X as [X,X,X]\n",
        "    X = np.stack([X, X, X], axis=-1)\n",
        "\n",
        "    # Standardize\n",
        "    mean = mean or X.mean()\n",
        "    X = X - mean\n",
        "    std = std or X.std()\n",
        "    Xstd = X / (std + eps)\n",
        "    _min, _max = Xstd.min(), Xstd.max()\n",
        "    norm_max = norm_max or _max\n",
        "    norm_min = norm_min or _min\n",
        "    if (_max - _min) > eps:\n",
        "        # Normalize to [0, 255]\n",
        "        V = Xstd\n",
        "        V[V < norm_min] = norm_min\n",
        "        V[V > norm_max] = norm_max\n",
        "        V = 255 * (V - norm_min) / (norm_max - norm_min)\n",
        "        V = V.astype(np.uint8)\n",
        "    else:\n",
        "        # Just zero\n",
        "        V = np.zeros_like(Xstd, dtype=np.uint8)\n",
        "    return V\n",
        "\n",
        "\n",
        "class TestDataset(data.Dataset):\n",
        "    def __init__(self, df: pd.DataFrame, clip: np.ndarray,\n",
        "                 img_size=224, melspectrogram_parameters={}):\n",
        "        self.df = df\n",
        "        self.clip = clip\n",
        "        self.img_size = img_size\n",
        "        self.melspectrogram_parameters = melspectrogram_parameters\n",
        "        \n",
        "    def __len__(self):\n",
        "        return len(self.df)\n",
        "    \n",
        "    def __getitem__(self, idx: int):\n",
        "        SR = 32000\n",
        "        sample = self.df.loc[idx, :]\n",
        "        site = sample.site\n",
        "        row_id = sample.row_id\n",
        "        \n",
        "        if site == \"site_3\":\n",
        "            y = self.clip.astype(np.float32)\n",
        "            len_y = len(y)\n",
        "            start = 0\n",
        "            end = SR * 5\n",
        "            images = []\n",
        "            while len_y > start:\n",
        "                y_batch = y[start:end].astype(np.float32)\n",
        "                if len(y_batch) != (SR * 5):\n",
        "                    break\n",
        "                start = end\n",
        "                end = end + SR * 5\n",
        "                \n",
        "                melspec = librosa.feature.melspectrogram(y_batch,\n",
        "                                                         sr=SR,\n",
        "                                                         **self.melspectrogram_parameters)\n",
        "                melspec = librosa.power_to_db(melspec).astype(np.float32)\n",
        "                image = mono_to_color(melspec)\n",
        "                height, width, _ = image.shape\n",
        "                image = cv2.resize(image, (int(width * self.img_size / height), self.img_size))\n",
        "                image = np.moveaxis(image, 2, 0)\n",
        "                image = (image / 255.0).astype(np.float32)\n",
        "                images.append(image)\n",
        "            images = np.asarray(images)\n",
        "            return images, row_id, site\n",
        "        else:\n",
        "            end_seconds = int(sample.seconds)\n",
        "            start_seconds = int(end_seconds - 5)\n",
        "            \n",
        "            start_index = SR * start_seconds\n",
        "            end_index = SR * end_seconds\n",
        "            \n",
        "            y = self.clip[start_index:end_index].astype(np.float32)\n",
        "\n",
        "            melspec = librosa.feature.melspectrogram(y, sr=SR, **self.melspectrogram_parameters)\n",
        "            melspec = librosa.power_to_db(melspec).astype(np.float32)\n",
        "\n",
        "            image = mono_to_color(melspec)\n",
        "            height, width, _ = image.shape\n",
        "            image = cv2.resize(image, (int(width * self.img_size / height), self.img_size))\n",
        "            image = np.moveaxis(image, 2, 0)\n",
        "            image = (image / 255.0).astype(np.float32)\n",
        "\n",
        "            return image, row_id, site"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BKHHAUf-pyWv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_model(config: dict, weights_path: str):\n",
        "    model = ResNet(**config)\n",
        "    checkpoint = torch.load(weights_path)\n",
        "    model.load_state_dict(checkpoint[\"model_state_dict\"])\n",
        "    device = torch.device(\"cuda\")\n",
        "    model.to(device)\n",
        "    model.eval()\n",
        "    return model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0NP9dFMap3FX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def prediction_for_clip(test_df: pd.DataFrame, \n",
        "                        clip: np.ndarray, \n",
        "                        model: ResNet, \n",
        "                        mel_params: dict, \n",
        "                        threshold=0.55):\n",
        "\n",
        "    dataset = TestDataset(df=test_df, \n",
        "                          clip=clip,\n",
        "                          img_size=224,\n",
        "                          melspectrogram_parameters=mel_params)\n",
        "    loader = data.DataLoader(dataset, batch_size=1, shuffle=False)\n",
        "    device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "    \n",
        "    model.eval()\n",
        "    prediction_dict = {}\n",
        "    for image, row_id, site in progress_bar(loader):\n",
        "        site = site[0]\n",
        "        row_id = row_id[0]\n",
        "        if site in {\"site_1\", \"site_2\"}:\n",
        "            image = image.to(device)\n",
        "\n",
        "            with torch.no_grad():\n",
        "                prediction = model(image)\n",
        "                proba = prediction[\"multilabel_proba\"].detach().cpu().numpy().reshape(-1)\n",
        "\n",
        "            events = proba >= threshold\n",
        "            labels = np.argwhere(events).reshape(-1).tolist()\n",
        "\n",
        "        else:\n",
        "            # to avoid prediction on large batch\n",
        "            image = image.squeeze(0)\n",
        "            batch_size = 16\n",
        "            whole_size = image.size(0)\n",
        "            if whole_size % batch_size == 0:\n",
        "                n_iter = whole_size // batch_size\n",
        "            else:\n",
        "                n_iter = whole_size // batch_size + 1\n",
        "                \n",
        "            all_events = set()\n",
        "            for batch_i in range(n_iter):\n",
        "                batch = image[batch_i * batch_size:(batch_i + 1) * batch_size]\n",
        "                if batch.ndim == 3:\n",
        "                    batch = batch.unsqueeze(0)\n",
        "\n",
        "                batch = batch.to(device)\n",
        "                with torch.no_grad():\n",
        "                    prediction = model(batch)\n",
        "                    proba = prediction[\"multilabel_proba\"].detach().cpu().numpy()\n",
        "                    \n",
        "                events = proba >= threshold\n",
        "                for i in range(len(events)):\n",
        "                    event = events[i, :]\n",
        "                    labels = np.argwhere(event).reshape(-1).tolist()\n",
        "                    for label in labels:\n",
        "                        all_events.add(label)\n",
        "                        \n",
        "            labels = list(all_events)\n",
        "        if len(labels) == 0:\n",
        "            prediction_dict[row_id] = \"nocall\"\n",
        "        else:\n",
        "            labels_str_list = list(map(lambda x: INV_BIRD_CODE[x], labels))\n",
        "            label_string = \" \".join(labels_str_list)\n",
        "            prediction_dict[row_id] = label_string\n",
        "    return prediction_dict"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ohZPIpipp83N",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def prediction(test_df: pd.DataFrame,\n",
        "               test_audio: Path,\n",
        "               model_config: dict,\n",
        "               mel_params: dict,\n",
        "               weights_path: str,\n",
        "               threshold=0.5):\n",
        "    model = get_model(model_config, weights_path)\n",
        "    unique_audio_id = test_df.audio_id.unique()\n",
        "\n",
        "    warnings.filterwarnings(\"ignore\")\n",
        "    prediction_dfs = []\n",
        "    for audio_id in unique_audio_id:\n",
        "        with timer(f\"Loading {audio_id}\", logger):\n",
        "            clip, _ = librosa.load(test_audio / (audio_id + \".mp3\"),\n",
        "                                   sr=TARGET_SR,\n",
        "                                   mono=True,\n",
        "                                   res_type=\"kaiser_fast\")\n",
        "        \n",
        "        test_df_for_audio_id = test_df.query(\n",
        "            f\"audio_id == '{audio_id}'\").reset_index(drop=True)\n",
        "        with timer(f\"Prediction on {audio_id}\", logger):\n",
        "            prediction_dict = prediction_for_clip(test_df_for_audio_id,\n",
        "                                                  clip=clip,\n",
        "                                                  model=model,\n",
        "                                                  mel_params=mel_params,\n",
        "                                                  threshold=threshold)\n",
        "        row_id = list(prediction_dict.keys())\n",
        "        birds = list(prediction_dict.values())\n",
        "        prediction_df = pd.DataFrame({\n",
        "            \"row_id\": row_id,\n",
        "            \"birds\": birds\n",
        "        })\n",
        "        prediction_dfs.append(prediction_df)\n",
        "    \n",
        "    prediction_df = pd.concat(prediction_dfs, axis=0, sort=False).reset_index(drop=True)\n",
        "    return prediction_df"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eTtt-WjAp_R2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "submission = prediction(test_df=test,\n",
        "                        test_audio=test_audio,\n",
        "                        model_config=model_config,\n",
        "                        mel_params=melspectrogram_parameters,\n",
        "                        weights_path=weights_path,\n",
        "                        threshold=0.85)\n",
        "submission.to_csv(\"submission.csv\", index=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JYkPSZBTqFDO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "submission"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}