ismir2021lbd_using_the_synctoolbox_for_an_experiment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Demo: Using The Sync Toolbox for an Experiment                                                      on High-Resolution Music Alignment\n",
    "\n",
    "\n",
    "Music synchronization aims to automatically align multiple music representations such as audio recordings, MIDI files, and sheet music. For this task, we have recently published the **Sync Toolbox**[1], an open-source Python package for efficient, robust, and accurate music synchronization. This work combines spectral flux used as onset features with conventional chroma features to increase the alignment accuracy. We conduct some experiments within the **Sync Toolbox** framework to show that our approach preserves the accuracy compared with another high-resolution approach while being computationally simpler. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Loading some modules and defining some constants used later\n",
    "import IPython.display as ipd\n",
    "import matplotlib.pyplot as plt\n",
    "import librosa.display\n",
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd\n",
    "import scipy.interpolate\n",
    "\n",
    "from synctoolbox.dtw.mrmsdtw import sync_via_mrmsdtw\n",
    "from synctoolbox.dtw.utils import compute_optimal_chroma_shift, shift_chroma_vectors, make_path_strictly_monotonic\n",
    "from synctoolbox.feature.chroma import pitch_to_chroma, quantize_chroma, quantized_chroma_to_CENS\n",
    "from synctoolbox.feature.dlnco import pitch_onset_features_to_DLNCO\n",
    "from synctoolbox.feature.novelty import spectral_flux\n",
    "from synctoolbox.feature.pitch import audio_to_pitch_features\n",
    "from synctoolbox.feature.pitch_onset import audio_to_pitch_onset_features\n",
    "from synctoolbox.feature.utils import estimate_tuning\n",
    "%matplotlib inline\n",
    "\n",
    "Fs = 22050\n",
    "FEATURE_RATE = 50\n",
    "STEP_WEIGHTS = np.array([1.5, 1.5, 2.0])\n",
    "THRESHOLD_REC = 10 ** 6\n",
    "FIG_SIZE = (9, 3)\n",
    "GAMMA = 10.0\n",
    "AUDIO_DIR = 'winterreise/01_RawData/audio_wav/'\n",
    "MEASURE_ANN_DIR = 'winterreise/02_Annotations/ann_audio_measure/'\n",
    "FILENAME_PREFIX = 'Schubert_D911-'\n",
    "FIGURE_DIR = 'figs'\n",
    "\n",
    "# Create a directory for the figures, if not exists\n",
    "if not os.path.exists(FIGURE_DIR):\n",
    "    os.makedirs(FIGURE_DIR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download the latest version of the Schubert Winterreise Dataset (SWD)\n",
    "**Schubert Winterreise Dataset (SWD)**[2] comprises several representations of the song cycle *Winterreise* D911 (Op. 89), which consists of 24 songs composed for solo voice with piano accompaniment. For our experiments, we focus on the  music recordings by the baritones Gerhard Hüsch and Randall Scarlata and the corresponding measure annotations. These two versions are publicly available, which allows reproducing all our experiments based on open-source code and open-source data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!apt-get install unzip wget\n",
    "!wget \"https://zenodo.org/record/5139893/files/Schubert_Winterreise_Dataset_v2-0.zip?download=1\" -O winterreise.zip\n",
    "!unzip winterreise -d winterreise\n",
    "!rm -r winterreise.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_chroma_features_from_audio(audio,\n",
    "                                   tuning_offset,\n",
    "                                   Fs=Fs,\n",
    "                                   feature_rate=FEATURE_RATE,\n",
    "                                   verbose=False):\n",
    "    \n",
    "    f_pitch = audio_to_pitch_features(f_audio=audio, \n",
    "                                      Fs=Fs, \n",
    "                                      tuning_offset=tuning_offset, \n",
    "                                      feature_rate=feature_rate, \n",
    "                                      verbose=verbose)\n",
    "    f_chroma = pitch_to_chroma(f_pitch=f_pitch)\n",
    "    f_chroma_quantized = quantize_chroma(f_chroma=f_chroma)\n",
    "    \n",
    "    return f_chroma_quantized\n",
    "\n",
    "def get_spectral_flux_from_audio(audio,\n",
    "                                 feature_sequence_length,\n",
    "                                 gamma=GAMMA, # log compression param\n",
    "                                 Fs=Fs,\n",
    "                                 feature_rate=FEATURE_RATE):\n",
    "    f_novelty = spectral_flux(audio, Fs=Fs, feature_rate=feature_rate, gamma=gamma)\n",
    "        \n",
    "            \n",
    "    if f_novelty.size < feature_sequence_length:\n",
    "        # The feature sequence length of the chroma features are not same as the novelty curve \n",
    "        # due to the padding while the computation of STFT for chroma features and\n",
    "        # the differentiation in spectral flux\n",
    "        diff = feature_sequence_length - f_novelty.size\n",
    "        pad = int(diff / 2)\n",
    "        f_novelty = np.concatenate((np.zeros(pad), f_novelty, np.zeros(pad)))\n",
    "       \n",
    "    return f_novelty.reshape(1, -1)\n",
    "\n",
    "\n",
    "def get_DLNCO_features_from_audio(audio,\n",
    "                                  tuning_offset,\n",
    "                                  feature_sequence_length,\n",
    "                                  Fs=Fs,\n",
    "                                  feature_rate=FEATURE_RATE,\n",
    "                                  verbose=False):\n",
    "    f_pitch_onset = audio_to_pitch_onset_features(f_audio=audio, \n",
    "                                                  Fs=Fs, \n",
    "                                                  tuning_offset=tuning_offset, \n",
    "                                                  verbose=verbose)\n",
    "    \n",
    "    f_DLNCO = pitch_onset_features_to_DLNCO(f_peaks=f_pitch_onset, \n",
    "                                            feature_rate=feature_rate, \n",
    "                                            feature_sequence_length=feature_sequence_length, \n",
    "                                            visualize=verbose)\n",
    "    \n",
    "    return f_DLNCO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Alignment with MrMsDTW\n",
    "\n",
    "We now perform alignment using MrMsDTW[3] using three settings as input features:\n",
    "* Chroma\n",
    "* Chroma & DLNCO [4]\n",
    "* Chroma & Spectral Flux [5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wp_dict = dict()\n",
    "\n",
    "for song_id in range(1, 25):\n",
    "    if song_id < 10:\n",
    "        song_id = '0' + str(song_id)\n",
    "    else:\n",
    "        song_id = str(song_id)\n",
    "        \n",
    "    filename_1 = FILENAME_PREFIX + song_id + '_HU33'\n",
    "    filename_2 = FILENAME_PREFIX + song_id + '_SC06'\n",
    "    \n",
    "    print(f\"\\nRunning for the Song ID {song_id} in SWD.\")\n",
    "    \n",
    "    # read audio\n",
    "    audio_1, _ = librosa.load(os.path.join(AUDIO_DIR, filename_1 + '.wav'), sr=Fs)\n",
    "    audio_2, _ = librosa.load(os.path.join(AUDIO_DIR, filename_2 + '.wav'), sr=Fs)\n",
    "    \n",
    "    # estimate tuning\n",
    "    tuning_offset_1 = estimate_tuning(audio_1, Fs)\n",
    "    tuning_offset_2 = estimate_tuning(audio_2, Fs)\n",
    "\n",
    "    # generate chroma features\n",
    "    f_chroma_quantized_1 = get_chroma_features_from_audio(audio=audio_1,\n",
    "                                                          tuning_offset=tuning_offset_1)\n",
    "    \n",
    "    f_chroma_quantized_2 = get_chroma_features_from_audio(audio=audio_2,\n",
    "                                                          tuning_offset=tuning_offset_2)\n",
    "    \n",
    "    # generate novelty features (i.e. spectral flux)\n",
    "    f_sf_1 = get_spectral_flux_from_audio(audio=audio_1, \n",
    "                                          feature_sequence_length=f_chroma_quantized_1.shape[1])\n",
    "    \n",
    "    f_sf_2 = get_spectral_flux_from_audio(audio=audio_2, \n",
    "                                          feature_sequence_length=f_chroma_quantized_2.shape[1])\n",
    "    \n",
    " \n",
    "    # generate DLNCO features\n",
    "    f_DLNCO_1 = get_DLNCO_features_from_audio(audio=audio_1, \n",
    "                                              tuning_offset=tuning_offset_1,\n",
    "                                              feature_sequence_length=f_chroma_quantized_1.shape[1])\n",
    "    \n",
    "    f_DLNCO_2 = get_DLNCO_features_from_audio(audio=audio_2, \n",
    "                                              tuning_offset=tuning_offset_2,\n",
    "                                              feature_sequence_length=f_chroma_quantized_2.shape[1])\n",
    "    \n",
    "    # compute the optimal chroma shift and shift the chroma-based features of the second recording\n",
    "    opt_chroma_shift = compute_optimal_chroma_shift(quantized_chroma_to_CENS(f_chroma_quantized_1, \n",
    "                                                                             201, 50, \n",
    "                                                                             FEATURE_RATE)[0], \n",
    "                                                    quantized_chroma_to_CENS(f_chroma_quantized_2, \n",
    "                                                                             201, 50, \n",
    "                                                                             FEATURE_RATE)[0])\n",
    "    \n",
    "    f_chroma_quantized_2 = shift_chroma_vectors(f_chroma_quantized_2, opt_chroma_shift)\n",
    "    f_DLNCO_2 = shift_chroma_vectors(f_DLNCO_2, opt_chroma_shift)\n",
    "\n",
    "    \n",
    "    # run MrMsDTW for chroma\n",
    "    wp_chroma = sync_via_mrmsdtw(f_chroma1=f_chroma_quantized_1, \n",
    "                                 f_chroma2=f_chroma_quantized_2, \n",
    "                                 input_feature_rate=FEATURE_RATE, \n",
    "                                 step_weights=STEP_WEIGHTS, \n",
    "                                 threshold_rec=THRESHOLD_REC, \n",
    "                                 verbose=False)\n",
    "    \n",
    "    \n",
    "    # run MrMsDTW for chroma & DLNCO\n",
    "    wp_chroma_dlnco = sync_via_mrmsdtw(f_chroma1=f_chroma_quantized_1, \n",
    "                                       f_onset1=f_DLNCO_1, \n",
    "                                       f_chroma2=f_chroma_quantized_2, \n",
    "                                       f_onset2=f_DLNCO_2, \n",
    "                                       input_feature_rate=FEATURE_RATE, \n",
    "                                       step_weights=STEP_WEIGHTS, \n",
    "                                       threshold_rec=THRESHOLD_REC, \n",
    "                                       verbose=False)\n",
    "    \n",
    "    \n",
    "    # run MrMsDTW for chroma & spectral flux\n",
    "    wp_chroma_sf = sync_via_mrmsdtw(f_chroma1=f_chroma_quantized_1, \n",
    "                                    f_onset1=f_sf_1, \n",
    "                                    f_chroma2=f_chroma_quantized_2, \n",
    "                                    f_onset2=f_sf_2, \n",
    "                                    input_feature_rate=FEATURE_RATE, \n",
    "                                    step_weights=STEP_WEIGHTS, \n",
    "                                    threshold_rec=THRESHOLD_REC, \n",
    "                                    verbose=False)\n",
    "        \n",
    "        \n",
    "    wp_dict[song_id] = dict()\n",
    "    wp_dict[song_id]['wp_chroma'] = wp_chroma\n",
    "    wp_dict[song_id]['wp_chroma_dlnco'] = wp_chroma_dlnco\n",
    "    wp_dict[song_id]['wp_chroma_sf'] = wp_chroma_sf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "For the evaluation, we utilize the pairwise alignment error $\\epsilon_{P}$ from [6]. Given two versions of the same music piece with the time-continuous axes $[0, T_{1}]$ and $[0, T_{2}]$, the monotonous alignment can be modeled as a function\n",
    "\t\n",
    "$$\\mathcal{A}: [0, T_{1}] \\rightarrow [0, T_{2}].$$\n",
    "\t\n",
    "The pairwise alignment error $\\epsilon_{P}$ for a given alignment of two recording is specified as the mean over the values\n",
    "\n",
    "$$\\epsilon_{P}(g_{1}):=|\\mathcal{A}(g_{1}) - g_{2}|,$$\n",
    "\n",
    "where $(g_{1}, g_{2}) \\in [0, T_{1}] \\times [0, T_{2}]$ indicates the ground-truth pairs of measure annotations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_stats(wp, \n",
    "              measure_ann_filepath_1,\n",
    "              measure_ann_filepath_2,\n",
    "              feature_rate=FEATURE_RATE,\n",
    "              tolerances=[30, 50, 100, 150, 200, 300, 500, 1000]): # tolerances in milliseconds\n",
    "    wp = make_path_strictly_monotonic(wp)\n",
    "    \n",
    "    measure_ann_1 = pd.read_csv(filepath_or_buffer=measure_ann_filepath_1, delimiter=';')['start']\n",
    "    measure_ann_2 = pd.read_csv(filepath_or_buffer=measure_ann_filepath_2, delimiter=';')['start']\n",
    "\n",
    "    measure_positions_1_transferred_to_2 = scipy.interpolate.interp1d(wp[0] / feature_rate, \n",
    "                                                                      wp[1] / feature_rate, \n",
    "                                                                      kind='linear')(measure_ann_1)\n",
    "\n",
    "    absolute_errors_at_measures = np.abs(measure_positions_1_transferred_to_2 - measure_ann_2)\n",
    "    \n",
    "    misalignments = np.zeros(len(tolerances))\n",
    "    \n",
    "    for idx, tolerance in enumerate(tolerances):  # in milliseconds\n",
    "        misalignments[idx] = np.mean((absolute_errors_at_measures>tolerance/1000.0))\n",
    "        \n",
    "    mean = np.mean(absolute_errors_at_measures) * 1000.0\n",
    "    std = np.std(absolute_errors_at_measures) * 1000.0\n",
    "    \n",
    "    return mean, std, np.array(misalignments), absolute_errors_at_measures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stats_dict = dict()\n",
    "TOLERANCES = [30, 50, 100, 150, 200, 300, 500, 1000]\n",
    "\n",
    "for song_id in wp_dict:\n",
    "    wp_chroma_dlnco = wp_dict[song_id]['wp_chroma_dlnco']\n",
    "    wp_chroma_sf = wp_dict[song_id]['wp_chroma_sf']\n",
    "    wp_chroma = wp_dict[song_id]['wp_chroma']\n",
    "\n",
    "    filename_1 = FILENAME_PREFIX + song_id + '_HU33'\n",
    "    filename_2 = FILENAME_PREFIX + song_id + '_SC06'\n",
    "    \n",
    "    stats_dict[song_id] = dict()\n",
    "    \n",
    "    mean, std, misalignments, err = get_stats(wp = wp_chroma, \n",
    "                                              measure_ann_filepath_1 = os.path.join(MEASURE_ANN_DIR, filename_1 + '.csv'),\n",
    "                                              measure_ann_filepath_2 = os.path.join(MEASURE_ANN_DIR, filename_2 + '.csv'),\n",
    "                                              tolerances = TOLERANCES)\n",
    "    \n",
    "    \n",
    "    stats_dict[song_id]['chroma'] = dict()\n",
    "    stats_dict[song_id]['chroma']['mean'] = mean\n",
    "    stats_dict[song_id]['chroma']['std'] = std\n",
    "    stats_dict[song_id]['chroma']['misalignments'] = misalignments\n",
    "    stats_dict[song_id]['chroma']['absolute_errors'] = err\n",
    "    \n",
    "\n",
    "    mean, std, misalignments, err = get_stats(wp=wp_chroma_dlnco, \n",
    "                                              measure_ann_filepath_1=os.path.join(MEASURE_ANN_DIR, filename_1 + '.csv'),\n",
    "                                              measure_ann_filepath_2=os.path.join(MEASURE_ANN_DIR, filename_2 + '.csv'),\n",
    "                                              tolerances=TOLERANCES)\n",
    "    \n",
    "    stats_dict[song_id]['chroma_dlnco'] = dict()\n",
    "    stats_dict[song_id]['chroma_dlnco']['mean'] = mean\n",
    "    stats_dict[song_id]['chroma_dlnco']['std'] = std\n",
    "    stats_dict[song_id]['chroma_dlnco']['misalignments'] = misalignments\n",
    "    stats_dict[song_id]['chroma_dlnco']['absolute_errors'] = err\n",
    "    \n",
    "    mean, std, misalignments, err= get_stats(wp=wp_chroma_sf, \n",
    "                                             measure_ann_filepath_1=os.path.join(MEASURE_ANN_DIR, filename_1 + '.csv'),\n",
    "                                             measure_ann_filepath_2=os.path.join(MEASURE_ANN_DIR, filename_2 + '.csv'),\n",
    "                                             tolerances=TOLERANCES)\n",
    "    \n",
    "    stats_dict[song_id]['chroma_sf'] = dict()\n",
    "    stats_dict[song_id]['chroma_sf']['mean'] = mean\n",
    "    stats_dict[song_id]['chroma_sf']['std'] = std\n",
    "    stats_dict[song_id]['chroma_sf']['misalignments'] = misalignments\n",
    "    stats_dict[song_id]['chroma_sf']['absolute_errors'] = err"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Misalignment Rates per Song in the SWD Dataset\n",
    "In addition to the pair-wise alignment error, one may also consider the misalignment rate from [6], which identifies the percentage of measure positions in an alignment with an error above a given threshold $\\tau$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rows = pd.MultiIndex.from_product([stats_dict.keys()],\n",
    "                                   names=['Song ID'])\n",
    "columns = pd.MultiIndex.from_product([['Chroma', \n",
    "                                       'Chroma & DLNCO', \n",
    "                                       'Chroma & Spectral Flux'], TOLERANCES],\n",
    "                                     names=['Feature Type', '$\\u03C4$ (ms)'])\n",
    "data = np.zeros((len(stats_dict), len(misalignments) * 3))\n",
    "for row_idx, song_id in enumerate(stats_dict):\n",
    "    data[row_idx, :len(misalignments)] = stats_dict[song_id]['chroma']['misalignments'] * 100\n",
    "    data[row_idx, len(misalignments):2*len(misalignments)] = stats_dict[song_id]['chroma_dlnco']['misalignments'] * 100\n",
    "    data[row_idx, 2*len(misalignments):3*len(misalignments)] = stats_dict[song_id]['chroma_sf']['misalignments'] * 100\n",
    "\n",
    "df = pd.DataFrame(data, index=rows, columns=columns)\n",
    "with pd.option_context('display.float_format', '{:0.2f}'.format):\n",
    "    ipd.display(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chroma_means = np.zeros(len(stats_dict))\n",
    "chroma_std = np.zeros(len(stats_dict))\n",
    "\n",
    "chroma_dlnco_means = np.zeros(len(stats_dict))\n",
    "chroma_dlnco_std = np.zeros(len(stats_dict))\n",
    "\n",
    "chroma_sf_means = np.zeros(len(stats_dict))\n",
    "chroma_sf_std = np.zeros(len(stats_dict))\n",
    "\n",
    "for idx, song_id in enumerate(stats_dict):\n",
    "    chroma_means[idx] = stats_dict[song_id]['chroma']['mean']\n",
    "    chroma_std[idx] = stats_dict[song_id]['chroma']['std']\n",
    "    chroma_dlnco_means[idx] = stats_dict[song_id]['chroma_dlnco']['mean']\n",
    "    chroma_dlnco_std[idx] = stats_dict[song_id]['chroma_dlnco']['std']\n",
    "    chroma_sf_means[idx] = stats_dict[song_id]['chroma_sf']['mean']\n",
    "    chroma_sf_std[idx] = stats_dict[song_id]['chroma_sf']['std']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SONG_LABELS=['01. Gute Nacht', '02. Die Wetterfahne', '03. Gefrorne Tränen', '04. Erstarrung',\n",
    "             '05. Der Lindenbaum', '06. Wasserflut', '07. Auf dem Flusse', '08. Rückblick',\n",
    "             '09. Irrlicht', '10. Rast', '11. Frühlingstraum', '12. Einsamkeit',\n",
    "             '13. Die Post', '14. Der greise Kopf', '15. Die Krähe', '16. Letzte Hoffnung',\n",
    "             '17. Im Dorfe', '18. Der stürmische Morgen', '19. Täuschung', '20. Der Wegweiser',\n",
    "             '21. Das Wirtshaus', '22. Muth', '23. Die Nebensonnen', '24. Der Leiermann']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Errorbar Plot for Misalignment Rates per Feature, given a Threshold $\\tau$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,4))\n",
    "\n",
    "x_axis = (np.array(TOLERANCES) / 1000).astype('str')\n",
    "\n",
    "plt.errorbar(x=x_axis,\n",
    "             y=df['Chroma'].mean(), \n",
    "             yerr=df['Chroma'].std(),\n",
    "             marker='x', \n",
    "             markersize=7,\n",
    "             alpha=0.75,\n",
    "             fmt='gray',\n",
    "             linestyle=':',\n",
    "             capsize=3,\n",
    "             elinewidth=2,\n",
    "             label='Chroma');\n",
    "\n",
    "plt.errorbar(x=x_axis,\n",
    "             y=df['Chroma & DLNCO'].mean(), \n",
    "             yerr=df['Chroma & DLNCO'].std(),\n",
    "             marker='^', \n",
    "             markersize=7,\n",
    "             fmt='red',\n",
    "             linestyle='--',\n",
    "             capsize=3,\n",
    "             elinewidth=2,\n",
    "             label='Chroma & DLNCO');\n",
    "\n",
    "plt.errorbar(x=x_axis,\n",
    "             y=df['Chroma & Spectral Flux'].mean(), \n",
    "             yerr=df['Chroma & Spectral Flux'].std(),\n",
    "             marker='o', \n",
    "             linestyle='-',\n",
    "             alpha=0.6,\n",
    "             markersize=7,\n",
    "             elinewidth=2,\n",
    "             capsize=3,\n",
    "             label='Chroma & Spectral Flux' )\n",
    "\n",
    "plt.xticks(x_axis);\n",
    "plt.ylabel('Misalignment rate (%)', fontsize=14)\n",
    "plt.xlabel('Threshold $\\u03C4$ (seconds)', fontsize=14)\n",
    "plt.legend(fontsize=13, loc='upper right')\n",
    "plt.grid(linestyle=':');\n",
    "plt.savefig(f'{FIGURE_DIR}/misalignment_rates_errorbar.pdf', dpi=300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,4))\n",
    "\n",
    "x_axis = (np.array(TOLERANCES) / 1000).astype('str')\n",
    "x_axis_arr = np.arange(len(TOLERANCES))\n",
    "BAR_WIDTH = 0.275\n",
    "\n",
    "plt.bar(x=x_axis_arr,\n",
    "        height=df['Chroma'].mean(), \n",
    "        yerr=df['Chroma'].std(),\n",
    "        width=BAR_WIDTH,\n",
    "        color='gray',\n",
    "        label='Chroma',\n",
    "        alpha=0.75);\n",
    "        \n",
    "plt.bar(x=x_axis_arr + BAR_WIDTH,\n",
    "        height=df['Chroma & DLNCO'].mean(), \n",
    "        yerr=df['Chroma & DLNCO'].std(),\n",
    "        width=BAR_WIDTH,\n",
    "        alpha=0.75,\n",
    "        color='red',\n",
    "        label='Chroma & DLNCO')\n",
    "\n",
    "\n",
    "        \n",
    "plt.bar(x=x_axis_arr + 2 * BAR_WIDTH,\n",
    "        height=df['Chroma & Spectral Flux'].mean(), \n",
    "        yerr=df['Chroma & Spectral Flux'].std(),\n",
    "        width=BAR_WIDTH,\n",
    "        alpha=0.75,\n",
    "        color='#1f77b4',\n",
    "        label='Chroma & Spectral Flux')\n",
    "\n",
    "\n",
    "plt.xticks([r + BAR_WIDTH for r in np.arange(len(TOLERANCES))], x_axis)\n",
    "\n",
    "plt.ylabel('Misalignment rate (%)', fontsize=14)\n",
    "plt.xlabel('Threshold $\\u03C4$ (seconds)', fontsize=14)\n",
    "plt.legend(fontsize=13, loc='upper right')\n",
    "plt.grid(linestyle=':');\n",
    "plt.savefig(f'{FIGURE_DIR}/misalignment_rates_bar.pdf', dpi=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Errorbar Plot for Mean Alignment Error per Song"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,4))\n",
    "\n",
    "\n",
    "plt.errorbar(x=np.arange(1, len(stats_dict)+1) - 0.15,\n",
    "             y=chroma_means / 1000, \n",
    "             yerr=chroma_std / 1000,\n",
    "             marker='x', \n",
    "             markersize=6,\n",
    "             fmt='gray',\n",
    "             alpha=0.75,\n",
    "             linestyle='',\n",
    "             capsize=3,\n",
    "             elinewidth=2,\n",
    "             label='Chroma');\n",
    "\n",
    "plt.errorbar(x=np.arange(1, len(stats_dict)+1),\n",
    "             y=chroma_dlnco_means / 1000, \n",
    "             yerr=chroma_dlnco_std / 1000,\n",
    "             marker='^', \n",
    "             markersize=6,\n",
    "             fmt='red',\n",
    "             linestyle='',\n",
    "             capsize=3,\n",
    "             elinewidth=2,\n",
    "             label='Chroma & DLNCO');\n",
    "\n",
    "plt.errorbar(x=np.arange(1, len(stats_dict)+1) + 0.15,\n",
    "             y=chroma_sf_means / 1000, \n",
    "             yerr=chroma_sf_std / 1000,\n",
    "             marker='o', \n",
    "             linestyle='',\n",
    "             markersize=6,\n",
    "             elinewidth=2,\n",
    "             capsize=3,\n",
    "             label='Chroma & Spectral Flux' )\n",
    "\n",
    "plt.legend(fontsize=13, loc='upper right')\n",
    "plt.grid(linestyle=':');\n",
    "plt.ylabel('Mean Alignment Error (sec)', fontsize=14)\n",
    "plt.xlabel('Song Number in SWD', fontsize=14)\n",
    "\n",
    "SHOW_SONG_LABELS = False\n",
    "\n",
    "if SHOW_SONG_LABELS:\n",
    "    labels = SONG_LABELS\n",
    "    rotation = 90\n",
    "else:\n",
    "    labels = None\n",
    "    rotation = None\n",
    "    \n",
    "plt.xticks(np.arange(1, 25, 1), \n",
    "           labels=labels, \n",
    "           rotation=rotation);\n",
    "plt.savefig(f'{FIGURE_DIR}/mean_error_errorbar.pdf', dpi=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Boxplot for Mean Alignment Error per Song"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12, 4))\n",
    "chroma_errors = [stats_dict[song_id]['chroma']['absolute_errors'] for song_id in stats_dict]#\n",
    "chroma_sf_errors = [stats_dict[song_id]['chroma_sf']['absolute_errors'] for song_id in stats_dict]#\n",
    "chroma_dlnco_errors = [stats_dict[song_id]['chroma_dlnco']['absolute_errors'] for song_id in stats_dict]#\n",
    "\n",
    "SHOW_OUTLIERS = True\n",
    "SHOW_MEANS = True\n",
    "\n",
    "\n",
    "if SHOW_OUTLIERS:\n",
    "    sym = '+'\n",
    "else:\n",
    "    sym = ''\n",
    "\n",
    "c = 'gray'\n",
    "bp1 = plt.boxplot(chroma_errors, \n",
    "                  widths = 0.20,\n",
    "                  sym=sym,\n",
    "                  boxprops=dict(color=c, alpha=0.75),\n",
    "                  capprops=dict(color=c),\n",
    "                  whiskerprops=dict(color=c),\n",
    "                  medianprops=dict(color=c),\n",
    "                  positions=np.arange(0, 24) - 0.25,\n",
    "                  showmeans=SHOW_MEANS,\n",
    "                  meanprops={\"marker\":\"o\",\n",
    "                             \"markerfacecolor\":\"white\", \n",
    "                             \"markeredgecolor\":\"gray\",\n",
    "                             \"markersize\":\"5\"},\n",
    "                  flierprops={\"markersize\":\"4\",\n",
    "                             \"markerfacecolor\":\"white\",\n",
    "                             \"markeredgecolor\":\"gray\"});\n",
    "\n",
    "c = 'red'\n",
    "bp2 = plt.boxplot(chroma_dlnco_errors, \n",
    "                  widths = 0.20,\n",
    "                  sym=sym,\n",
    "                  boxprops=dict(color=c),\n",
    "                  capprops=dict(color=c),\n",
    "                  whiskerprops=dict(color=c),\n",
    "                  medianprops=dict(color=c),\n",
    "                  positions=np.arange(0, 24),\n",
    "                  showmeans=SHOW_MEANS,\n",
    "                  meanprops={\"marker\":\"o\",\n",
    "                             \"markerfacecolor\":\"white\", \n",
    "                             \"markeredgecolor\":\"red\",\n",
    "                             \"markersize\":\"5\"},\n",
    "                  flierprops={\"markersize\":\"4\",\n",
    "                             \"markerfacecolor\":\"white\",\n",
    "                             \"markeredgecolor\":\"red\"});\n",
    "\n",
    "c = '#1f77b4'\n",
    "bp3 = plt.boxplot(chroma_sf_errors, \n",
    "                  widths = 0.20,\n",
    "                  sym=sym,\n",
    "                  boxprops=dict(color=c),\n",
    "                  capprops=dict(color=c),\n",
    "                  whiskerprops=dict(color=c),\n",
    "                  medianprops=dict(color=c),\n",
    "                  positions=np.arange(0, 24) + 0.25,\n",
    "                  showmeans=SHOW_MEANS,\n",
    "                  meanprops={\"marker\":\"o\",\n",
    "                             \"markerfacecolor\":\"white\", \n",
    "                             \"markeredgecolor\":c,\n",
    "                             \"markersize\":\"5\"},\n",
    "                  flierprops={\"markersize\":\"4\",\n",
    "                             \"markerfacecolor\":\"white\",\n",
    "                             \"markeredgecolor\":c});\n",
    "\n",
    "\n",
    "\n",
    "plt.xticks(ticks=np.arange(0,24), labels=stats_dict.keys());\n",
    "plt.legend([bp1[\"boxes\"][0], bp2[\"boxes\"][0], bp3[\"boxes\"][0]], \n",
    "           ['Chroma', \n",
    "            'Chroma & DLNCO',\n",
    "            'Chroma & Spectral Flux'], \n",
    "           loc='upper right',\n",
    "           fontsize=13);\n",
    "plt.grid(linestyle=':');\n",
    "plt.ylabel('Alignment Error (sec)', fontsize=14)\n",
    "plt.xlabel('Song Number in SWD', fontsize=14)\n",
    "plt.ylim([0, 0.55]);\n",
    "plt.savefig(f'{FIGURE_DIR}/error_boxplot.pdf', dpi=300);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References\n",
    "\n",
    "[1] M. Müller, Y. Özer, M. Krause, T. Prätzlich, and J. Driedger, “Sync toolbox: A python package for efficient, robust, and accurate music synchronization,” Journal of Open Source Software, vol. 6, no. 64, p. 3434, 2021. [Online]. Available: https://doi.org/10.21105/joss.03434\n",
    "\n",
    "[2] C. Weiß, F. Zalkow, V. Arifi-Müller, M. Müller, H. V.Koops, A. Volk, and H. Grohganz, “Schubert Winterreise dataset: A multimodal scenario for music analysis,” ACM Journal on Computing and Cultural Heritage (JOCCH), vol. 14, no. 2, pp. 25:1–18, 2021.\n",
    "\n",
    "[3] T. Prätzlich, J. Driedger, and M. Müller, “Memory-restricted multiscale dynamic time warping,” in Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Shanghai, China, March 2016, pp. 569–573.\n",
    "\n",
    "[4] S. Ewert, M. Müller, and P. Grosche, “High resolution audio synchronization using chroma onset features,”\n",
    "in Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Taipei, Taiwan, Apr. 2009, pp. 1869–1872.\n",
    "\n",
    "[5] P. Grosche, M. Müller, and S. Ewert, “Combination of onset-features with applications to high-resolution\n",
    "music synchronization,” in Proceedings of the International Conference on Acoustics (NAG/DAGA), 2009, pp. 357–360.\n",
    "\n",
    "[6] T. Prätzlich and M. Müller, “Triple-based analysis of music alignments without the need of ground-truth annotations,” in Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Shanghai, China, March 2016, pp. 266–270."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}