notebooks/work-in-progress/2018-09_PianoTranscription/Onsets_and_Frames_mdda.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gYGMczS-7rkJ"
   },
   "source": [
    "Copyright 2017 Google LLC.\n",
    "\n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "you may not use this file except in compliance with the License.\n",
    "You may obtain a copy of the License at\n",
    "\n",
    "https://www.apache.org/licenses/LICENSE-2.0\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software\n",
    "distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "See the License for the specific language governing permissions and\n",
    "limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Updated by Martin Andrews (mdda @ GitHub) to include cleaner file upload/download code.  Repo is MIT licensed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IiIFvcvy7dH5"
   },
   "source": [
    "# Onsets and Frames: Dual-Objective Piano Transcription\n",
    "\n",
    "### ___Curtis Hawthorne, Erich Elsen, Jialin Song, Adam Roberts, Ian Simon, Colin Raffel, Jesse Engel, Sageev Oore, Douglas Eck___ ([arXiv](https://goo.gl/magenta/onsets-frames-paper)) ([code](https://goo.gl/magenta/onsets-frames-code))\n",
    "\n",
    "Onsets and Frames is an automatic piano music transcription model. This notebook demonstrates running the model on user-supplied recordings. For more details on the architecture of the model, see our [arXiv paper](https://goo.gl/magenta/onsets-frames-paper).\n",
    "\n",
    "___\n",
    "\n",
    "This colab notebook is self-contained and should run natively on google cloud. The code and checkpoints can be downloaded separately and run locally, which is recommended if you want to train your own model. Details on how to do this can be found in the [GitHub repo](https://goo.gl/magenta/onsets-frames-code)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "j79eR9mp_oaH"
   },
   "source": [
    "# Environment Setup\n",
    "\n",
    "Includes package installation for sequence synthesis and downloading pretrained checkpoint. May take a few minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "nzGyqJja7I0O"
   },
   "outputs": [],
   "source": [
    "#@title Setup Environment\n",
    "\n",
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "\n",
    "import glob\n",
    "\n",
    "print('Copying checkpoint from GCS...')\n",
    "!rm -r /content/onsets-frames\n",
    "!mkdir /content/onsets-frames\n",
    "!gsutil -q -m cp -R gs://magentadata/models/onsets_frames_transcription/* /content/onsets-frames/\n",
    "!unzip -o /content/onsets-frames/checkpoint.zip -d /content/onsets-frames\n",
    "CHECKPOINT_DIR = '/content/onsets-frames/train'\n",
    "  \n",
    "print('Installing dependencies...')\n",
    "!apt-get update -qq && apt-get install -qq libfluidsynth1 fluid-soundfont-gm build-essential libasound2-dev libjack-dev ffmpeg  \n",
    "!pip install pyfluidsynth pretty_midi\n",
    "\n",
    "if glob.glob('/content/onsets-frames/magenta*.whl'):\n",
    "  !pip install -q /content/onsets-frames/magenta*.whl\n",
    "else:\n",
    "  !pip install -q magenta\n",
    "\n",
    "# Hack to allow python to pick up the newly-installed fluidsynth lib.\n",
    "import ctypes.util\n",
    "\n",
    "orig_find_library = ctypes.util.find_library\n",
    "def proxy_find_library(lib):\n",
    "  if lib == 'fluidsynth':\n",
    "    return 'libfluidsynth.so.1'\n",
    "  else:\n",
    "    return orig_find_library(lib)\n",
    "\n",
    "ctypes.util.find_library = proxy_find_library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RE-Z_a6UCAf2"
   },
   "source": [
    "# Model Initializiation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "both",
    "colab": {},
    "colab_type": "code",
    "id": "n8GJ5pRc6biH"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import librosa\n",
    "import numpy as np\n",
    "\n",
    "from google.colab import files\n",
    "\n",
    "from magenta.common import tf_utils\n",
    "from magenta.music import audio_io\n",
    "import magenta.music as mm\n",
    "from magenta.models.onsets_frames_transcription import model\n",
    "from magenta.models.onsets_frames_transcription import constants\n",
    "from magenta.models.onsets_frames_transcription import data\n",
    "from magenta.models.onsets_frames_transcription import infer_util\n",
    "from magenta.music import midi_io\n",
    "from magenta.protobuf import music_pb2\n",
    "\n",
    "## Define model and load checkpoint\n",
    "## Only needs to be run once.\n",
    "\n",
    "acoustic_checkpoint = tf.train.latest_checkpoint(CHECKPOINT_DIR)\n",
    "print('acoustic_checkpoint=' + acoustic_checkpoint)\n",
    "hparams =  tf_utils.merge_hparams(\n",
    "      constants.DEFAULT_HPARAMS, model.get_default_hparams())\n",
    "\n",
    "with tf.Graph().as_default():\n",
    "  examples = tf.placeholder(tf.string, [None])\n",
    "\n",
    "  num_dims = constants.MIDI_PITCHES\n",
    "\n",
    "  batch, iterator = data.provide_batch(\n",
    "      batch_size=1,\n",
    "      examples=examples,\n",
    "      hparams=hparams,\n",
    "      is_training=False,\n",
    "      truncated_length=0)\n",
    "\n",
    "  model.get_model(batch, hparams, is_training=False)\n",
    "\n",
    "  session = tf.Session()\n",
    "  saver = tf.train.Saver()\n",
    "  saver.restore(session, acoustic_checkpoint)\n",
    "\n",
    "  onset_probs_flat = tf.get_default_graph().get_tensor_by_name(\n",
    "      'onsets/onset_probs_flat:0')\n",
    "  frame_probs_flat = tf.get_default_graph().get_tensor_by_name(\n",
    "     'frame_probs_flat:0')\n",
    "  velocity_values_flat = tf.get_default_graph().get_tensor_by_name(\n",
    "     'velocity/velocity_values_flat:0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qFiGvUBaCM9_"
   },
   "source": [
    "# Getting the Audio Files\n",
    "\n",
    "One key part of the transcription is having fairly clean audio files as input.  To do this conveniently, we create two folders for audio : ```./orig``` and ```./audio```.  The ```./orig``` folder is where the new files are uploaded (either from your machine, or by grabbing them out of YouTube videos).  \n",
    "\n",
    "Then, using the ```! cp ./orig/XYZ ./audio/``` cell below, you can pick each individual audio file that you want to experiment with, without having to repeatedly doing the import steps (and you can easily switch back-and-forth too).\n",
    "\n",
    "The original Magenta code has been modified to read the files from the ```./audio``` folder.  \n",
    "\n",
    "\n",
    "## Upload Audio\n",
    "\n",
    "Run the following cell to upload audio files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rEFMoYFRmwLX"
   },
   "outputs": [],
   "source": [
    "! rm ./audio/*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6te27kh0ZSY1"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "if not os.path.exists('./audio'):\n",
    "  os.mkdir('./audio')\n",
    "\n",
    "if not os.path.exists('./orig'):\n",
    "  os.mkdir('./orig')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CHDkREz6rOCr"
   },
   "outputs": [],
   "source": [
    "uploaded = files.upload()\n",
    "\n",
    "for fn in uploaded.keys():\n",
    "  print('User uploaded file \"{name}\" with length {length} bytes'.format(\n",
    "      name=fn, length=len(uploaded[fn])))\n",
    "  open(os.path.join('./orig', fn), 'w').write(uploaded[fn])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3Xq4IN5DpyE7"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dAb3qYu4bcW8"
   },
   "source": [
    "## Download Audio from YouTube\n",
    "\n",
    "Run the following cell to download audio files from YouTube."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dlcy9HJOZSQr"
   },
   "outputs": [],
   "source": [
    "# !apt-get -qq install youtube-dl\n",
    "# !apt-get remove youtube-dl \n",
    "! pip install youtube-dl  # More recent version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: The best audio tends to be from studio recordings, rather than concerts.  This is probably because the Magenta training set was done 'dry' (i.e. without reverb).   So there should also be the opportunity to create a better model by adding some (synthetic) reverb and background noises to the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "R1KhGlzZZSGm"
   },
   "outputs": [],
   "source": [
    "# https://stackoverflow.com/questions/49804874/dowload-the-best-quality-audio-file-with-youtube-dl\n",
    "\n",
    "# Scarlatti : \n",
    "yt=\"https://www.youtube.com/watch?v=yIAk61xEZ80\"\n",
    "\n",
    "# Scarlatti Guitar (not piano) :\n",
    "#yt=\"https://www.youtube.com/watch?v=Q-1O6A8P5mM\"\n",
    "\n",
    "# Satie Gnossen (a bit quiet) : \n",
    "#yt=\"https://www.youtube.com/watch?v=IUAF3abGY2M\"\n",
    "\n",
    "# Kissin Chopin Etude  Etude Op. 10 No. 4\n",
    "#yt=\"https://www.youtube.com/watch?v=0TZy4Va97xQ\"\n",
    "\n",
    "# Kissin Winter Wind (perhaps concert recordings aren't the best...) :\n",
    "#yt=\"https://www.youtube.com/watch?v=Zsks5L2QPO0\"\n",
    "\n",
    "! youtube-dl --extract-audio --audio-format wav {yt}\n",
    "\n",
    "! mv *.wav ./orig/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6eHH2oAYfZNK"
   },
   "outputs": [],
   "source": [
    "! ls -l ./orig\n",
    "! rm ./audio/*\n",
    "\n",
    "# ! cp ./orig/401* ./audio/  # Peterson Someone To Watch Over Me\n",
    "# ! cp ./orig/23* ./audio/    # Chopin Winter Wind\n",
    "! cp \"./orig/Scarlatti Sonate K.455, Yuja Wang-yIAk61xEZ80.wav\" ./audio/ \n",
    "\n",
    "! ls -l ./audio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Load the audio into TF-ready format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "both",
    "colab": {},
    "colab_type": "code",
    "id": "AxmkX4Tu5UJd"
   },
   "outputs": [],
   "source": [
    "#for fn in uploaded.keys():\n",
    "#  print('User uploaded file \"{name}\" with length {length} bytes'.format(\n",
    "#      name=fn, length=len(uploaded[fn])))\n",
    "#  open(fn, 'w').write(uploaded[fn])\n",
    "\n",
    "to_process = []\n",
    "for fname in os.listdir('./audio'):\n",
    "  fn = os.path.join('./audio', fname)\n",
    "  \n",
    "  raw_audio, _sample_rate = librosa.core.load(fn, sr=hparams.sample_rate)\n",
    "  print(raw_audio.shape)\n",
    "  \n",
    "  # These two lines restrict the audio to the first 30seconds\n",
    "  #   Remove/modify them if you need a different segment/the whole sample.\n",
    "  raw_audio = raw_audio[int(hparams.sample_rate* 0.) :\n",
    "                        int(hparams.sample_rate*30.)]\n",
    "  print(raw_audio.shape)\n",
    "\n",
    "  wav_data = audio_io.samples_to_wav_data(\n",
    "      librosa.util.normalize(raw_audio),\n",
    "      hparams.sample_rate)\n",
    "  \n",
    "  example = tf.train.Example(features=tf.train.Features(feature={\n",
    "      'id':\n",
    "          tf.train.Feature(bytes_list=tf.train.BytesList(\n",
    "              value=[fn.encode('utf-8')]\n",
    "          )),\n",
    "      'sequence':\n",
    "          tf.train.Feature(bytes_list=tf.train.BytesList(\n",
    "              value=[music_pb2.NoteSequence().SerializeToString()]\n",
    "          )),\n",
    "      'audio':\n",
    "          tf.train.Feature(bytes_list=tf.train.BytesList(\n",
    "              value=[wav_data]\n",
    "          )),\n",
    "      'velocity_range':\n",
    "          tf.train.Feature(bytes_list=tf.train.BytesList(\n",
    "              value=[music_pb2.VelocityRange().SerializeToString()]\n",
    "          )),\n",
    "  }))\n",
    "  to_process.append(example.SerializeToString())\n",
    "  print('Processing complete for', fn)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RcrvMINkmdQh"
   },
   "outputs": [],
   "source": [
    "# Create an iterator over the files\n",
    "session.run(iterator.initializer, {examples: to_process})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lMimufLfNMMq"
   },
   "source": [
    "# Inference\n",
    "\n",
    "Run the following cell to transcribe the files you uploaded. Each time it runs it will transcribe one of the uploaded files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Z5SYRvIm8gq5"
   },
   "outputs": [],
   "source": [
    "filenames, frame_logits, onset_logits, velocity_values = session.run([\n",
    "    batch.filenames,\n",
    "    frame_probs_flat,\n",
    "    onset_probs_flat,\n",
    "    velocity_values_flat\n",
    "])\n",
    "\n",
    "print('Inference complete for', filenames[0])\n",
    "\n",
    "frame_predictions = frame_logits > .5\n",
    "\n",
    "onset_predictions = onset_logits > .5\n",
    "\n",
    "sequence_prediction = infer_util.pianoroll_to_note_sequence(\n",
    "    frame_predictions,\n",
    "    frames_per_second=data.hparams_frames_per_second(hparams),\n",
    "    min_duration_ms=0,\n",
    "    onset_predictions=onset_predictions,\n",
    "    velocity_values=velocity_values)\n",
    "\n",
    "mm.plot_sequence(sequence_prediction)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KkabcqQSkRRk"
   },
   "outputs": [],
   "source": [
    "mm.play_sequence(sequence_prediction, mm.midi_synth.fluidsynth)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1h4JXhKZIzjc"
   },
   "outputs": [],
   "source": [
    "frame_predictions.shape # (8099, 88)  Booleans \n",
    "#onset_predictions.shape # (8099, 88)  Booleans\n",
    "#velocity_values.shape  # (8099, 88)  # values range :-0.51426625 ...  1.3687868"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FFDVTcMwj5VX"
   },
   "source": [
    "Optionally run the following cell to download a MIDI version of the inferred transcription."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3JnQW-Gaj-5d"
   },
   "outputs": [],
   "source": [
    "midi_filename = (filenames[0] + '.mid').replace(' ', '_')\n",
    "midi_io.sequence_proto_to_midi_file(sequence_prediction, midi_filename)\n",
    "\n",
    "files.download(midi_filename)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Onsets-and-Frames_mdda.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 2",
   "name": "python2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}