ddsp/colab/tutorials/4_core_functions.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "NpJd3dlOCStH"
      },
      "source": [
        "\u003ca href=\"https://colab.research.google.com/github/magenta/ddsp/blob/master/ddsp/colab/tutorials/4_core_functions.ipynb\" target=\"_parent\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/\u003e\u003c/a\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "hMqWDc_m6rUC"
      },
      "source": [
        "\n",
        "##### Copyright 2020 Google LLC.\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "VNhgka4UKNjf"
      },
      "outputs": [],
      "source": [
        "# Copyright 2020 Google LLC. All Rights Reserved.\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License.\n",
        "# =============================================================================="
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ZFIqwYGbZ-df"
      },
      "source": [
        "# DDSP Core Functions\n",
        "\n",
        "This notebook provides some simple demonstrations of using DDSP primitives for synthesis, filtering, and interpolation. Keep in mind that all of components are fully differentiable and can be integrated with neural networks and end-2-end learning. \n",
        "\n",
        "While the preferred API is to use the Synthesizer and Effect Processors that are built around these central components, it is of course possible to call the core functions directly as well."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "y2CDNnLs6DLJ"
      },
      "outputs": [],
      "source": [
        "#@title Install and import dependencies\n",
        "\n",
        "%tensorflow_version 2.x\n",
        "!pip install -qU ddsp\n",
        "\n",
        "# Ignore a bunch of deprecation warnings\n",
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")\n",
        "\n",
        "import ddsp\n",
        "import ddsp.training\n",
        "from ddsp.colab.colab_utils import (play, specplot, transfer_function, \n",
        "                                    plot_impulse_responses, DEFAULT_SAMPLE_RATE)\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "import tensorflow_datasets as tfds\n",
        "\n",
        "sample_rate = DEFAULT_SAMPLE_RATE  # 16000"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "iZ6aqiNiYGHW"
      },
      "source": [
        "# Generation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "AiCIG1x5bxkh"
      },
      "source": [
        "## `oscillator_bank()`\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "EvbLU56AWG-D"
      },
      "source": [
        "\n",
        "Synthesize audio with an array of sinusoidal oscillators. Frequencies and amplitudes must be provided at audio rate.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "LL5mf1ECkO4V"
      },
      "source": [
        "### Ex: Simple harmonic sound"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "LkyUtCmfi_7O"
      },
      "outputs": [],
      "source": [
        "n_samples = int(sample_rate * 4.0)\n",
        "n_components = 3\n",
        "\n",
        "# Amplitudes [n_batch, n_samples, n_components].\n",
        "# Linearly decay in time.\n",
        "amps = np.linspace(0.3, 0.0, n_samples)\n",
        "amps = np.tile(amps[np.newaxis, :, np.newaxis], [1, 1, n_components])\n",
        "\n",
        "# Frequencies in Hz [n_batch, n_samples, n_components].\n",
        "frequencies = np.ones([1, n_samples, 1]) * np.array([[[220, 440, 660]]])\n",
        "\n",
        "# Sythesize.\n",
        "audio = ddsp.core.oscillator_bank(frequencies, amps, sample_rate)\n",
        "\n",
        "# Listen.\n",
        "play(audio)\n",
        "specplot(audio)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "4x1YxNHkNlr4"
      },
      "source": [
        "### Ex: Random frequencies"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "v_wl-sVgNlr7"
      },
      "outputs": [],
      "source": [
        "n_samples = int(sample_rate * 4.0)\n",
        "n_components = 6\n",
        "n_frames = 100\n",
        "\n",
        "# Amplitudes [n_batch, n_samples, n_components].\n",
        "# Linearly decay in time.\n",
        "amps = np.linspace(0.3, 0.0, n_samples)\n",
        "amps = np.tile(amps[np.newaxis, :, np.newaxis], [1, 1, n_components])\n",
        "\n",
        "# Frequencies in Hz [n_batch, n_samples, n_components].\n",
        "frequencies = []\n",
        "for _ in range(n_components):\n",
        "  f_start = np.random.uniform(20, 4000)\n",
        "  f_end = np.random.uniform(20, 4000)\n",
        "  frequencies.append(np.linspace(f_start, f_end, n_frames))\n",
        "frequencies = np.stack(frequencies).T[np.newaxis, ...] \n",
        "frequencies = ddsp.core.resample(frequencies, n_samples)\n",
        "\n",
        "# Sythesize.\n",
        "audio = ddsp.core.oscillator_bank(frequencies, amps, sample_rate)\n",
        "\n",
        "# Listen.\n",
        "play(audio)\n",
        "specplot(audio)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "XpoIxVsfvxyy"
      },
      "source": [
        "### Ex: Swarm of sinusoids\n",
        "\n",
        "Just for fun..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "JprtBl2Wvxy2"
      },
      "outputs": [],
      "source": [
        "def smooth(x, window_size=2000):\n",
        "  \"\"\"Smooth signal with box filter. For random frequency modulation.\"\"\"\n",
        "  window = np.ones(window_size) / window_size\n",
        "  return np.convolve(window, x, mode='same')\n",
        "\n",
        "n_samples = int(sample_rate * 6.0)\n",
        "n_components = 100\n",
        "\n",
        "# Time points for the frequency ramp.\n",
        "n_start = int(sample_rate * 1.5)\n",
        "n_stop = int(sample_rate * 4.0)\n",
        "n_ramp = n_stop - n_start\n",
        "n_level = n_samples - n_stop\n",
        "\n",
        "# Amplitudes [n_batch, n_samples, n_components].\n",
        "# Decrease amplitude for higher components.\n",
        "amps = np.ones([1, n_samples, 1])\n",
        "amps = amps * np.logspace(0, -2, n_components)[np.newaxis, np.newaxis, :]\n",
        "# Fade in at the start, out at end.\n",
        "amps[:, :n_start, :] *= np.logspace(-2, 0, n_start)[np.newaxis, :, np.newaxis]\n",
        "amps[:, -2000:, :] *= np.logspace(0, -2, 2000)[np.newaxis, :, np.newaxis]\n",
        "\n",
        "# Frequencies in Hz [n_batch, n_samples, n_components].\n",
        "# Sweep frequencies from random initial frequenices to fixed final frequencies.\n",
        "freq_initial = np.random.uniform(low=240.0, high=280.0, size=10)\n",
        "harmonics = np.arange(1, 11)\n",
        "f0 = np.array([0.5, 1, 1, 2.5, 3, 3.5, 4, 4.5, 5, 5.25])\n",
        "freq_final = 150 * f0\n",
        "\n",
        "# Treat each frequency sweep separately.\n",
        "frequencies = []\n",
        "\n",
        "for i, f in zip(freq_initial, freq_final):\n",
        "  # Sweep the frequency.\n",
        "  freq = np.concatenate(\n",
        "      [i * np.ones(n_start), np.linspace(i, f, n_ramp), f * np.ones(n_level),])\n",
        "  # Modulate the frequency.\n",
        "  d_freq = smooth(np.concatenate([\n",
        "      np.random.uniform(low=0.1, high=1.9, size=n_start),\n",
        "      np.random.uniform(low=0.5, high=1.5, size=n_ramp + n_level),\n",
        "  ]))\n",
        "  freq *= d_freq\n",
        "  # Add harmonics for each fundamental.\n",
        "  frequencies.append([freq * h for h in harmonics])\n",
        "\n",
        "# Rearrange to [n_batch, n_samples, n_components].\n",
        "frequencies = np.transpose(np.stack(frequencies), (2, 1, 0))\n",
        "frequencies = np.reshape(frequencies, [1, n_samples, -1])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "34QCUicI-WaE"
      },
      "outputs": [],
      "source": [
        "# Sythesize.\n",
        "audio = ddsp.core.oscillator_bank(frequencies, amps, sample_rate)\n",
        "\n",
        "# Listen.\n",
        "audio /= np.abs(audio).max()\n",
        "play(audio)\n",
        "specplot(audio)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "kQgyvYMkhKHS"
      },
      "source": [
        "## `linear_lookup()`\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "A-t0xjI4hKHa"
      },
      "source": [
        "Synthesize audio with an array of sinusoidal oscillators. Frequencies and amplitudes must be provided at audio rate.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "q9ngxn8pD0K6"
      },
      "source": [
        "### Ex: Sinusoidal lookup\n",
        "\n",
        "As a simple example, lookup from a sin-wave wavetable produces a sin wave at the lookup frequency"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "FvhBNNdB-fSI"
      },
      "outputs": [],
      "source": [
        "n_samples = int(sample_rate * 1.0)\n",
        "n_wavetable = 2048\n",
        "n_cycles = 440\n",
        "\n",
        "# Sin wave\n",
        "wavetable = tf.sin(tf.linspace(0.0, 2.0 * np.pi, n_wavetable))\n",
        "wavetable = wavetable[tf.newaxis, tf.newaxis, :]\n",
        "\n",
        "phase = tf.linspace(0.0, n_cycles, n_samples) % 1.0\n",
        "phase = phase[tf.newaxis, :, tf.newaxis]\n",
        "\n",
        "output = ddsp.core.linear_lookup(phase, wavetable)\n",
        "target = np.sin(np.linspace(0.0, 2.0 * np.pi * n_cycles, n_samples))\n",
        "\n",
        "# For plotting.\n",
        "output = output[0]\n",
        "phase = phase[0, :, 0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "fHLNIa_bANKi"
      },
      "outputs": [],
      "source": [
        "# Plot the results\n",
        "plt.figure(figsize=(12, 6))\n",
        "\n",
        "plt.subplot(121)\n",
        "plt.plot(wavetable[0, 0, :])\n",
        "plt.title('Wavetable')\n",
        "\n",
        "plt.subplot(122)\n",
        "plt.plot(output[:200], label='Output')\n",
        "plt.plot(phase[:200], label='Oscillator Phase')\n",
        "plt.plot(target[:200], label='Target')\n",
        "plt.title('Wavetable lookup')\n",
        "plt.legend(loc='lower left')\n",
        "\n",
        "print('Target')\n",
        "play(target)\n",
        "print('Output')\n",
        "play(output)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "abg8rdAkIoID"
      },
      "source": [
        "There are small aritfacts due to the linear interpolation and implicit resampling of the signal"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "zUNRp0FwIuKd"
      },
      "outputs": [],
      "source": [
        "plt.plot(target[:200] - output[:200])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "8uHuGn7JEIR5"
      },
      "source": [
        "You can also use any arbitrary waveform as the lookup signal to get more interesting outputs"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "WPuFyX9fCHJy"
      },
      "outputs": [],
      "source": [
        "modulation =  tf.linspace(0.0, 0.5, n_samples)\n",
        "modulation = modulation[tf.newaxis, :, tf.newaxis]\n",
        "phase2 = (tf.sin(np.pi * phase[tf.newaxis, :, tf.newaxis]) + modulation)**2.0 % 1.0\n",
        "output2 = ddsp.core.linear_lookup(phase2, wavetable)\n",
        "\n",
        "# For plotting\n",
        "output2 = output2[0]\n",
        "phase2 = phase2[0, :, 0]\n",
        "\n",
        "# Plot the results\n",
        "plt.figure(figsize=(6, 6))\n",
        "\n",
        "plt.plot(output2[:200], label='Output')\n",
        "plt.plot(phase2[:200], label='Oscillator Phase')\n",
        "plt.title('Wavetable lookup')\n",
        "plt.ylim(-1.5, 1.5)\n",
        "plt.legend(loc='lower left')\n",
        "\n",
        "print('Output')\n",
        "play(output2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "aXg73zyno-4U"
      },
      "source": [
        "### Ex: Wavetable Synthesis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "4upeES-sNb25"
      },
      "source": [
        "We can also use this linear lookup to build a wavetable synthesizer. Here, we pass in a series of wavetables (one for each timestep) and look up from the changing wavetables over time"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Wiwn7bFQLuB2"
      },
      "outputs": [],
      "source": [
        "n_secs = 3\n",
        "n_samples = int(sample_rate * n_secs)\n",
        "n_wavetable = 2048\n",
        "n_cycles = 110 * n_secs\n",
        "\n",
        "phase = tf.linspace(0.0, n_cycles, n_samples) % 1.0\n",
        "phase = phase[tf.newaxis, :, tf.newaxis]\n",
        "\n",
        "# Sin wave\n",
        "wavetable_sin = tf.sin(tf.linspace(0.0, 2.0 * np.pi, n_wavetable))\n",
        "wavetable_sin = wavetable_sin[tf.newaxis, tf.newaxis, :]\n",
        "\n",
        "# Square wave\n",
        "wavetable_square = tf.cast(wavetable_sin \u003e 0.0, tf.float32) * 2.0 - 1.0\n",
        "\n",
        "# Combine them\n",
        "wavetables = tf.concat([wavetable_sin, wavetable_square], axis=1)\n",
        "wavetables = ddsp.core.resample(wavetables, n_samples)\n",
        "\n",
        "wavetables *= 0.5\n",
        "\n",
        "output_multiwave = ddsp.core.linear_lookup(phase, wavetables)\n",
        "\n",
        "# For plotting\n",
        "wavetables = wavetables[0]\n",
        "output_multiwave = output_multiwave[0]\n",
        "phase = phase[0, :, 0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "vIEvGgEsMRoa"
      },
      "outputs": [],
      "source": [
        "# Plot the results\n",
        "plt.figure(figsize=(12, 6))\n",
        "\n",
        "plt.subplot(121)\n",
        "plt.plot(wavetables[0, :])\n",
        "plt.plot(wavetables[16000, :])\n",
        "plt.plot(wavetables[32000, :])\n",
        "plt.title('Wavetable')\n",
        "\n",
        "plt.subplot(122)\n",
        "plt.plot(output_multiwave[:200], label='Output')\n",
        "plt.plot(phase[:200], label='Oscillator Phase')\n",
        "plt.title('Wavetable lookup')\n",
        "plt.legend(loc='lower left')\n",
        "\n",
        "print('Output')\n",
        "play(output_multiwave)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "7A3AasLQuIAZ"
      },
      "source": [
        "## `wavetable_synthesis()`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "mhqX9FYtpqBZ"
      },
      "source": [
        "We also have a convenience function to make wavetable synthesis easier. `wavetable_synthesis()` takes a frame-based frequency and amplitude of the oscillator."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "gQCjpkgarsqk"
      },
      "outputs": [],
      "source": [
        "n_frames = 100\n",
        "frequencies = 110 * tf.linspace(1.5, 1, n_frames)[tf.newaxis, :, tf.newaxis]\n",
        "amplitudes = 0.5 * tf.linspace(0.7, 0.001, n_frames)[tf.newaxis, :, tf.newaxis]\n",
        "\n",
        "n_secs = 3\n",
        "n_samples = int(sample_rate * n_secs)\n",
        "n_wavetable = 2048\n",
        "\n",
        "# Sin wave\n",
        "wavetable_sin = tf.sin(tf.linspace(0.0, 2.0 * np.pi, n_wavetable))\n",
        "wavetable_sin = wavetable_sin[tf.newaxis, tf.newaxis, :]\n",
        "\n",
        "# Square wave\n",
        "wavetable_square = tf.cast(wavetable_sin \u003e 0.0, tf.float32) * 2.0 - 1.0\n",
        "\n",
        "# Combine them\n",
        "wavetables = tf.concat([wavetable_sin, wavetable_square, wavetable_sin], axis=1)\n",
        "wavetables = ddsp.core.resample(wavetables, n_samples)\n",
        "\n",
        "output_multiwave = ddsp.core.wavetable_synthesis(frequencies, \n",
        "                                                 amplitudes, \n",
        "                                                 wavetables, \n",
        "                                                 n_samples=n_samples,\n",
        "                                                 sample_rate=sample_rate)\n",
        "\n",
        "# For plotting\n",
        "wavetables = wavetables[0]\n",
        "output_multiwave = output_multiwave[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "moBlBkPGrsqm"
      },
      "outputs": [],
      "source": [
        "# Plot the results\n",
        "plt.figure(figsize=(12, 6))\n",
        "\n",
        "plt.subplot(121)\n",
        "plt.plot(wavetables[0, :])\n",
        "plt.plot(wavetables[16000, :])\n",
        "plt.plot(wavetables[32000, :])\n",
        "plt.title('Wavetable')\n",
        "\n",
        "print('Output')\n",
        "play(output_multiwave)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "n5L0bdsntwEE"
      },
      "source": [
        "## `variable_length_delay()`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "O5FIZSvrvAy_"
      },
      "source": [
        "If we instead treat a moving window as a \"wavetable\" we can implement a variable time delay in a forward pass using `linear_lookup()`. Variable time delays the key component to time modulation effects such as [vibrato](https://en.wikipedia.org/wiki/Vibrato), [chorus](https://en.wikipedia.org/wiki/Chorus_effect), and [flanging](https://en.wikipedia.org/wiki/Flanging). "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "IzzaWKxVkYms"
      },
      "outputs": [],
      "source": [
        "# Get a single example from NSynth.\n",
        "# Takes a few seconds to load from GCS.\n",
        "data_provider = ddsp.training.data.NSynthTfds(split='train')\n",
        "batch = data_provider.get_batch(batch_size=1, shuffle=False).skip(1)\n",
        "audio = next(iter(tfds.as_numpy(batch)))['audio']\n",
        "\n",
        "specplot(audio)\n",
        "play(audio)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "2XlkznvlT0ai"
      },
      "outputs": [],
      "source": [
        "n_samples = audio.shape[1]\n",
        "n_seconds = n_samples / sample_rate\n",
        "\n",
        "def sin_phase(mod_rate):\n",
        "  phase = tf.sin(tf.linspace(0.0, mod_rate * n_seconds * 2.0 * np.pi, n_samples))\n",
        "  phase = (phase[tf.newaxis, :, tf.newaxis] + 1.0) / 2.0  # Scale to [0, 1.0]\n",
        "  return phase"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "i2-KD36rkmfw"
      },
      "source": [
        "### Ex. Flanger"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Pfx1qz_uSS8J"
      },
      "outputs": [],
      "source": [
        "mod_rate = 0.25  # Hz\n",
        "mod_ms = 1.5\n",
        "center_ms = 0.0\n",
        "delay_ms = mod_ms + center_ms\n",
        "max_length = int(sample_rate / 1000.0 * delay_ms)\n",
        "\n",
        "phase = sin_phase(mod_rate) * (mod_ms / delay_ms) + (center_ms / delay_ms)\n",
        "\n",
        "audio_wet = ddsp.core.variable_length_delay(phase, \n",
        "                                            audio, \n",
        "                                            max_length=max_length)\n",
        "\n",
        "audio_out = 0.5 * (audio + audio_wet)\n",
        "\n",
        "# Listen.\n",
        "play(audio_out)\n",
        "specplot(audio_out)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "X26Camb_TMGL"
      },
      "source": [
        "### Ex. Chorus"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "ncvGgEf8TMGO"
      },
      "outputs": [],
      "source": [
        "mod_rate = 2.0  # Hz\n",
        "mod_ms = 1.0\n",
        "center_ms = 25.0\n",
        "delay_ms = mod_ms + center_ms\n",
        "max_length = int(sample_rate / 1000.0 * delay_ms)\n",
        "\n",
        "phase = sin_phase(mod_rate) * (mod_ms / delay_ms) + (center_ms / delay_ms)\n",
        "\n",
        "audio_wet = ddsp.core.variable_length_delay(phase, \n",
        "                                            audio, \n",
        "                                            max_length=max_length)\n",
        "\n",
        "audio_out = 0.5 * (audio + audio_wet)\n",
        "\n",
        "# Listen.\n",
        "play(audio_out)\n",
        "specplot(audio_out)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "u1UEha2zTNc5"
      },
      "source": [
        "### Ex. Vibrato"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "NYSfiNFvTNc7"
      },
      "outputs": [],
      "source": [
        "mod_rate = 1.0  # Hz\n",
        "mod_ms = 20.0\n",
        "center_ms = 00.0\n",
        "delay_ms = mod_ms + center_ms\n",
        "max_length = int(sample_rate / 1000.0 * delay_ms)\n",
        "\n",
        "phase = sin_phase(mod_rate) * (mod_ms / delay_ms) + (center_ms / delay_ms)\n",
        "\n",
        "audio_wet = ddsp.core.variable_length_delay(phase, \n",
        "                                            audio, \n",
        "                                            max_length=max_length)\n",
        "\n",
        "audio_out = audio_wet\n",
        "\n",
        "# Listen.\n",
        "play(audio_out)\n",
        "specplot(audio_out)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Sh0Yd1u_Xq9P"
      },
      "source": [
        "# Filtering\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "RDT5FM0lVeaO"
      },
      "source": [
        "Time-varying differentiable linear filters (parameterized in frequency space). Impulse responses are designed by `sinc_impulse_response()` and `frequency_impulse_reponse()` and then applied by `fft_convolve()`. \n",
        "\n",
        "`sinc_filter()` and `frequency_filter()` are thin wrappers around filter design and `fft_convolve()`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "nYhd919a8vid"
      },
      "source": [
        "## `fft_convolve()`\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "7a0y33qnVnFf"
      },
      "source": [
        "Time-varying filter. Given audio [batch, n_samples], and a series of impulse responses [batch, n_frames, n_impulse_response], splits the audio into frames, applies filters, and then overlap-and-adds audio back together. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "fmgwdYujl-cR"
      },
      "source": [
        "#### Ex: Low-pass sweep"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "-tLry0oimFZo"
      },
      "outputs": [],
      "source": [
        "## Low-pass sweep in Hertz.\n",
        "noise = np.random.uniform(-0.5, 0.5, [1, sample_rate *4])\n",
        "f_cutoff = np.linspace(0., 1.0, 200)[np.newaxis, :, np.newaxis]\n",
        "\n",
        "ir = ddsp.core.sinc_impulse_response(f_cutoff , 2048)\n",
        "filtered = ddsp.core.fft_convolve(noise, ir)\n",
        "\n",
        "specplot(noise)\n",
        "specplot(filtered)\n",
        "play(noise)\n",
        "play(filtered)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "SaFiXN_dk45P"
      },
      "source": [
        "## `sinc_impulse_response()`\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "DPl1DRcgVgWh"
      },
      "source": [
        "Simple FIR low-pass filter design using sinc functions."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "VHn_Xvzm9rLW"
      },
      "source": [
        "### Ex: Brick-wall filter\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "sy45Gpjg9rLa"
      },
      "outputs": [],
      "source": [
        "# Brick-wall filter\n",
        "f_cutoff = 4000\n",
        "window_size = 2000\n",
        "\n",
        "# True filter.\n",
        "impulse_response = ddsp.core.sinc_impulse_response(f_cutoff, \n",
        "                                                   window_size, \n",
        "                                                   sample_rate)\n",
        "\n",
        "# Ideal brick-wall filter\n",
        "half_nyquist = int(window_size / 2)\n",
        "desired_magnitudes = np.concatenate([np.ones([half_nyquist]), \n",
        "                                     np.zeros([half_nyquist]) + 1e-6], axis=0) \n",
        "\n",
        "plot_impulse_responses(impulse_response, desired_magnitudes)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "i1qjbB-FQuml"
      },
      "outputs": [],
      "source": [
        "## Normalized frequency [0, 1] works as well, without needing sample_rate.\n",
        "f_cutoff = 0.5\n",
        "\n",
        "# True filter.\n",
        "impulse_response = ddsp.core.sinc_impulse_response(f_cutoff, window_size)\n",
        "\n",
        "plot_impulse_responses(impulse_response, desired_magnitudes)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "YNtT-wVw9rLd"
      },
      "outputs": [],
      "source": [
        "# Changing window size changes the time-frequency characteristics.\n",
        "impulse_response = ddsp.core.sinc_impulse_response(f_cutoff, window_size=250)\n",
        "\n",
        "plot_impulse_responses(impulse_response, desired_magnitudes)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "S7CmzEUJOeoD"
      },
      "source": [
        "## `sinc_filter()`\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ltffSmcPVqB3"
      },
      "source": [
        "Thin wrapper around `sinc_impulse_response()` and `fft_convolve`. Filter audio with a low-pass filter.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "9gs6AHwjU-kf"
      },
      "source": [
        "### Ex: Bandlimited Upsampling\n",
        "\n",
        "Let's start with a triangle wave at 100 Hz, sampled at 10kHz."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "j6hABvRjZCgG"
      },
      "outputs": [],
      "source": [
        "original_sample_rate = 10000\n",
        "n_samples = sample_rate + 1\n",
        "\n",
        "# Let's start with a triangle wave at 100 Hz.\n",
        "time = tf.linspace(0.0, 1.0, n_samples)\n",
        "signal = (tf.linspace(0.0, 100.0, n_samples) % 1.0) - 0.5\n",
        "\n",
        "# Look at FFT of signal.\n",
        "frequencies, magnitudes = transfer_function(signal[tf.newaxis, tf.newaxis, :],\n",
        "                                            sample_rate=original_sample_rate)\n",
        "\n",
        "plt.figure(figsize=(12, 6))\n",
        "plt.subplot(121)\n",
        "plt.plot(time[:200], signal[:200])\n",
        "plt.title('Amplitude (time)')\n",
        "plt.subplot(122)\n",
        "plt.semilogy(frequencies, magnitudes[0, 0, :])\n",
        "plt.title('Magnitude (frequency)')\n",
        "\n",
        "print('Original')\n",
        "play(signal, sample_rate=original_sample_rate)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "z1sVjCM4E6z9"
      },
      "source": [
        "If we naively double the sample rate to 20kHz, we introduce upsampling artifacts."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "YJumSQa7U-kl"
      },
      "outputs": [],
      "source": [
        "upsample = 2\n",
        "upsample_rate = int(original_sample_rate * upsample)\n",
        "n_upsample = int(n_samples * upsample)\n",
        "\n",
        "time_up = tf.linspace(0.0, 1.0, n_upsample)\n",
        "\n",
        "# Box upsampling\n",
        "signal_up = tf.compat.v1.image.resize_nearest_neighbor(\n",
        "    signal[tf.newaxis, :, tf.newaxis, tf.newaxis], [n_upsample, 1]\n",
        "    )[0, :, 0, 0]\n",
        "frequencies_up, magnitudes_up = transfer_function(signal_up[tf.newaxis, tf.newaxis, :],\n",
        "                                                      sample_rate=upsample_rate)\n",
        "\n",
        "# Bilinear upsampling\n",
        "signal_up_bl = ddsp.core.resample(signal[tf.newaxis, :, tf.newaxis], n_upsample)[0, :, 0]\n",
        "frequencies_up_bl, magnitudes_up_bl = transfer_function(signal_up_bl[tf.newaxis, tf.newaxis, :],\n",
        "                                                        sample_rate=upsample_rate)\n",
        "\n",
        "plt.figure(figsize=(12, 6))\n",
        "plt.subplot(121)\n",
        "plt.semilogy(frequencies_up, magnitudes_up[0, 0, :], label='box upsample')\n",
        "plt.semilogy(frequencies_up_bl, magnitudes_up_bl[0, 0, :], label='bilinear upsample')\n",
        "plt.semilogy(frequencies, magnitudes[0, 0, :], label='original')\n",
        "plt.ylim(1e-3, 1e4)\n",
        "plt.title('Magnitude (frequency)')\n",
        "plt.legend()\n",
        "\n",
        "print('Box upsample')\n",
        "play(signal_up, sample_rate=upsample_rate)\n",
        "print('Bilinear upsample')\n",
        "play(signal_up_bl, sample_rate=upsample_rate)\n",
        "print('Original')\n",
        "play(signal, sample_rate=original_sample_rate)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "IcMe0kxUIT3M"
      },
      "source": [
        "By applying a brick-wall low-pass filter as above, we can remove aliasing artifacts."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "9QLx0ukAKEQ3"
      },
      "outputs": [],
      "source": [
        "n_frequencies = 1024\n",
        "half_nyquist = int(n_frequencies / 2)\n",
        "\n",
        "# Bandpass filters, [n_batch, n_frames, n_frequencies].\n",
        "cutoff_frequency = tf.ones([1, 1, 1]) * 0.5 \n",
        "\n",
        "signal_filt = ddsp.core.sinc_filter(signal_up[tf.newaxis, :],\n",
        "                               cutoff_frequency,\n",
        "                               window_size=1024)[0]\n",
        "\n",
        "frequencies_filt, magnitudes_filt = transfer_function(signal_filt[tf.newaxis, tf.newaxis, :],\n",
        "                                                      sample_rate=upsample_rate)\n",
        "\n",
        "\n",
        "plt.figure(figsize=(12, 6))\n",
        "plt.subplot(121)\n",
        "plt.semilogy(frequencies_up, magnitudes_up[0, 0, :], label='box upsample')\n",
        "plt.semilogy(frequencies_filt, magnitudes_filt[0, 0, :], label='anti-aliased')\n",
        "plt.ylim(1e-3, 1e4)\n",
        "plt.title('Magnitude (frequency)')\n",
        "plt.legend()\n",
        "\n",
        "print('Box upsample')\n",
        "play(signal_up, sample_rate=upsample_rate)\n",
        "print('Anti-aliased')\n",
        "play(signal_filt, sample_rate=upsample_rate)\n",
        "print('Original')\n",
        "play(signal, sample_rate=original_sample_rate)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "fkqK-kpscG-K"
      },
      "source": [
        "## `frequency_impulse_response()`\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "sve4ONRvVjCH"
      },
      "source": [
        "FIR filter design method used by `ddsp.frequency_filter()`. Uses the frequency sampling method of filter design as described [here]( https://ccrma.stanford.edu/~jos/sasp/Windowing_Desired_Impulse_Response.html)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "qmldSx4Nkwaj"
      },
      "source": [
        "### Ex: Arbitrary filter design\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "y--EYnEmcKQz"
      },
      "outputs": [],
      "source": [
        "# Brick-wall filter\n",
        "n_frequencies = 512\n",
        "half_nyquist = int(n_frequencies / 2)\n",
        "\n",
        "# Bandpass filters, [n_batch, n_frames, n_frequencies].\n",
        "magnitudes = (tf.linspace(1.0, 0.001, n_frequencies) + \n",
        "              0.1 * tf.sin(tf.linspace(0.0, 2.0 * np.pi * 8, n_frequencies)))\n",
        "magnitudes = magnitudes[tf.newaxis, tf.newaxis, :]\n",
        "desired_magnitudes = magnitudes[0, 0, :]\n",
        "\n",
        "# Designed filter.\n",
        "impulse_response = ddsp.core.frequency_impulse_response(magnitudes, window_size=0)\n",
        "\n",
        "plot_impulse_responses(impulse_response, desired_magnitudes)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "fz-Gql9kdg7t"
      },
      "outputs": [],
      "source": [
        "# Changing window size changes the time-frequency characteristics.\n",
        "impulse_response = ddsp.core.frequency_impulse_response(magnitudes, \n",
        "                                                        window_size=80)\n",
        "\n",
        "plot_impulse_responses(impulse_response, desired_magnitudes)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "4YVbRBdrKd4z"
      },
      "source": [
        "## `frequency_filter()`\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "8_WZotWTV7zi"
      },
      "source": [
        "\n",
        "Thin wrapper around `frequency_impulse_response()` and `fft_convolve`. Filter audio with a finite impulse response linear time-varying filter, designed using the frequency sampling method.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "sZaffuekKd41"
      },
      "source": [
        "### Ex: Arbitrary time-varying filter"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "l04gWYOnXvcn"
      },
      "source": [
        "Let's try a time-invariant filter. The `magnitudes` have a single frame and `n_frequency` bands linearly spaced between 0 and Nyquist."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "K3deVwbgKd42"
      },
      "outputs": [],
      "source": [
        "n_samples = int(sample_rate * 4.0)\n",
        "n_frequencies = 1000\n",
        "\n",
        "# White noise.\n",
        "audio_in = tf.random.uniform([1, n_samples], -0.5, 0.5)\n",
        "\n",
        "# Bandpass filters, [n_batch, n_frames, n_frequencies].\n",
        "magnitudes = tf.sin(tf.linspace(0.0, 10.0, n_frequencies))**4.0\n",
        "magnitudes = magnitudes[tf.newaxis, tf.newaxis, :]\n",
        "\n",
        "# Filter.\n",
        "audio_out = ddsp.core.frequency_filter(audio_in, magnitudes)\n",
        "\n",
        "# Listen.\n",
        "print('Original')\n",
        "play(audio_in)\n",
        "specplot(audio_in)\n",
        "\n",
        "print('Filtered')\n",
        "play(audio_out)\n",
        "specplot(audio_out)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "2ImwUm8aYERR"
      },
      "source": [
        "`ddsp.fir_filter()` uses the frequency sampling method of filter design as described [here]( https://ccrma.stanford.edu/~jos/sasp/Windowing_Desired_Impulse_Response.html).\n",
        "\n",
        "Reducing `n_frequencies` thus reduces frequency resolution.\n",
        "\n",
        "`window_size` crops the impulse responses to also determine the time-frequency tradeoff.\n",
        "`window_size` must be \u003e the fft_size which is the power of 2 \u003e= `n_frequencies` * 2.\n",
        "Setting `window_size` \u003c 1, automatically sets it at `n_frequencies`.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Dui_su3nWT0T"
      },
      "outputs": [],
      "source": [
        "# Fewer frequencies, less frequency resolution.\n",
        "n_frequencies = 32\n",
        "\n",
        "# Bandpass filters, [n_batch, n_frames, n_frequencies].\n",
        "magnitudes = tf.sin(tf.linspace(0.0, 10.0, n_frequencies))**4.0\n",
        "magnitudes = magnitudes[tf.newaxis, tf.newaxis, :]\n",
        "\n",
        "# Filter.\n",
        "audio_out = ddsp.core.frequency_filter(audio_in, magnitudes, window_size=0)\n",
        "\n",
        "# Listen.\n",
        "print('Less frequency resolution')\n",
        "play(audio_out)\n",
        "specplot(audio_out)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "sxnbxZsQYrQb"
      },
      "outputs": [],
      "source": [
        "# Smaller window_size, less frequency resolution (more temporal resolution).\n",
        "n_frequencies = 1000\n",
        "\n",
        "# Bandpass filters, [n_batch, n_frames, n_frequencies].\n",
        "magnitudes = tf.sin(tf.linspace(0.0, 10.0, n_frequencies))**4.0\n",
        "magnitudes = magnitudes[tf.newaxis, tf.newaxis, :]\n",
        "\n",
        "# Filter.\n",
        "audio_out = ddsp.core.frequency_filter(audio_in, magnitudes, window_size=32)\n",
        "\n",
        "# Listen.\n",
        "print('Smaller window')\n",
        "play(audio_out)\n",
        "specplot(audio_out)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "-xJw4QDmPJ8k"
      },
      "outputs": [],
      "source": [
        "# Now let's try a time-varying filter.\n",
        "n_frames = 250\n",
        "n_frequencies = 1000\n",
        "\n",
        "# Bandpass filters, [n_batch, n_frames, n_frequencies].\n",
        "magnitudes = [tf.sin(tf.linspace(0.0, w, n_frequencies))**4.0 for w in np.linspace(4.0, 40.0, n_frames)]\n",
        "magnitudes = tf.stack(magnitudes)\n",
        "magnitudes = magnitudes[tf.newaxis, :, :]\n",
        "\n",
        "# Filter.\n",
        "audio_out = ddsp.core.frequency_filter(audio_in, magnitudes)\n",
        "\n",
        "# Listen.\n",
        "print('Time-varying Filter')\n",
        "play(audio_out)\n",
        "specplot(audio_out)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "TrvuULKxUDNu"
      },
      "outputs": [],
      "source": [
        "# The filter equally spaces the frames in time, so if you don't have enough, you'll hear transitions.\n",
        "n_frames = 15\n",
        "\n",
        "# Bandpass filters, [n_batch, n_frames, n_frequencies].\n",
        "magnitudes = [tf.sin(tf.linspace(0.0, w, n_frequencies))**4.0 for w in np.linspace(4.0, 40.0, n_frames)]\n",
        "magnitudes = tf.stack(magnitudes)\n",
        "magnitudes = magnitudes[tf.newaxis, :, :]\n",
        "\n",
        "# Filter.\n",
        "audio_out = ddsp.core.frequency_filter(audio_in, magnitudes)\n",
        "\n",
        "# Listen.\n",
        "print('Time-varying Filter, Low temporal resolution')\n",
        "play(audio_out)\n",
        "specplot(audio_out)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "fxWs2f_ETNmy"
      },
      "source": [
        "# Resampling"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ySKvgrNTTREO"
      },
      "source": [
        "Many functions require controls to be provided at the audio sample rate, but often one will want the network to output controls at a coarser rate."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "mi438VbbTizE"
      },
      "source": [
        "## `resample()`\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "sNn0o9ahUBin"
      },
      "source": [
        "Simple bilinear upsampling of control signal based on `tf.image.resize()`.\n",
        "\n",
        "With `add_endpoint=False`, uses the last timestep as the endpoint, producing `n_frames - 1` segments, each with a length of `n_timesteps / (n_frames - 1)`.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "3VW3shCdT7SY"
      },
      "outputs": [],
      "source": [
        "n_coarse = 9\n",
        "n_fine = 16000\n",
        "\n",
        "coarse = 1.0 - np.sin(np.linspace(0, np.pi, n_coarse))[np.newaxis, :, np.newaxis]\n",
        "fine = ddsp.core.resample(coarse, n_fine, add_endpoint=False)\n",
        "\n",
        "plt.plot(np.linspace(0, n_fine, n_coarse), coarse[0, :, 0], 'o', label='coarse')\n",
        "plt.plot(np.linspace(0, n_fine, n_fine), fine[0, :, 0], label='fine')\n",
        "plt.title('Bilinear upsampling ({} points, {} intervals)'.format(n_coarse, n_coarse - 1))\n",
        "plt.legend(loc='lower right')\n",
        "_ = plt.ylim(-0.1, 1.1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "-OwAfm5LImt_"
      },
      "source": [
        "With `add_endpoint=True`, holds the last timestep for an additional step as the endpoint. \n",
        "Then, `n_timesteps` is divided evenly into `n_frames` segments of size `n_timesteps / n_frames`. This is the default behavior, as it matches the default behavior of `fft_convolve`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "XvVjDue3XNnr"
      },
      "outputs": [],
      "source": [
        "fine = ddsp.core.resample(coarse, n_fine)\n",
        "\n",
        "n_adjusted = int(n_fine / n_coarse * (n_coarse - 1))\n",
        "plt.plot(np.linspace(0, n_adjusted, n_coarse), coarse[0, :, 0], 'o', label='coarse')\n",
        "plt.plot(np.linspace(0, n_fine, n_fine), fine[0, :, 0], label='fine')\n",
        "plt.title('Bilinear upsampling ({} points, {} intervals)'.format(n_coarse, n_coarse))\n",
        "plt.legend(loc='lower right')\n",
        "_ = plt.ylim(-0.1, 1.1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "joKFtUJpKMo1"
      },
      "source": [
        "You can also do cubic interpolation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "WVKi0KmKKPas"
      },
      "outputs": [],
      "source": [
        "fine = ddsp.core.resample(coarse, n_fine, method='cubic', add_endpoint=False)\n",
        "\n",
        "plt.plot(np.linspace(0, n_fine, n_coarse), coarse[0, :, 0], 'o', label='coarse')\n",
        "plt.plot(np.linspace(0, n_fine, n_fine), fine[0, :, 0], label='fine')\n",
        "plt.title('Bicubic upsampling ({} points, {} intervals)'.format(n_coarse, n_coarse - 1))\n",
        "plt.legend(loc='lower right')\n",
        "_ = plt.ylim(-0.1, 1.1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "clHwTkJfY8Xn"
      },
      "source": [
        "Resampling also works for downsampling"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "QVEy1lTNY_4E"
      },
      "outputs": [],
      "source": [
        "n_coarse = 9\n",
        "n_fine = 16000\n",
        "\n",
        "fine = 1.0 - np.sin(np.linspace(0, np.pi, n_fine))[np.newaxis, :, np.newaxis]\n",
        "coarse = ddsp.core.resample(fine, n_coarse, add_endpoint=False)\n",
        "\n",
        "plt.plot(np.linspace(0, n_coarse, n_fine), fine[0, :, 0],  label='fine')\n",
        "plt.plot(np.linspace(0, n_coarse, n_coarse), coarse[0, :, 0], 'o', label='coarse')\n",
        "plt.title('Bilinear downsampling ({} points, {} intervals)'.format(n_coarse, n_coarse - 1))\n",
        "plt.legend(loc='lower right')\n",
        "plt.xlim(-0.5, 10.5)\n",
        "_ = plt.ylim(-0.1, 1.1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "lsnURhiHKt2D"
      },
      "source": [
        "For downsampling add endpoint interpolates up to an added endpoint, which actually removes an endpoint from the downsampled signal. This still results in the same number of points and segments."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "R3A7MeciayQv"
      },
      "outputs": [],
      "source": [
        "n_intervals = (n_fine - 1)\n",
        "n_forward = int(n_coarse / n_fine * n_intervals)\n",
        "\n",
        "fine = 1.0 - np.sin(np.linspace(0, np.pi, n_fine))[np.newaxis, :, np.newaxis]\n",
        "coarse = ddsp.core.resample(fine, n_coarse)\n",
        "\n",
        "plt.plot(np.linspace(0, n_coarse, n_fine), fine[0, :, 0], label='fine')\n",
        "plt.plot(np.linspace(0, n_coarse - 1, n_coarse), coarse[0, :, 0], 'o', label='coarse')\n",
        "plt.title('Bilinear downsampling ({} points, {} intervals)'.format(n_coarse, n_coarse))\n",
        "plt.legend(loc='lower right')\n",
        "plt.xlim(-0.5, 10.5)\n",
        "_ = plt.ylim(-0.1, 1.1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "e4FAB5cRTntw"
      },
      "source": [
        "## `upsample_with_windows()`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "AwJvbCdLW1hA"
      },
      "source": [
        "Upsample signal with overlapping hann windows (like an inverse STFT). Good for smooth amplitude envelopes. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "xgHOaCQeW1hB"
      },
      "outputs": [],
      "source": [
        "n_coarse = 5\n",
        "n_fine = 16000\n",
        "\n",
        "coarse = 1.0 - np.sin(np.linspace(0, np.pi, n_coarse))[np.newaxis, :, np.newaxis]\n",
        "fine = ddsp.core.upsample_with_windows(coarse, n_fine, add_endpoint=False)\n",
        "\n",
        "plt.plot(np.linspace(0, n_fine, n_coarse), coarse[0, :, 0], 'o', label='coarse')\n",
        "plt.plot(np.linspace(0, n_fine, n_fine), fine[0, :, 0], label='fine')\n",
        "plt.title('Upsample with windows ({} points, {} intervals)'.format(n_coarse, n_coarse - 1))\n",
        "plt.legend(loc='lower right')\n",
        "_ = plt.ylim(-0.1, 1.1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "nBlswhEZMPoh"
      },
      "source": [
        "`add_endpoint` has the same behavior and defaults to True, as it matches the behavior of `fft_convolve`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "55gjj454gTQi"
      },
      "outputs": [],
      "source": [
        "coarse = 1.0 - np.sin(np.linspace(0, np.pi, n_coarse))[np.newaxis, :, np.newaxis]\n",
        "fine = ddsp.core.upsample_with_windows(coarse, n_fine)\n",
        "\n",
        "n_intervals = (n_coarse - 1)\n",
        "n_forward = int(n_fine / n_coarse * n_intervals)\n",
        "plt.plot(np.linspace(0, n_forward, n_coarse), coarse[0, :, 0], 'o', label='coarse')\n",
        "plt.plot(np.linspace(0, n_fine, n_fine), fine[0, :, 0], label='fine')\n",
        "plt.title('Upsample with windows ({} points, {} intervals)'.format(n_coarse, n_coarse))\n",
        "plt.legend(loc='lower right')\n",
        "_ = plt.ylim(-0.1, 1.1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "7MZdFKvaMYFM"
      },
      "source": [
        "You can also call `upsample_with_windows()` by calling `resample(method='window')`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "I1cHRCpsMjBj"
      },
      "outputs": [],
      "source": [
        "fine = ddsp.core.resample(coarse, n_fine, method='window')\n",
        "\n",
        "plt.plot(np.linspace(0, n_forward, n_coarse), coarse[0, :, 0], 'o', label='coarse')\n",
        "plt.plot(np.linspace(0, n_fine, n_fine), fine[0, :, 0], label='fine')\n",
        "plt.title('Upsample with windows ({} points, {} intervals)'.format(n_coarse, n_coarse))\n",
        "plt.legend(loc='lower right')\n",
        "_ = plt.ylim(-0.1, 1.1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "RDDgrs6QMzLl"
      },
      "source": [
        "The Hann window transitions are smooth like bicubic, but more gradual and don't overshoot."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "wmruPmaEM7wO"
      },
      "outputs": [],
      "source": [
        "fine = ddsp.core.resample(coarse, n_fine, method='cubic')\n",
        "\n",
        "plt.plot(np.linspace(0, n_forward, n_coarse), coarse[0, :, 0], 'o', label='coarse')\n",
        "plt.plot(np.linspace(0, n_fine, n_fine), fine[0, :, 0], label='fine')\n",
        "plt.title('Bicubic upsampling ({} points, {} intervals)'.format(n_coarse, n_coarse))\n",
        "plt.legend(loc='lower right')\n",
        "_ = plt.ylim(-0.1, 1.1)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "hMqWDc_m6rUC",
        "iZ6aqiNiYGHW",
        "AiCIG1x5bxkh",
        "LL5mf1ECkO4V",
        "4x1YxNHkNlr4",
        "XpoIxVsfvxyy",
        "kQgyvYMkhKHS",
        "q9ngxn8pD0K6",
        "aXg73zyno-4U",
        "7A3AasLQuIAZ",
        "n5L0bdsntwEE",
        "i2-KD36rkmfw",
        "X26Camb_TMGL",
        "u1UEha2zTNc5",
        "Sh0Yd1u_Xq9P",
        "nYhd919a8vid",
        "fmgwdYujl-cR",
        "SaFiXN_dk45P",
        "VHn_Xvzm9rLW",
        "S7CmzEUJOeoD",
        "9gs6AHwjU-kf",
        "fkqK-kpscG-K",
        "qmldSx4Nkwaj",
        "4YVbRBdrKd4z",
        "sZaffuekKd41",
        "fxWs2f_ETNmy",
        "mi438VbbTizE",
        "e4FAB5cRTntw"
      ],
      "last_runtime": {},
      "name": "4_core_functions.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}