Copy_of_Predict_Shakespeare_with_Cloud_TPUs_and_Keras.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy of Predict Shakespeare with Cloud TPUs and Keras",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [
        "N6ZDpd9XzFeN"
      ],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "accelerator": "TPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/SinKasula/BookInfoDesktopApp/blob/master/Copy_of_Predict_Shakespeare_with_Cloud_TPUs_and_Keras.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "N6ZDpd9XzFeN"
      },
      "cell_type": "markdown",
      "source": [
        "##### Copyright 2018 The TensorFlow Hub Authors.\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");"
      ]
    },
    {
      "metadata": {
        "cellView": "form",
        "colab_type": "code",
        "id": "KUu4vOt5zI9d",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Copyright 2018 The TensorFlow Hub Authors. All Rights Reserved.\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License.\n",
        "# =============================================================================="
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "innBbve1LdjE",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "edfbxDDh2AEs"
      },
      "cell_type": "markdown",
      "source": [
        "## Predict Shakespeare with Cloud TPUs and Keras"
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "RNo1Vfghpa8j"
      },
      "cell_type": "markdown",
      "source": [
        "## Overview\n",
        "\n",
        "This example uses [tf.keras](https://www.tensorflow.org/guide/keras) to build a *language model* and train it on a Cloud TPU. This language model predicts the next character of text given the text so far. The trained model can generate new snippets of text that read in a similar style to the text training data.\n",
        "\n",
        "The model trains for 10 epochs and completes in approximately 5 minutes.\n",
        "\n",
        "This notebook is hosted on GitHub. To view it in its original repository, after opening the notebook, select **File > View on GitHub**."
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "dgAHfQtuhddd"
      },
      "cell_type": "markdown",
      "source": [
        "## Learning objectives\n",
        "\n",
        "In this Colab, you will learn how to:\n",
        "*   Build a two-layer, forward-LSTM model.\n",
        "*   Convert a `tf.keras` model to an equivalent TPU version and then use the standard Keras methods to train: `fit`, `predict`, and `evaluate`.\n",
        "*   Use the trained model to make predictions and generate your own Shakespeare-esque play.\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "QrprJD-R-410"
      },
      "cell_type": "markdown",
      "source": [
        "## Instructions"
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "_I0RdnOSkNmi"
      },
      "cell_type": "markdown",
      "source": [
        "<h3>  &nbsp;&nbsp;Train on TPU&nbsp;&nbsp; <a href=\"https://cloud.google.com/tpu/\"><img valign=\"middle\" src=\"https://raw.githubusercontent.com/GoogleCloudPlatform/tensorflow-without-a-phd/master/tensorflow-rl-pong/images/tpu-hexagon.png\" width=\"50\"></a></h3>\n",
        "\n",
        "   1. On the main menu, click Runtime and select **Change runtime type**. Set \"TPU\" as the hardware accelerator.\n",
        "   1. Click Runtime again and select **Runtime > Run All**. You can also run the cells manually with Shift-ENTER. "
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "kYxeFuKCUx9d"
      },
      "cell_type": "markdown",
      "source": [
        "TPUs are located in Google Cloud, for optimal performance, they read data directly from Google Cloud Storage (GCS)"
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "Lvo0t7XVIkWZ"
      },
      "cell_type": "markdown",
      "source": [
        "## Data, model, and training"
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "xzpUtDMqmA-x"
      },
      "cell_type": "markdown",
      "source": [
        "In this example, you train the model on the combined works of William Shakespeare, then use the model to compose a play in the style of *The Great Bard*:\n",
        "\n",
        "<blockquote>\n",
        "Loves that led me no dumbs lack her Berjoy's face with her to-day.  \n",
        "The spirits roar'd; which shames which within his powers  \n",
        "\tWhich tied up remedies lending with occasion,  \n",
        "A loud and Lancaster, stabb'd in me  \n",
        "\tUpon my sword for ever: 'Agripo'er, his days let me free.  \n",
        "\tStop it of that word, be so: at Lear,  \n",
        "\tWhen I did profess the hour-stranger for my life,  \n",
        "\tWhen I did sink to be cried how for aught;  \n",
        "\tSome beds which seeks chaste senses prove burning;  \n",
        "But he perforces seen in her eyes so fast;  \n",
        "And _  \n",
        "</blockquote>\n"
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "KRQ6Fjra3Ruq"
      },
      "cell_type": "markdown",
      "source": [
        "### Download data\n",
        "\n",
        "Download *The Complete Works of William Shakespeare* as a single text file from [Project Gutenberg](https://www.gutenberg.org/). You use snippets from this file as the *training data* for the model. The *target* snippet is offset by one character."
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "j8sIXh1DEDDd",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!wget --show-progress --continue -O /content/shakespeare.txt http://www.gutenberg.org/files/100/100-0.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "AbL6cqCl7hnt"
      },
      "cell_type": "markdown",
      "source": [
        "### Build the data generator"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "E3V4V-Jxmuv3",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import six\n",
        "import tensorflow as tf\n",
        "import time\n",
        "import os\n",
        "\n",
        "# This address identifies the TPU we'll use when configuring TensorFlow.\n",
        "TPU_WORKER = 'grpc://' + os.environ['COLAB_TPU_ADDR']\n",
        "\n",
        "SHAKESPEARE_TXT = '/content/shakespeare.txt'\n",
        "\n",
        "tf.logging.set_verbosity(tf.logging.INFO)\n",
        "\n",
        "def transform(txt, pad_to=None):\n",
        "  # drop any non-ascii characters\n",
        "  output = np.asarray([ord(c) for c in txt if ord(c) < 255], dtype=np.int32)\n",
        "  if pad_to is not None:\n",
        "    output = output[:pad_to]\n",
        "    output = np.concatenate([\n",
        "        np.zeros([pad_to - len(txt)], dtype=np.int32),\n",
        "        output,\n",
        "    ])\n",
        "  return output\n",
        "\n",
        "def training_generator(seq_len=100, batch_size=1024):\n",
        "  \"\"\"A generator yields (source, target) arrays for training.\"\"\"\n",
        "  with tf.gfile.GFile(SHAKESPEARE_TXT, 'r') as f:\n",
        "    txt = f.read()\n",
        "\n",
        "  tf.logging.info('Input text [%d] %s', len(txt), txt[:50])\n",
        "  source = transform(txt)\n",
        "  while True:\n",
        "    offsets = np.random.randint(0, len(source) - seq_len, batch_size)\n",
        "\n",
        "    # Our model uses sparse crossentropy loss, but Keras requires labels\n",
        "    # to have the same rank as the input logits.  We add an empty final\n",
        "    # dimension to account for this.\n",
        "    yield (\n",
        "        np.stack([source[idx:idx + seq_len] for idx in offsets]),\n",
        "        np.expand_dims(\n",
        "            np.stack([source[idx + 1:idx + seq_len + 1] for idx in offsets]),\n",
        "            -1),\n",
        "    )\n",
        "\n",
        "six.next(training_generator(seq_len=10, batch_size=1))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "Bbb05dNynDrQ"
      },
      "cell_type": "markdown",
      "source": [
        "### Build the model\n",
        "\n",
        "The model is defined as a two-layer, forward-LSTM—with two changes from the `tf.keras` standard LSTM definition:\n",
        "\n",
        "1. Define the input `shape` of the model to comply with the [XLA compiler](https://www.tensorflow.org/performance/xla/)'s static shape requirement.\n",
        "2. Use `tf.train.Optimizer` instead of a standard Keras optimizer (Keras optimizer support is still experimental)."
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "yLEM-fLJlEEt",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "EMBEDDING_DIM = 512\n",
        "\n",
        "def lstm_model(seq_len=100, batch_size=None, stateful=True):\n",
        "  \"\"\"Language model: predict the next word given the current word.\"\"\"\n",
        "  source = tf.keras.Input(\n",
        "      name='seed', shape=(seq_len,), batch_size=batch_size, dtype=tf.int32)\n",
        "\n",
        "  embedding = tf.keras.layers.Embedding(input_dim=256, output_dim=EMBEDDING_DIM)(source)\n",
        "  lstm_1 = tf.keras.layers.LSTM(EMBEDDING_DIM, stateful=stateful, return_sequences=True)(embedding)\n",
        "  lstm_2 = tf.keras.layers.LSTM(EMBEDDING_DIM, stateful=stateful, return_sequences=True)(lstm_1)\n",
        "  predicted_char = tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(256, activation='softmax'))(lstm_2)\n",
        "  model = tf.keras.Model(inputs=[source], outputs=[predicted_char])\n",
        "  model.compile(\n",
        "      optimizer=tf.train.RMSPropOptimizer(learning_rate=0.01),\n",
        "      loss='sparse_categorical_crossentropy',\n",
        "      metrics=['sparse_categorical_accuracy'])\n",
        "  return model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "VzBYDJI0_Tfm"
      },
      "cell_type": "markdown",
      "source": [
        "### Train the model\n",
        "\n",
        "The `tf.contrib.tpu.keras_to_tpu_model` function converts a `tf.keras` model to an equivalent TPU version. You then use the standard Keras methods to train: `fit`, `predict`, and `evaluate`."
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "ExQ922tfzSGA",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "tf.keras.backend.clear_session()\n",
        "\n",
        "training_model = lstm_model(seq_len=100, batch_size=128, stateful=False)\n",
        "\n",
        "tpu_model = tf.contrib.tpu.keras_to_tpu_model(\n",
        "    training_model,\n",
        "    strategy=tf.contrib.tpu.TPUDistributionStrategy(\n",
        "        tf.contrib.cluster_resolver.TPUClusterResolver(TPU_WORKER)))\n",
        "\n",
        "tpu_model.fit_generator(\n",
        "    training_generator(seq_len=100, batch_size=1024),\n",
        "    steps_per_epoch=100,\n",
        "    epochs=10,\n",
        ")\n",
        "tpu_model.save_weights('/tmp/bard.h5', overwrite=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "TCBtcpZkykSf"
      },
      "cell_type": "markdown",
      "source": [
        "### Make predictions with the model\n",
        "\n",
        "Use the trained model to make predictions and generate your own Shakespeare-esque play.\n",
        "Start the model off with a *seed* sentence, then generate 250 characters from it. The model makes five predictions from the initial seed."
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "tU7M-EGGxR3E",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "BATCH_SIZE = 5\n",
        "PREDICT_LEN = 250\n",
        "\n",
        "# Keras requires the batch size be specified ahead of time for stateful models.\n",
        "# We use a sequence length of 1, as we will be feeding in one character at a \n",
        "# time and predicting the next character.\n",
        "prediction_model = lstm_model(seq_len=1, batch_size=BATCH_SIZE, stateful=True)\n",
        "prediction_model.load_weights('/tmp/bard.h5')\n",
        "\n",
        "# We seed the model with our initial string, copied BATCH_SIZE times\n",
        "\n",
        "seed_txt = 'Looks it not like the king?  Verily, we must go! '\n",
        "seed = transform(seed_txt)\n",
        "seed = np.repeat(np.expand_dims(seed, 0), BATCH_SIZE, axis=0)\n",
        "\n",
        "# First, run the seed forward to prime the state of the model.\n",
        "prediction_model.reset_states()\n",
        "for i in range(len(seed_txt) - 1):\n",
        "  prediction_model.predict(seed[:, i:i + 1])\n",
        "\n",
        "# Now we can accumulate predictions!\n",
        "predictions = [seed[:, -1:]]\n",
        "for i in range(PREDICT_LEN):\n",
        "  last_word = predictions[-1]\n",
        "  next_probits = prediction_model.predict(last_word)[:, 0, :]\n",
        "  \n",
        "  # sample from our output distribution\n",
        "  next_idx = [\n",
        "      np.random.choice(256, p=next_probits[i])\n",
        "      for i in range(BATCH_SIZE)\n",
        "  ]\n",
        "  predictions.append(np.asarray(next_idx, dtype=np.int32))\n",
        "  \n",
        "\n",
        "for i in range(BATCH_SIZE):\n",
        "  print('PREDICTION %d\\n\\n' % i)\n",
        "  p = [predictions[j][i] for j in range(PREDICT_LEN)]\n",
        "  generated = ''.join([chr(c) for c in p])\n",
        "  print(generated)\n",
        "  print()\n",
        "  assert len(generated) == PREDICT_LEN, 'Generated text too short'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "2a5cGsSTEBQD"
      },
      "cell_type": "markdown",
      "source": [
        "## What's next\n",
        "\n",
        "* Learn about [Cloud TPUs](https://cloud.google.com/tpu/docs) that Google designed and optimized specifically to speed up and scale up ML workloads for training and inference and to enable ML engineers and researchers to iterate more quickly.\n",
        "* Explore the range of [Cloud TPU tutorials and Colabs](https://cloud.google.com/tpu/docs/tutorials) to find other examples that can be used when implementing your ML project.\n",
        "\n",
        "On Google Cloud Platform, in addition to GPUs and TPUs available on pre-configured [deep learning VMs](https://cloud.google.com/deep-learning-vm/),  you will find [AutoML](https://cloud.google.com/automl/)*(beta)* for training custom models without writing code and [Cloud ML Engine](https://cloud.google.com/ml-engine/docs/) which will allows you to run parallel trainings and hyperparameter tuning of your custom models on powerful distributed hardware.\n"
      ]
    }
  ]
}