tf_quant_finance/examples/jupyter_notebooks/Introduction_to_TensorFlow_Part_1_-_Basics.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EheA5_j_cEwc"
      },
      "source": [
        "##### Copyright 2019 Google LLC.\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YCriMWd-pRTP"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\"); { display-mode: \"form\" }\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OvRwFTkqcp1e"
      },
      "source": [
        "# Introduction to TensorFlow Part 1 - Basics\n",
        "\n",
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/google/tf-quant-finance/blob/master/tf_quant_finance/examples/jupyter_notebooks/Introduction_to_TensorFlow_Part_1_-_Basics.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/google/tf-quant-finance/blob/master/tf_quant_finance/examples/jupyter_notebooks/Introduction_to_TensorFlow_Part_1_-_Basics.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView source on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "uG8UAZXjeUhf"
      },
      "outputs": [],
      "source": [
        "#@title Upgrade to TensorFlow 2.5+\n",
        "!pip install --upgrade tensorflow"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nPX9m7Q_w_8p"
      },
      "outputs": [],
      "source": [
        "#@title Install Libraries for this colab\n",
        "!pip install matplotlib\n",
        "import tensorflow as tf\n",
        "# Load utility to save the graph in a tensorboard-friendly way\n",
        "from tensorflow.python.summary.writer.writer import FileWriter\n",
        "from PIL import Image\n",
        "%load_ext tensorboard\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "obZHjPQScSqd"
      },
      "source": [
        "# What this notebook covers\n",
        "\n",
        "The aim of this mini course is to get you up and running with TensorFlow.\n",
        "It's not just a walkthrough. To get the most out of this, you should follow along by running the examples below. We cover just the basics of TensorFlow.\n",
        "  - Underlying concepts of Tensors, shapes, Operations, graphs etc.\n",
        "  - The functionality available in the core TensorFlow API. \n",
        "  - Not really concerned with Machine Learning. While TensorFlow is widely associated with ML, it can (and was originally designed to) be used to accelerate most traditional numerical calculations."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5lgZQ_LDdaaL"
      },
      "source": [
        "# Tips And References\n",
        "\n",
        "- The TensorFlow API reference is indispensable while you learn the ropes. [Tensorflow Python API](https://www.tensorflow.org/api_docs/python/tf)\n",
        "- You will need to refer to it through this course.\n",
        "- This notebook uses Google Colab - allowing you to run code interactively. See [the introduction](https://colab.research.google.com/notebooks/welcome.ipynb) for more information.\n",
        "- In Colab you can use the autocomplete feature to see the available functions and\n",
        "  the usage.\n",
        "  - **To see all the names available** under some module (e.g. tf.constant under tf), type the name of the module followed by a period and hit **Ctrl+Space**.\n",
        "  - **To see the usage instructions** for a function, type the function name followed by an open bracket\n",
        "    and hit **Tab**.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "cTfswCaHTzyt"
      },
      "outputs": [],
      "source": [
        "#@title\n",
        "%%html\n",
        "\u003cdiv style=\"text-align:center; font-size: 100px; opacity: 0.9; color:orange\"\u003eLet's Begin\u003c/div\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OEaJQIEKUjlo"
      },
      "source": [
        "# What is TensorFlow\n",
        "\n",
        "TensorFlow is a system designed for efficient numerical computations.\n",
        "- The core of TensorFlow is written in C++.\n",
        "- The client API is most commonly developed in Python and other languages are supported.\n",
        "- A TensorFlow program describes the computation as a sequence of operations.\n",
        "- The operations (also called Ops) act on zero or more inputs. The inputs are called Tensors.\n",
        "- The outputs of ops are also Tensors.\n",
        "- You can think of tensors as the data and the ops as the functions that act on that data.\n",
        "- Output of one op can be sent as input to another op. \n",
        "- A TensorFlow program is a directed **graph** whose edges are tensors and the nodes are Ops.\n",
        "- The tensors 'flow' along the edges, hence, TensorFlow.\n",
        "\n",
        "Let us consider an example.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Cvldwun6UnQv"
      },
      "outputs": [],
      "source": [
        "# Define a function that returns sin(a + b). We add names for each of the\n",
        "# tensor and operation\n",
        "def compute_sin(a, b):\n",
        "  \"\"\"Computes sine of a sum of two numbers.\"\"\"\n",
        "  # a, b, c and d below are tensors.\n",
        "  a = tf.convert_to_tensor(a, name = \"a\")\n",
        "  b = tf.convert_to_tensor(b, name = \"b\")\n",
        "\n",
        "  # This creates an output tensor of the 'Op' tf.add.\n",
        "  # The tensor's name is 'addition', which is what you'll see in the graph\n",
        "  # below. And we store the tensor in a python variable called 'c'\n",
        "  c = tf.add(a, b, name=\"addition\")\n",
        "\n",
        "  # The Op 'tf.sin' consumes the input tensor 'c' and returns another tensor 'd'\n",
        "  d = tf.sin(c, name=\"sin\")\n",
        "  return d\n",
        "\n",
        "# Returns sin(1.0 + 2.0)\n",
        "compute_sin(1.0, 2.0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ha3C_k6sEXmm"
      },
      "outputs": [],
      "source": [
        "# Computation graph visualization\n",
        "g = tf.Graph()  # This is the graph to be saved\n",
        "with g.as_default():\n",
        "  with tf.name_scope('main_graph'):\n",
        "    c = compute_sin(1.0, 2.0)  # Add calculations to the graph \n",
        "  writer = FileWriter(\"logs\", g)\n",
        "  writer.close()\n",
        "\n",
        "# Invoke tensorboard to visualize the graph\n",
        "%tensorboard --logdir=logs\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4lgGTx_ZhiEv"
      },
      "source": [
        "# Execution Modes"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JunoqOWDhrQO"
      },
      "source": [
        "TensorFlow has two modes of execution: deferred and (since v2) eager."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4QaWnNepmYXE"
      },
      "source": [
        "## Deferred Execution"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oyNGMXWvmbUA"
      },
      "source": [
        "In order to access the deferred execution, eager execution has to be \n",
        "disabled\n",
        "```python\n",
        "tf.compat.v1.disable_eager_execution()\n",
        "```\n",
        "\n",
        "With deferred mode, you build up a graph of operations, so this tensorflow code\n",
        "```python\n",
        "a = tf.constant(1.0)\n",
        "b = tf.constant(2.0)\n",
        "c = tf.add(a, b)\n",
        "```\n",
        "is roughly equivalent to this python code\n",
        "```python\n",
        "a = 1\n",
        "b = 2\n",
        "def c():\n",
        "  return a + b\n",
        "```\n",
        "In the python code, `c` does not hold the value 3: it holds a function. In order to get the value 3, you need to execute the function:\n",
        "```python\n",
        "c() # returns the value 3\n",
        "```\n",
        "\n",
        "In the same way with TensorFlow in deferred (graph) mode, `c` doesn't hold a tensor with the value 3, it holds an execution graph that can be executed inside a *Session* to obtain the value 3, as per below"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "J3fwFJ8Fk71I"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "g = tf.Graph()\n",
        "\n",
        "with g.as_default():\n",
        "  a = tf.constant([1.0], name = \"a\")\n",
        "  b = tf.constant([2.0], name = \"b\")\n",
        "  c = tf.add(a, b, name=\"addition\")\n",
        "\n",
        "print(c)\n",
        "with tf.compat.v1.Session(graph=g) as sess:\n",
        "  result = sess.run(c)\n",
        "print(result)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-OdRJWrXlRvQ"
      },
      "source": [
        "\n",
        "\n",
        "While the graph is being built, TensorFlow does little more than check that the operations have been passed the right numbers of arguments and that they're of the right type and shape. Once the graph has been built, it can be \n",
        "*   serialized, stored, repeatedly executed\n",
        "*   analyzed and optimised: removing duplicate operations, hoisting constants up levels etc.\n",
        "*   split across multiple CPUs, GPUs and TPUs \n",
        "*   etc.\n",
        "This allows for a lot of flexibility and performance for long calculations. However when manually experimenting this mode does get in the way of the REPL behaviour that python developers are used to. Thus in TensorFlow v2, eager execution was added.\n",
        "\n",
        "In order to bridge the performance gap incurred by optimization done by graph\n",
        "mode, one can wrap calculations into `tf.function` which makes the computations\n",
        "executed in a single graph scope.\n",
        "\n",
        "`tf.compat.v1.Session` and `tf.function` are discussed in more detail [later on](#scrollTo=KGYeF4K1JLIJ)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CeUrh0P1nSUD"
      },
      "source": [
        "## Eager Execution"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5FQFnqS1oHyl"
      },
      "source": [
        "Eager execution works in a more intuitive fashion. Operations are executed immediately, rather than being stored as deferred instructions. With eager mode enabled, the following\n",
        "```python\n",
        "import tensorflow as tf\n",
        "\n",
        "a = tf.constant([1.0], name = \"a\")\n",
        "b = tf.constant([2.0], name = \"b\")\n",
        "c = tf.add(a, b, name=\"addition\")\n",
        "\n",
        "print (c)\n",
        "```\n",
        "would produce the output\n",
        "```python\n",
        "tf.Tensor([3.], shape=(1,), dtype=float32)\n",
        "```\n",
        "\n",
        "Eager execution is enabled by defualt. Note that the choice of execution mode is an irrevocable, global one. It must be done before any TensorFlow method is called, once your process is using TensorFlow in one mode, it cannot switch to the other without restarting the kernel.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4VEpFc4IOhok"
      },
      "source": [
        "# Tensors and Shapes\n",
        "\n",
        "- A tensor is just an $n$-dimensional matrix.\n",
        "- The *rank* of a tensor is the number of dimensions it has (alternatively, the number of indices you have to specify to get at an element).\n",
        "- A vector is a rank $\\color{blue} 1$ tensor.\n",
        "- A matrix is a rank $\\color{blue} 2$ tensor.\n",
        "- Tensor is characterized by its shape and the data type of its elements. \n",
        "- The shape is a specification of the number of dimensions and the length of the tensor in each of those dimensions.\n",
        "- Shape is described by an integer vector giving the lengths in each dimension.\n",
        "- For example, $\\left[\\begin{array}{cc} 1 \u0026 0 \\\\ 0 \u0026 1 \\\\ 1 \u0026 1 \\end{array}\\right]$ is tensor of shape [3, 2].\n",
        "- On the other hand, $\\left[\\begin{array}{cc} [1] \u0026 [0] \\\\ [0] \u0026 [1] \\\\ [1] \u0026 [1] \\end{array}\\right]$ is a tensor of shape [3, 2, 1].\n",
        "- The shape is read starting from the \"outside\" and moving in until you reach  \n",
        "  an elementary object (e.g. number or string).\n",
        "- Note that Tensors are not just arbitrary arrays. For example, $[1, [2]]$ is \n",
        "  not a Tensor and has no unambiguous shape.\n",
        "- TensorFlow shapes are almost the same as numpy shapes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "code",
        "id": "BQSOWFLdX_KI"
      },
      "outputs": [],
      "source": [
        "#@title Fun with shapes\n",
        "\n",
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "\n",
        "# This is equivalent to a 0-rank tensor (i.e. a scalar).\n",
        "x = np.array(2.0)\n",
        "t = tf.constant(x)\n",
        "print (\"Shape of %s = %s\\n\" % (x, t.shape))\n",
        "\n",
        "# A rank 1 tensor. Shape = [5]\n",
        "x = np.array([1, 2, 3, 4, 5])\n",
        "t = tf.constant(x)\n",
        "print (\"Shape of %s: %s\\n\" % (x, t.shape))\n",
        "\n",
        "# A rank 2 tensor. Shape = [5, 1]\n",
        "x = np.array([[1], [2], [3], [4], [5]])\n",
        "t = tf.constant(x)\n",
        "print (\"Shape of %s: %s\\n\" % (x, t.shape))\n",
        "\n",
        "# A rank 2 tensor. Shape = [1, 5]\n",
        "x = np.array([[1, 2, 3, 4, 5]])\n",
        "t = tf.constant(x)\n",
        "print (\"Shape of %s: %s\\n\" % (x, t.shape))\n",
        "\n",
        "# A rank 3 tensor. Shape = [2, 1, 2]\n",
        "x = np.array(\n",
        "    [\n",
        "        [ [0, 0] ],\n",
        "        [ [0, 0] ]\n",
        "    ])\n",
        "\n",
        "t = tf.constant(x)\n",
        "print (\"Shape of %s: %s\\n\" % (x, t.shape))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "code",
        "id": "ins_jc86dCUa"
      },
      "outputs": [],
      "source": [
        "#@title Shape Quiz\n",
        "\n",
        "# to-do: Fill in an array of shape [1, 2, 1, 2] in the variable x. \n",
        "# The values you choose don't matter but the shape does.\n",
        "\n",
        "x = np.array([])\n",
        "t = tf.constant(x, name = \"t\")\n",
        "\n",
        "\n",
        "if t.shape == [1, 2, 1, 2]:\n",
        "  print (\"Success!\")\n",
        "else:\n",
        "  print (\"Shape was %s. Try again\"%t.shape)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "JAxpR7UUw7WP"
      },
      "outputs": [],
      "source": [
        "#@title Solution: Shape Quiz. Double click to reveal\n",
        "\n",
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "\n",
        "# The values you choose don't matter but the shape does.\n",
        "\n",
        "x = np.array([ [[[0, 0]], [[0, 0]]] ] )\n",
        "t = tf.constant(x)\n",
        "\n",
        "if t.shape == [1, 2, 1, 2]:\n",
        "  print (\"Success!\")\n",
        "else:\n",
        "  print (\"Shape was %s. Try again\"%t.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DKAeuSfXtQzy"
      },
      "source": [
        "## Shape And Reshape\n",
        "\n",
        "Most Tensorflow operations preserve the tensor shapes or modify it in obvious ways.  \n",
        "However you often need to rearrange the shape to fit the problem at hand. \n",
        "\n",
        "There are number of shape related ops in TensorFlow that you can make use of.  \n",
        "First we have these ops to give us information about the tensor shape\n",
        "\n",
        "| Name | Description |\n",
        "|---   | ---         |\n",
        "|[tf.shape](https://www.tensorflow.org/api_docs/python/tf/shape) | Returns the shape of the tensor |\n",
        "|[tf.size](https://www.tensorflow.org/api_docs/python/tf/size) | Returns the total number of elements in the tensor |\n",
        "|[tf.rank](https://www.tensorflow.org/api_docs/python/tf/rank) | Returns the tensor rank |\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "code",
        "id": "w2w0oN2cwOMw"
      },
      "outputs": [],
      "source": [
        "#@title Shape Information Ops\n",
        "\n",
        "import numpy as np\n",
        "\n",
        "# These examples are a little silly because we already know\n",
        "# the shapes. \n",
        "x = tf.constant(np.zeros([2, 2, 3, 12]))\n",
        "\n",
        "shape_x = tf.shape(x, name=\"my_shape\")\n",
        "print(\"Shape of x: %s\" % shape_x)\n",
        "\n",
        "rank_x = tf.rank(x)\n",
        "print(\"Rank of x: %s\" % rank_x)\n",
        "\n",
        "size_x = tf.size(x)\n",
        "print(\"Size of x: %s\" % size_x)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mIonnbVFwQUq"
      },
      "source": [
        "\n",
        "NB: The hawkeyed amongst us would have noticed that there seem to be two different  \n",
        "shape methods. In the examples on the previous slide we saw tensor.shape property  \n",
        "and above we saw tf.shape(tensor). There are subtle differences between the two  \n",
        "which we will discuss more when we talk about placeholders.  \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HqlH8CXJztlu"
      },
      "source": [
        "\n",
        "## Reshaping Continued\n",
        "\n",
        "Coming back to the ops that you can use to modify the shape, the following table\n",
        "lists some of them.\n",
        "\n",
        "\n",
        "|  Name | Description  |\n",
        "|---    |:---|\n",
        "|[tf.reshape](https://www.tensorflow.org/api_docs/python/tf/reshape)| Reshapes a tensor while preserving number of elements |\n",
        "|[tf.squeeze](https://www.tensorflow.org/api_docs/python/tf/squeeze)| \"Squeezes out\" dimensions of length 1|\n",
        "|[tf.expand\\_dims](https://www.tensorflow.org/api_docs/python/tf/expand_dims)| Inverse of squeeze. Expands the dimension by 1|\n",
        "|[tf.transpose](https://www.tensorflow.org/api_docs/python/tf/transpose)| Permutes the dimensions. For matrices, performs usual matrix transpose.|\n",
        "|[tf.meshgrid](https://www.tensorflow.org/api_docs/python/tf/meshgrid) | Effectively creates an N dimensional grid from N one dimensional arrays. |\n",
        "\n",
        "The following example demonstrates the use of the reshape op."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "code",
        "id": "ffFoO2v90hte"
      },
      "outputs": [],
      "source": [
        "#@title Reshaping Tensors\n",
        "\n",
        "import numpy as np\n",
        "\n",
        "# Create a constant tensor of shape [12]\n",
        "x = tf.constant(np.arange(1, 13))\n",
        "print(\"x = %s\\n\" % x)\n",
        "\n",
        "# Reshape this to [2, 6]. Note how the elements get laid out.\n",
        "x_2_6 = tf.reshape(x, [2, -1])\n",
        "print(\"x_2_6 = %s\\n\" % x_2_6)\n",
        "\n",
        "# Further rearrange x_2_6 to [3, 4]\n",
        "x_3_4 = tf.reshape(x_2_6, [3, 4])\n",
        "print(\"x_3_4 = %s\\n\" % x_3_4)\n",
        "\n",
        "# In fact you don't have to specify the full shape. You can leave\n",
        "# one component of the shape unspecified by setting it to -1.\n",
        "# This component will then be computed automatically by preserving the\n",
        "# total size.\n",
        "x_12_1 = tf.reshape(x_3_4, [-1, 1])\n",
        "print(\"x_12_1 = %s\\n\" % x_12_1)\n",
        "\n",
        "# What happens when are too many or too few elements?\n",
        "# You get an error!\n",
        "#x_wrong = tf.reshape(x_3_4, [4, 5])\n",
        "#print(\"x_wrong = %s\" % x_12_1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f8tSDpt71xDJ"
      },
      "source": [
        "The next set of examples show how to use the squeeze and expand_dims ops."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "code",
        "id": "TC18sa0q3Npp"
      },
      "outputs": [],
      "source": [
        "#@title Squeezing and Expanding Tensors\n",
        "\n",
        "import numpy as np\n",
        "\n",
        "# Create a tensor where the second and fourth dimension is of length 1.\n",
        "x = tf.constant(np.reshape(np.arange(1, 5), [2, 1, 2, 1]))\n",
        "print(\"Shape of x = %s\" % x.shape)\n",
        "\n",
        "# Now squeeze out all the dimensions of length 1\n",
        "x_squeezed = tf.squeeze(x)\n",
        "print(\"\\nShape of x_squeezed = %s\" % x_squeezed.shape)\n",
        "\n",
        "# You can control which dimension you squeeze\n",
        "x_squeeze_partial = tf.squeeze(x,3)\n",
        "print(\"\\nShape of x_squeeze_partial = %s\" % x_squeeze_partial.shape)\n",
        "\n",
        "# Expand_dims works in reverse to add dimensions of length one.\n",
        "# Think of this as just adding brackets [] somewhere in the tensor.\n",
        "\n",
        "y = tf.constant([[1, 2],[3, 4]])\n",
        "y_2 = tf.expand_dims(y, 2)\n",
        "y_3 = tf.expand_dims(y_2, 2)\n",
        "print(\"\\nShape of y = %s\" % y.shape)\n",
        "print(\"\\nShape of y_2 = %s\" % y_2.shape)\n",
        "print(\"\\nShape of y_3 = %s\" % y_3.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hXK-pENJuDlL"
      },
      "source": [
        "* The transpose op deserves a bit of explanation. \n",
        "* For matrices, it does the usual transpose operation.\n",
        "* For higher rank tensors, it allows you to permute the dimensions by specifying the permutation you want.\n",
        "\n",
        "Examples will (hopefully) make this clearer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "code",
        "id": "D1XjQPmz23dJ"
      },
      "outputs": [],
      "source": [
        "#@title Transposing tensors\n",
        "\n",
        "# Create a matrix\n",
        "x = tf.constant([[1, 2], [3, 4]])\n",
        "\n",
        "x_t = tf.transpose(x)\n",
        "\n",
        "print(\"X:\\n%s\\n\" % x)\n",
        "\n",
        "print(\"transpose(X):\\n%s\\n\" % x_t)\n",
        "\n",
        "# Now try this for a higher rank tensor. \n",
        "\n",
        "# Create a tensor of shape [3, 2, 1]\n",
        "y = tf.constant([[[1],[2]], [[3],[4]], [[5],[6]]])\n",
        "\n",
        "print(\"Shape of Y: %s\\n\" % y.shape)\n",
        "print(\"Y:\\n%s\\n\" % y)\n",
        "\n",
        "# Flip the first two dimensions\n",
        "y_t12 = tf.transpose(y, [1, 0, 2])\n",
        "\n",
        "print(\"Shape of Y with the first two dims flipped: %s\\n\" % y_t12.shape)\n",
        "print(\"transpose(Y, 1 \u003c-\u003e 2):\\n%s\\n\" % y_t12)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PwIi8A-6-HVG"
      },
      "source": [
        "## Quiz: Create a Grid\n",
        "\n",
        "We mentioned the tf.meshgrid op above but didn't use it. In this quiz you will  \n",
        "use it to do something we will find useful later on.\n",
        "\n",
        "Suppose we are given a set of x coordinates, say, [1, 2, 3] and another set of  \n",
        "y coordinates e.g. [1, 2, 3]. We want to create the \"grid\" formed from these\n",
        "coordinates as shown in the following diagram.\n",
        "\n",
        "tf.meshgrid allows you to do this but it will produce the X and Y coordinates of  \n",
        "the grid separately. Your task below is to create a tensor of complex numbers\n",
        "such that Z = X + j Y represents points on the grid (e.g. the lower left most point   \n",
        "will have Z = 1 + j while the top right one has Z = 3 + 3j.\n",
        "\n",
        "You should put your code in the function **create_grid** and run the cell when you are done.\n",
        "If it works, you will see a plot of the grid that you produced.\n",
        "\n",
        "Hints:\n",
        " * Experiment with tf.meshgrid to get X and Y of the right shape needed for the grid.\n",
        " * Join the separate X and Y using tf.complex(x, y)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "Yd0BBxEzTbS9"
      },
      "outputs": [],
      "source": [
        "#@title \n",
        "\n",
        "%%html\n",
        "\u003cdiv style=\"text-align:center; font-size: 40px; opacity: 0.9; color:blue\"\u003e\u003cp\u003eExample Grid\u003c/p\u003e\n",
        "\u003csvg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"400px\" height=\"300px\" viewBox=\"0 0 400 300\" preserveAspectRatio=\"xMidYMid meet\" \u003e\u003crect id=\"svgEditorBackground\" x=\"0\" y=\"0\" width=\"300\" height=\"300\" style=\"fill: none; stroke: none;\"/\u003e\n",
        "  \u003cline x1=\"106.26899719238736\" y1=\"50.779850959776496\" x2=\"107.26899719238736\" y2=\"236.77943420410023\" stroke=\"black\" style=\"stroke-width: 1px; fill: none;\"/\u003e\n",
        "  \u003cline x1=\"174.26887512207486\" y1=\"50.779850959776496\" x2=\"174.26887512207486\" y2=\"236.77943420410023\" stroke=\"black\" style=\"stroke-width: 1px; fill: none;\"/\u003e\n",
        "  \u003cline x1=\"240.26852416992642\" y1=\"50.779850959776496\" x2=\"240.26852416992642\" y2=\"236.77943420410023\" stroke=\"black\" style=\"stroke-width: 1px; fill: none;\"/\u003e\n",
        "  \u003ctext fill=\"black\" x=\"101.269\" y=\"271.779\" style=\"font-family: Arial; font-size: 20px;\" \u003e\n",
        "    \u003ctspan x=\"100.271\" y=\"256.785\" dy=\"\" dx=\"\"\u003e1\u003c/tspan\u003e\n",
        "    \u003ctspan x=\"169.271\" dy=\"0\" y=\"\" dx=\"\"\u003e2\u003c/tspan\u003e\n",
        "    \u003ctspan x=\"234.271\" dy=\"0\" y=\"\" dx=\"\"\u003e3\u003c/tspan\u003e\n",
        "  \u003c/text\u003e\n",
        "  \u003cline x1=\"62.26904296875455\" y1=\"209.77961730956898\" x2=\"320.26831054687955\" y2=\"209.77961730956898\" stroke=\"black\" style=\"stroke-width: 1px; fill: none;\"/\u003e\n",
        "  \u003cline x1=\"62.26904296875455\" y1=\"153.77967834472523\" x2=\"320.26831054687955\" y2=\"153.77967834472523\" stroke=\"black\" style=\"stroke-width: 1px; fill: none;\"/\u003e\n",
        "  \u003cline x1=\"62.26904296875455\" y1=\"99.77981567382679\" x2=\"320.26831054687955\" y2=\"99.77981567382679\" stroke=\"black\" style=\"stroke-width: 1px; fill: none;\"/\u003e\n",
        "  \u003ctext fill=\"black\" x=\"42.269\" y=\"215.78\" id=\"e523_texte\" style=\"font-family: Arial; font-size: 20px;\" \u003e1\u003c/text\u003e\n",
        "  \u003ctext fill=\"black\" x=\"42.269\" y=\"156.78\" id=\"e552_texte\" style=\"font-family: Arial; font-size: 20px;\" \u003e2\u003c/text\u003e\n",
        "  \u003ctext fill=\"black\" x=\"41.269\" y=\"105.78\" id=\"e564_texte\" style=\"font-family: Arial; font-size: 20px;\" \u003e3\u003c/text\u003e\n",
        "  \u003ccircle id=\"e616_circle\" cx=\"105.26899719238736\" cy=\"99.77981567382679\" stroke=\"black\" style=\"stroke-width: 1px;\" r=\"3.25248\" fill=\"khaki\"/\u003e\n",
        "  \u003ccircle id=\"e628_circle\" cx=\"173.26887512207486\" cy=\"99.77981567382679\" stroke=\"black\" style=\"stroke-width: 1px;\" r=\"3.25248\" fill=\"khaki\"/\u003e\n",
        "  \u003ccircle id=\"e640_circle\" cx=\"240.26852416992642\" cy=\"99.77981567382679\" stroke=\"black\" style=\"stroke-width: 1px;\" r=\"3.25248\" fill=\"khaki\"/\u003e\n",
        "  \u003ccircle id=\"e652_circle\" cx=\"240.26852416992642\" cy=\"153.77967834472523\" stroke=\"black\" style=\"stroke-width: 1px;\" r=\"3.25248\" fill=\"khaki\"/\u003e\n",
        "  \u003ccircle id=\"e664_circle\" cx=\"241.26850891113736\" cy=\"208.77961730956898\" stroke=\"black\" style=\"stroke-width: 1px;\" r=\"3.25248\" fill=\"khaki\"/\u003e\n",
        "  \u003ccircle id=\"e676_circle\" cx=\"174.26887512207486\" cy=\"153.77967834472523\" stroke=\"black\" style=\"stroke-width: 1px;\" r=\"3.25248\" fill=\"khaki\"/\u003e\n",
        "  \u003ccircle id=\"e688_circle\" cx=\"106.26899719238736\" cy=\"153.77967834472523\" stroke=\"black\" style=\"stroke-width: 1px;\" r=\"3.25248\" fill=\"khaki\"/\u003e\n",
        "  \u003ccircle id=\"e700_circle\" cx=\"107.26899719238736\" cy=\"208.77961730956898\" stroke=\"black\" style=\"stroke-width: 1px;\" r=\"3.25248\" fill=\"khaki\"/\u003e\n",
        "  \u003ccircle id=\"e712_circle\" cx=\"174.26887512207486\" cy=\"209.77961730956898\" stroke=\"black\" style=\"stroke-width: 1px;\" r=\"3.25248\" fill=\"khaki\"/\u003e\n",
        "  \u003ctext fill=\"black\" x=\"111.269\" y=\"199.78\" id=\"e749_texte\" style=\"font-family: Arial; font-size: 16px;\" dy=\"\" dx=\"\" \u003e(1,1)\u003c/text\u003e\n",
        "  \u003ctext fill=\"black\" x=\"174.269\" y=\"201.78\" id=\"e835_texte\" style=\"font-family: Arial; font-size: 16px;\" \u003e(2,1)\u003c/text\u003e\n",
        "  \u003ctext fill=\"black\" x=\"107.269\" y=\"90.7798\" id=\"e847_texte\" style=\"font-family: Arial; font-size: 16px;\" \u003e(1,3)\u003c/text\u003e\n",
        "  \u003ctext fill=\"black\" x=\"108.269\" y=\"145.78\" id=\"e859_texte\" style=\"font-family: Arial; font-size: 16px;\" dy=\"\" dx=\"\" \u003e(1,2)\u003c/text\u003e\n",
        "  \u003ctext fill=\"black\" x=\"174.269\" y=\"145.78\" id=\"e967_texte\" style=\"font-family: Arial; font-size: 16px;\" \u003e(2,2)\u003c/text\u003e\n",
        "  \u003ctext fill=\"black\" x=\"175.269\" y=\"92.7798\" id=\"e994_texte\" style=\"font-family: Arial; font-size: 16px;\" \u003e(2,3)\u003c/text\u003e\n",
        "  \u003ctext fill=\"black\" x=\"240.269\" y=\"200.78\" id=\"e1021_texte\" style=\"font-family: Arial; font-size: 16px;\" \u003e(3,1)\u003c/text\u003e\n",
        "  \u003ctext fill=\"black\" x=\"241.269\" y=\"145.78\" id=\"e1048_texte\" style=\"font-family: Arial; font-size: 16px;\" \u003e(3,2)\u003c/text\u003e\n",
        "  \u003ctext fill=\"black\" x=\"241.269\" y=\"92.7798\" id=\"e1075_texte\" style=\"font-family: Arial; font-size: 16px;\" \u003e(3,3)\u003c/text\u003e\n",
        "  \u003ctext fill=\"black\" x=\"176.269\" y=\"284.779\" id=\"e1257_texte\" style=\"font-family: Arial; font-size: 20px;\" \u003ex\u003c/text\u003e\n",
        "  \u003ctext fill=\"black\" x=\"11.269\" y=\"157.78\" id=\"e1272_texte\" style=\"font-family: Arial; font-size: 20px;\" \u003ey\u003c/text\u003e\n",
        "\u003c/svg\u003e\n",
        "\u003c/div\u003e"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "code",
        "id": "-KSr-wgh6KZb"
      },
      "outputs": [],
      "source": [
        "#@title Reshaping Quiz\n",
        "\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "def create_grid(x, y):\n",
        "  \"\"\"Creates a grid on the complex plane from x and y.\n",
        "\n",
        "  Given a set of x and y coordinates as rank 1 tensors \n",
        "  of sizes n and m respectively, returns a complex tensor \n",
        "  of shape [n, m] containing points on the grid formed by\n",
        "  intersection of horizontal and vertical lines rooted at\n",
        "  those x and y values.\n",
        "  \n",
        "  Args:\n",
        "    x: A float32 or float64 tensor of shape [n]\n",
        "    y: A tensor of the same data type as x and shape [m].\n",
        "    \n",
        "  Returns:\n",
        "    A complex tensor with shape [n, m].\n",
        "  \"\"\"\n",
        "  raise NotImplementedError()\n",
        "\n",
        "coords = tf.constant([1.0, 2.0, 3.0])\n",
        "square_grid = create_grid(coords, coords)\n",
        "\n",
        "def test():\n",
        "  x_p = np.array([1.0, 2.0, 3.0])\n",
        "  y_p = np.array([5.0, 6.0, 7.0, 8.0])\n",
        "  grid = create_grid(tf.constant(x_p),  tf.constant(y_p))\n",
        "  n_p = x_p.size * y_p.size\n",
        "  x = np.reshape(np.real(grid), [n_p])\n",
        "  y = np.reshape(np.imag(grid), [n_p])\n",
        "  plt.plot(x, y, 'ro')\n",
        "  plt.xlim((x_p.min() - 1.0, x_p.max() + 1.0))\n",
        "  plt.ylim((y_p.min() - 1.0, y_p.max() + 1.0))\n",
        "  plt.ylabel('Imaginary')\n",
        "  plt.xlabel('Real')\n",
        "  plt.show()\n",
        "  \n",
        "test()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "Bmlur4rUv5mn"
      },
      "outputs": [],
      "source": [
        "#@title Reshaping Quiz - Solution. Double click to reveal\n",
        "\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import tensorflow as tf\n",
        "\n",
        "def create_grid(x, y):\n",
        "  \"\"\"Creates a grid on the complex plane from x and y.\n",
        "\n",
        "  Given a set of x and y coordinates as rank 1 tensors \n",
        "  of sizes n and m respectively, returns a complex tensor \n",
        "  of shape [n, m] containing points on the grid formed by\n",
        "  intersection of horizontal and vertical lines rooted at\n",
        "  those x and y values.\n",
        "  \n",
        "  Args:\n",
        "    x: A float32 or float64 tensor of shape [n]\n",
        "    y: A tensor of the same data type as x and shape [m].\n",
        "    \n",
        "  Returns:\n",
        "    A complex tensor with shape [n, m].\n",
        "  \"\"\"\n",
        "  \n",
        "  X, Y = tf.meshgrid(x, y)\n",
        "  return tf.complex(X, Y)\n",
        "\n",
        "coords = tf.constant([1.0, 2.0, 3.0])\n",
        "square_grid = create_grid(coords, coords)\n",
        "\n",
        "def test():\n",
        "  x_p = np.array([1.0, 2.0, 3.0])\n",
        "  y_p = np.array([5.0, 6.0, 7.0, 8.0])\n",
        "  grid = create_grid(tf.constant(x_p),  tf.constant(y_p))\n",
        "  n_p = x_p.size * y_p.size\n",
        "  x = np.reshape(np.real(grid), [n_p])\n",
        "  y = np.reshape(np.imag(grid), [n_p])\n",
        "  plt.plot(x, y, 'ro')\n",
        "  plt.xlim((x_p.min() - 1.0, x_p.max() + 1.0))\n",
        "  plt.ylim((y_p.min() - 1.0, y_p.max() + 1.0))\n",
        "  plt.ylabel('Imaginary')\n",
        "  plt.xlabel('Real')\n",
        "  plt.show()\n",
        "  \n",
        "test()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ko6yVr6lf-Za"
      },
      "source": [
        "# Tensors vs. Numpy Arrays\n",
        "\n",
        "- Tensors can be created by wrapping numpy arrays (as above) or even python lists.\n",
        "- You can use all the Numpy methods to create arrays that can be wrapped as a tensor.\n",
        "- Most TensorFlow ops will accept a numpy array directly but they will convert it to\n",
        "  a tensor implicitly.\n",
        "- For many Numpy methods, there are analogous TensorFlow methods which we will see later.\n",
        "  - Example: np.zeros $\\leftrightarrow$ tf.zeros\n",
        "- However, a tensor is *not* the same as a numpy array.\n",
        "  - Tensors are more like \"pointers\" to the data.\n",
        "  - In eager mode, tensors don't have their values until they are evaluated. Numpy arrays are eagerly evalauted.\n",
        "  - You can't convert a Tensor back to a numpy array without evaluating the graph. In eager mode you can do so by invoking `numpy()` method on the tensor.\n",
        "\n",
        "The following examples clarify this."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "O81YdutWgfJ_"
      },
      "outputs": [],
      "source": [
        "# Import TensorFlow and numpy.\n",
        "import numpy as np\n",
        "\n",
        "# You can make a tensor out of a python list.\n",
        "tensor_of_ones = tf.constant([1, 1, 1], dtype=tf.int64)\n",
        "\n",
        "# You can also use numpy methods to generate the arrays.\n",
        "tensor_of_twos = tf.constant(np.repeat(2, [3]))\n",
        "\n",
        "# TensorFlow Ops (tf.add) accept tensors ...\n",
        "tensor_of_threes = tf.add(tensor_of_ones, tensor_of_twos)\n",
        "\n",
        "# ... and (sometimes) also the numpy array directly.\n",
        "tensor_of_threes_1 = tf.add(np.array([1, 1, 1]), \n",
        "                            tensor_of_twos)\n",
        "\n",
        "# You can check that the tensor\n",
        "print(\"Type: %s\" % type(tensor_of_threes)) # This is not an array! \n",
        "                                           # It is a tensor.\n",
        "# You can check that the tensor\n",
        "print(\"Numpy conversion: %s\" % tensor_of_threes.numpy())\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xWE1sdr-g0Wl"
      },
      "source": [
        "## How does it work?\n",
        "- **tf.constant** creates a tensor from some constant data.\n",
        "- In addition to supplying python lists, you may also supply numpy arrays.\n",
        "- Much easier to create higher dimensional data with numpy arrays.\n",
        "- **tf.add** is an example of an ***Op***. It takes two tensor arguments and returns a tensor.\n",
        "- Tensors have a definite type, e.g. int32, float64, bool etc. \n",
        "  - By default, TF will use the type of the supplied numpy array. \n",
        "  - For python lists, TF will try to infer the type but you are better off supplying the type explicitly using \"dtype=\" arg.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ih_hf9_Z57gV"
      },
      "source": [
        "## Tensor data types\n",
        "- Other than shape, tensors are characterized by the type of data elements it contains.\n",
        "- Useful to keep in mind the following commonly used types\n",
        "  - Integer types: tf.int32, tf.int64\n",
        "  - Float types: tf.float32, tf.float64\n",
        "  - Boolean type: tf.bool\n",
        "  - Complex type: tf.complex64, tf.complex128\n",
        "- Many tensor creation ops (e.g. `tf.constant`, `tf.convert_to_tensor` etc.) accept an optional *dtype* argument.\n",
        "- TensorFlow does not do automatic type conversions. For example, the following code causes an error."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "code",
        "id": "zBxaxkfG5Bfb"
      },
      "outputs": [],
      "source": [
        "#@title Strict Types in TensorFlow\n",
        "\n",
        "int32_tensor = tf.constant([1, 1], dtype=tf.int32)\n",
        "int64_tensor = tf.constant([2, 2], dtype=tf.int64)\n",
        "try_mix_types = tf.add(int32_tensor, int64_tensor)  # Causes a TypeError"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X7NSx8LXJF-h"
      },
      "source": [
        "- Occasionally, we need to convert one type to another (e.g. int32 -\u003e float32). \n",
        "- There are a few explicit conversion ops:\n",
        "  - **[tf.to\\_double](https://www.tensorflow.org/api_docs/python/tf/to_double)**: Convert to float64.\n",
        "  - **[tf.to\\_float](https://www.tensorflow.org/api_docs/python/tf/to_float)**: Convert to float32.\n",
        "  - **[tf.to\\_int64](https://www.tensorflow.org/api_docs/python/tf/to_int64)**: Convert to int64.\n",
        "  - **[tf.to\\_int32](https://www.tensorflow.org/api_docs/python/tf/to_int32)**: Convert to int32.\n",
        "- If you need conversion to something that isn't listed you can use the more general: **[tf.cast](https://www.tensorflow.org/api_docs/python/tf/cast)**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "code",
        "id": "amrbSgo67bWD"
      },
      "outputs": [],
      "source": [
        "#@title Casting from one type to another\n",
        "\n",
        "# Make sure this is an int32 tensor by explicitly specifying type.\n",
        "# NB: In this particular case, even if you left out the type, TF\n",
        "# will infer it as an int32.\n",
        "int32_tensor = tf.constant([1, 1], dtype=tf.int32)\n",
        "\n",
        "int64_tensor = tf.constant([2, 2], dtype=tf.int64)\n",
        "\n",
        "casted_to_64 = tf.cast(int32_tensor, dtype=tf.int64)\n",
        "\n",
        "# This is OK.\n",
        "added = tf.add(casted_to_64, int64_tensor)\n",
        "\n",
        "\n",
        "# As an example of tf.cast, consider casting to boolean\n",
        "zero_one = tf.constant([1.0, 0.0, 1.0])  # Inferred as tf.float32\n",
        "print(\"Type of zero_ones = %s\" % repr(zero_one.dtype))\n",
        "\n",
        "zero_one_bool = tf.cast(zero_one, tf.bool)\n",
        "print(\"Type of zero_ones_bool = %s\" % repr(zero_one_bool.dtype))\n",
        "\n",
        "# Another example of cast: Convert real numbers to Complex\n",
        "real_tensor = tf.constant([1.0, 1.0])\n",
        "cplx_tensor = tf.cast(real_tensor, tf.complex64)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "91kbGxwvwnMY"
      },
      "source": [
        "# Creating Tensors\n",
        "- We have already seen that tf.constant creates a tensor from supplied data.\n",
        "- Some other useful functions are in the table below. Use the Colab auto complete\n",
        "  feature to see their usage instructions.\n",
        "  \n",
        "## Constant Tensors\n",
        "\n",
        "\n",
        "|  Name | Description  |\n",
        "|---    |:---|\n",
        "|  [tf.zeros](https://www.tensorflow.org/api_docs/python/tf/zeros) | Creates a constant tensor of zeros of a given shape and type.  |\n",
        "|  [tf.zeros\\_like](https://www.tensorflow.org/api_docs/python/tf/zeros_like) |  Creates a constant tensor of zeros of the same shape as the input tensor. |\n",
        "| [tf.ones](https://www.tensorflow.org/api_docs/python/tf/ones)  | Creates a constant tensor of ones of a given shape and type.  |\n",
        "| [tf.ones\\_like](https://www.tensorflow.org/api_docs/python/tf/ones_like)  | Creates a constant tensor of  ones of the same shape as the input tensor.  |\n",
        "| [tf.linspace](https://www.tensorflow.org/api_docs/python/tf/linspace)  | Creates an evenly spaced tensor of values between supplied end points.  |\n",
        "\n",
        "The following example demonstrates some of these ops."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "code",
        "id": "xUF03k1wwsxO"
      },
      "outputs": [],
      "source": [
        "#@title Creating Constant Tensors without numpy\n",
        "\n",
        "# Create a bunch of zeros of a specific shape and type.\n",
        "x = tf.zeros([2, 2], dtype=tf.float64)\n",
        "print(\"tf.zeros example: %s\" % x)\n",
        "\n",
        "# tf.zeros_like is pretty useful. It creates a zero tensors which is \n",
        "# shaped like some other tensor you supply.\n",
        "x = tf.constant([[[1], [2]]])\n",
        "zeros_like_x = tf.zeros_like(x, dtype=tf.float32)\n",
        "\n",
        "print(\"Shape(x) = %s \\nShape(zeros_like_x) = %s\" % \n",
        "      (x.shape, zeros_like_x.shape))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jv3ykCFZcNOl"
      },
      "source": [
        "\n",
        "## Random Tensors\n",
        "\n",
        "A common need is to create tensors with specific shape but with randomly distributed entries. TF provides a  \n",
        "few methods for these.\n",
        "\n",
        "  |  Name | Description  |\n",
        "  |---    |:---|\n",
        "  |  [tf.random.normal](https://www.tensorflow.org/api_docs/python/tf/random/normal) | Generates a constant tensor with independent normal entries.  |\n",
        "  |  [tf.random.uniform](https://www.tensorflow.org/api_docs/python/tf/random/uniform) |  Generates a constant tensor with uniformly distributed elements. |\n",
        "  |  [tf.random.gamma](https://www.tensorflow.org/api_docs/python/tf/random/gamma) |  Generates a constant tensor with gamma distributed elements. |\n",
        "  |  [tf.random.shuffle](https://www.tensorflow.org/api_docs/python/tf/random/shuffle) |  Takes an input tensor and randomly permutes the entries along the first dimension. |\n",
        "  \n",
        "Let us see some of these in action."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "code",
        "id": "bNtwR-WNu-R9"
      },
      "outputs": [],
      "source": [
        "#@title Creating Random Tensors\n",
        "\n",
        "# Create a matrix with normally distributed entries.\n",
        "x = tf.random.normal([1, 3], mean=1.0, stddev=4.0, dtype=tf.float64)\n",
        "print(\"A random normal tensor: %s\" % x)\n",
        "\n",
        "# Randomly shuffle the first dimension of a tensor.\n",
        "r = tf.random.shuffle([1, 2, 3, 4])\n",
        "print(\"Random shuffle of [1,2,3,4]: %s\" % r)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KGYeF4K1JLIJ"
      },
      "source": [
        "\n",
        "# Sessions and `tf.function`\n",
        "\n",
        "We have discussed above that TensorFlow has an executor accessible through `tf.compat.v1.Session`. But what does it do? \n",
        "\n",
        "- When you write TensorFlow ops or tensors, you are adding them to the \"graph\".\n",
        "- It does not immediately evaluate anything. It only performs some sanity checks  \n",
        "  on your ops.\n",
        "- Recall: a tensor itself is not the value. It is a container for the data that will be  \n",
        "  generated when it is evaluated.\n",
        "- After creating the graph you have to explicitly ask for one or more of the tensors  \n",
        "  to be evaluated.\n",
        "- Let's see this in action:\n",
        "- The argument you supply to eval is called a **Session**. \n",
        "- The session is an object that creates/controls/talks to the C++ runtime that will  \n",
        "  actually run your computation.\n",
        "- The client (i.e. your python session) transfers the graph information to the session  \n",
        "  to be evaluated.\n",
        "- The session evaluates **the relevant part of your graph** and returns the value to  \n",
        "  your client for you to enjoy."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "code",
        "id": "DpH-eJ3N9ywq"
      },
      "outputs": [],
      "source": [
        "# Evaluating Tensors\n",
        "\n",
        "# Create graph \n",
        "g = tf.Graph()\n",
        "\n",
        "with g.as_default():\n",
        "  x = tf.constant([1., 1.])\n",
        "\n",
        "# Check that x is not actually a list.\n",
        "print(\"Type of 'x': %s\" % type(x)) # It is a tensor\n",
        "\n",
        "# Evaluate the tensor to actually make TF to do the computation.\n",
        "# Note we need to bind Session to the graph by passing it as an argument\n",
        "with tf.compat.v1.Session(graph=g) as sess:\n",
        "  x_values = sess.run(x)\n",
        "print(\"Value of 'x_values': %s\\nType of x_values: %s\" % (x_values,\n",
        "                                                         type(x_values)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nEPNRGcUL32a"
      },
      "source": [
        "* When you eval a tensor, you are telling TF that you want it to go ahead and run the computation needed to get a value for that tensor. At that point, TF figures out what other operations and tensors it needs to evaluate to be able to give you what you want.\n",
        "  * This extra step may seem annoying but it is (part of) what makes TF powerful. \n",
        "    It allows TF to evaluate only those ops that are directly needed for the output.\n",
        "* In the usage above, it is rather inconvenient that we can only evaluate\n",
        "  one tensor at a time. There are two way to avoid this.\n",
        "* Create a session variable and hold on to it for use with multiple evals.  \n",
        "  The following example demonstrates this:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "code",
        "id": "j9RDED45m4jR"
      },
      "outputs": [],
      "source": [
        "#@title Using Sessions\n",
        "\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "g = tf.Graph()\n",
        "with g.as_default():\n",
        "  # Create a 1-tensor with uniform entries between -pi and pi.\n",
        "  x = tf.linspace(-np.pi, np.pi, 20)\n",
        "\n",
        "  y = tf.sin(x) + tf.random.uniform([20], minval=-0.5, maxval=0.5)\n",
        "\n",
        "# Create session object which we will use multiple times.\n",
        "sess = tf.compat.v1.Session(graph=g)\n",
        "\n",
        "plt.plot(sess.run(x), sess.run(y), 'ro')\n",
        "\n",
        "# A session is a resource (like a file) and you must close it when\n",
        "# you are done.\n",
        "sess.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nvJAa7kDpYAT"
      },
      "source": [
        "* In the above method, it is still inconvenient to have to call eval on each tensor separately.\n",
        "* This can be avoided by using the method \"run\" on sessions as follows"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RYVh6hkKqAdh"
      },
      "outputs": [],
      "source": [
        "# Continuation of the above example so you must run that first.\n",
        "\n",
        "sess = tf.compat.v1.Session(graph=g)\n",
        "\n",
        "# Session.run evaluates one or more tensors supplied as a list\n",
        "# or a tuple. It returns their values as numpy arrays which you\n",
        "# may capture by assigning to a variable.\n",
        "x_v, y_v = sess.run((x, y))\n",
        "\n",
        "plt.plot(x_v, y_v, 'ro')\n",
        "\n",
        "sess.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RXnSBQDfq_UE"
      },
      "source": [
        "* It is pretty easy to forget to close the sessions so the best idea is to use them as context managers. This is the most common way of using sessions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YsGxvUmPrTVw"
      },
      "outputs": [],
      "source": [
        "#@title Sessions as Context managers\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "g = tf.Graph()\n",
        "with g.as_default():\n",
        "  x = tf.linspace(-5.0, 5.0, 1000)\n",
        "  y = tf.nn.sigmoid(x)\n",
        "\n",
        "with tf.compat.v1.Session(graph=g) as sess:\n",
        "  x_v, y_v = sess.run([x, y])\n",
        "  plt.plot(x_v, y_v)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZGuI3sy4VmRJ"
      },
      "source": [
        "Normally one would develop in eager mode. If the calculations are complex, graph\n",
        "optimization might provide enormous benefits. In this case, `tf.function` should\n",
        "be used."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HDjxC81AV6YB"
      },
      "outputs": [],
      "source": [
        "\n",
        "def eager_execution(a, b):\n",
        "  a = tf.convert_to_tensor(a, name=\"a\")\n",
        "  b = tf.convert_to_tensor(b, name=\"b\")\n",
        "  c = a + b\n",
        "  print(\"Tensor `c` is evaluated inside the function call: \", c)\n",
        "  d = c**2\n",
        "  return d\n",
        "\n",
        "@tf.function\n",
        "def graph_execution(a, b):\n",
        "  a = tf.convert_to_tensor(a, name=\"a\")\n",
        "  b = tf.convert_to_tensor(b, name=\"b\")\n",
        "  c = a + b\n",
        "  print(\"Evaluation of `c` is deferred: \", c)\n",
        "  d = c**2\n",
        "  return d\n",
        "\n",
        "print(\"Eager execution: \", eager_execution(1.0, 2.0))\n",
        "print(\"-------------\")\n",
        "print(\"Graph execution: \", graph_execution(1.0, 2.0))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bs68matUwq-P"
      },
      "source": [
        "# Example: Random Matrices\n",
        "\n",
        "Please restart your runtime if you are in the graph mode (check if\n",
        "`tf.executing_eagerly()` is True).\n",
        "Let us put together a few of the ops we have seen so far (and a few we haven't) into a longer example.\n",
        "\n",
        "A random matrix is a matrix whose entries are (usually independently) randomly distributed drawn from some chosen distribution.\n",
        "\n",
        "In this example, we will approximate the distribution of the **determinant** of a random $n \\times n$ matrix.\n",
        "\n",
        "The steps we will follow are:\n",
        "* Generate a sample of matrices of a desired size.\n",
        "* Compute their determinant.\n",
        "* Plot the histogram."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "X2VbLUYFDDui"
      },
      "outputs": [],
      "source": [
        "# Dimension of matrix to generate.\n",
        "n = 10\n",
        "\n",
        "# Number of samples to generate.\n",
        "sample_size = 100000\n",
        "\n",
        "# Using suggestion above let us build a graph which we wrap in a `tf.function`.\n",
        "\n",
        "def random_matrix_generator():\n",
        "  # We will generate matrices with elements uniformly drawn from (-1, 1).\n",
        "  # TensorFlow provides a whole bunch of methods to generate random tensors\n",
        "  # of a given shape and here we will use the random_uniform method.\n",
        "  samples = tf.random.uniform(shape=[sample_size, n, n], minval=-1, maxval=1)\n",
        "\n",
        "  # There is also an Op to generate matrix determinant. It requires that you pass\n",
        "  # it a tensor of shape [...., N, N]. This ensures that the last two dimensions\n",
        "  # can be interpreted as a matrix.\n",
        "  # Can you guess what the shape of the resulting determinants is?\n",
        "  dets_sample = tf.linalg.det(samples)\n",
        "\n",
        "  print('Determinant shape: ', dets_sample.shape)\n",
        "\n",
        "  # While we are at it, we might as well compute some summary stats.\n",
        "  dets_mean = tf.reduce_mean(dets_sample)\n",
        "  dets_var = tf.reduce_mean(tf.square(dets_sample)) - tf.square(dets_mean)\n",
        "  return dets_sample, dets_mean, dets_var\n",
        "\n",
        "# Wrap computations in `tf.function`\n",
        "random_matrix_generator_optimized = tf.function(random_matrix_generator)\n",
        "# Evaluate the determinants and plot a histogram.\n",
        "det_vals, mean, var = random_matrix_generator_optimized()\n",
        "# Plot a beautiful histogram.\n",
        "plt.hist(det_vals, 50, density=1, facecolor='green', alpha=0.75)\n",
        "plt.xlabel('Det(Unif(%d))' % n)\n",
        "plt.ylabel('Probability')\n",
        "plt.title(r'$\\mathrm{Random\\ Matrix\\ Determinant\\ Distribution:}\\ \\mu = %f,\\ \\sigma^2=%f$' % (mean, var))\n",
        "plt.grid(True)\n",
        "plt.show()\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fs_xfO1KDqHK"
      },
      "source": [
        "In this example, we used some ops such as tf.reduce_mean and tf.square which we will discuss more later."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nLMPM1NxwNas"
      },
      "source": [
        "# Maths Ops\n",
        "\n",
        "- There is a whole suite of commonly needed math ops built in.\n",
        "- We have already seen binary ops such as tf.add (addition) and tf.mul (multiplication).\n",
        "- Five can also be accessed as inline operators on tensors.\n",
        "- The inline form of the op allows you to e.g. write x + y instead of tf.add(x, y).\n",
        "\n",
        "| Name | Description | Inline form |\n",
        "| --- | --- | --- |\n",
        "| [tf.math.add](https://www.tensorflow.org/api_docs/python/tf/math/add) | Adds two tensors element wise | + |\n",
        "| [tf.math.subtract](https://www.tensorflow.org/api_docs/python/tf/math/subtract) | Subtracts two tensors element wise | - |\n",
        "| [tf.math.multiply](https://www.tensorflow.org/api_docs/python/tf/math/multiply) | Multiplies two tensors element wise | * |\n",
        "| [tf.math.divide](https://www.tensorflow.org/api_docs/python/tf/math/divide) | Divides two tensors element wise | / |\n",
        "| [tf.math.mod](https://www.tensorflow.org/api_docs/python/tf/math/floormod) | Computes the remainder of division element wise | % |\n",
        "\n",
        "\n",
        "- Note that the behaviour of \"/\" and \"//\" varies depending on python version and presence of `from __future__ import division`, to match how division behaves with ordinary python scalars.\n",
        "- The following table lists some more commonly needed functions:\n",
        "\n",
        "| Name | Description |\n",
        "| --- | --- |\n",
        "| [tf.math.exp](https://www.tensorflow.org/api_docs/python/tf/math/exp) | The exponential of the argument element wise. |\n",
        "| [tf.math.log](https://www.tensorflow.org/api_docs/python/tf/math/log) | The natural log element wise |\n",
        "| [tf.math.sqrt](https://www.tensorflow.org/api_docs/python/tf/math/sqrt) | Square root element wise |\n",
        "| [tf.math.round](https://www.tensorflow.org/api_docs/python/tf/math/round) | Rounds to the nearest integer element wise |\n",
        "| [tf.math.maximum](https://www.tensorflow.org/api_docs/python/tf/math/maximum) | Maximum of two tensors element wise. |\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7amE3PPMRrOC"
      },
      "source": [
        "# Matrix Ops\n",
        "\n",
        "* Matrices are rank 2 tensors. There is a suite of ops for doing matrix manipulations which we briefly discuss.\n",
        "\n",
        "| Name | Description |\n",
        "| --- | --- |\n",
        "| [tf.linalg.matrix_diag](https://www.tensorflow.org/api_docs/python/tf/linalg/diag) | Creates a tensor from its diagonal |\n",
        "| [tf.linalg.trace](https://www.tensorflow.org/api_docs/python/tf/linalg/trace) | Computes the sum of the diagonal elements of a matrix. |\n",
        "| [tf.linalg.matrix\\_determinant](https://www.tensorflow.org/api_docs/python/tf/linalg/det) | Computes the determinant of a matrix (square only) |\n",
        "| [tf.linalg.matmul](https://www.tensorflow.org/api_docs/python/tf/linalg/matmul) | Multiplies two matrices |\n",
        "| [tf.linalg.matrix\\_inverse](https://www.tensorflow.org/api_docs/python/tf/linalg/inv) | Computes the inverse of the matrix (square only) |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "whITNpIu4_pL"
      },
      "source": [
        "## Quiz: Normal Density\n",
        "\n",
        "\n",
        "- In the following mini-codelab, you are asked to compute the normal density using the ops you have seen so far.\n",
        "- You first generate a sample of points at which you will evaluate the density.\n",
        "- The points are generated using a normal distribution (need not be the same one whose density you are evaluating).\n",
        "- This is done by the function **generate\\_normal\\_draws** below.\n",
        "- The function **normal\\_density\\_at** computes the density at any given set of points.\n",
        "- You have to complete the code of these two functions so they work as expected.\n",
        "- Execute the code and check that the test passes.\n",
        "\n",
        "### Hints\n",
        "- Recall that the normal density is given by  \n",
        "  $f(x) = \\frac{1}{\\sqrt{2\\pi\\sigma^2}} e^{-\\frac{(x-\\mu)^2}{2\\sigma^2}}$\n",
        "- Here $\\mu$ is the mean of the distribution and $\\sigma \u003e 0$ is the standard deviation.\n",
        "- Pay attention to the data types mentioned in the function documentations. You should ensure that your implementations respect the data types stated."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "code",
        "id": "YK94ZbOYOj3L"
      },
      "outputs": [],
      "source": [
        "#@title Mini-codelab: Compute the normal density.\n",
        "\n",
        "import numpy as np\n",
        "import numpy.testing as npt\n",
        "from scipy import stats\n",
        "\n",
        "def generate_normal_draws(shape, mean=0.0, stddev=1.0):\n",
        "  \"\"\"Generates a tensor drawn from a 1D normal distribution.\n",
        "  \n",
        "    Creates a constant tensor of the supplied shape whose elements are drawn\n",
        "    independently from a normal distribution with the supplied parameters.\n",
        "\n",
        "    Args:\n",
        "      shape: An int32 tensor. Specifies the shape of the return value.\n",
        "      mean: A float32 value. The mean of the normal distribution. \n",
        "      stddev: A positive float32 value. The standard deviation of the \n",
        "        distribution.\n",
        "\n",
        "    Returns:\n",
        "      A constant float32 tensor whose elements are normally distributed.\n",
        "  \"\"\"\n",
        "  \n",
        "  # to-do: Complete this function.\n",
        "  pass\n",
        "\n",
        "def normal_density_at(x, mean=0.0, stddev=1.0):\n",
        "  \"\"\"Computes the normal density at the supplied points.\n",
        "  \n",
        "    Args:\n",
        "      x: A float32 tensor at which the density is to be computed.\n",
        "      mean: A float32. The mean of the distribution.\n",
        "      stddev: A positive float32. The standard deviation of the distribution.\n",
        "      \n",
        "    Returns:\n",
        "      A float32 tensor of the normal density evaluated at the supplied points.\n",
        "  \"\"\"\n",
        "  \n",
        "  # to-do: Complete this function. As a reminder, the normal density is\n",
        "  # f(x) = exp(-(x-mu)^2/(2*stddev^2)) / sqrt(2 pi stddev^2).\n",
        "  # The value of pi can be accessed as np.pi.\n",
        "  pass\n",
        "\n",
        "def test():\n",
        "  mu, sd = 1.1, 2.1\n",
        "  x = generate_normal_draws([2, 3, 5], mean=mu, stddev=sd)\n",
        "  pdf = normal_density_at(x)\n",
        "  npt.assert_array_equal(x.shape, [2,3,5], 'Shape is incorrect')\n",
        "  norm = stats.norm()\n",
        "  npt.assert_allclose(pdf, norm.pdf(x), atol=1e-6)\n",
        "  print (\"All good!\")\n",
        "    \n",
        "test()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "ItfVVFcRwd3Z"
      },
      "outputs": [],
      "source": [
        "#@title Mini-codelab Solution: Compute the normal density. Double-click to reveal\n",
        "\n",
        "import numpy as np\n",
        "import numpy.testing as npt\n",
        "from scipy import stats\n",
        "\n",
        "def generate_normal_draws(shape, mean=0.0, stddev=1.0):\n",
        "  \"\"\"Generates a tensor drawn from a 1D normal distribution.\n",
        "  \n",
        "    Creates a constant tensor of the supplied shape whose elements are drawn\n",
        "    independently from a normal distribution with the supplied parameters.\n",
        "\n",
        "    Args:\n",
        "      shape: An int32 tensor. Specifies the shape of the return value.\n",
        "      mean: A float32 value. The mean of the normal distribution. \n",
        "      stddev: A positive float32 value. The standard deviation of the \n",
        "        distribution.\n",
        "\n",
        "    Returns:\n",
        "      A constant float32 tensor whose elements are normally distributed.\n",
        "  \"\"\"\n",
        "  return tf.random.normal(shape=shape, mean=mean, stddev=stddev)\n",
        "\n",
        "def normal_density_at(x, mean=0.0, stddev=1.0):\n",
        "  \"\"\"Computes the normal density at the supplied points.\n",
        "\n",
        "    Args:\n",
        "      x: A float32 tensor at which the density is to be computed.\n",
        "      mean: A float32. The mean of the distribution.\n",
        "      stddev: A positive float32. The standard deviation of the distribution.\n",
        "\n",
        "    Returns:\n",
        "      A float32 tensor of the normal density evaluated at the supplied points.\n",
        "  \"\"\"\n",
        "  stddev = tf.convert_to_tensor(stddev, dtype=x.dtype)\n",
        "  normalization = 1.0 / tf.math.sqrt(2.0 * np.pi * stddev * stddev)\n",
        "  return tf.math.exp(-tf.math.square((x - mean) / stddev) / 2.0) * normalization\n",
        "\n",
        "def test():\n",
        "  mu, sd = 1.1, 2.1\n",
        "  x = generate_normal_draws([2, 3, 5], mean=mu, stddev=sd)\n",
        "  pdf = normal_density_at(x)\n",
        "  npt.assert_array_equal(x.shape, [2,3,5], 'Shape is incorrect')\n",
        "  norm = stats.norm()\n",
        "  npt.assert_allclose(pdf, norm.pdf(x), atol=1e-6)\n",
        "  print (\"All good!\")\n",
        "    \n",
        "test()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RFXAyuV2Fyyx"
      },
      "source": [
        "# Logical And Comparison Ops\n",
        "\n",
        "- Tensorflow has the full complement of logical operators you would expect.\n",
        "- These are also overloaded so you can use their inline version.\n",
        "- The ops most frequently used are as follows:\n",
        "\n",
        "| Name | Description | Inline form |\n",
        "| --- | --- | --- |\n",
        "| [tf.equal](https://www.tensorflow.org/api_docs/python/tf/math/equal) | Element wise equality | **None** |\n",
        "| [tf.less](https://www.tensorflow.org/api_docs/python/tf/math/less) | Element wise less than | \u003c |\n",
        "| [tf.less\\_equal](https://www.tensorflow.org/api_docs/python/tf/math/less_equal) | Element wise less than or equal to | \u003c= |\n",
        "| [tf.greater](https://www.tensorflow.org/api_docs/python/tf/math/greater) |  Element wise greater than | \u003e |\n",
        "| [tf.greater\\_equal](https://www.tensorflow.org/api_docs/python/tf/math/greater_equal) | Element wise greater than or equal to | \u003e= |\n",
        "| [tf.logical\\_and](https://www.tensorflow.org/api_docs/python/tf/math/logical_and) | Element wise And | \u0026 |\n",
        "| [tf.logical\\_or](https://www.tensorflow.org/api_docs/python/tf/math/logical_or) | Element wise Or | \u0026#124; |\n",
        "\n",
        "- Note that tf.equal doesn't have an inline form. Comparing two tensors with == will use the default python comparison. It will **not** call tf.equal.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nhy1UXXJI8VY"
      },
      "source": [
        "## Note about Broadcasting\n",
        "\n",
        "- All the binary operators described above expect their operands to be of the same\n",
        "  shape up to broadcasting.\n",
        "- Broadcasting attempts to find a larger shape that would render the two arguments compatible.\n",
        "  - Tensorflow's broadcasting behaviour is like Numpy's.\n",
        "  - Example: [ 1, 2, 3 ] \u003e 0. The LHS is tensor of shape [3] while the right hand side can be\n",
        "    promoted to [ 0, 0, 0] which makes it compatible.\n",
        "  - More non trivial example: tf.equal([[1,2], [2, 3]], [2,3]). The LHS is shape [2,2] right is shape [2]. The RHS gets broadcasted so that it looks like [[2,3],[2,3]] and the comparison  is performed element wise.\n",
        "  - These are the most common case and we will make extensive use of this below.\n",
        "  - The full set of rules for broadcasting are available [here](https://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "code",
        "id": "XTI8NXvfOzAx"
      },
      "outputs": [],
      "source": [
        "#@title Comparison Ops Examples\n",
        "\n",
        "a = tf.constant([1.0, 1.0])\n",
        "b = tf.constant([2.0, 2.0])\n",
        "c = tf.constant([1.0, 2.0])\n",
        "d = 3.0\n",
        "\n",
        "print (\"Inputs:\\na: %s\\nb: %s\\nc: %s\\nd: %s\\n\" % (a, b, c, d))\n",
        "# Less-than op. Tests if the first argument is less than the second argument\n",
        "# component wise.\n",
        "a_less_than_b = a \u003c b\n",
        "b_greater_than_c = b \u003e c\n",
        "\n",
        "# Simple broadcasting in action\n",
        "a_less_than_d = a \u003c d\n",
        "\n",
        "# More complex broadcasting\n",
        "a2 = tf.constant([[1,2],[2,3]])\n",
        "b2 = tf.constant([1,3])\n",
        "c2 = tf.equal(a2, b2)\n",
        "\n",
        "# Note that there is no inline form for tf.equals. If you do b == c, you will\n",
        "# not get what you think you might.\n",
        "b_equal_to_c = tf.equal(b, c)\n",
        "\n",
        "outputs = (a_less_than_b, b_greater_than_c, a_less_than_d,  b_equal_to_c)\n",
        "\n",
        "print(\"Outputs:\\na \u003c b: %s\\nb \u003e c: %s\\na \u003c d: %s\\nb == c: %s\\n\" % (outputs))\n",
        "\n",
        "\n",
        "print(\"Complex Broadcasting\")\n",
        "print(\"%s == %s =\u003e %s\" % (a2.numpy(), b2.numpy(), c2.numpy()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3fxepp087YzU"
      },
      "source": [
        "# Aggregations and Scans\n",
        "\n",
        "Most of the ops we have seen so far, act on the input tensors in an element wise manner. Another important set of operators allow you to do aggregations on a whole tensor as well as scan the tensor. \n",
        "\n",
        "- Aggregations (or reductions) act on a tensor and produce a reduced dimension tensor. The main ops here are \n",
        "\n",
        "| Name | Description |\n",
        "| --- | --- |\n",
        "| [tf.reduce\\_sum](https://www.tensorflow.org/api_docs/python/tf/math/reduce_sum) | Sum of elements along all or some dimensions. |\n",
        "| [tf.reduce\\_mean](https://www.tensorflow.org/api_docs/python/tf/math/reduce_mean) | Average of elements along all or some dimensions. |\n",
        "| [tf.reduce\\_min](https://www.tensorflow.org/api_docs/python/tf/math/reduce_min) | Minimum of elements along all or some dimensions. |\n",
        "| [tf.reduce\\_max](https://www.tensorflow.org/api_docs/python/tf/math/reduce_max) | Maximum of elements along all or some dimensions. |\n",
        "\n",
        "- and for boolean tensors only\n",
        "\n",
        "| Name | Description |\n",
        "| --- | --- |\n",
        "| [tf.reduce\\_any](https://www.tensorflow.org/api_docs/python/tf/math/reduce_any) | Result of logical OR along all or some dimensions. |\n",
        "| [tf.reduce\\_all](https://www.tensorflow.org/api_docs/python/tf/math/reduce_all) | Result of logical AND along all or some dimensions. |\n",
        "\n",
        "\n",
        "- Scan act on a tensor and produce a tensor of the same dimension.\n",
        "\n",
        "| Name | Description |\n",
        "| --- | --- |\n",
        "| [tf.cumsum](https://www.tensorflow.org/api_docs/python/tf/math/cumsum) | Cumulative sum of elements along an axis. |\n",
        "| [tf.cumprod](https://www.tensorflow.org/api_docs/python/tf/math/cumprod) | Cumulative product of elements along an axis. |\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iL0fRYg0Eipa"
      },
      "source": [
        "## Codelab: Estimating $\\pi$\n",
        "\n",
        "In this short codelab, we will use an age old method to estimate the value of $\\pi$.\n",
        "The idea is very simple: Throw darts at a square and check what fraction lies inside the\n",
        "inscribed circle (see diagram).\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "ivz5TMw7JOS6"
      },
      "outputs": [],
      "source": [
        "#@title\n",
        "%%html\n",
        "\u003csvg width=\"210\" height=\"210\"\u003e\n",
        "  \u003crect x1=\"0\" y1=\"0\" width=\"200\" height=\"200\" stroke=\"blue\" fill=\"red\"\n",
        "       fill-opacity=\"0.5\" stroke-opacity=\"0.8\"/\u003e\n",
        "  \u003ccircle cx=\"100\" cy=\"100\" r=\"99\" fill=\"green\" stroke=\"rgba(0,20,0,0.7)\" stroke-width=\"2\"/\u003e\n",
        "  \n",
        "  \u003ccircle cx=\"188\" cy=\"49\" r=\"3\" fill=\"blue\" stroke=\"rgba(0,20,0,0.7)\" stroke-width=\"2\"/\u003e\n",
        "  \u003ccircle cx=\"113\" cy=\"130\" r=\"3\" fill=\"blue\" stroke=\"rgba(0,20,0,0.7)\" stroke-width=\"2\"/\u003e\n",
        "  \u003ccircle cx=\"44\" cy=\"78\" r=\"3\" fill=\"blue\" stroke=\"rgba(0,20,0,0.7)\" stroke-width=\"2\"/\u003e\n",
        "  \u003ccircle cx=\"116\" cy=\"131\" r=\"3\" fill=\"blue\" stroke=\"rgba(0,20,0,0.7)\" stroke-width=\"2\"/\u003e\n",
        "  \u003ccircle cx=\"189\" cy=\"188\" r=\"3\" fill=\"blue\" stroke=\"rgba(0,20,0,0.7)\" stroke-width=\"2\"/\u003e\n",
        "  \u003ccircle cx=\"126\" cy=\"98\" r=\"3\" fill=\"blue\" stroke=\"rgba(0,20,0,0.7)\" stroke-width=\"2\"/\u003e\n",
        "  \u003ccircle cx=\"18\" cy=\"42\" r=\"3\" fill=\"blue\" stroke=\"rgba(0,20,0,0.7)\" stroke-width=\"2\"/\u003e\n",
        "  \u003ccircle cx=\"146\" cy=\"62\" r=\"3\" fill=\"blue\" stroke=\"rgba(0,20,0,0.7)\" stroke-width=\"2\"/\u003e\n",
        "  \u003ccircle cx=\"13\" cy=\"139\" r=\"3\" fill=\"blue\" stroke=\"rgba(0,20,0,0.7)\" stroke-width=\"2\"/\u003e\n",
        "  \u003ccircle cx=\"157\" cy=\"94\" r=\"3\" fill=\"blue\" stroke=\"rgba(0,20,0,0.7)\" stroke-width=\"2\"/\u003e\n",
        "\n",
        "  \u003cline x1=\"100\" y1=\"100\" x2=\"170\" y2=\"170\" stroke=\"black\"/\u003e\n",
        "  \u003ctext x=\"144\" y=\"130\" text-anchor=\"middle\"\u003e\u003ctspan baseline-shift=\"sub\" font-size=\"normal\"\u003e1\u003c/tspan\u003e\u003c/text\u003e\n",
        "\u003c/svg\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BvY-YaWGJNwZ"
      },
      "source": [
        "\n",
        "The steps to estimate it are:\n",
        "\n",
        "  * Generate $n$ samples of pairs of uniform variates $(x, y)$ drawn from $[-1, 1]$.\n",
        "  * Compute the fraction $f_n$ that lie inside the unit circle, i.e. have $x^2+y^2 \\leq 1$.\n",
        "  * Estimate $\\pi \\approx = 4 f_n$, because \n",
        "    * The area of the unit circle is $\\pi$\n",
        "    * The area of the rectangle is $4$ \n",
        "    * $f_{\\infty} = \\frac{\\pi}{4}$.\n",
        "  \n",
        "\n",
        "Your task is to complete the functions **generate_sample** and **compute_fraction** below.\n",
        "They correspond to the first and the second steps described above.  \n",
        "The last step is already done for you in the function estimate_pi.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "code",
        "id": "GOmuCgXvEK2J"
      },
      "outputs": [],
      "source": [
        "#@title Codelab: Estimating Pi\n",
        "\n",
        "import numpy as np\n",
        "\n",
        "\n",
        "def generate_sample(size):\n",
        "  \"\"\"Sample a tensor from the uniform distribution.\n",
        "  \n",
        "  Creates a tensor of shape [size, 2] containing independent uniformly\n",
        "  distributed numbers drawn between [-1.0, 1.0].\n",
        "  \n",
        "  Args:\n",
        "    size: A positive integer. The number of samples to generate.\n",
        "  \n",
        "  Returns:\n",
        "    A tensor of data type tf.float64 and shape [size, 2].\n",
        "  \"\"\"\n",
        "  raise NotImplementedError()\n",
        "\n",
        "\n",
        "def compute_fraction(sample):\n",
        "  \"\"\"The fraction of points inside the unit circle.\n",
        "  \n",
        "  Computes the fraction of points that satisfy\n",
        "  sample[0]^2 + sample[1]^2 \u003c= 1.\n",
        "  \n",
        "  Args:\n",
        "    sample: A float tensor of shape [n, 2].\n",
        "    \n",
        "  Returns:\n",
        "    The fraction of n that lie inside the unit circle.\n",
        "  \"\"\"\n",
        "  raise NotImplementedError()\n",
        "\n",
        "def estimate_pi(num_samples):\n",
        "  sample = generate_sample(num_samples)\n",
        "  f_t = compute_fraction(sample)\n",
        "  error = np.abs(np.pi / 4 - f_t) / (np.pi / 4)\n",
        "  print (\"Estimate: %.5f, Error: %.3f%%\" % (4 * f_t, error * 100.))\n",
        "\n",
        "estimate_pi(100000)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "qk-FPhTg0r1m"
      },
      "outputs": [],
      "source": [
        "#@title Codelab Solution: Estimating Pi. Double click to reveal\n",
        "\n",
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "\n",
        "def generate_sample(size):\n",
        "  \"\"\"Sample a tensor from the uniform distribution.\n",
        "  \n",
        "  Creates a tensor of shape [size, 2] containing independent uniformly\n",
        "  distributed numbers drawn between [-1.0, 1.0].\n",
        "  \n",
        "  Args:\n",
        "    size: A positive integer. The number of samples to generate.\n",
        "  \n",
        "  Returns:\n",
        "    A tensor of data type tf.float64 and shape [size, 2].\n",
        "  \"\"\"\n",
        "  \n",
        "  return tf.random.uniform(shape=[size, 2], minval=-1.0, maxval=1.0, \n",
        "                           dtype=tf.float64)  \n",
        "\n",
        "\n",
        "def compute_fraction(sample):\n",
        "  \"\"\"The fraction of points inside the unit circle.\n",
        "  \n",
        "  Computes the fraction of points that satisfy\n",
        "  sample[0]^2 + sample[1]^2 \u003c= 1.\n",
        "  \n",
        "  Args:\n",
        "    sample: A float tensor of shape [n, 2].\n",
        "    \n",
        "  Returns:\n",
        "    The fraction of n that lie inside the unit circle.\n",
        "  \"\"\"\n",
        "\n",
        "  sq_distance = tf.math.reduce_sum(tf.square(sample), 1)\n",
        "  in_circle = tf.cast(sq_distance \u003c= 1.0, dtype=sq_distance.dtype)\n",
        "  return tf.reduce_mean(in_circle)\n",
        "\n",
        "\n",
        "def estimate_pi(num_samples):\n",
        "  sample = generate_sample(num_samples)\n",
        "  f_t = compute_fraction(sample)\n",
        "  error = np.abs(np.pi / 4 - f_t) / (np.pi / 4)\n",
        "  print (\"Estimate: %.5f, Error: %.3f%%\" % (4 * f_t, error * 100.))\n",
        "\n",
        "estimate_pi(100000)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XzODzQo--s1t"
      },
      "outputs": [],
      "source": [
        "#@title Aggregation/Scan examples\n",
        "\n",
        "# Generate a tensor of gamma-distributed values and aggregate them\n",
        "x = tf.random.gamma([100, 10], 0.5)\n",
        "\n",
        "# Adds all the elements\n",
        "x_sum = tf.math.reduce_sum(x)\n",
        "\n",
        "# Adds along the first axis.\n",
        "x_sum_0 = tf.math.reduce_sum(x, 0)\n",
        "\n",
        "# Maximum along the first axis\n",
        "x_max_0 = tf.math.reduce_max(x, 0)\n",
        "\n",
        "# Cumulative sum for x_max_0:\n",
        "x_max_cumsum = tf.math.cumsum(x_max_0)\n",
        "\n",
        "print(\"Total Sum: %s\\n\" % x_sum)\n",
        "print(\"Partial Sum: %s\\n\" % x_sum_0)\n",
        "print(\"Maximum: %s\\n\" % x_max_0)\n",
        "print(\"Cumulative sum of x_max_0: %s\\n\" % x_max_cumsum)\n",
        "  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "whdNCwBXDxOb"
      },
      "source": [
        "# Mixing and Locating Elements (tf.where)\n",
        "\n",
        "\n",
        "\n",
        "We often need to be able to mix two tensors based on the values in another tensor.\n",
        "The where_v2 op is particularly useful in this context. It has two major uses:\n",
        "\n",
        "\n",
        "- **tf.where(Condition, T, F)**: Allows you to mix and match elements of two tensors based on a boolean tensor.\n",
        "  - All tensors must be of the same shape (or broadcastable to same).\n",
        "  - T and F must have the same data type.\n",
        "  - Picks elements of T where Condition is true and F where Condition is False.\n",
        "  - Example: tf.where_v2([True, False], [1, 2], [3, 4]) $\\rightarrow$ [1, 4].\n",
        "\n",
        "- **tf.where(tensor)**: Alternatively, if T and F aren't supplied, then the op returns locations of elements which are true.\n",
        "  - Example: tf.where_v2([1, 2, 3, 4] \u003e 2) $\\rightarrow$ [[2], [3]]\n",
        "\n",
        "### Example:\n",
        "Let's see them in action. We will create tensor of integers between 1 and 50 and set\n",
        "all multiples of 3 to 0."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iLsiuIc1N2ev"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "# Create a tensor with numbers between 1 and 50\n",
        "nums = tf.constant(np.arange(1, 51))\n",
        "\n",
        "# tf.mod(x, y) gives the remainder of the division x / y. \n",
        "# Find all multiples of 3.\n",
        "to_replace = tf.equal(nums % 3, 0)\n",
        "\n",
        "# First form of where_v2: if to_replace is true, tf.where_v2 picks the element \n",
        "# from the first tensor and otherwise, from the second tensor.\n",
        "result = tf.where(to_replace, tf.zeros_like(nums), nums)\n",
        "print(result)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zZpaNMluhTqH"
      },
      "outputs": [],
      "source": [
        "# Now let's confirm that we did indeed set the right numbers to zero. \n",
        "# This is where the second form of tf.where_v2 helps us. It will find all the \n",
        "# indices where its first argument is true.\n",
        "# Keep in mind that tensors are zero indexed (i.e. the first element has \n",
        "# index 0) so we will need to add a 1 to the result.\n",
        "zero_locations = tf.where(tf.equal(result, 0)) + 1\n",
        "\n",
        "print(tf.transpose(zero_locations))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fk9Qsm7ol-x8"
      },
      "source": [
        "# Slicing and Joining\n",
        "\n",
        "There are a number of ops which allow you to take parts of a tensor as well join multiple tensors together. \n",
        "\n",
        "Before we discuss those ops, let's look at how we can use the usual array indexing to access parts of a tensor.\n",
        "\n",
        "## Indexing\n",
        "\n",
        "* Even though tensors are not arrays in the usual sense, you can still index into them.\n",
        "\n",
        "* The indexing produces tensors which may be evaluated or consumed further in the usual way.\n",
        "\n",
        "* Indexing is a short cut for writing a explicit op (just like you can write x + y instead of tf.add(x, y)).\n",
        "\n",
        "* Tensorflow's indexing works similarly to Numpy's."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "code",
        "id": "thsRwNogmH1I"
      },
      "outputs": [],
      "source": [
        "#@title Indexing\n",
        "\n",
        "x = tf.constant([[1, 2, 3], [4, 5, 6]])\n",
        "\n",
        "# Get a tensor containing only the first component of x.\n",
        "x_0 = x[0]\n",
        "\n",
        "# A tensor of the first two elements of the first row.\n",
        "x_0_12 = x[0, 0:2] \n",
        "\n",
        "print(\"x_0: %s\" % x_0)\n",
        "print(\"x_0_12: %s\" % x_0_12)\n",
        "  \n",
        "# You can also do this more generally with the tf.slice op which is useful\n",
        "# if the indices you want are themselves tensors.\n",
        "\n",
        "x_slice = tf.slice(x, [0, 0], [1, 2])\n",
        "\n",
        "print(\"With tf.slice: %s\" % x_slice)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lbPBvYBEqnTO"
      },
      "source": [
        "Coming back to the ops that are available for tailoring tensors, here are a few of them\n",
        "\n",
        "| Name | Description |\n",
        "| --- | --- |\n",
        "| [tf.slice](https://www.tensorflow.org/api_docs/python/tf/slice) | Take a contiguous slice out of a tensor. |\n",
        "| [tf.split](https://www.tensorflow.org/api_docs/python/tf/split) | Split a tensor into equal pieces along a dimension |\n",
        "| [tf.tile](https://www.tensorflow.org/api_docs/python/tf/tile) | Tile a tensor by copying and concatenating it |\n",
        "| [tf.pad](https://www.tensorflow.org/api_docs/python/tf/pad) | Pads a tensor |\n",
        "| [tf.concat](https://www.tensorflow.org/api_docs/python/tf/concat) | Concatenate tensors along a dimension |\n",
        "| [tf.stack](https://www.tensorflow.org/api_docs/python/tf/stack) | Stacks n tensors of rank R into one tensor of rank R+1 |\n",
        "\n",
        "\n",
        "Let's briefly look at these ops in action."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "code",
        "id": "15r4UHb7raYM"
      },
      "outputs": [],
      "source": [
        "#@title Slicing and Joining Examples\n",
        "\n",
        "x = tf.constant([[1, 2, 3], [4, 5, 6], [7, 8, 9]])\n",
        "\n",
        "# Slice takes a starting index somewhere in the tensor and a size in each\n",
        "# dimension that you want to keep. It allows you to pass tensors for the start\n",
        "# position and the sizes. Note that the shape of the result is same as size arg.\n",
        "start_index = tf.constant([1, 1])\n",
        "size = tf.constant([1, 2])\n",
        "x_slice = tf.slice(x, start_index, size)\n",
        "print(\"tf.slice\")\n",
        "print(\"x[1:2, 1:3] = %s\" % x_slice)\n",
        "\n",
        "\n",
        "# Split splits the tensor along any given dimension. The return value is a list\n",
        "# of tensors (and not just one tensor).\n",
        "pieces = tf.split(x, 3, 0)\n",
        "\n",
        "print(\"\\ntf.split\")\n",
        "print(pieces)\n",
        "  \n",
        "# Tile makes a bigger tensor out of your tensor by tiling copies of it in the\n",
        "# dimensions you specify. \n",
        "\n",
        "y = tf.constant([[1, 2], [3, 4]])\n",
        "\n",
        "tiled = tf.tile(y, [2, 2])\n",
        "\n",
        "print(\"\\n tf.tile\")\n",
        "print(\"Y:\\n%s\\n\" % y)\n",
        "print(\"Y tiled twice in both dims:\\n%s\\n\" % tiled)\n",
        "  \n",
        "# Pad has a few modes of operation but the simplest one is where you pad a \n",
        "# tensor with zeros (the default mode). You specify the amount of padding you\n",
        "# want at the top and at the bottom of each dimension. In this example, we will\n",
        "# pad y defined above with zero asymmetrically\n",
        "\n",
        "padded = tf.pad(y, paddings=[[1, 2], [3, 4]])\n",
        "print(\"\\n tf.pad\")\n",
        "print(\"Y with padding:\\n%s\\n\" % padded)\n",
        "\n",
        "# Concat simply concatenates two tensors of the same rank along some axis.\n",
        "x = tf.constant([[1], [2]])\n",
        "y = tf.constant([[3], [4]])\n",
        "x_y = tf.concat([x, y], 0)\n",
        "print(\"\\n tf.concat\")\n",
        "print(\"Concat X and Y:\\n%s\\n\" % x_y)\n",
        "\n",
        "# Pack is quite useful when you have a bunch of tensors and you want to join\n",
        "# them into a higher rank tensor. Let's take the same x and y as above.\n",
        "stacked = tf.stack([x, y], axis=0)\n",
        "print(\"\\n tf.stacked\")\n",
        "print(\"Stacked X and Y:\\n%s\\n\" % stacked)\n",
        "print(\"Shape X: %s, Shape Y: %s, Shape of Stacked_0: %s\" % \n",
        "      (x.shape, y.shape, stacked.shape))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "enb8GAOcioNX"
      },
      "source": [
        "# Codelab: Distribution of Bernoulli Random Matrices\n",
        "\n",
        "It's time to flex those tensorflow muscles. Using the ops we have seen so far, let us\n",
        "reconsider the distribution of the   \n",
        "determinant. As it happens, mathematicians focus a lot more on random matrices whose\n",
        "entries are either -1 or 1.  \n",
        "They worry about questions regarding the singularity of a random Bernoulli matrix.\n",
        "\n",
        "In this exercise, you are asked to generate random matrices whose entries are either +1 or -1  \n",
        "with probability p for +1 (and 1-p for -1). \n",
        "The function *bernoulli_matrix_sample* needs to return a tensor of such  \n",
        "matrices.\n",
        "\n",
        "Once that is done, you can run the rest of the code to see the plot of the empirical distribution for the determinant.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7O0q0AzcQvPn"
      },
      "outputs": [],
      "source": [
        "#@title Imports And Setup: Run Me First!\n",
        "\n",
        "\n",
        "import seaborn as sns\n",
        "sns.set(color_codes=True)\n",
        "\n",
        "def plot_det_distribution(sample_tensor, p=0.5):\n",
        "  \"\"\"Plots the distribution of the determinant of the supplied tensor.\n",
        "  \n",
        "  Computes the determinant of the supplied sample of matrices and plots its\n",
        "  histogram.\n",
        "  \n",
        "  Args:\n",
        "    sample_tensor: A tensor of shape [sample_size, n, n].\n",
        "    p: The probability of generating a +1. Used only for display.\n",
        "  \n",
        "  Returns:\n",
        "    The mean and the variance of the determinant sample as a tuple.\n",
        "  \"\"\"\n",
        "\n",
        "  dets_sample = tf.linalg.det(sample_tensor)\n",
        "  dets_uniq, _, counts = tf.unique_with_counts(dets_sample)\n",
        "  dets_mean = tf.reduce_mean(dets_sample)\n",
        "  dets_var = tf.reduce_mean(tf.square(dets_sample)) - tf.square(dets_mean)\n",
        "  \n",
        "  # Get values from the eager tensors\n",
        "  det_vals, count_vals, mean, var = (dets_uniq.numpy(),\n",
        "                                     counts.numpy(),\n",
        "                                     dets_mean.numpy(),\n",
        "                                     dets_var.numpy())\n",
        "  num_bins = min(len(det_vals), 50)\n",
        "  \n",
        "  plt.hist(det_vals, num_bins, weights=count_vals, density=1, facecolor='green', \n",
        "           alpha=0.75)\n",
        "  plt.xlabel('Det(Bern(p=%.2g))' % p)\n",
        "  plt.ylabel('Probability')\n",
        "  plt.title(r'$\\mathrm{Determinant\\ Distribution:}\\ \\mu = %.2g,\\ \\sigma^2=%.2g$'\n",
        "            % (mean, var))\n",
        "  plt.grid(True)\n",
        "  plt.show()\n",
        "  return mean, var"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-kXbtvnOrYxt"
      },
      "outputs": [],
      "source": [
        "#@title Codelab: Bernoulli Matrix Distribution\n",
        "\n",
        "# NB: Run the Setup and Imports above first.\n",
        "\n",
        "def bernoulli_matrix_sample(n, size, p=0.5):\n",
        "  \"\"\"Generate a sample of matrices with entries +1 or -1.\n",
        "  \n",
        "  Generates matrices whose elements are independently drawn from {-1, 1}\n",
        "  with probability {1-p, p} respectively.\n",
        "  \n",
        "  Args:\n",
        "    n: The dimension of the (square) matrix to generate. An integer.\n",
        "    size: The number of samples to generate.\n",
        "    p: The probability of drawing +1.\n",
        "\n",
        "  Returns:\n",
        "    A tf.Tensor object of shape [size, n, n] and data type float64.\n",
        "  \"\"\"\n",
        "  # Tensorflow provides a number of distributions to generate random tensors.\n",
        "  # This includes uniform, normal and gamma. The Tensorflow API docs are an\n",
        "  # excellent reference for this and many other topics.\n",
        "  # https://www.tensorflow.org/api_docs/python/tf/random\n",
        "  # \n",
        "  # Unfortunately, however, there is no bernoulli sampler in base tensorflow.\n",
        "  # There is one in one of the libraries but we will discuss that later.\n",
        "  # For now, you need to use a uniform sampler to generate the desired sample.\n",
        "  gen_shape = [size, n, n]\n",
        "  draws = tf.random.uniform(shape=gen_shape, dtype=tf.float64)\n",
        "  ones = tf.ones(shape=gen_shape)\n",
        "  raise NotImplementedError()\n",
        "  \n",
        "\n",
        "prob_1 = 0.5\n",
        "sample = bernoulli_matrix_sample(5, 1000000, prob_1)\n",
        "plot_det_distribution(sample, prob_1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "vxtGJwgA39Hs"
      },
      "outputs": [],
      "source": [
        "#@title Codelab Solution: Bernoulli Matrix Distribution - Double click to reveal\n",
        "# NB: Run the Setup and Imports above first.\n",
        "\n",
        "@tf.function\n",
        "def bernoulli_matrix_sample(n, size, p=0.5):\n",
        "  \"\"\"Generate a sample of matrices with entries +1 or -1.\n",
        "  \n",
        "  Generates matrices whose elements are independently drawn from {-1, 1}\n",
        "  with probability {1-p, p} respectively.\n",
        "  \n",
        "  Args:\n",
        "    n: The dimension of the (square) matrix to generate. An integer.\n",
        "    size: The number of samples to generate.\n",
        "    p: The probability of drawing +1.\n",
        "\n",
        "  Returns:\n",
        "    A tf.Tensor object of shape [size, n, n] and data type float64.\n",
        "  \"\"\"\n",
        "\n",
        "  # Tensorflow provides a number of distributions to generate random tensors.\n",
        "  # This includes uniform, normal and gamma. The Tensorflow API docs are an\n",
        "  # excellent reference for this and many other topics.\n",
        "  # https://www.tensorflow.org/api_docs/python/tf/random\n",
        "  # \n",
        "  # Unfortunately, however, there is no bernoulli sampler in base tensorflow.\n",
        "  # There is one in one of the libraries but we will discuss that later.\n",
        "  # For now, you need to use a uniform sampler to generate the desired sample.\n",
        "\n",
        "  gen_shape = [size, n, n]\n",
        "  ones = tf.ones(shape=gen_shape)\n",
        "  draws = tf.random.uniform(shape=gen_shape, dtype=tf.float64)\n",
        "  return tf.where(draws \u003c= p, ones, -ones)\n",
        "\n",
        "prob_1 = 0.5\n",
        "sample = bernoulli_matrix_sample(5, 1000000, prob_1)\n",
        "plot_det_distribution(sample, prob_1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LLXPws4VMQXP"
      },
      "source": [
        "# Control Flow\n",
        "## tf.cond\n",
        "\n",
        "The guide below is relevant only for the deferred execution mode (i.e., either with\n",
        "disabled eager execution or inside `tf.function`). For the sake of determinism\n",
        "assume that the eager mode is disabled.\n",
        "```python\n",
        "tf.compat.v1.disable_eager_execution()\n",
        "```\n",
        "In Python (like in most imperative languages), we have the *if-else* construct which\n",
        "allows us to do different things based on the value of some variable.\n",
        "\n",
        "The equivalent construct in Tensorflow is the **tf.cond** op.\n",
        "Consider the following (very contrived) example:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5Bcg2gdVPON2"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "tf.compat.v1.disable_eager_execution()\n",
        "\n",
        "# Create a vector of 10 iid normal variates.\n",
        "x = tf.random.normal([10], name=\"x\")\n",
        "\n",
        "# If the average of the absolute values of x is greater than 1, we \n",
        "# return a tensor of 0's otherwise a tensor of 1's\n",
        "# Note that the predicate must return a boolean scalar.\n",
        "w = tf.cond(tf.reduce_mean(tf.abs(x)) \u003e 1.0,\n",
        "            lambda: tf.zeros_like(x, name=\"Zeros\"),\n",
        "            lambda: tf.ones_like(x, name=\"Ones\"), name=\"w\")\n",
        "\n",
        "print(w)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LyKNu8qs7TTz"
      },
      "source": [
        "Some things to note here:\n",
        "- The predicate must be a scalar tensor (or a value convertible to a scalar tensor).\n",
        "- The two branches are provided as a Python function taking no arguments and returning\n",
        "  one or more tensors\n",
        "- Both branches must return the same number and type of tensors.\n",
        "- The evaluation model is lazy. The branch not taken is not evaluated.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AYgST51T5xzb"
      },
      "source": [
        "# Inputting Data\n",
        "\n",
        "So far we have used data that we generated on the fly. Real world problems typically come with external data sources.\n",
        "\n",
        "If the data set is of small to medium size, we can load it into the python session using the usual file APIs. \n",
        "\n",
        "If we are using a Tensorflow pipeline to process this data, we need to feed this data in somehow. \n",
        "\n",
        "Tensorflow provides a couple of mechanisms to do this.\n",
        "\n",
        "The simplest way is through the feeding mechanism which we consider first.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7TCleoAT57Za"
      },
      "source": [
        "## Feed Mechanism\n",
        "\n",
        "\n",
        "We have seen that Tensorflow computation is basically graph evaluation. Tensorflow allows you to\n",
        "\"cut\" the graph at some edge and replace the tensor on that edge with some value that you can \"feed\".\n",
        "\n",
        "This can be done with any tensor, whether they are constants or variables. You do this by passing an\n",
        "override value for that tensor when doing Session.run() through an argument called \"feed_dict\".\n",
        "\n",
        "Let's consider an example"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-OqVLda_SQGX"
      },
      "outputs": [],
      "source": [
        "g = tf.Graph()\n",
        "with g.as_default():\n",
        "  # Build a simple graph.\n",
        "  x = tf.constant(4.0)\n",
        "\n",
        "  # y = √x\n",
        "  y = tf.sqrt(x)\n",
        "\n",
        "  # z = x^2\n",
        "  z = tf.square(x)\n",
        "\n",
        "  # w = √x + x^2\n",
        "  w = y + z\n",
        "\n",
        "\n",
        "with tf.compat.v1.Session(graph=g) as sess:\n",
        "  print(\"W by default: %s\\n\" % sess.run(w))\n",
        "\n",
        "  # By default y should evaluate to sqrt(4) = 2. \n",
        "  # We cut that part of the graph and set y to 10.\n",
        "  print(\"(W|y=10) = %s\" % sess.run(w, feed_dict={y: 10.0}))\n",
        "  \n",
        "  # You can also replace z at the same time.\n",
        "  print(\"(W|y=10, z=1) = %s\" % sess.run(w, feed_dict={y: 10.0, z: 1.0}))\n",
        "  \n",
        "  # At this point, you can generate the values to be fed in any manner\n",
        "  # you like, including calling functions.\n",
        "  print(\"(W|y=random,z=1) = %s\" % sess.run(\n",
        "      w, feed_dict={y: np.random.rand(), z: 1.0}))\n",
        "  \n",
        "  # What you cannot do, however, is supply a value which would be inconsistent\n",
        "  # with the expected shape or type of the original tensor. This is true even\n",
        "  # if you stay consistent with the relevant bit of the graph.\n",
        "  # In this (non-)example, we attempt to replace both y and z with a vector\n",
        "  # and Tensorflow doesn't like that.\n",
        "  \n",
        "  #print(\"(W|y=[random],z=[1])=%s\" % sess.run(w,feed_dict={y: [0.0], z: [1.0]}))\n",
        "  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9S4easdKcS_p"
      },
      "source": [
        "* So we see that while we can replace the value of any tensor, we cannot change the shape or the type of the tensor.\n",
        "* The feed value must be concrete object and not a tensor. So, python lists, numpy arrays or scalars are OK."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VolnQaKMVkRw"
      },
      "source": [
        "## Placeholders\n",
        "\n",
        "**Placeholders are relevant only in deferred execution**. When using `tf.function`,\n",
        "the inputs to the function become the placeholders.\n",
        "\n",
        "* The feed mechanism is a convenient, if somewhat *ad hoc* way to input data. \n",
        "* While you can replace anything, it is not usually a good idea to replace arbitrary tensors except for debugging.\n",
        "* Tensorflow provides **tf.compat.v1.placeholder** objects whose only job is to be fed data.\n",
        "* They can be bound to data only at run time.\n",
        "* They are defined by their shape and data type. At run time they expect to be fed a concrete\n",
        "  object of that shape and type.\n",
        "* It is an error to not supply a required placeholder (though there is a way to specify defaults).\n",
        "\n",
        "Let us see them in action:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5vz5WA7FaJrm"
      },
      "outputs": [],
      "source": [
        "g = tf.Graph()\n",
        "\n",
        "with g.as_default():\n",
        "  # Define a placeholder. You need to define its type and shape and these will be\n",
        "  # enforced when you supply the data.\n",
        "  x = tf.compat.v1.placeholder(tf.float32, shape=(10, 10))  # A square matrix\n",
        "\n",
        "  y = tf.linalg.det(x)\n",
        "\n",
        "with tf.compat.v1.Session(graph=g) as sess:\n",
        "  value_to_feed = np.random.rand(10, 10)\n",
        "  print(sess.run(y, feed_dict={x: value_to_feed}))\n",
        "\n",
        "  # You can check that if you do not feed the value of x, you get an error.\n",
        "  #sess.run(y)  ## InvalidArgumentError"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aohpV9d_n0nJ"
      },
      "source": [
        "## Shapes Revisited\n",
        "\n",
        "### The Problem\n",
        "* Placeholders are commonly used as a slot where you can enter your data for training.\n",
        "* Data is typically supplied in batches suitable for use with stochastic gradient descent or some variant thereof.\n",
        "* Pretty inconvenient to hard code the batch size.\n",
        "* But placeholder definition requires a shape!\n",
        "\n",
        "### The Solution\n",
        "* Allow shapes which are potentially unknown at graph building time but will be known at run time.\n",
        "* This is done by setting one or more dimensions in a shape to None.\n",
        "* For example, a shape of [None, 4] indicates that we plan to have a matrix with 4 columns but some unknown number of rows.\n",
        "* An obvious point: constants cannot be defined with unknown shape.  \n",
        "\n",
        "Let's look at some examples with partially specified shapes for placeholders."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Jva5ZjzcqdGs"
      },
      "outputs": [],
      "source": [
        "g = tf.Graph()\n",
        "with g.as_default():\n",
        "  # Defines a placeholder with unknown number of rows and 2 columns.\n",
        "  x = tf.compat.v1.placeholder(tf.float32, shape=[None, 2])\n",
        "\n",
        "  # You can do almost everything that you can do with a fully specified shape\n",
        "  # tensor. Here we compute the sum of squares of elements of x.\n",
        "  y = tf.reduce_sum(x * x)\n",
        "\n",
        "with tf.compat.v1.Session(graph=g) as sess:\n",
        "  # When evaluating, you can specify any value of x compatible with the shape\n",
        "\n",
        "  # A 2 x 2 matrix is OK\n",
        "  print(\"2x2 input: %s\" % sess.run(y, feed_dict={x: [[1, 2], [3, 4]]}))\n",
        "  \n",
        "  # A 3 x 2 matrix is also OK\n",
        "  print(\"3x2 input: %s\" % sess.run(y, feed_dict={x: [[1, 2], [3, 4], [5, 6]]}))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PhErwoaDvSPQ"
      },
      "source": [
        "* This seems absolutely awesome, so is there a downside to this?\n",
        "* Yes! \n",
        "  * Unspecified shapes allow you to write ops which may fail at run time even though  \n",
        "    they seem OK at graph building time as the following example demonstrates.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qyxGN3EsvQcR"
      },
      "outputs": [],
      "source": [
        "# Continuation of the previous example. Run that first.\n",
        "with g.as_default():\n",
        "  # This seems OK because while a shape of [None, 2] is not always square, it\n",
        "  # could be square. So Tensorflow is OK with it.\n",
        "  z = tf.linalg.det(x * x)\n",
        "\n",
        "with tf.compat.v1.Session(graph=g) as sess:\n",
        "  # With a 2x2 matrix we have no problem evaluating z\n",
        "  print(\"Det([2x2]): %s\" % sess.run(z, feed_dict={x:[[1, 2], [3, 4]]}))\n",
        "\n",
        "  # But with 3x2 matrix we obviously get an error\n",
        "  #print(\"Det([3x2]): %s\" % sess.run(z, feed_dict={x:[[1, 2], [3, 4], [1, 4]]}))\n",
        "  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C__U_b_ixoLL"
      },
      "source": [
        "### tf.shape vs tensor.get_shape\n",
        "\n",
        "Earlier we encountered two different ways to get the shape of a tensor.\n",
        "Now we can see the difference between these two. \n",
        "\n",
        "* **tensor.get_shape()**: Returns the statically determined shape of a tensor. It is possible that this is only partially known.\n",
        "* **tf.shape(tensor)**: Returns the **actual** fully specified shape of the tensor but is guaranteed to be known only at run time.\n",
        "\n",
        "Let's see the difference in action:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WJ3DeMGxyk7Y"
      },
      "outputs": [],
      "source": [
        "g = tf.Graph()\n",
        "with g.as_default():\n",
        "  x = tf.compat.v1.placeholder(tf.int32, [None, None])\n",
        "  # This is a tensor so we have to evaluate it to get its value.\n",
        "  x_s = tf.shape(x)\n",
        "\n",
        "with tf.compat.v1.Session(graph=g) as sess:\n",
        "  print(\"Static shape of x: %s\" % x.get_shape())\n",
        "  print(\"Runtime shape of x: %s\" % sess.run(x_s, feed_dict={x: [[1],[2]]}))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NLIeB4TR6FMF"
      },
      "source": [
        "## Reading Files\n",
        "\n",
        "* While data can be fed in through placeholders, it would be still more efficient if we could just ask Tensorflow to directly read from data files.\n",
        "\n",
        "* There is a large, well developed framework in TF to do this. \n",
        "\n",
        "* To get an idea of the steps involved, tensorflow.org has this to say about it:\n",
        "\n",
        "\n",
        "\n",
        "\u003e A typical pipeline for reading records from files has the following stages:\n",
        "\u003e 1.   The list of filenames\n",
        "1.   Optional filename shuffling\n",
        "1.   Optional epoch limit\n",
        "1.   Filename queue\n",
        "1.   A Reader for the file format\n",
        "1.   A decoder for a record read by the reader\n",
        "1.   Optional preprocessing\n",
        "1.   Example queue\n",
        "\n",
        "\n",
        "* However, if you are not setting up a large scale distributed tensorflow job, you can get away with using standard python IO along with placeholders.\n",
        "\n",
        "In the following example, we read a small csv StringIO object using numpy and bind the data to a placeholder.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6F9rcP1-LoS2"
      },
      "outputs": [],
      "source": [
        "\n",
        "# We'll use StringIO (to avoid external file handling in colab) to fake a CSV \n",
        "# file containing two integer columns labeled x and y. In reality, you'd be \n",
        "# using something like \n",
        "# with open(\"path/to/csv_file\") as csv_file:\n",
        "import numpy as np\n",
        "from io import StringIO\n",
        "csv_file = StringIO(u\"\"\"x,y\n",
        "0,1\n",
        "1,2\n",
        "2,4\n",
        "3,8\n",
        "4,16\n",
        "5,32\"\"\")\n",
        "\n",
        "g = tf.Graph()\n",
        "with g.as_default():\n",
        "  x = tf.compat.v1.placeholder(tf.int32, shape=(None))\n",
        "  y = tf.compat.v1.placeholder(tf.int32, shape=(None))\n",
        "\n",
        "  z = x + y\n",
        "\n",
        "# There are many ways to read the data in using standard python utilities.\n",
        "# Here we use the numpy method to directly read into a numpy array.\n",
        "data = np.genfromtxt(csv_file, dtype='int32', delimiter=',', skip_header=True)\n",
        "\n",
        "print(\"x: %s\" % data[:, 0])\n",
        "print(\"y: %s\" % data[:, 1])\n",
        "\n",
        "# Now we can evaluate the tensor z using the loaded data to replace the \n",
        "# placeholders x and y\n",
        "with tf.compat.v1.Session(graph=g) as sess:\n",
        "  print(\"z: %s\" % sess.run(z, feed_dict={x: data[:,0], y: data[:, 1]}))"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "nLMPM1NxwNas"
      ],
      "last_runtime": {
        "build_target": "//learning/deepmind/dm_python:dm_notebook3",
        "kind": "private"
      },
      "name": "Introduction To Tensorflow",
      "private_outputs": true,
      "provenance": [
        {
          "file_id": "",
          "timestamp": 1568989563947
        },
        {
          "file_id": "https://github.com/google/tf-quant-finance/blob/master/tf_quant_finance/examples/jupyter_notebooks/Introduction_to_TensorFlow_Part_1_-_Basics.ipynb",
          "timestamp": 1568127057499
        },
        {
          "file_id": "0B-bJrnlfFt2cYVFNU2VnYlJQSlE",
          "timestamp": 1568027793834
        }
      ],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}