intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Going deeper with Tensorflow\n",
    "\n",
    "We will start our course with [Tensorflow](https://www.tensorflow.org/) intro.\n",
    "\n",
    "To install tensorflow on your machine run:\n",
    "\n",
    "* `pip install tensorflow` -- **cpu-only** version for Linux & Mac OSX\n",
    "* if you want GPU support try -- `pip install tensorflow-gpu`\n",
    "* also see docs here: [TF install page](https://www.tensorflow.org/install/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "from sklearn.metrics import roc_auc_score, mean_squared_error\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "gpu_options = tf.GPUOptions(allow_growth=True, per_process_gpu_memory_fraction=0.1)\n",
    "s = tf.InteractiveSession(config=tf.ConfigProto(gpu_options=gpu_options))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Lets start\n",
    "\n",
    "Implement simple function using numpy to calculate sum of squares from 0 to N-1\n",
    "\n",
    "\n",
    "**Hint:**\n",
    "use `numpy.arange(N)` to get array from 0 to N-1 inclusive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def sum_squares(N):\n",
    "    return <student.Implement_me()>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "sum_squares(10**8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Tensoflow teaser\n",
    "\n",
    "Doing the very same thing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#I gonna be your function parameter\n",
    "N = tf.placeholder('int64', name=\"input_to_your_function\")\n",
    "\n",
    "#i am a recipe on how to produce sum of squares of arange of N given N\n",
    "result = tf.reduce_sum((tf.range(N)**2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "662921401752298880\n",
      "CPU times: user 1.24 s, sys: 2.78 s, total: 4.02 s\n",
      "Wall time: 3.19 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#example of computing the same as sum_squares\n",
    "print(result.eval({N:10**8}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How does it work?\n",
    "1. define placeholders where you'll send inputs;\n",
    "2. make symbolic graph: a recipe for mathematical transformation of those placeholders;\n",
    "3. compute outputs of your graph with particular values for each placeholder\n",
    "  * output.eval({placeholder:value}) \n",
    "  * s.run(output, {placeholder:value})\n",
    "\n",
    "* So far there are two main entities: \"placeholder\" and \"transformation\"\n",
    "* Both can be numbers, vectors, matrices, tensors, etc.\n",
    "* Both can be int32/64, floats of booleans (uint8) of various size.\n",
    "\n",
    "* You can define new transformations as an arbitrary operation on placeholders and other transformations\n",
    " * tf.reduce_sum(tf.arange(N)\\**2) are 3 sequential transformations of placeholder N\n",
    " * There's a tensorflow symbolic version for every numpy function\n",
    "   * `a+b, a/b, a**b, ...` behave just like in numpy\n",
    "   * np.mean -> tf.reduce_mean\n",
    "   * np.arange -> tf.range\n",
    "   * np.cumsum -> tf.cumsum\n",
    "   * If if you can't find the op you need, see the [docs](https://www.tensorflow.org/api_docs/python).\n",
    " \n",
    " \n",
    "Still confused? We gonna fix that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Default placeholder that can be arbitrary float32 scalar, vertor, matrix, etc.\n",
    "arbitrary_input = tf.placeholder('float32')\n",
    "\n",
    "#Input vector of arbitrary length\n",
    "input_vector = tf.placeholder('float32',shape=(None,))\n",
    "\n",
    "#Input vector that _must_ have 10 elements and integer type\n",
    "fixed_vector = tf.placeholder('int32',shape=(10,))\n",
    "\n",
    "#Matrix of arbitrary n_rows and 15 columns (e.g. a minibatch your data table)\n",
    "input_matrix = tf.placeholder('float32',shape=(None,15))\n",
    "\n",
    "#You can generally use None whenever you don't need a specific shape\n",
    "input1 = tf.placeholder('float64',shape=(None,100,None))\n",
    "input2 = tf.placeholder('int32',shape=(None,None,3,224,224))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#elementwise multiplication\n",
    "double_the_vector = input_vector*2\n",
    "\n",
    "#elementwise cosine\n",
    "elementwise_cosine = tf.cos(input_vector)\n",
    "\n",
    "#difference between squared vector and vector itself\n",
    "vector_squares = input_vector**2 - input_vector\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[1 point max]**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Practice time: create two vectors of type float32\n",
    "my_vector = <student.init_float32_vector()>\n",
    "my_vector2 = <student.init_one_more_such_vector()>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Write a transformation(recipe):\n",
    "#(vec1)*(vec2) / (sin(vec1) +1)\n",
    "my_transformation = <student.implementwhatwaswrittenabove()>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(my_transformation)\n",
    "#it's okay, it's a symbolic graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#\n",
    "dummy = np.arange(5).astype('float32')\n",
    "\n",
    "my_transformation.eval({my_vector:dummy,my_vector2:dummy[::-1]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing graphs\n",
    "\n",
    "It's often useful to visualize the computation graph when debugging or optimizing. \n",
    "Interactive visualization is where tensorflow really shines as compared to other frameworks. \n",
    "\n",
    "There's a special instrument for that, called Tensorboard. You can launch it from console:\n",
    "\n",
    "```tensorboard --logdir=/tmp/tboard --port=7007```\n",
    "\n",
    "If you're pathologically afraid of consoles, try this:\n",
    "\n",
    "```os.system(\"tensorboard --logdir=/tmp/tboard --port=7007 &\"```\n",
    "\n",
    "_(but don't tell anyone we taught you that)_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# launch tensorflow the ugly way, uncomment if you need that\n",
    "import os\n",
    "port = 7007\n",
    "\n",
    "#!killall tensorboard\n",
    "os.system(\"tensorboard --logdir=./tboard --port=%d &\" % port)\n",
    "\n",
    "# show graph to tensorboard\n",
    "writer = tf.summary.FileWriter(\"./tboard\", graph=tf.get_default_graph())\n",
    "writer.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One basic functionality of tensorboard is drawing graphs. One you've run the cell above, go to `localhost:7007` in your browser and switch to _graphs_ tab in the topbar. \n",
    "\n",
    "Here's what you should see:\n",
    "\n",
    "<img src=\"https://www.tensorflow.org/images/graph_vis_animation.gif\" width=780>\n",
    "\n",
    "Tensorboard also allows you to draw graphs (e.g. learning curves), record images & audio ~~and play flash games~~. This is useful when monitoring learning progress and catching some training issues.\n",
    "\n",
    "One researcher said:\n",
    "```\n",
    "If you spent last four hours of your worktime watching as your algorithm prints numbers and draws figures, you're probably doing deep learning wrong.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can read more on tensorboard usage [here](https://www.tensorflow.org/get_started/graph_viz)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Do It Yourself\n",
    "\n",
    "__[2 points max]__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Quest #1 - implement a function that computes a mean squared error of two input vectors\n",
    "# Your function has to take 2 vectors and return a single number\n",
    "\n",
    "<student.define_inputs_and_transformations()>\n",
    "\n",
    "mse =<student.define_transformation()>\n",
    "\n",
    "compute_mse = lambda vector1, vector2: <how to run you graph?>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Tests\n",
    "for n in [1,5,10,10**3]:\n",
    "    \n",
    "    elems = [np.arange(n),np.arange(n,0,-1), np.zeros(n),\n",
    "             np.ones(n),np.random.random(n),np.random.randint(100,size=n)]\n",
    "    \n",
    "    for el in elems:\n",
    "        for el_2 in elems:\n",
    "            true_mse = np.array(mean_squared_error(el,el_2))\n",
    "            my_mse = compute_mse(el,el_2)\n",
    "            if not np.allclose(true_mse,my_mse):\n",
    "                print('Wrong result:')\n",
    "                print('mse(%s,%s)' % (el,el_2))\n",
    "                print(\"should be: %f, but your function returned %f\" % (true_mse,my_mse))\n",
    "                raise ValueError(\"Что-то не так\")\n",
    "\n",
    "print(\"All tests passed\")    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# variables\n",
    "\n",
    "The inputs and transformations have no value outside function call. This isn't too comfortable if you want your model to have parameters (e.g. network weights) that are always present, but can change their value over time.\n",
    "\n",
    "Tensorflow solves this with `tf.Variable` objects.\n",
    "* You can assign variable a value at any time in your graph\n",
    "* Unlike placeholders, there's no need to explicitly pass values to variables when `s.run(...)`-ing\n",
    "* You can use variables the same way you use transformations \n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#creating shared variable\n",
    "shared_vector_1 = tf.Variable(initial_value=np.ones(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#initialize variable(s) with initial values\n",
    "s.run(tf.global_variables_initializer())\n",
    "\n",
    "#evaluating shared variable (outside symbolicd graph)\n",
    "print(\"initial value\", s.run(shared_vector_1))\n",
    "\n",
    "# within symbolic graph you use them just as any other inout or transformation, not \"get value\" needed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#setting new value\n",
    "s.run(shared_vector_1.assign(np.arange(5)))\n",
    "\n",
    "#getting that new value\n",
    "print(\"new value\", s.run(shared_vector_1))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# tf.gradients - why graphs matter\n",
    "* Tensorflow can compute derivatives and gradients automatically using the computation graph\n",
    "* Gradients are computed as a product of elementary derivatives via chain rule:\n",
    "\n",
    "$$ {\\partial f(g(x)) \\over \\partial x} = {\\partial f(g(x)) \\over \\partial g(x)}\\cdot {\\partial g(x) \\over \\partial x} $$\n",
    "\n",
    "It can get you the derivative of any graph as long as it knows how to differentiate elementary operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "my_scalar = tf.placeholder('float32')\n",
    "\n",
    "scalar_squared = my_scalar**2\n",
    "\n",
    "#a derivative of scalar_squared by my_scalar\n",
    "derivative = tf.gradients(scalar_squared, my_scalar)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = np.linspace(-3,3)\n",
    "x_squared, x_squared_der = s.run([scalar_squared,derivative],\n",
    "                                 {my_scalar:x})\n",
    "\n",
    "plt.plot(x, x_squared,label=\"x^2\")\n",
    "plt.plot(x, x_squared_der, label=\"derivative\")\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Why that rocks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[2 points max]**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "my_vector = tf.placeholder('float32',[None])\n",
    "\n",
    "#Compute the gradient of the next weird function over my_scalar and my_vector\n",
    "#warning! Trying to understand the meaning of that function may result in permanent brain damage\n",
    "\n",
    "weird_psychotic_function = tf.reduce_mean((my_vector+my_scalar)**(1+tf.nn.moments(my_vector,[0])[1]) + 1./ tf.atan(my_scalar))/(my_scalar**2 + 1) + 0.01*tf.sin(2*my_scalar**1.5)*(tf.reduce_sum(my_vector)* my_scalar**2)*tf.exp((my_scalar-4)**2)/(1+tf.exp((my_scalar-4)**2))*(1.-(tf.exp(-(my_scalar-4)**2))/(1+tf.exp(-(my_scalar-4)**2)))**2\n",
    "\n",
    "der_by_scalar = <student.compute_grad_over_scalar()>\n",
    "der_by_vector = <student.compute_grad_over_vector()>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Plotting your derivative\n",
    "scalar_space = np.linspace(1, 7, 100)\n",
    "\n",
    "y = [s.run(weird_psychotic_function, {my_scalar:x, my_vector:[1, 2, 3]})\n",
    "     for x in scalar_space]\n",
    "\n",
    "plt.plot(scalar_space, y, label='function')\n",
    "\n",
    "y_der_by_scalar = [s.run(der_by_scalar, {my_scalar:x, my_vector:[1, 2, 3]})\n",
    "     for x in scalar_space]\n",
    "\n",
    "plt.plot(scalar_space, y_der_by_scalar, label='derivative')\n",
    "plt.grid()\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Almost done - optimizers\n",
    "\n",
    "While you can perform gradient descent by hand with automatic grads from above, tensorflow also has some optimization methods implemented for you.\n",
    "\n",
    "You can try different optimizers (eg. GradientDescentOptimizer, MomentumOptimizer, AdamOptimizer) and combine their trajectories on one plot (don't forget the legend). **[1 extra point]**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_guess = tf.Variable(np.zeros(2,dtype='float32'))\n",
    "y_true = tf.range(1,3,dtype='float32')\n",
    "\n",
    "loss = tf.reduce_mean((y_guess - y_true + tf.random_normal([2]))**2) \n",
    "\n",
    "optimizer = tf.train.MomentumOptimizer(0.01,0.9).minimize(loss,var_list=y_guess)\n",
    "\n",
    "#same, but more detailed:\n",
    "#updates = [[tf.gradients(loss,y_guess)[0], y_guess]]\n",
    "#optimizer = tf.train.MomentumOptimizer(0.01,0.9).apply_gradients(updates)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from IPython.display import clear_output\n",
    "\n",
    "s.run(tf.global_variables_initializer())\n",
    "\n",
    "guesses = [s.run(y_guess)]\n",
    "\n",
    "for _ in range(100):\n",
    "    s.run(optimizer)\n",
    "    guesses.append(s.run(y_guess))\n",
    "    \n",
    "    clear_output(True)\n",
    "    plt.plot(*zip(*guesses),marker='.')\n",
    "    plt.scatter(*s.run(y_true),c='red')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic regression example\n",
    "Implement the regular logistic regression training algorithm\n",
    "\n",
    "Tips:\n",
    "* Use a shared variable for weights\n",
    "* X and y are potential inputs\n",
    "* Compile 2 functions:\n",
    " * `train_function(X, y)` - returns error and computes weights' new values __(through updates)__\n",
    " * `predict_fun(X)` - just computes probabilities (\"y\") given data\n",
    " \n",
    " \n",
    "We shall train on a two-class MNIST dataset\n",
    "* please note that target `y` are `{0,1}` and not `{-1,1}` as in some formulae"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[5 points max]**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_digits\n",
    "mnist = load_digits(2)\n",
    "\n",
    "X,y = mnist.data, mnist.target\n",
    "\n",
    "print(\"y [shape - %s]:\" % (str(y.shape)), y[:10])\n",
    "print(\"X [shape - %s]:\" % (str(X.shape)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print('X:\\n',X[:3,:10])\n",
    "print('y:\\n',y[:10])\n",
    "plt.imshow(X[0].reshape([8,8]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# inputs and shareds\n",
    "weights = <student.code_variable()>\n",
    "input_X = <student.code_placeholder()>\n",
    "input_y = <student.code_placeholder()>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "predicted_y = <predicted probabilities for input_X>\n",
    "loss = <logistic loss (scalar, mean over sample)>\n",
    "\n",
    "optimizer = <optimizer that minimizes loss>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_function = <compile function that takes X and y, returns log loss and updates weights>\n",
    "predict_function = <compile function that takes X and computes probabilities of y>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for i in range(5):\n",
    "    <run optimizer operation>\n",
    "    loss_i = <compute loss at iteration i>\n",
    "    \n",
    "    print(\"loss at iter %i:%.4f\" % (i, loss_i))\n",
    "    \n",
    "    print(\"train auc:\",roc_auc_score(y_train, predict_function(X_train)))\n",
    "    print(\"test auc:\",roc_auc_score(y_test, predict_function(X_test)))\n",
    "\n",
    "    \n",
    "print (\"resulting weights:\")\n",
    "plt.imshow(weights.eval().reshape([8,8]))\n",
    "plt.colorbar()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}