extra_tensorflow_reproducibility.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow Reproducibility"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/ageron/.virtualenvs/ml/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "from __future__ import division, print_function, unicode_literals\n",
    "\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Checklist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Do not run TensorFlow on the GPU.\n",
    "2. Beware of multithreading, and make TensorFlow single-threaded.\n",
    "3. Set all the random seeds.\n",
    "4. Eliminate any other source of variability."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Do Not Run TensorFlow on the GPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some operations (like `tf.reduce_sum()`) have favor performance over precision, and their outputs may vary slightly across runs. To get reproducible results, make sure TensorFlow runs on the CPU:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Beware of Multithreading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because floats have limited precision, the order of execution matters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.4285714285714286"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "2. * 5. / 7."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.4285714285714284"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "2. / 7. * 5."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should make sure TensorFlow runs your ops on a single thread:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = tf.ConfigProto(intra_op_parallelism_threads=1,\n",
    "                        inter_op_parallelism_threads=1)\n",
    "\n",
    "with tf.Session(config=config) as sess:\n",
    "    #... this will run single threaded\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The thread pools for all sessions are created when you create the first session, so all sessions in the rest of this notebook will be single-threaded:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.Session() as sess:\n",
    "    #... also single-threaded!\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set all the random seeds!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Python's built-in `hash()` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n', 'k', 'l', 'h', 'r', 'a', 'i', 't', 'd', 's', 'g', 'T', ' ', 'y', 'e', 'u'}\n",
      "{'n', 'k', 'l', 'h', 'r', 'a', 'i', 't', 'd', 's', 'g', 'T', ' ', 'y', 'e', 'u'}\n"
     ]
    }
   ],
   "source": [
    "print(set(\"Try restarting the kernel and running this again\"))\n",
    "print(set(\"Try restarting the kernel and running this again\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since Python 3.3, the result will be different every time, unless you start Python with the `PYTHONHASHSEED` environment variable set to `0`:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```shell\n",
    "PYTHONHASHSEED=0 python\n",
    "```\n",
    "\n",
    "```pycon\n",
    ">>> print(set(\"Now the output is stable across runs\"))\n",
    "{'n', 'b', 'h', 'o', 'i', 'a', 'r', 't', 'p', 'N', 's', 'c', ' ', 'l', 'e', 'w', 'u'}\n",
    ">>> exit()\n",
    "```\n",
    "\n",
    "```shell\n",
    "PYTHONHASHSEED=0 python\n",
    "```\n",
    "```pycon\n",
    ">>> print(set(\"Now the output is stable across runs\"))\n",
    "{'n', 'b', 'h', 'o', 'i', 'a', 'r', 't', 'p', 'N', 's', 'c', ' ', 'l', 'e', 'w', 'u'}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, you could set this environment variable system-wide, but that's probably not a good idea, because this automatic randomization was [introduced for security reasons](http://ocert.org/advisories/ocert-2011-003.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, setting the environment variable from within Python (e.g., using `os.environ[\"PYTHONHASHSEED\"]=\"0\"`) will not work, because Python reads it upon startup. For Jupyter notebooks, you have to start the Jupyter server like this:\n",
    "\n",
    "```shell\n",
    "PYTHONHASHSEED=0 jupyter notebook\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "if os.environ.get(\"PYTHONHASHSEED\") != \"0\":\n",
    "    raise Exception(\"You must set PYTHONHASHSEED=0 when starting the Jupyter server to get reproducible results.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Python Random Number Generators (RNGs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6394267984578837\n",
      "0.025010755222666936\n",
      "\n",
      "0.6394267984578837\n",
      "0.025010755222666936\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "random.seed(42)\n",
    "print(random.random())\n",
    "print(random.random())\n",
    "\n",
    "print()\n",
    "\n",
    "random.seed(42)\n",
    "print(random.random())\n",
    "print(random.random())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NumPy RNGs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.3745401188473625\n",
      "0.9507143064099162\n",
      "\n",
      "0.3745401188473625\n",
      "0.9507143064099162\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "np.random.seed(42)\n",
    "print(np.random.rand())\n",
    "print(np.random.rand())\n",
    "\n",
    "print()\n",
    "\n",
    "np.random.seed(42)\n",
    "print(np.random.rand())\n",
    "print(np.random.rand())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TensorFlow RNGs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TensorFlow's behavior is more complex because of two things:\n",
    "* you create a graph, and then you execute it. The random seed must be set before you create the random operations.\n",
    "* there are two seeds: one at the graph level, and one at the individual random operation level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.63789964\n",
      "0.8774011\n",
      "\n",
      "0.63789964\n",
      "0.8774011\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "tf.set_random_seed(42)\n",
    "rnd = tf.random_uniform(shape=[])\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(rnd.eval())\n",
    "    print(rnd.eval())\n",
    "\n",
    "print()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(rnd.eval())\n",
    "    print(rnd.eval())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Every time you reset the graph, you need to set the seed again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.63789964\n",
      "0.8774011\n",
      "\n",
      "0.63789964\n",
      "0.8774011\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "tf.set_random_seed(42)\n",
    "rnd = tf.random_uniform(shape=[])\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(rnd.eval())\n",
    "    print(rnd.eval())\n",
    "\n",
    "print()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(rnd.eval())\n",
    "    print(rnd.eval())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you create your own graph, it will ignore the default graph's seed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5718187\n",
      "0.6233171\n",
      "\n",
      "0.32140207\n",
      "0.46593904\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "tf.set_random_seed(42)\n",
    "\n",
    "graph = tf.Graph()\n",
    "with graph.as_default():\n",
    "    rnd = tf.random_uniform(shape=[])\n",
    "\n",
    "with tf.Session(graph=graph):\n",
    "    print(rnd.eval())\n",
    "    print(rnd.eval())\n",
    "\n",
    "print()\n",
    "\n",
    "with tf.Session(graph=graph):\n",
    "    print(rnd.eval())\n",
    "    print(rnd.eval())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You must set its own seed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.63789964\n",
      "0.8774011\n",
      "\n",
      "0.63789964\n",
      "0.8774011\n"
     ]
    }
   ],
   "source": [
    "graph = tf.Graph()\n",
    "with graph.as_default():\n",
    "    tf.set_random_seed(42)\n",
    "    rnd = tf.random_uniform(shape=[])\n",
    "\n",
    "with tf.Session(graph=graph):\n",
    "    print(rnd.eval())\n",
    "    print(rnd.eval())\n",
    "\n",
    "print()\n",
    "\n",
    "with tf.Session(graph=graph):\n",
    "    print(rnd.eval())\n",
    "    print(rnd.eval())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you set the seed after the random operation is created, the seed has no effet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.087068915\n",
      "0.6322479\n",
      "\n",
      "0.17158246\n",
      "0.2868148\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "rnd = tf.random_uniform(shape=[])\n",
    "\n",
    "tf.set_random_seed(42) # BAD, NO EFFECT!\n",
    "with tf.Session() as sess:\n",
    "    print(rnd.eval())\n",
    "    print(rnd.eval())\n",
    "\n",
    "print()\n",
    "\n",
    "tf.set_random_seed(42) # BAD, NO EFFECT!\n",
    "with tf.Session() as sess:\n",
    "    print(rnd.eval())\n",
    "    print(rnd.eval())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A note about operation seeds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also set a seed for each individual random operation. When you do, it is combined with the graph seed into the final seed used by that op. The following table summarizes how this works:\n",
    "\n",
    "| Graph seed | Op seed |  Resulting seed                |\n",
    "|------------|---------|--------------------------------|\n",
    "| None       | None    | Random                         |\n",
    "| graph_seed | None    | f(graph_seed, op_index)        |\n",
    "| None       | op_seed | f(default_graph_seed, op_seed) |\n",
    "| graph_seed | op_seed | f(graph_seed, op_seed)         |\n",
    "\n",
    "* `f()` is a deterministic function.\n",
    "* `op_index = graph._last_id` when there is a graph seed, different random ops without op seeds will have different outputs. However, each of them will have the same sequence of outputs at every run.\n",
    "\n",
    "In eager mode, there is a global seed instead of graph seed (since there is no graph in eager mode)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.95227146\n",
      "0.95227146\n",
      "0.55099714\n",
      "0.8960779\n",
      "0.8960779\n",
      "0.54318357\n",
      "\n",
      "0.95227146\n",
      "0.95227146\n",
      "0.6398845\n",
      "0.8960779\n",
      "0.8960779\n",
      "0.24617589\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "rnd1 = tf.random_uniform(shape=[], seed=42)\n",
    "rnd2 = tf.random_uniform(shape=[], seed=42)\n",
    "rnd3 = tf.random_uniform(shape=[])\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(rnd1.eval())\n",
    "    print(rnd2.eval())\n",
    "    print(rnd3.eval())\n",
    "    print(rnd1.eval())\n",
    "    print(rnd2.eval())\n",
    "    print(rnd3.eval())\n",
    "\n",
    "print()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(rnd1.eval())\n",
    "    print(rnd2.eval())\n",
    "    print(rnd3.eval())\n",
    "    print(rnd1.eval())\n",
    "    print(rnd2.eval())\n",
    "    print(rnd3.eval())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following example, you may think that all random ops will have the same random seed, but `rnd3` will actually have a different seed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4163028\n",
      "0.4163028\n",
      "0.96100175\n",
      "0.033224702\n",
      "0.033224702\n",
      "0.17637014\n",
      "\n",
      "0.4163028\n",
      "0.4163028\n",
      "0.96100175\n",
      "0.033224702\n",
      "0.033224702\n",
      "0.17637014\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "tf.set_random_seed(42)\n",
    "\n",
    "rnd1 = tf.random_uniform(shape=[], seed=42)\n",
    "rnd2 = tf.random_uniform(shape=[], seed=42)\n",
    "rnd3 = tf.random_uniform(shape=[])\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(rnd1.eval())\n",
    "    print(rnd2.eval())\n",
    "    print(rnd3.eval())\n",
    "    print(rnd1.eval())\n",
    "    print(rnd2.eval())\n",
    "    print(rnd3.eval())\n",
    "\n",
    "print()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(rnd1.eval())\n",
    "    print(rnd2.eval())\n",
    "    print(rnd3.eval())\n",
    "    print(rnd1.eval())\n",
    "    print(rnd2.eval())\n",
    "    print(rnd3.eval())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Estimators API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tip**: in a Jupyter notebook, you probably want to set the random seeds regularly so that you can come back and run the notebook from there (instead of from the beginning) and still get reproducible outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "random.seed(42)\n",
    "np.random.seed(42)\n",
    "tf.set_random_seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you use the Estimators API, make sure to create a `RunConfig` and set its `tf_random_seed`, then pass it to the constructor of your estimator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Using temporary folder as model directory: /var/folders/wy/h39t6kb11pnbb0pzhksd_fqh0000gn/T/tmp2xxrubio\n",
      "INFO:tensorflow:Using config: {'_model_dir': '/var/folders/wy/h39t6kb11pnbb0pzhksd_fqh0000gn/T/tmp2xxrubio', '_tf_random_seed': 42, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x11dba7da0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
     ]
    }
   ],
   "source": [
    "my_config = tf.estimator.RunConfig(tf_random_seed=42)\n",
    "\n",
    "feature_cols = [tf.feature_column.numeric_column(\"X\", shape=[28 * 28])]\n",
    "dnn_clf = tf.estimator.DNNClassifier(hidden_units=[300, 100], n_classes=10,\n",
    "                                     feature_columns=feature_cols,\n",
    "                                     config=my_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try it on MNIST:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "(X_train, y_train), (X_test, y_test) = tf.keras.datasets.mnist.load_data()\n",
    "X_train = X_train.astype(np.float32).reshape(-1, 28*28) / 255.0\n",
    "y_train = y_train.astype(np.int32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, the `numpy_input_fn` does not allow us to set the seed when `shuffle=True`, so we must shuffle the data ourself and set `shuffle=False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into /var/folders/wy/h39t6kb11pnbb0pzhksd_fqh0000gn/T/tmp2xxrubio/model.ckpt.\n",
      "INFO:tensorflow:loss = 73.945915, step = 1\n",
      "INFO:tensorflow:global_step/sec: 348.999\n",
      "INFO:tensorflow:loss = 21.020527, step = 101 (0.287 sec)\n",
      "INFO:tensorflow:global_step/sec: 431.365\n",
      "INFO:tensorflow:loss = 8.926933, step = 201 (0.232 sec)\n",
      "INFO:tensorflow:global_step/sec: 438.11\n",
      "INFO:tensorflow:loss = 2.3184745, step = 301 (0.228 sec)\n",
      "INFO:tensorflow:global_step/sec: 437.696\n",
      "INFO:tensorflow:loss = 10.654381, step = 401 (0.228 sec)\n",
      "INFO:tensorflow:global_step/sec: 452.808\n",
      "INFO:tensorflow:loss = 4.2829914, step = 501 (0.221 sec)\n",
      "INFO:tensorflow:global_step/sec: 450.062\n",
      "INFO:tensorflow:loss = 2.497019, step = 601 (0.222 sec)\n",
      "INFO:tensorflow:global_step/sec: 451.86\n",
      "INFO:tensorflow:loss = 3.9215999, step = 701 (0.221 sec)\n",
      "INFO:tensorflow:global_step/sec: 442.86\n",
      "INFO:tensorflow:loss = 3.8031044, step = 801 (0.226 sec)\n",
      "INFO:tensorflow:global_step/sec: 444.581\n",
      "INFO:tensorflow:loss = 3.9209557, step = 901 (0.225 sec)\n",
      "INFO:tensorflow:global_step/sec: 439.603\n",
      "INFO:tensorflow:loss = 5.506338, step = 1001 (0.227 sec)\n",
      "INFO:tensorflow:global_step/sec: 444.545\n",
      "INFO:tensorflow:loss = 2.6690354, step = 1101 (0.225 sec)\n",
      "INFO:tensorflow:global_step/sec: 445.176\n",
      "INFO:tensorflow:loss = 6.559507, step = 1201 (0.225 sec)\n",
      "INFO:tensorflow:global_step/sec: 443.365\n",
      "INFO:tensorflow:loss = 5.707597, step = 1301 (0.225 sec)\n",
      "INFO:tensorflow:global_step/sec: 447.822\n",
      "<<314 more lines>>\n",
      "INFO:tensorflow:loss = 0.48648793, step = 17101 (0.227 sec)\n",
      "INFO:tensorflow:global_step/sec: 454.872\n",
      "INFO:tensorflow:loss = 0.49331194, step = 17201 (0.220 sec)\n",
      "INFO:tensorflow:global_step/sec: 443.025\n",
      "INFO:tensorflow:loss = 0.32060045, step = 17301 (0.226 sec)\n",
      "INFO:tensorflow:global_step/sec: 440.069\n",
      "INFO:tensorflow:loss = 0.13167329, step = 17401 (0.227 sec)\n",
      "INFO:tensorflow:global_step/sec: 448.211\n",
      "INFO:tensorflow:loss = 0.05688939, step = 17501 (0.223 sec)\n",
      "INFO:tensorflow:global_step/sec: 450.458\n",
      "INFO:tensorflow:loss = 0.36213198, step = 17601 (0.222 sec)\n",
      "INFO:tensorflow:global_step/sec: 428.842\n",
      "INFO:tensorflow:loss = 0.36243188, step = 17701 (0.233 sec)\n",
      "INFO:tensorflow:global_step/sec: 456.734\n",
      "INFO:tensorflow:loss = 0.20977254, step = 17801 (0.219 sec)\n",
      "INFO:tensorflow:global_step/sec: 432.647\n",
      "INFO:tensorflow:loss = 0.09754325, step = 17901 (0.231 sec)\n",
      "INFO:tensorflow:global_step/sec: 389.941\n",
      "INFO:tensorflow:loss = 0.03494991, step = 18001 (0.256 sec)\n",
      "INFO:tensorflow:global_step/sec: 434.925\n",
      "INFO:tensorflow:loss = 0.17031653, step = 18101 (0.230 sec)\n",
      "INFO:tensorflow:global_step/sec: 445.735\n",
      "INFO:tensorflow:loss = 0.3200203, step = 18201 (0.224 sec)\n",
      "INFO:tensorflow:global_step/sec: 444.929\n",
      "INFO:tensorflow:loss = 0.18385477, step = 18301 (0.225 sec)\n",
      "INFO:tensorflow:global_step/sec: 445.546\n",
      "INFO:tensorflow:loss = 0.20921718, step = 18401 (0.225 sec)\n",
      "INFO:tensorflow:global_step/sec: 450.454\n",
      "INFO:tensorflow:loss = 0.01868303, step = 18501 (0.222 sec)\n",
      "INFO:tensorflow:global_step/sec: 445.762\n",
      "INFO:tensorflow:loss = 0.051421717, step = 18601 (0.224 sec)\n",
      "INFO:tensorflow:global_step/sec: 445.921\n",
      "INFO:tensorflow:loss = 0.047041617, step = 18701 (0.224 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 18750 into /var/folders/wy/h39t6kb11pnbb0pzhksd_fqh0000gn/T/tmp2xxrubio/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.46282205.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.estimator.canned.dnn.DNNClassifier at 0x11711b748>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "indices = np.random.permutation(len(X_train))\n",
    "X_train_shuffled = X_train[indices]\n",
    "y_train_shuffled = y_train[indices]\n",
    "\n",
    "input_fn = tf.estimator.inputs.numpy_input_fn(\n",
    "    x={\"X\": X_train_shuffled}, y=y_train_shuffled, num_epochs=10, batch_size=32, shuffle=False)\n",
    "dnn_clf.train(input_fn=input_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final loss should be exactly 0.46282205."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of using the `numpy_input_fn()` function (which cannot reproducibly shuffle the dataset at each epoch), you can create your own input function using the Data API and set its shuffling seed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_dataset(X, y=None, n_epochs=1, batch_size=32,\n",
    "                   buffer_size=1000, seed=None):\n",
    "    dataset = tf.data.Dataset.from_tensor_slices(({\"X\": X}, y))\n",
    "    dataset = dataset.repeat(n_epochs)\n",
    "    dataset = dataset.shuffle(buffer_size, seed=seed)\n",
    "    return dataset.batch(batch_size)\n",
    "\n",
    "input_fn=lambda: create_dataset(X_train, y_train, seed=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Using temporary folder as model directory: /var/folders/wy/h39t6kb11pnbb0pzhksd_fqh0000gn/T/tmpawwl1lf0\n",
      "INFO:tensorflow:Using config: {'_model_dir': '/var/folders/wy/h39t6kb11pnbb0pzhksd_fqh0000gn/T/tmpawwl1lf0', '_tf_random_seed': 42, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x137e1c6d8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into /var/folders/wy/h39t6kb11pnbb0pzhksd_fqh0000gn/T/tmpawwl1lf0/model.ckpt.\n",
      "INFO:tensorflow:loss = 80.279686, step = 1\n",
      "INFO:tensorflow:global_step/sec: 161.253\n",
      "INFO:tensorflow:loss = 16.09288, step = 101 (0.621 sec)\n",
      "INFO:tensorflow:global_step/sec: 433.582\n",
      "INFO:tensorflow:loss = 5.605775, step = 201 (0.231 sec)\n",
      "INFO:tensorflow:global_step/sec: 447.561\n",
      "INFO:tensorflow:loss = 12.584702, step = 301 (0.224 sec)\n",
      "INFO:tensorflow:global_step/sec: 442.148\n",
      "INFO:tensorflow:loss = 2.089463, step = 401 (0.226 sec)\n",
      "INFO:tensorflow:global_step/sec: 434.492\n",
      "INFO:tensorflow:loss = 9.2258215, step = 501 (0.230 sec)\n",
      "INFO:tensorflow:global_step/sec: 447.994\n",
      "INFO:tensorflow:loss = 8.11821, step = 601 (0.223 sec)\n",
      "INFO:tensorflow:global_step/sec: 442.723\n",
      "INFO:tensorflow:loss = 0.653025, step = 701 (0.226 sec)\n",
      "INFO:tensorflow:global_step/sec: 425.438\n",
      "INFO:tensorflow:loss = 4.331424, step = 801 (0.235 sec)\n",
      "INFO:tensorflow:global_step/sec: 444.471\n",
      "INFO:tensorflow:loss = 1.55325, step = 901 (0.225 sec)\n",
      "INFO:tensorflow:global_step/sec: 436.037\n",
      "INFO:tensorflow:loss = 5.208349, step = 1001 (0.229 sec)\n",
      "INFO:tensorflow:global_step/sec: 433.071\n",
      "INFO:tensorflow:loss = 0.80289483, step = 1101 (0.231 sec)\n",
      "INFO:tensorflow:global_step/sec: 436.717\n",
      "INFO:tensorflow:loss = 3.1879468, step = 1201 (0.229 sec)\n",
      "INFO:tensorflow:global_step/sec: 452.687\n",
      "INFO:tensorflow:loss = 5.55963, step = 1301 (0.221 sec)\n",
      "INFO:tensorflow:global_step/sec: 446.2\n",
      "INFO:tensorflow:loss = 12.830038, step = 1401 (0.224 sec)\n",
      "INFO:tensorflow:global_step/sec: 450.525\n",
      "INFO:tensorflow:loss = 6.8311796, step = 1501 (0.222 sec)\n",
      "INFO:tensorflow:global_step/sec: 452.967\n",
      "INFO:tensorflow:loss = 1.635078, step = 1601 (0.221 sec)\n",
      "INFO:tensorflow:global_step/sec: 453.743\n",
      "INFO:tensorflow:loss = 1.9616288, step = 1701 (0.220 sec)\n",
      "INFO:tensorflow:global_step/sec: 450.01\n",
      "INFO:tensorflow:loss = 1.4227519, step = 1801 (0.222 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1875 into /var/folders/wy/h39t6kb11pnbb0pzhksd_fqh0000gn/T/tmpawwl1lf0/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 1.0556093.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.estimator.canned.dnn.DNNClassifier at 0x11dba7a20>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random.seed(42)\n",
    "np.random.seed(42)\n",
    "tf.set_random_seed(42)\n",
    "\n",
    "my_config = tf.estimator.RunConfig(tf_random_seed=42)\n",
    "\n",
    "feature_cols = [tf.feature_column.numeric_column(\"X\", shape=[28 * 28])]\n",
    "dnn_clf = tf.estimator.DNNClassifier(hidden_units=[300, 100], n_classes=10,\n",
    "                                     feature_columns=feature_cols,\n",
    "                                     config=my_config)\n",
    "dnn_clf.train(input_fn=input_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final loss should be exactly 1.0556093."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "indices = np.random.permutation(len(X_train))\n",
    "X_train_shuffled = X_train[indices]\n",
    "y_train_shuffled = y_train[indices]\n",
    "\n",
    "input_fn = tf.estimator.inputs.numpy_input_fn(\n",
    "    x={\"X\": X_train_shuffled}, y=y_train_shuffled,\n",
    "    num_epochs=10, batch_size=32, shuffle=False)\n",
    "dnn_clf.train(input_fn=input_fn)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Keras API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you use the Keras API, all you need to do is set the random seed any time you clear the session:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "60000/60000 [==============================] - 5s 78us/step - loss: 0.5929 - acc: 0.8450\n",
      "Epoch 2/10\n",
      "60000/60000 [==============================] - 4s 75us/step - loss: 0.2804 - acc: 0.9199\n",
      "Epoch 3/10\n",
      "60000/60000 [==============================] - 4s 74us/step - loss: 0.2276 - acc: 0.9350\n",
      "Epoch 4/10\n",
      "60000/60000 [==============================] - 4s 74us/step - loss: 0.1933 - acc: 0.9449\n",
      "Epoch 5/10\n",
      "60000/60000 [==============================] - 4s 74us/step - loss: 0.1682 - acc: 0.9518\n",
      "Epoch 6/10\n",
      "60000/60000 [==============================] - 4s 74us/step - loss: 0.1490 - acc: 0.9573\n",
      "Epoch 7/10\n",
      "60000/60000 [==============================] - 4s 74us/step - loss: 0.1332 - acc: 0.9622\n",
      "Epoch 8/10\n",
      "60000/60000 [==============================] - 5s 75us/step - loss: 0.1202 - acc: 0.9658\n",
      "Epoch 9/10\n",
      "60000/60000 [==============================] - 4s 75us/step - loss: 0.1090 - acc: 0.9693\n",
      "Epoch 10/10\n",
      "60000/60000 [==============================] - 4s 75us/step - loss: 0.1000 - acc: 0.9716\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1379fff98>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "\n",
    "random.seed(42)\n",
    "np.random.seed(42)\n",
    "tf.set_random_seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(300, activation=\"relu\"),\n",
    "    keras.layers.Dense(100, activation=\"relu\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\"),\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"sgd\",\n",
    "              metrics=[\"accuracy\"])\n",
    "model.fit(X_train, y_train, epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should get exactly 97.16% accuracy on the training set at the end of training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Eliminate other sources of variability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, `os.listdir()` returns file names in an order that depends on how the files were indexed by the file system:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['my_test_foo_1',\n",
       " 'my_test_foo_6',\n",
       " 'my_test_foo_8',\n",
       " 'my_test_foo_9',\n",
       " 'my_test_foo_7',\n",
       " 'my_test_foo_0',\n",
       " 'my_test_foo_5',\n",
       " 'my_test_foo_2',\n",
       " 'my_test_foo_3',\n",
       " 'my_test_foo_4']"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "for i in range(10):\n",
    "    with open(\"my_test_foo_{}\".format(i), \"w\"):\n",
    "        pass\n",
    "\n",
    "[f for f in os.listdir() if f.startswith(\"my_test_foo_\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['my_test_bar_4',\n",
       " 'my_test_bar_3',\n",
       " 'my_test_bar_2',\n",
       " 'my_test_bar_5',\n",
       " 'my_test_bar_0',\n",
       " 'my_test_bar_7',\n",
       " 'my_test_bar_9',\n",
       " 'my_test_bar_8',\n",
       " 'my_test_bar_6',\n",
       " 'my_test_bar_1']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "for i in range(10):\n",
    "    with open(\"my_test_bar_{}\".format(i), \"w\"):\n",
    "        pass\n",
    "\n",
    "[f for f in os.listdir() if f.startswith(\"my_test_bar_\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should sort the file names before you use them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "filenames = os.listdir()\n",
    "filenames.sort()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['my_test_foo_0',\n",
       " 'my_test_foo_1',\n",
       " 'my_test_foo_2',\n",
       " 'my_test_foo_3',\n",
       " 'my_test_foo_4',\n",
       " 'my_test_foo_5',\n",
       " 'my_test_foo_6',\n",
       " 'my_test_foo_7',\n",
       " 'my_test_foo_8',\n",
       " 'my_test_foo_9']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[f for f in filenames if f.startswith(\"my_test_foo_\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "for f in os.listdir():\n",
    "    if f.startswith(\"my_test_foo_\") or f.startswith(\"my_test_bar_\"):\n",
    "        os.remove(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I hope you enjoyed this notebook. If you do not get reproducible results, or if they are different than mine, then please [file an issue](https://github.com/ageron/handson-ml/issues) on github, specifying what version of Python, TensorFlow, and NumPy you are using, as well as your O.S. version. Thank you!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to learn more about Deep Learning and TensorFlow, check out my book [Hands-On Machine Learning with Scitkit-Learn and TensorFlow](http://homl.info/amazon), O'Reilly. You can also follow me on twitter [@aureliengeron](https://twitter.com/aureliengeron) or watch my videos on YouTube at [youtube.com/c/AurelienGeron](https://www.youtube.com/c/AurelienGeron)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}