20180403_ExponentialMovingAveraged.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# tensorflow使用一些总结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2018-4-3 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (1)问题\n",
    "- 将tf.random_normal传入给tf.constant发生错误,**TypeError: List of Tensors when single Tensor expected**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "# a = tf.constant(tf.random_normal([2, 2]))     运行该代码出现错误\n",
    "# print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (2)出现问题原因\n",
    "[参考文章](https://stackoverflow.com/questions/35661677/tensorflow-generating-a-random-constant/35662013#35662013)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1 首先查看tf.random_normal函数定义:\n",
    "\n",
    "`def random_normal(shape,\n",
    "                  mean=0.0,\n",
    "                  stddev=1.0,\n",
    "                  dtype=dtypes.float32,\n",
    "                  seed=None,\n",
    "                  name=None):\n",
    "`\n",
    "- Returns: A tensor of the specified shape filled with random normal values.\n",
    "\n",
    "#### 2 查看tf.constant函数定义:\n",
    "` def constant(value, dtype=None, shape=None, name=\"Const\", verify_shape=False) `\n",
    "- value:          **A constant value (or list)** of output type \\`dtype\\`.\n",
    "- Returns: A Constant Tensor.\n",
    "\n",
    "**结论:** 因为tf.random_normal返回值是一个Tensor,但是tf.constat传入的形参是list二者类型是不匹配的,所以出现错误"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (3)解决"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 方法1:\n",
    "####  Use NumPy to generate the random value and put it in a tf.constant()<br/>\n",
    "`\n",
    "some_test = tf.constant(\n",
    "    np.random.normal(loc=0.0, scale=1.0, size=(2, 2)).astype(np.float32))\n",
    "`\n",
    "### 方法2:\n",
    "#### (Potentially faster, as it can use the GPU to generate the random numbers) Use TensorFlow to generate the random value and put it in a tf.Variable:\n",
    "`\n",
    "some_test = tf.Variable(\n",
    "    tf.random_normal([2, 2], mean=0.0, stddev=1.0, dtype=tf.float32)\n",
    "sess.run(some_test.initializer) \n",
    "`\n",
    "\n",
    "#### 查看tf.Variable函数定义:\n",
    "`\n",
    "def __init__(self,\n",
    "               initial_value=None,\n",
    "               trainable=True,\n",
    "               collections=None,\n",
    "               validate_shape=True,\n",
    "               caching_device=None,\n",
    "               name=None,\n",
    "               variable_def=None,\n",
    "               dtype=None,\n",
    "               expected_shape=None,\n",
    "               import_scope=None,\n",
    "               constraint=None):\n",
    "               ...\n",
    "               )\n",
    "`\n",
    "- args: initial_value: A \\`Tensor\\`, or Python object convertible to a \\`Tensor\\`,which is the initial value for the Variable.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2018-4-11 tensorflow一些常见库函数使用"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. tf.nn.moments返回数据的均值和方差\n",
    "`\n",
    "tf.nn.moments(x, axes, shift=None, name=None, keep_dims=False)\n",
    "`\n",
    "\n",
    "- x是输入张量\n",
    "- axes是在哪个维度上求解， 即想要 normalize的维度, [0] 代表 batch 维度，如果是图像数据，可以传入 [0, 1, 2]，相当于求[batch, height, width] 的均值/方差，注意不要加入channel 维度。\n",
    "- name: Name used to scope the operations that compute the moments.\n",
    "- keep_dims: produce moments with the same dimensionality as the input.\n",
    "- return: 该函数返回两个张量，均值mean和方差variance。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 38.20000076  47.79999924  60.59999847  50.09999847  35.09999847]\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "#batch = np.array(np.random.randint(1, 100,size=(5,5))) 一开始错误没有进行类型转换出现data type not understood,表示moments处理的数据需要是tf.float32类型的\n",
    "batch = np.array(np.random.randint(1, 100, size=(10, 5)))\n",
    "batch = tf.cast(batch, tf.float32)  # 如果直接在np使用dtype=np.float32类型转换的时候,还是会出现data type not understand错误,因此我们使用tf.cast进行强制类型转换\n",
    "#print(batch)\n",
    "mm, vv = tf.nn.moments(batch, axes=[0])\n",
    "with tf.Session() as sess:\n",
    "    print(sess.run(mm))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. tf.train.ExponentialMovingAverage(decay)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Some training algorithms, such as GradientDescent and Momentum often benefit from maintaining a moving average of variables during optimization. Using the moving averages for evaluations often improve results significantly. \n",
    " tensorflow \n",
    "- 官网上对于这个方法功能的介绍。GradientDescent 和 Momentum 方式的训练 都能够从 ExponentialMovingAverage 方法中获益。\n",
    "\n",
    " \n",
    " \n",
    "$$shadowVariable = decay * shadowVariable + (1 - decay) * variable $$\n",
    " \n",
    " - 一般取decay=0.999或者0.9999"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 这里参考博客 :[tensorflow学习笔记(三十三):ExponentialMovingAverage](https://blog.csdn.net/u012436149/article/details/56484572)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.1\n",
      "1.29\n",
      "1.561\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "w = tf.Variable(1.0)\n",
    "ema = tf.train.ExponentialMovingAverage(0.9)   # 这里的相当于decay=0.9\n",
    "update = tf.assign_add(w, 1.0)   # w = w + 1.0\n",
    " \n",
    "with tf.control_dependencies([update]):   # 只有里面当里面为op才可以执行update数据\n",
    "    #返回一个op,这个op用来更新moving_average,i.e. shadow value\n",
    "    ema_op = ema.apply([w])#这句和下面那句不能调换顺序 ,创建影子变量并进行指数滑动平均\n",
    "# 以 w 当作 key， 获取 shadow value 的值\n",
    "ema_val = ema.average(w)#参数不能是list，有点蛋疼,获取当前影子变量的值,并作为之后输出,这里的average或得的key应该和apply保持一致?\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    tf.global_variables_initializer().run()\n",
    "    for i in range(3):\n",
    "        sess.run(ema_op)\n",
    "        print(sess.run(ema_val))\n",
    "        \n",
    "# 创建一个时间序列 1 2 3 4\n",
    "#输出：\n",
    "#1.1      =0.9*1 + 0.1*2  # 一开始的时候默认影子变量和变量值是一样的,如果没有定义则是\n",
    "#1.29     =0.9*1.1+0.1*3\n",
    "#1.561    =0.9*1.29+0.1*4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.20000005]\n",
      "[0.48000011]\n",
      "[0.8320002]\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "w = tf.Variable(1.0)\n",
    "ema = tf.train.ExponentialMovingAverage(0.9)\n",
    "update = tf.assign_add(w, 1.0)\n",
    "\n",
    "ema_op = ema.apply([update])\n",
    "with tf.control_dependencies([ema_op]):\n",
    "    ema_val = tf.identity(ema.average(update)) #一个identity搞定,因为average不是返回的一个op,不然ema_op就不会执行了,所以使用identily直接就搞定了\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    tf.global_variables_initializer().run()\n",
    "    for i in range(3):\n",
    "        print(sess.run([ema_val]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用 ExponentialMovingAveraged parameters\n",
    "- 假设我们使用了ExponentialMovingAverage方法训练了神经网络， 在**test**阶段，如何使用 ExponentialMovingAveraged parameters呢？ 官网也给出了答案,使用Saver模型\n",
    "\n",
    "####  1.Create a Saver that loads variables from their saved shadow values.\n",
    "`\n",
    "shadow_var0_name = ema.average_name(var0)\n",
    "shadow_var1_name = ema.average_name(var1)\n",
    "saver = tf.train.Saver({shadow_var0_name: var0, shadow_var1_name: var1})\n",
    "saver.restore(...checkpoint filename...)\n",
    "`\n",
    "#### 2. var0 and var1 now hold the moving average values\n",
    "\n",
    "- Returns a map of names to Variables to restore.\n",
    "`\n",
    "variables_to_restore = ema.variables_to_restore()\n",
    "saver = tf.train.Saver(variables_to_restore)\n",
    "...\n",
    "saver.restore(...checkpoint filename...)\n",
    "`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 这里要注意的一个问题是，用于保存的saver可不能这么写，[参考]( http://blog.csdn.net/u012436149/article/details/56665612)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  3.  tf.tf.assign(ref, value, validate_shape=None, use_locking=None, name=None)\n",
    "\n",
    "- 将　value 赋值给　ref，并输出 ref，即  **ref = value**；\n",
    "\n",
    "- 这使得需要使用复位值的连续操作变简单\n",
    "\n",
    "- return: Same as “ref”. Returned as a convenience for operations that want to use the new value after the variable has been reset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "a = tf.Variable(tf.constant(1.0), dtype=tf.float32)\n",
    "b = tf.assign(a,2.0)  # assign可以用于分配数值\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    print(sess.run(b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4 tf.assign_add(ref,value,use_locking=None,name=None)\n",
    "\n",
    "- Update 'ref' by adding 'value' to it.\n",
    "\n",
    "- 更新ref的值，通过增加value，即：ref = ref + value；\n",
    "\n",
    "- This operation outputs \"ref\" after the update is done. This makes it easier to chain operations that need to use the reset value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "a = tf.Variable(tf.constant(1.0), dtype=tf.float32)\n",
    "a = tf.assign_add(a,2.0)  # assign可以用于分配数值\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    print(sess.run(a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5 tf.identity(input,name=None)\n",
    "\n",
    "- Return a tensor with the same shape and contents as input.返回一个具有相同形状张量和内容作为输入；\n",
    "\n",
    "- Args:\n",
    " input: A Tensor.\n",
    " name: A name for the operation (optional).\n",
    "\n",
    "- Returns:\n",
    " A Tensor. Has the same type as input."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6 tf.control_dependencies(control_inputs)\n",
    "- tf.control_dependencies()设计是用来控制计算流图的，给图中的某些计算指定顺序。比如：我们想要获取参数更新后的值，那么我们可以这么组织我们的代码。自己的理解：如果不是tf的tensor，并且没有加入到整个图中，则不会执行；\n",
    "\n",
    "- Wrapper for Graph.control_dependencies() using the default graph.\n",
    "\n",
    "- 在有些机器学习程序中我们想要指定某些操作执行的**依赖关系**，这时我们可以使用tf.control_dependencies()来实现。 \n",
    " control_dependencies(control_inputs)返回一个控制依赖的上下文管理器，使用with关键字可以让在这个上下文环境中的操作都在control_inputs 执行。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 测试代码1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n",
      "2.0\n",
      "3.0\n",
      "4.0\n",
      "5.0\n"
     ]
    }
   ],
   "source": [
    "x = tf.Variable(0.0)\n",
    "#返回一个op，表示给变量x加1的操作\n",
    "x_plus_1 = tf.assign_add(x, 1)\n",
    "  \n",
    "#control_dependencies的意义是，在执行with包含的内容（在这里就是 y = x）前\n",
    "#先执行control_dependencies中的内容（在这里就是 x_plus_1）\n",
    "with tf.control_dependencies([x_plus_1]):  # 表明并没有执行x_plus_1这个操作,y仅仅是x复制都是0\n",
    "    y = x + 0.0\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    for i in range(5):\n",
    "        print(sess.run(y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 测试代码2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.Variable 'Variable_21:0' shape=() dtype=float32_ref>\n",
      "Tensor(\"add_1:0\", shape=(), dtype=float32)\n",
      "1.0\n",
      "2.0\n",
      "3.0\n",
      "4.0\n",
      "5.0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "x = tf.Variable(0.0)\n",
    "print(x)  # 一开始x仅仅是一个变量\n",
    "x_plus_1 = tf.assign_add(x, 1)\n",
    "with tf.control_dependencies([x_plus_1]):\n",
    "    y = x + 0.0  # 这个时候变成立一个node?就可以执行了?\n",
    "    print(y) #z=tf.identity(x,name='x')\n",
    "init = tf.global_variables_initializer()\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    for i in range(5):\n",
    "        print(sess.run(y))\n",
    "\n",
    "# 可以看到当y定义为节点的输出后，就可以顺利执行操作了，此时y成为节点的输出，可以被图识别。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 测试代码3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n",
      "2.0\n",
      "3.0\n",
      "4.0\n",
      "5.0\n"
     ]
    }
   ],
   "source": [
    "x = tf.Variable(0.0)\n",
    "x_plus_1 = tf.assign_add(x, 1)\n",
    "  \n",
    "with tf.control_dependencies([x_plus_1]):\n",
    "    y = tf.identity(x)#修改部分\n",
    "init = tf.initialize_all_variables()\n",
    "  \n",
    "with tf.Session() as session:\n",
    "    init.run()\n",
    "    for i in range(5):\n",
    "        print(y.eval())\n",
    "        \n",
    "# 查询y为：Tensor(\"Identity_1:0\", shape=(), dtype=float32)，和节点联系起来了。\n",
    "#tf.identity是返回了一个一模一样新的tensor，再control_dependencies的作用块下，需要增加一个新节点到gragh中。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 测试代码4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n",
      "2.0\n",
      "3.0\n",
      "4.0\n",
      "5.0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf  \n",
    "x = tf.Variable(0.0)  \n",
    "x_plus = tf.assign_add(x, 1)  \n",
    "with tf.control_dependencies([x_plus]):#只有当内部为操作时以来才会生效  \n",
    "    #y = tf.identity(x)#将该语句变为操作  \n",
    "    y = x  \n",
    "    update = tf.group(y)#将该语句变为操作  \n",
    "init = tf.global_variables_initializer()  \n",
    "with tf.Session() as session:  \n",
    "    init.run()  \n",
    "    for i in range(5):  \n",
    "        session.run(update)  \n",
    "        print(y.eval())  \n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 解释:\n",
    "- 从上面的四个小小的测试我们可以知道,tensorflow下的control_dependencies函数中包含的语句必须是一个op或者说是一种操作,而不能是tensor,所以,一开始的话我们使用y=x相当于是tensor变量,并没有执行x_puls_1语句,从而进一步导致输出的y全是0,第二个测试y=x+0.0表示是一种op,是可以正确执行的;第三种比较通用,y=tf.identity(x)其中identity返回的是和x一样的变量但是他是一个op,因此是可以执行x_plus_1语句的,进而y值更新了,输出正确的结果了.第四种使用group把其变成了op了"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7   在TensorFlow中，Session.run()和Tensor.eval()有什么区别？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 7.1 如果你有一个Tensor t, 调用t.eval()相当于调用了tf.get_default_session().run(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 可以按照一下方式设置成默认会话:\n",
    "import tensorflow as tf\n",
    "t = tf.constant(42)\n",
    "sess = tf.Session()\n",
    "with sess.as_default():\n",
    "    assert sess is tf.get_default_session()  # 结果证明二者等价\n",
    "    assert t.eval() == sess.run(t)  # 结果证明二者也是等价的"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 7.2 另外一点sess.run是可以在相同的步骤输出多个值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "504.0\n",
      "504.0\n",
      "[504.0, 504.0]\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "t = tf.constant(42.0)\n",
    "u = tf.constant(12.0)\n",
    "tu = tf.multiply(t, u)\n",
    "ut = tf.multiply(u, t)\n",
    "# sess = tf.Session()\n",
    "with tf.Session() as sess:\n",
    "    print(tu.eval())\n",
    "    print(ut.eval())\n",
    "    print(sess.run([tu, ut]))   # sess.run(?) 可以一次性输出多个值"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8 如何正确理解关键字\"with\"与上下文管理器\n",
    "[参考文章](https://foofish.net/with-and-context-manager.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 自己一些理解:\n",
    "\n",
    "- 一般在文件中我们打开一个文件,然后对文件进行处理最后把他关闭,极端情况之下出现打开文件过多的情况; 当我们在链接数据库的时候,如果我们链接的\n",
    " 数据之后没有断开,极端的情况下出现链接数据库过多的情况,这个时候我们也是需要断开数据库的. 也就是我们在使用某些资源之后,可能需要**断开资源**\n",
    " \n",
    "- 那么with可以帮助我们断开这种链接,相当于是**try/finally** 语句,本质上是由于上下文管理器的原因.\n",
    "\n",
    "- 任何实现了 __enter__() 和 __exit__() 方法的对象都可称之为上下文管理器，上下文管理器对象可以使用 with 关键字。显然，文件（file）对象也实现了上下文管理器。此外，Python 还提供了一个 contextmanager 装饰器，更进一步简化上下管理器的实现方式。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9  tf.InteractiveSession()与tf.Session()的区别\n",
    "[参考文章](https://blog.csdn.net/yeqiang19910412/article/details/78651849)\n",
    "\n",
    "* tf.InteractiveSession():它能让你在运行图的时候，插入一些计算图，这些计算图是由某些操作(operations)构成的。这对于工作在交互式环境中的人们来说非常便利，比如使用IPython。\n",
    "\n",
    "* tf.Session():需要在启动session之前构建整个计算图，然后启动该计算图。\n",
    "\n",
    "* 意思就是在我们使用tf.InteractiveSession()来构建会话的时候，我们可以先构建一个session然后再定义操作（operation），如果我们使用tf.Session()来构建会话我们需要在会话构建之前定义好全部的操作（operation）然后再构建会话。\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}