example-notebook/Tutorial_03_1 The usage of  name_scope and variable_scope.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow 变量命名管理机制（一）\n",
    "\n",
    "初学者经常会遇到类似于下面的错误，特别是在使用 notebook 写代码的时候：\n",
    "\n",
    " <font color='red'>ValueError: </font>Variable XXX... already exists, disallowed.<br/>\n",
    " \n",
    "现在我们来探讨一下 TensorFlow 的命名管理机制，相信看完后你就能够很好地解决这个问题了。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')  # 不打印 warning \n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "# 设置GPU按需增长\n",
    "config = tf.ConfigProto()\n",
    "config.gpu_options.allow_growth = True\n",
    "sess = tf.Session(config=config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.使用 tf.Variable() 和 tf.get_variable() 创建变量"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 使用 tf.Variable() 创建变量，如果 name 一样的话会自动处理。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "v1 <tf.Variable 'v:0' shape=(1,) dtype=float32_ref>\n",
      "v2 <tf.Variable 'v_1:0' shape=(1,) dtype=float32_ref>\n",
      "v3 <tf.Variable 'v_2:0' shape=(2,) dtype=float32_ref>\n"
     ]
    }
   ],
   "source": [
    "v1 = tf.Variable(initial_value=[1.0], name='v')\n",
    "v2 = tf.Variable(initial_value=[2.0], name='v')\n",
    "v3 = tf.Variable(initial_value=[1.0, 2.0], name='v')\n",
    "print('v1', v1)\n",
    "print('v2', v2)\n",
    "print('v3', v3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 使用 tf.get_variable() 创建变量，不能用一样的 name "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gv1 <tf.Variable 'gv:0' shape=(2, 3) dtype=float32_ref>\n"
     ]
    }
   ],
   "source": [
    "gv1 = tf.get_variable(name='gv', shape=[2,3], initializer=tf.truncated_normal_initializer())\n",
    "# gv2 = tf.get_variable(name='gv', shape=[2,3], initializer=tf.truncated_normal_initializer())   # ValueError: Variable gv already exists\n",
    "print('gv1', gv1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果非要使用一样的 name 的话，那么需要结合 tf.variable_scope 并设置参数 reuse=True，后面会介绍。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 tf.Variable() 和 tf.get_variable() 同时创建变量，会自动处理。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "var1 <tf.Variable 'var:0' shape=(1,) dtype=float32_ref>\n",
      "var2 <tf.Variable 'var_1:0' shape=(2, 3) dtype=float32_ref>\n",
      "var3 <tf.Variable 'var_2:0' shape=(1,) dtype=float32_ref>\n"
     ]
    }
   ],
   "source": [
    "var1 = tf.Variable(initial_value=[1.0], name='var', trainable=False)\n",
    "var2 = tf.get_variable(name='var', shape=[2,3])\n",
    "var3 = tf.Variable(initial_value=[1.0], name='var')\n",
    "# var4 = tf.get_variable(name='var', shape=[2,3])  # 报错\n",
    "print('var1', var1)\n",
    "print('var2', var2)\n",
    "print('var3', var3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意上面，如果 tf.get_variable() 的名字和 tf.Variable() 的名字冲突的话也是会自动处理的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 使用 tf.placeholder() 创建占位符\n",
    "tf.placeholder 创建的并不是变量，这里也提一下只是想看看它的命名是什么样子的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ph1: Tensor(\"ph:0\", shape=(1, 3), dtype=float32)\n",
      "ph2: Tensor(\"ph_1:0\", shape=(2, 3), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "ph1 = tf.placeholder(dtype=tf.float32, shape=[1,3], name='ph')\n",
    "ph2 = tf.placeholder(dtype=tf.float32, shape=[2,3], name='ph')\n",
    "\n",
    "print('ph1:', ph1)\n",
    "print('ph2:', ph2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到我们用 tf.placeholder 挖了两个坑，坑名都是 'ph'，然后 TensorFlow 自动处理了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.5 获取全部的变量和 trainable 变量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 <tf.Variable 'v:0' shape=(1,) dtype=float32_ref>\n",
      "1 <tf.Variable 'v_1:0' shape=(1,) dtype=float32_ref>\n",
      "2 <tf.Variable 'v_2:0' shape=(2,) dtype=float32_ref>\n",
      "3 <tf.Variable 'gv:0' shape=(2, 3) dtype=float32_ref>\n",
      "4 <tf.Variable 'var:0' shape=(1,) dtype=float32_ref>\n",
      "5 <tf.Variable 'var_1:0' shape=(2, 3) dtype=float32_ref>\n",
      "6 <tf.Variable 'var_2:0' shape=(1,) dtype=float32_ref>\n"
     ]
    }
   ],
   "source": [
    "all_vars = tf.global_variables()\n",
    "for i in range(len(all_vars)):\n",
    "    print(i, all_vars[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 <tf.Variable 'v:0' shape=(1,) dtype=float32_ref>\n",
      "1 <tf.Variable 'v_1:0' shape=(1,) dtype=float32_ref>\n",
      "2 <tf.Variable 'v_2:0' shape=(2,) dtype=float32_ref>\n",
      "3 <tf.Variable 'gv:0' shape=(2, 3) dtype=float32_ref>\n",
      "4 <tf.Variable 'var_1:0' shape=(2, 3) dtype=float32_ref>\n",
      "5 <tf.Variable 'var_2:0' shape=(1,) dtype=float32_ref>\n"
     ]
    }
   ],
   "source": [
    "all_trainable_vars = tf.trainable_variables()\n",
    "for i in range(len(all_trainable_vars)):\n",
    "    print(i, all_trainable_vars[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "tf.global_variables() 和 tf.trainable_variables() 用来获取**所有的变量**和**所有可训的变量**。\n",
    "\n",
    "这个在后期的时候会用到，比如：我们定义两个不同的优化器，它们各自优化部分参数，这时候我们需要给它们传入各自需要优化的变量的list，这时候它们就只优化list中的变量，其他变量它就不算梯度了。\n",
    "\n",
    "注意上面 trainable_variables() 中少了一个 'var:0' 变量，这是因为我们把它设置成了 trainable=False, 那么在优化的过程中就不会对它求梯度了。\n",
    "\n",
    "如果现在不太理解这个也没有关系，后面的学习中要用到这个的时候就能明白了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 使用 tf.name_scope() 和 tf.variable_scope() 管理命名空间\n",
    "\n",
    "为了避免和上面的名称搞混了，这里建议 **Restart Kernel 然后再继续试验。**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')  # 不打印 warning \n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "# 设置GPU按需增长\n",
    "config = tf.ConfigProto()\n",
    "config.gpu_options.allow_growth = True\n",
    "sess = tf.Session(config=config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 使用 tf.name_scope()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ns_v1 <tf.Variable 'ns1/v:0' shape=(1,) dtype=float32_ref>\n",
      "ns_gv1 <tf.Variable 'v:0' shape=(2, 3) dtype=float32_ref>\n",
      "ns_v2 <tf.Variable 'ns1/v_1:0' shape=(1,) dtype=float32_ref>\n"
     ]
    }
   ],
   "source": [
    "with tf.name_scope('ns1') as ns:\n",
    "    ns_v1 = tf.Variable(initial_value=[1.0], name='v')\n",
    "    ns_gv1 = tf.get_variable(name='v', shape=[2,3])\n",
    "    ns_v2 = tf.Variable(initial_value=[1.0], name='v')\n",
    "    \n",
    "print('ns_v1', ns_v1)\n",
    "print('ns_gv1', ns_gv1)\n",
    "print('ns_v2', ns_v2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "结果就很清晰了，在 tf.name_scope() 里边：\n",
    "- 使用 tf.Variable() 创建的变量，会自动地在 name 前面加上 scope name\n",
    "- **使用 tf.get_variable() 创建的变量，并不受 name_scope 的影响**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 使用 tf.variable_scope()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vs_v1 <tf.Variable 'vs1/v:0' shape=(1,) dtype=float32_ref>\n",
      "vs_gv1 <tf.Variable 'vs1/v_1:0' shape=(2, 3) dtype=float32_ref>\n"
     ]
    }
   ],
   "source": [
    "with tf.variable_scope('vs1') as vs:\n",
    "    vs_v1 = tf.Variable(initial_value=[1.0], name='v')\n",
    "    vs_gv1 = tf.get_variable(name='v', shape=[2,3])\n",
    "    # vs_gv2 = tf.get_variable(name='v', shape=[1,3])  # ValueError: Variable vs1/v already exists\n",
    "\n",
    "print('vs_v1', vs_v1)\n",
    "print('vs_gv1', vs_gv1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从上面的结果可以看到 tf.variable_scope 和 tf.name_scope 的区别了吧：\n",
    "- 使用 tf.Variable() 创建的变量，会自动地在 name 前面加上 variable scope name\n",
    "- 使用 tf.get_variable() 创建的变量，也会自动地在 name 前面加上 variable scope name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 tf.variable_scope() 中设置 reuse=True\n",
    "在开始之前，先来看看 python 中怎么判断两个变量指向的对象是不是一样的。我们直接使用 **is** 关键字来判断就行了，注意不能使用 == 来判断，和 java, C++ 中不同，python 中的 == 只是判断左右两边的内容是否一致，而不是判断左右两边是不是同一个对象。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a is b:  False\n",
      "a is c:  True\n",
      "a: [555, 2, 3]\n",
      "b: [1, 666, 3]\n",
      "c: [555, 2, 3]\n"
     ]
    }
   ],
   "source": [
    "a = [1,2,3]\n",
    "b = [1,2,3]\n",
    "c = a\n",
    "print('a is b: ', a is b)\n",
    "print('a is c: ', a is c)\n",
    "\n",
    "c[0] = 555  # 修改 c，a也会变化：因为 a 和 c 指向同一个对象\n",
    "b[1] = 666  # 修改 b, a不受影响：因为 a 和 b 指向两个不同的对象\n",
    "print('a:', a)\n",
    "print('b:', b)\n",
    "print('c:', c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面的结果显示，a 和 c 是指向内存中的同一个对象的，**如果修改c的内容，a 也会一起变**。而 b 虽然和 a 的内容一模一样，但是他们并不是一个对象，如果修改 b 的内容，并不会影响到 a。\n",
    "\n",
    "**在 TensorFlow 中，我们是通过变量的 name 来区别不同的对象。** tf.variable_scope() 中的 reuse 就是是我们为了实现变量的共享。\n",
    "\n",
    "在上面，我们已经创建了一个 variable_scope 它的 name='vs1'；在 'vs1' 里边，我们创建了一个变量：name='v'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vs_gv2 <tf.Variable 'vs1/v2:0' shape=(2, 3) dtype=float32_ref>\n"
     ]
    }
   ],
   "source": [
    "with tf.variable_scope('vs1') as vs:\n",
    "    vs_gv2 = tf.get_variable(name='v2', shape=[2,3]) \n",
    "    # vs_gv3 = tf.get_variable(name='v', shape=[2,3])  # ValueError: Variable vs1/v already exists  \n",
    "    \n",
    "print('vs_gv2', vs_gv2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vs_gv3 <tf.Variable 'vs1/v_1:0' shape=(2, 3) dtype=float32_ref>\n",
      "vs_gv3 is vs_gv1:  True\n"
     ]
    }
   ],
   "source": [
    "with tf.variable_scope('vs1', reuse=True) as vs:\n",
    "    vs_gv3 = tf.get_variable(name='v', shape=[2,3])  \n",
    "    \n",
    "print('vs_gv3', vs_gv3)\n",
    "print('vs_gv3 is vs_gv1: ', vs_gv3 is vs_gv1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面发现了没有，当我们要**在同一个 variable_scope 下面‘定义’一个已经存在的变量 name='v' 的时候，需要设置 reuse=True**，实际上两次‘定义’指向同一个内存对象。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.总结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面主要介绍了 tf.Variable() 和 tf.get_variable() 两种创建变量的方式。实际上，TensorFlow 更加鼓励使用 t.get_variable() 来创建对象，只是它稍微麻烦一丢丢。此外，我们还探索了 tf.name_scope() 和 tf.variable_scope() 两种命名空间管理。一般来说：\n",
    "- 方式一：tf.name_scope() 常常和 tf.Variable() 一起来进行变量名称管理。\n",
    "- 方式二：tf.variable_scope() 常常和 tf.get_variable() 一起来进行变量名称管理和**实现权值变量共享。**\n",
    "\n",
    "在以后的学习中，我们应该尽量用方式二来创建变量。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.实用示例\n",
    "下面我们将通过两个简单的例子来感受一下 TensorFlow 的变量命名管理。\n",
    "\n",
    "为了避免前面和前面的变量搞混，建议先**restart kernel**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')  # 不打印 warning \n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "# 设置GPU按需增长\n",
    "config = tf.ConfigProto()\n",
    "config.gpu_options.allow_growth = True\n",
    "sess = tf.Session(config=config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fc(X, out_size):\n",
    "    \"\"\"全连接层。\n",
    "    Args:\n",
    "        X: 2-D tensor, [batch_size, in_size]\n",
    "        out_size: the size of output tensor.\n",
    "    Returns:\n",
    "        h_fc: 2-D tensor, [batch_size, out_size].\n",
    "    \"\"\"\n",
    "    in_size = X.shape[1]  # 特征维度\n",
    "    W = tf.get_variable('weight', shape=[in_size, out_size], initializer=tf.truncated_normal_initializer())\n",
    "    b = tf.get_variable('bias', [out_size], initializer=tf.zeros_initializer())\n",
    "    h_fc = tf.nn.relu(tf.nn.xw_plus_b(X, W, b), name='relu')\n",
    "    return h_fc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 先来看一个正确的例子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "var 0:  <tf.Variable 'fc1/weight:0' shape=(50, 64) dtype=float32_ref>\n",
      "var 1:  <tf.Variable 'fc1/bias:0' shape=(64,) dtype=float32_ref>\n",
      "var 2:  <tf.Variable 'fc2/weight:0' shape=(64, 32) dtype=float32_ref>\n",
      "var 3:  <tf.Variable 'fc2/bias:0' shape=(32,) dtype=float32_ref>\n"
     ]
    }
   ],
   "source": [
    "batch_size = 128  \n",
    "feature_size = 50  # 特征维度\n",
    "fc1_size = 64      # 第一个全连接层的神经元个数\n",
    "fc2_size = 32      # 第二个全连接层的神经元个数\n",
    "X_input = tf.placeholder(dtype=tf.float32, shape=[batch_size, feature_size], name='X_input')\n",
    "\n",
    "with tf.variable_scope('fc1') as vs:\n",
    "    h_fc1 = fc(X_input, out_size=fc1_size)\n",
    "with tf.variable_scope('fc2') as vs:\n",
    "    h_fc2 = fc(h_fc1, out_size=fc2_size)\n",
    "\n",
    "all_vars = tf.global_variables()\n",
    "for i in range(len(all_vars)):\n",
    "    print('var {}: '.format(i), all_vars[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 再看看一个错误的例子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "Variable weight already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Originally defined at:\n\n  File \"<ipython-input-2-2a6c9c5d485a>\", line 10, in fc\n    W = tf.get_variable('weight', shape=[in_size, out_size], initializer=tf.truncated_normal_initializer())\n  File \"<ipython-input-4-368830f9fb3e>\", line 7, in <module>\n    h_fc1 = fc(X_input, out_size=fc1_size)\n  File \"/usr/local/lib/python3.5/dist-packages/IPython/core/interactiveshell.py\", line 2963, in run_code\n    exec(code_obj, self.user_global_ns, self.user_ns)\n",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-4-368830f9fb3e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0mh_fc1\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_input\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mout_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mfc1_size\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m \u001b[0mh_fc2\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mh_fc1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mout_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mfc2_size\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      9\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[0mall_vars\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mglobal_variables\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-2-2a6c9c5d485a>\u001b[0m in \u001b[0;36mfc\u001b[0;34m(X, out_size)\u001b[0m\n\u001b[1;32m      8\u001b[0m     \"\"\"\n\u001b[1;32m      9\u001b[0m     \u001b[0min_size\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m  \u001b[0;31m# 特征维度\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 10\u001b[0;31m     \u001b[0mW\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_variable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'weight'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mshape\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0min_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mout_size\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minitializer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtruncated_normal_initializer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     11\u001b[0m     \u001b[0mb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_variable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'bias'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mout_size\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minitializer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros_initializer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     12\u001b[0m     \u001b[0mh_fc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrelu\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mxw_plus_b\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mW\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'relu'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variable_scope.py\u001b[0m in \u001b[0;36mget_variable\u001b[0;34m(name, shape, dtype, initializer, regularizer, trainable, collections, caching_device, partitioner, validate_shape, use_resource, custom_getter, constraint)\u001b[0m\n\u001b[1;32m   1295\u001b[0m       \u001b[0mpartitioner\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mpartitioner\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalidate_shape\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mvalidate_shape\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1296\u001b[0m       \u001b[0muse_resource\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0muse_resource\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcustom_getter\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcustom_getter\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1297\u001b[0;31m       constraint=constraint)\n\u001b[0m\u001b[1;32m   1298\u001b[0m get_variable_or_local_docstring = (\n\u001b[1;32m   1299\u001b[0m     \"\"\"%s\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variable_scope.py\u001b[0m in \u001b[0;36mget_variable\u001b[0;34m(self, var_store, name, shape, dtype, initializer, regularizer, reuse, trainable, collections, caching_device, partitioner, validate_shape, use_resource, custom_getter, constraint)\u001b[0m\n\u001b[1;32m   1091\u001b[0m           \u001b[0mpartitioner\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mpartitioner\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalidate_shape\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mvalidate_shape\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1092\u001b[0m           \u001b[0muse_resource\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0muse_resource\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcustom_getter\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcustom_getter\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1093\u001b[0;31m           constraint=constraint)\n\u001b[0m\u001b[1;32m   1094\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1095\u001b[0m   def _get_partitioned_variable(self,\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variable_scope.py\u001b[0m in \u001b[0;36mget_variable\u001b[0;34m(self, name, shape, dtype, initializer, regularizer, reuse, trainable, collections, caching_device, partitioner, validate_shape, use_resource, custom_getter, constraint)\u001b[0m\n\u001b[1;32m    437\u001b[0m           \u001b[0mcaching_device\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcaching_device\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpartitioner\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mpartitioner\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    438\u001b[0m           \u001b[0mvalidate_shape\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mvalidate_shape\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0muse_resource\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0muse_resource\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 439\u001b[0;31m           constraint=constraint)\n\u001b[0m\u001b[1;32m    440\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    441\u001b[0m   def _get_partitioned_variable(\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variable_scope.py\u001b[0m in \u001b[0;36m_true_getter\u001b[0;34m(name, shape, dtype, initializer, regularizer, reuse, trainable, collections, caching_device, partitioner, validate_shape, use_resource, constraint)\u001b[0m\n\u001b[1;32m    406\u001b[0m           \u001b[0mtrainable\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtrainable\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcollections\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcollections\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    407\u001b[0m           \u001b[0mcaching_device\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcaching_device\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalidate_shape\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mvalidate_shape\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 408\u001b[0;31m           use_resource=use_resource, constraint=constraint)\n\u001b[0m\u001b[1;32m    409\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    410\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mcustom_getter\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variable_scope.py\u001b[0m in \u001b[0;36m_get_single_variable\u001b[0;34m(self, name, shape, dtype, initializer, regularizer, partition_info, reuse, trainable, collections, caching_device, validate_shape, use_resource, constraint)\u001b[0m\n\u001b[1;32m    745\u001b[0m                          \u001b[0;34m\"reuse=tf.AUTO_REUSE in VarScope? \"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    746\u001b[0m                          \"Originally defined at:\\n\\n%s\" % (\n\u001b[0;32m--> 747\u001b[0;31m                              name, \"\".join(traceback.format_list(tb))))\n\u001b[0m\u001b[1;32m    748\u001b[0m       \u001b[0mfound_var\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_vars\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    749\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mshape\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_compatible_with\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfound_var\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_shape\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: Variable weight already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Originally defined at:\n\n  File \"<ipython-input-2-2a6c9c5d485a>\", line 10, in fc\n    W = tf.get_variable('weight', shape=[in_size, out_size], initializer=tf.truncated_normal_initializer())\n  File \"<ipython-input-4-368830f9fb3e>\", line 7, in <module>\n    h_fc1 = fc(X_input, out_size=fc1_size)\n  File \"/usr/local/lib/python3.5/dist-packages/IPython/core/interactiveshell.py\", line 2963, in run_code\n    exec(code_obj, self.user_global_ns, self.user_ns)\n"
     ]
    }
   ],
   "source": [
    "batch_size = 128  \n",
    "feature_size = 50  # 特征维度\n",
    "fc1_size = 64      # 第一个全连接层的神经元个数\n",
    "fc2_size = 32      # 第二个全连接层的神经元个数\n",
    "X_input = tf.placeholder(dtype=tf.float32, shape=[batch_size, feature_size], name='X_input')\n",
    "\n",
    "h_fc1 = fc(X_input, out_size=fc1_size)\n",
    "h_fc2 = fc(h_fc1, out_size=fc2_size)\n",
    "\n",
    "all_vars = tf.global_variables()\n",
    "for i in range(len(all_vars)):\n",
    "    print('var {}: '.format(i), all_vars[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果你已经理解了 tf.variable_scope() 和 tf.get_variable() 的用法的话，相信你一下子就能明白为什么上面会出错了。在神经网络的结构定义的时候，我们应该养成比较好的习惯，在定义每个层的时候都放在一个 tf.variable_scope() 里边。\n",
    "\n",
    "在定义整个模型的时候，也应该把整个模型放在一个 tf.variable_scope('your_model_name') 里边，这样做有个好处就是后期你想要微调某些层或者把多个小模型联合成一个大模型的时候都可以很轻松地实现。\n",
    "\n",
    "如果你现在还不能很好地理解这里讲的内容的话也不用担心，再后面的学习中慢慢理解就行了。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}