09-theano/09.17-tensor-operator-and-elementwise-operations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Theano tensor 模块：操作符和逐元素操作"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 操作符"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using gpu device 1: Tesla C2075 (CNMeM is disabled)\n"
     ]
    }
   ],
   "source": [
    "import theano\n",
    "from theano import tensor as T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`tensor` 类型支持很多基本的操作："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 两个整形三维张量\n",
    "\n",
    "a, b = T.itensor3(\"a\"), T.itensor3(\"b\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 算术操作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(a + TensorConstant{3})\n",
      "(TensorConstant{3} - a)\n",
      "(a * TensorConstant{3.5})\n",
      "(TensorConstant{2.20000004768} / a)\n",
      "(TensorConstant{2.20000004768} // a)\n",
      "(TensorConstant{2.20000004768} ** a)\n",
      "mod(b, a)\n"
     ]
    }
   ],
   "source": [
    "print theano.pp(a + 3)      # T.add(a, 3) -> itensor3\n",
    "print theano.pp(3 - a)      # T.sub(3, a)\n",
    "print theano.pp(a * 3.5)    # T.mul(a, 3.5) -> ftensor3 or dtensor3 (depending on casting)\n",
    "print theano.pp(2.2 / a)    # T.truediv(2.2, a)\n",
    "print theano.pp(2.2 // a)   # T.intdiv(2.2, a)\n",
    "print theano.pp(2.2**a)     # T.pow(2.2, a)\n",
    "print theano.pp(b % a)      # T.mod(b, a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 比特操作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "and_(a, b)\n",
      "xor(a, TensorConstant{1})\n",
      "or_(a, b)\n",
      "invert(a)\n"
     ]
    }
   ],
   "source": [
    "print theano.pp(a & b)      # T.and_(a,b)    bitwise and (alias T.bitwise_and)\n",
    "print theano.pp(a ^ 1)      # T.xor(a,1)     bitwise xor (alias T.bitwise_xor)\n",
    "print theano.pp(a | b)      # T.or_(a,b)     bitwise or (alias T.bitwise_or)\n",
    "print theano.pp(~a)         # T.invert(a)    bitwise invert (alias T.bitwise_not)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 原地操作"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Theano` 不支持原地操作如 `+=` 等，`Theano` 的图优化解构会自动决定是否使用原地操作。如果需要更新变量的值，可以考虑使用共享变量 `theano.shared`。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 逐元素操作"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 类型转换"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`T.cast(x, dtype)` 用于类型转换："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = T.matrix()\n",
    "x_as_int = T.cast(x, 'int32')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`T.cast(x, dtype)` 的机制与 `numpy.asarray(x, dtype)` 的机制类似，只有 `dtype` 不同时才会创建新的变量： "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "print x_as_int is x\n",
    "print T.cast(x, theano.config.floatX) is x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "复数取实部，虚部，角度，模：\n",
    "\n",
    "- `T.real(a)`\n",
    "- `T.imag(a)`\n",
    "- `T.angle(a)`\n",
    "- `T.abs_(a)`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 比较"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Theano` 的比较操作也是逐元素的：\n",
    "\n",
    "- `T.lt(a, b)` : <\n",
    "- `T.gt(a, b)` : >\n",
    "- `T.le(a, b)` : <=\n",
    "- `T.ge(a, b)` : >=\n",
    "- `T.eq(a, b)` : ==\n",
    "- `T.neq(a, b)` : !=\n",
    "\n",
    "`Theano` 中没有 `bool` 类型，所有的 `bool` 类型都用 `int8` 表示。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "le(x, y)\n"
     ]
    }
   ],
   "source": [
    "x, y = T.dmatrices('x','y')\n",
    "\n",
    "print theano.pp(T.le(x, y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "除此之外，还有另一些与 `numpy` 类似的用法：\n",
    "\n",
    "- `T.isnan(a)` : 是否 NAN\n",
    "- `T.isinf(a)` : 是否 INF\n",
    "- `T.isclose(a, b)` ：浮点数是否接近\n",
    "- `T.allclose(a, b)` ：浮点数是否很接近"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 条件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`T.switch(cond, ift, iff)` 选择 `ift (if ture)` 和 `iff (if false)`。\n",
    "\n",
    "`T.where(cond, ift, iff)` 与 `switch` 一致。\n",
    "\n",
    "`T.clip(x, min, max)` 低于 `min` 的部分变成 `min`，超过 `max` 的部分变成 `max`。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数学操作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "maximum(a, b)\n",
      "minimum(a, b)\n",
      "(-a)\n",
      "inv(a)\n",
      "exp(a)\n",
      "log(a) log2(a) log10(a)\n",
      "sgn(a)\n",
      "floor(a)\n",
      "ceil(a)\n",
      "round_half_away_from_zero(a)\n",
      "int64(round_half_away_from_zero(a))\n",
      "sqr(a)\n",
      "sqrt(a)\n",
      "cos(a) sin(a) tan(a)\n",
      "cosh(a) sinh(a) tanh(a)\n",
      "erf(a) erfc(a)\n",
      "erfinv(a) erfcinv(a)\n",
      "gamma(a)\n",
      "gammaln(a)\n",
      "psi(a)\n"
     ]
    }
   ],
   "source": [
    "a, b = T.matrices(\"a\", \"b\")\n",
    "\n",
    "\n",
    "print theano.pp(T.maximum(a, b))  # max(a, b)\n",
    "print theano.pp(T.minimum(a, b))  # min(a, b)\n",
    "\n",
    "print theano.pp(T.neg(a)) # -a\n",
    "print theano.pp(T.inv(a)) # 1.0/a\n",
    "\n",
    "print theano.pp(T.exp(a)) \n",
    "print theano.pp(T.log(a)), theano.pp(T.log2(a)), theano.pp(T.log10(a))       # log10(a)\n",
    "\n",
    "print theano.pp(T.sgn(a))       # sgn(a)\n",
    "print theano.pp(T.floor(a))     # floor(a)\n",
    "print theano.pp(T.ceil(a))      # ceil(a)\n",
    "print theano.pp(T.round(a))     # round(a)\n",
    "print theano.pp(T.iround(a))    # iround(a)\n",
    "\n",
    "print theano.pp(T.sqr(a))   # sqr(a)\n",
    "print theano.pp(T.sqrt(a))  # sqrt(a)\n",
    "\n",
    "print theano.pp(T.cos(a)), theano.pp(T.sin(a)), theano.pp(T.tan(a))\n",
    "print theano.pp(T.cosh(a)), theano.pp(T.sinh(a)), theano.pp(T.tanh(a))         # tan(a)\n",
    "\n",
    "print theano.pp(T.erf(a)), theano.pp(T.erfc(a)) # erf(a), erfc(a)\n",
    "print theano.pp(T.erfinv(a)), theano.pp(T.erfcinv(a))\n",
    "\n",
    "print theano.pp(T.gamma(a))    # gamma(a)\n",
    "print theano.pp(T.gammaln(a))  # log(gamma(a))\n",
    "print theano.pp(T.psi(a))      # digamma(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "其中 `erf, erfc` 定义如下：\n",
    "https://en.wikipedia.org/wiki/Error_function\n",
    "\n",
    "$$\n",
    "\\operatorname{erf}(x) = \\frac{2}{\\sqrt\\pi} \\int_0^x e^{-t^2} dt\n",
    "$$\n",
    "$$\n",
    "\\begin{align}\n",
    "             \\operatorname{erfc}(x) & = 1-\\operatorname{erf}(x) \\\\\n",
    "                                    & = \\frac{2}{\\sqrt\\pi} \\int_x^{\\infty} e^{-t^2}\\,\\mathrm dt \\\\\n",
    "                                    & = e^{-x^2} \\operatorname{erfcx}(x)\n",
    "\\end{align} \n",
    "$$\n",
    "\n",
    "`erfinv, erfcinv` 为其反函数：1\n",
    "https://en.wikipedia.org/wiki/Error_function#Inverse_functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Broadcasting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](http://deeplearning.net/software/theano/_images/bcast.png)\n",
    "\n",
    "图示如上。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 线性代数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "矩阵乘法：`T.dot(x, y)`\n",
    "\n",
    "向量外积：`T.outer(x, y)`\n",
    "\n",
    "张量乘法：`tensordot(a, b, axes=2)`\n",
    "\n",
    "`axes` 参数表示 `a` `b` 对应要去掉的维度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2, 3, 4) (5, 6, 4, 3)\n",
      "(2, 5, 6)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "a = np.random.random((2,3,4))\n",
    "b = np.random.random((5,6,4,3))\n",
    "\n",
    "#tensordot\n",
    "c = np.tensordot(a, b, [[1,2],[3,2]])\n",
    "\n",
    "\n",
    "#loop replicating tensordot\n",
    "a0, a1, a2 = a.shape\n",
    "b0, b1, _, _ = b.shape\n",
    "cloop = np.zeros((a0,b0,b1))\n",
    "\n",
    "#loop over non-summed indices -- these exist\n",
    "#in the tensor product.\n",
    "for i in range(a0):\n",
    "    for j in range(b0):\n",
    "        for k in range(b1):\n",
    "            #loop over summed indices -- these don't exist\n",
    "            #in the tensor product.\n",
    "            for l in range(a1):\n",
    "                for m in range(a2):\n",
    "                    cloop[i,j,k] += a[i,l,m] * b[j,k,m,l]\n",
    "\n",
    "assert np.allclose(c, cloop)\n",
    "\n",
    "print a.shape, b.shape\n",
    "print c.shape"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}