docs/tutorials/main_formats.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Main Tensor Formats\n",
    "\n",
    "Three of the most popular tensor decompositions are supported in *tntorch*:\n",
    "\n",
    "- [CANDECOMP/PARAFAC (CP)](https://epubs.siam.org/doi/pdf/10.1137/07070111X)\n",
    "- [Tucker](https://epubs.siam.org/doi/pdf/10.1137/S0895479898346995)\n",
    "- [Tensor Train (TT)](https://epubs.siam.org/doi/abs/10.1137/090752286?journalCode=sjoce3)\n",
    "\n",
    "Those formats are all represented using $N$ *tensor cores* (one per tensor dimension, used for CP/TT) and, optionally, up to $N$ *factor matrices* (needed for Tucker).\n",
    "\n",
    "In an $N$-D tensor of shape $I_1 \\times \\dots \\times I_N$, each $n$-th core can come in four flavors:\n",
    "\n",
    "- $R^{\\mathrm{TT}}_n \\times I_n \\times R^{\\mathrm{TT}}_{n+1}$: a TT core.\n",
    "- $R^{\\mathrm{TT}}_n \\times S_n^{\\mathrm{Tucker}} \\times R^{\\mathrm{TT}}_{n+1}$: a TT-Tucker core, accompanied by an $I_n \\times S_n^{\\mathrm{Tucker}}$ factor matrix.\n",
    "- $I_n \\times R^{\\mathrm{CP}}_n$: a CP core. Conceptually, it works as if it were a 3D TT core of shape $R^{\\mathrm{CP}}_n \\times I_n \\times R^{\\mathrm{CP}}_n$ whose slices along the 2nd mode are all diagonal matrices.\n",
    "- $S_n^{\\mathrm{Tucker}} \\times R^{\\mathrm{CP}}_n$: a CP-Tucker core, accompanied by an $I_n \\times S_n^{\\mathrm{Tucker}}$ factor matrix. Conceptually, it works as a 3D TT-Tucker core.\n",
    "\n",
    "One tensor network can combine cores of different kinds. So all in all one may have TT, TT-Tucker, Tucker, CP, TT-CP, CP-Tucker, and TT-CP-Tucker tensors. We will show examples of all.\n",
    "\n",
    "*(see [this notebook](decompositions.ipynb) to decompose full tensors into those main formats)*\n",
    "\n",
    "*(see [this notebook](other_formats.ipynb) for other structured and custom decompositions)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TT\n",
    "\n",
    "Tensor train cores are represented in parentheses `( )`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5D TT tensor:\n",
       "\n",
       " 32  32  32  32  32\n",
       "  |   |   |   |   |\n",
       " (0) (1) (2) (3) (4)\n",
       " / \\ / \\ / \\ / \\ / \\\n",
       "1   5   5   5   5   1"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "torch.set_default_dtype(torch.float64)\n",
    "import tntorch as tn\n",
    "\n",
    "tn.rand([32]*5, ranks_tt=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TT-Tucker\n",
    "\n",
    "In this format, TT cores are compressed along their spatial dimension (2nd mode) using an accompanying Tucker factor. This was considered e.g. in the original [TT paper](https://epubs.siam.org/doi/abs/10.1137/090752286?journalCode=sjoce3)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5D TT-Tucker tensor:\n",
       "\n",
       " 32  32  32  32  32\n",
       "  |   |   |   |   |\n",
       "  6   6   6   6   6\n",
       " (0) (1) (2) (3) (4)\n",
       " / \\ / \\ / \\ / \\ / \\\n",
       "1   5   5   5   5   1"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tn.rand([32]*5, ranks_tt=5, ranks_tucker=6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is an example where only *some* cores have Tucker factors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5D TT-Tucker tensor:\n",
       "\n",
       "     32          32\n",
       " 32   |  32  32   |\n",
       "  |   6   |   |   7\n",
       " (0) (1) (2) (3) (4)\n",
       " / \\ / \\ / \\ / \\ / \\\n",
       "1   5   5   5   5   1"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tn.rand([32]*5, ranks_tt=5, ranks_tucker=[None, 6, None, None, 7])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: if you want to leave some factors fixed during gradient descent, simply set them to some PyTorch tensor that has requires_grad=False."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tucker\n",
    "\n",
    "\"Pure\" Tucker is technically speaking not supported, but is equivalent to a TT-Tucker tensor with full TT-ranks. The minimal necessary ranks are automatically computed and set up for you:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5D TT-Tucker tensor:\n",
       "\n",
       " 32  32  32  32  32\n",
       "  |   |   |   |   |\n",
       "  3   3   3   3   3\n",
       " (0) (1) (2) (3) (4)\n",
       " / \\ / \\ / \\ / \\ / \\\n",
       "1   3   9   9   3   1"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tn.rand([32]*5, ranks_tucker=3)  # Actually a TT-Tucker network, but just as expressive as a pure Tucker decomposition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In other words, all $32^5$ tensors of Tucker rank $3$ can be represented by a tensor that has the shape shown above, and vice versa."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CP\n",
    "\n",
    "CP factors are shown as cores in brackets `< >`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5D CP tensor:\n",
       "\n",
       " 32  32  32  32  32\n",
       "  |   |   |   |   |\n",
       " <0> <1> <2> <3> <4>\n",
       " / \\ / \\ / \\ / \\ / \\\n",
       "4   4   4   4   4   4"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tn.rand([32]*5, ranks_cp=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even though those factors work conceptually as 3D cores in a tensor train (every CP tensor [is a particular case](https://epubs.siam.org/doi/abs/10.1137/090752286?journalCode=sjoce3) of the TT format), they are stored in 2D as in a standard CP decomposition. In this case all cores have shape $32 \\times 4$.\n",
    "\n",
    "## TT-CP\n",
    "\n",
    "TT and CP cores can be combined by specifying lists of ranks for each format. You should provide $N-1$ TT ranks and $N$ CP ranks and use `None` so that they do not collide anywhere. Also note that consecutive CP ranks must coincide. Here is a tensor with 3 TT cores and 2 CP cores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5D TT-CP tensor:\n",
       "\n",
       " 32  32  32  32  32\n",
       "  |   |   |   |   |\n",
       " (0) (1) (2) <3> <4>\n",
       " / \\ / \\ / \\ / \\ / \\\n",
       "1   2   3   4   4   4"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tn.rand([32]*5, ranks_tt=[2, 3, None, None], ranks_cp=[None, None, None, 4, 4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is another example with 2 TT cores and 3 CP cores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5D TT-CP tensor:\n",
       "\n",
       " 32  32  32  32  32\n",
       "  |   |   |   |   |\n",
       " <0> (1) (2) <3> <4>\n",
       " / \\ / \\ / \\ / \\ / \\\n",
       "4   4   2   5   5   5"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tn.rand([32]*5, ranks_tt=[None, 2, None, None], ranks_cp=[4, None, None, 5, 5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CP-Tucker\n",
    "\n",
    "Similarly to TT-Tucker, this model restricts the columns along CP factors to live in a low-dimensional subspace. It is also known as a [canonical decomposition with linear constraints (*CANDELINC*)](https://link.springer.com/article/10.1007/BF02293596). [Compressing a Tucker core via CP](https://www.degruyter.com/downloadpdf/j/math.2007.5.issue-3/s11533-007-0018-0/s11533-007-0018-0.pdf) leads to an equivalent format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5D CP-Tucker tensor:\n",
       "\n",
       " 32  32  32  32  32\n",
       "  |   |   |   |   |\n",
       "  4   4   4   4   4\n",
       " <0> <1> <2> <3> <4>\n",
       " / \\ / \\ / \\ / \\ / \\\n",
       "2   2   2   2   2   2"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tn.rand([32]*5, ranks_cp=2, ranks_tucker=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TT-CP-Tucker\n",
    "\n",
    "Finally, we can combine all sorts of cores to get a hybrid of all 3 models:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5D TT-CP-Tucker tensor:\n",
       "\n",
       " 32      32  32    \n",
       "  |  32   |   |  32\n",
       "  5   |   5   5   |\n",
       " (0) (1) (2) <3> <4>\n",
       " / \\ / \\ / \\ / \\ / \\\n",
       "1   2   3   10  10  10"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tn.rand([32]*5, ranks_tt=[2, 3, None, None], ranks_cp=[None, None, None, 10, 10], ranks_tucker=[5, None, 5, 5, None])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}