einsum-tensors.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3426fbeb-cfd2-4a75-976a-1e8c8dee2335",
   "metadata": {},
   "source": [
    "# Implement tensor operations with PyTorch einsum: from basic to self-attention\n",
    "\n",
    "[PyTorch einsum](https://pytorch.org/docs/stable/generated/torch.einsum.html) is a helpful and practical tool in any machine learning engineer or researcher's toolkit. The syntax is pithy and self-documenting. This makes it both a great thinking tool and also easier coding and debugging. One can easily visualize the shapes of the input and output tensors in any deep learning model. \n",
    "\n",
    "In this article, we provide code using einsum and visualizations for several tensor operations, thinking of these operations as tensor compressions. These operations are commonly used in deep learning models. We start from simple and familiar operations like _rowsum_ that serve as building blocks. And gradually build up to defining multi-head attention, a core component of large language models. Where applicable I compare with more common ways of implementing these operations in PyTorch and why I think einsum is easier. See these [visualizations](https://www.tensors.net/tutorial-1) and the [original proposal](https://nlp.seas.harvard.edu/NamedTensor) for more discussion.\n",
    "\n",
    "A note on self-attention with einsum implementation: Like everything in PyTorch, there are mutiple ways to implement self-attention. llama3's [open-sourced attention](https://github.com/meta-llama/llama3/blob/14aab0428d3ec3a9596f1dea06d9c564f9c0e35f/llama/model.py#L90) for training LLMs on multiple GPUs in parallel uses modules like [ColumnParallelLinear](https://github.com/facebookresearch/fairscale/blob/5f484b3545f27eddb19d970fbe1d361b9c5f2b07/fairscale/nn/model_parallel/layers.py#L218) instead of Linear, but the underlying math remains the same. See also, [this efficient implementation](https://github.com/facebookresearch/xformers/blob/95f085abc3ba8cfaa2527250b8e274a95b10f7fe/xformers/components/attention/core.py#L208). With this einsum-based implementation of self-attention, I focus on readability and exposition. "
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdIAAADDCAYAAAA2nZC8AAAKp2lDQ1BJQ0MgUHJvZmlsZQAASImVlwdUk9kSgO//pzdaQihSQm+CdAJICT303myEJEAoMQZCtyGLK7CiiIiAuiKLIgquBZC1IIrYFsXeF2QRUNbFgg2V9wOHsLvvvPfOm5w598v8c2fm/udOzgQACo0jEqXCcgCkCTPEod5ujOiYWAZuBGAADpAAHWA43HQRKzjYHyAyt/5d3t8F0PR6y3Q61r8//68iz+OncwGAghGO56Vz0xA+jug4VyTOAABVj9h1sjJE03wJYZoYKRDhx9OcOMvj0xw/w2j0jE94qDvCygDgyRyOOBEAsi5iZ2RyE5E4ZA+EzYU8gRBh5DtwTktbyUMYyQsMER8RwtPxmfF/iZP4t5jx0pgcTqKUZ88yI3gPQboolZPzf76O/y1pqZK5HPqIkpPEPqHIitQF3U9Z6SdlYXxg0BwLeDP+M5wk8YmYY266e+wc8zgeftK9qYH+c5wg8GJL42Sww+eYn+4ZNsfilaHSXAlid9Ycc8TzeSUpEVJ7Ep8tjZ+bFB41x5mCyMA5Tk8J85v3cZfaxZJQaf18obfbfF4v6dnT0v9yXgFbujcjKdxHenbOfP18IWs+Znq0tDYe38Nz3idC6i/KcJPmEqUGS/35qd5Se3pmmHRvBnIh5/cGS99hMsc3eI6BB/AE/siHASKAJbBD1AqEAM8Mfvb0HQXuK0U5YkFiUgaDhXQZn8EWcs0WMizNLa0BmO7Z2Svx9v5ML0J0/LytsACAxVMIXJ+3BSB6dBtyffLmbXpIXXJIT3T/zpWIM2dt0+2E/BYQgSygARWgAXSAITBFarMFjsAVqdgXBIFwEAOWAy5IAmlADLJAPlgPikAJ2AK2g2qwB+wDB8BhcBS0gVPgHLgIroIb4A54BPrBEHgJxsF7MAlBEA6iQFRIBdKE9CATyBJiQs6QJ+QPhUIxUByUCAkhCZQPbYBKoHKoGtoLNUI/Qyehc9BlqA96AA1Ao9Ab6DOMgskwDVaH9eFFMBNmwX5wOLwMToRXwblwIbwZroLr4ENwK3wOvgrfgfvhl/AECqBIKDpKC2WKYqLcUUGoWFQCSoxagypGVaLqUM2oDlQP6haqHzWG+oTGoqloBtoU7Yj2QUeguehV6DXoUnQ1+gC6FX0BfQs9gB5Hf8NQMGoYE4wDho2JxiRisjBFmEpMA+YEphtzBzOEeY/FYulYA6wd1gcbg03G5mFLsbuwLdhObB92EDuBw+FUcCY4J1wQjoPLwBXhduIO4c7ibuKGcB/xJLwm3hLvhY/FC/EF+Er8QfwZ/E38MH6SIEfQIzgQggg8Qg6hjFBP6CBcJwwRJonyRAOiEzGcmExcT6wiNhO7iY+Jb0kkkjbJnhRCEpDWkapIR0iXSAOkT2QFsjHZnbyULCFvJu8nd5IfkN9SKBR9iisllpJB2UxppJynPKV8lKHKmMmwZXgya2VqZFplbsq8kiXI6smyZJfL5spWyh6TvS47JkeQ05dzl+PIrZGrkTspd09uQp4qbyEfJJ8mXyp/UP6y/IgCTkFfwVOBp1CosE/hvMIgFUXVobpTudQN1HpqN3WIhqUZ0Ni0ZFoJ7TCtlzauqKBorRipmK1Yo3hasZ+OouvT2fRUehn9KP0u/bOSuhJLia+0SalZ6abSB+UFyq7KfOVi5RblO8qfVRgqniopKltV2lSeqKJVjVVDVLNUd6t2q44toC1wXMBdULzg6IKHarCasVqoWp7aPrVrahPqGure6iL1nern1cc06BquGskaFRpnNEY1qZrOmgLNCs2zmi8YigwWI5VRxbjAGNdS0/LRkmjt1erVmtQ20I7QLtBu0X6iQ9Rh6iToVOh06YzrauoG6ObrNuk+1CPoMfWS9Hbo9eh90DfQj9LfqN+mP2KgbMA2yDVoMnhsSDF0MVxlWGd42whrxDRKMdpldMMYNrYxTjKuMb5uApvYmghMdpn0LcQstF8oXFi38J4p2ZRlmmnaZDpgRjfzNyswazN7tUh3UeyirYt6Fn0ztzFPNa83f2ShYOFrUWDRYfHG0tiSa1ljeduKYuVltdaq3eq1tYk133q39X0bqk2AzUabLpuvtna2Yttm21E7Xbs4u1q7e0waM5hZyrxkj7F3s19rf8r+k4OtQ4bDUYc/HU0dUxwPOo4sNljMX1y/eNBJ24njtNep35nhHOf8o3O/i5YLx6XO5ZmrjivPtcF1mGXESmYdYr1yM3cTu51w++Du4L7avdMD5eHtUezR66ngGeFZ7fnUS9sr0avJa9zbxjvPu9MH4+Pns9XnHludzWU3ssd97XxX+17wI/uF+VX7PfM39hf7dwTAAb4B2wIeB+oFCgPbgkAQO2hb0JNgg+BVwb+EYEOCQ2pCnodahOaH9oRRw1aEHQx7H+4WXhb+KMIwQhLRFSkbuTSyMfJDlEdUeVR/9KLo1dFXY1RjBDHtsbjYyNiG2Iklnku2LxlaarO0aOndZQbLspddXq66PHX56RWyKzgrjsVh4qLiDsZ94QRx6jgT8ez42vhxrjt3B/clz5VXwRvlO/HL+cMJTgnlCSOJTonbEkeTXJIqk8YE7oJqwetkn+Q9yR9SglL2p0ylRqW2pOHT4tJOChWEKcILKzVWZq/sE5mIikT9qxxWbV81LvYTN6RD6cvS2zNoyHB0TWIo+U4ykOmcWZP5MSsy61i2fLYw+1qOcc6mnOFcr9yf8tB53LyufK389fkDq1mr966B1sSv6Vqrs7Zw7dA673UH1hPXp6z/tcC8oLzg3YaoDR2F6oXrCge/8/6uqUimSFx0b6Pjxj3fo78XfN+7yWrTzk3finnFV0rMSypLvpRyS6/8YPFD1Q9TmxM295bZlu3egt0i3HJ3q8vWA+Xy5bnlg9sCtrVWMCqKK95tX7H9cqV15Z4dxB2SHf1V/lXtO3V3btn5pTqp+k6NW01LrVrtptoPu3i7bu523d28R31PyZ7PPwp+vL/Xe29rnX5d5T7svsx9z+sj63t+Yv7U2KDaUNLwdb9wf/+B0AMXGu0aGw+qHSxrgpskTaOHlh66cdjjcHuzafPeFnpLyRFwRHLkxc9xP9896ne06xjzWPNxveO1J6gniluh1pzW8baktv72mPa+k74nuzocO078YvbL/lNap2pOK54uO0M8U3hm6mzu2YlOUefYucRzg10ruh6djz5/+0LIhd5uv+5LF70unu9h9Zy95HTp1GWHyyevMK+0XbW92nrN5tqJX21+PdFr29t63e56+w37Gx19i/vO3HS5ee6Wx62Lt9m3r94JvNN3N+Lu/XtL7/Xf590feZD64PXDzIeTj9Y9xjwufiL3pPKp2tO634x+a+m37T894DFw7VnYs0eD3MGXv6f//mWo8DnleeWw5nDjiOXIqVGv0RsvlrwYeil6OTlW9If8H7WvDF8d/9P1z2vj0eNDr8Wvp96UvlV5u/+d9buuieCJp+/T3k9+KP6o8vHAJ+anns9Rn4cns77gvlR9Nfra8c3v2+OptKkpEUfMmRkFUIjCCQkAvNkPACUGAOoNAIhLZmfqGYFm/wfMEPhPPDt3z4gtAIc7AQh2BcCjc5b1ELOs66wt3BXAVlZSnZt/Z2b1aZE7BIBbDsvL339Au20d+IfMzvF/qfufK5BG/dv6L6CkBpneI4q0AAAAOGVYSWZNTQAqAAAACAABh2kABAAAAAEAAAAaAAAAAAACoAIABAAAAAEAAAHSoAMABAAAAAEAAADDAAAAAAVs3nsAACttSURBVHgB7Z0HfFTF9sdPCAFCC6EFDB2B0JEqRbCAKAL6QMCnIjYERPDJkyLFgorSVUQQqdI7gihVkQ4iAlKeL9ISIKFFSkJ65n/OvP8uWdJ2ly33bn7z+cS9ZcqZ7+D+7pk5d8dPcSIkEAABEAABEAABpwjkcaoUCoEACIAACIAACGgCecEBBEAABEAABEDAlkBiYiJ17dqVHnjgATp16hSdPXuW2rZtS9u3b6e3336bWrdubS0Aj9SKAgcgAAIg4PsE4uPj6fLlyxQbG5ttZ9PS0sjyl21GvmnJ5+6VwqtXr+Zkyl3d37Fjh7X8pk2b6OOPP6ahQ4dSYGAgdezYUQvoq6++SmXKlLHmkwMIqQ0OnIAACJiNwL59+6hixYr6r0GDBlSrVi19HBYWRk2aNKFKlSrp82+++cZsXXO5vSKgdevWpdKlS9Po0aOzrP/MmTPUqlUrKlmyJPn7+9OTTz6ZZd42bdroPGXLlqWWLVtqkc4yc7obUVFR2stLdynbwy+//FLb465xXLRoEfXq1ctqQ4UKFah+/fr6fNu2bfTggw/q4xo1alDVqlWt+eQAQmqDAycgAAJmI3Du3DnKmzcvbdmyhQ4dOkT9+/eniIgILaK//vorHTx4kKpXr66n58zWN1fbW6pUKRJPS1J23qM8fOzevZs6dOigva9169bRiRMnMpjz22+/kcVLfOqpp3QZacOe9Nlnn9GqVavsyarzFCpUiPLkcY9k3bx5k0aOHEki7pZkEdGYmBh9vXbt2vqWCKk8XKRP7rEqfQs4BgEQAAE3EhAvq1u3blStWrVMWylevDj17dvXbk8p00p86GJAQIBDvRk0aJAW3cmTJ2co9/nnn9Obb76Z4XpOF1JSUmj9+vU5ZbO5/9JLL5EIXu/evW2uu+Lk008/pSFDhlBCQgJdu3bNpkpZE5V1Uj8/P5vr6U8QbJSeBo5BAARMR0DWqyzeQlbGt2jRgtauXUtvvfWW/kIcMGAATZ06lWQq+Pnnnyf5Yp89ezb9/vvvJEEm9erVo+7du9M999yjPa5x48bpNUWZ4hTvV0SgcuXK1KVLF51H2j1+/DjNmDGDxIORacHChQvTsGHD9DrbxYsX9RRzSEgI7d27l4oVK6YDWaR9SxIb5s+fT/v379fC1bhxYxLxsHg/WdUv5WUq9ttvv6Xw8HCSKdY+ffpYpx/F81y9erX2FqWunFhZ7LF8Pvzww3qKU2yTNUOLxykzAcLqzmlOKZcdzwMHDtDgwYPp2LFjJJ5udHQ0tW/fnqSdjz76iITV/fffr6efZcxefvllkmn6Dz74QI/BQw89pD1lySv9kfXZp59+WvdbhF2uydhIOXuScJUxtUzdilcq42NJ6ad1LdcyfDJkJBAAARDwGQK8libvxisWSJs+/fHHH4oFVfEXrerUqZP+k3wcjam+//57XYbFVf3111+KBU6xICkWRRUXF6fYY9H3eQ1QDR8+XEldLLKKozh1GxzAo4oUKaJYiBV/savly5erOnXq6HssFoq9QF1+/Pjx6siRI+rZZ59VBQoUUEuXLtV5kpKSVPPmzRWLp2JRVDw1rfiLXbHAKPbCVHb179mzR/G0p+JIUm0vi53i4BjdL7GFxV/xWqf66aefdL38gKBtkfw5peeee06x8Kl58+bpMu+//761CHtwioNz1NatW/W91157zXovO56RkZFq7Nixuky/fv0UC5U6ffq0LmthFRoaqvtTtGhR1axZMyV8Jk6cqMvww4k+l3I8/aqv8bS+unTpkmKRVxMmTFA85Wy1JacDXhfVY3z9+nVdl/RHkvy7kPHihyIlfRMOWSV58kECARAAAZ8hkJWQSgd5ild/WfI6oRZM9mC08MmX6Ouvv65OnjypObAHqvN99913+pzXB/U5v/Kgz+U/jz32mGJPRn+p//e//9X3RaAXLlyoBWvJkiXWvCVKlFD33Xef9fzKlSuKpwoVr93qa+zJ6vIc8GLNI6IiQs+eoMqufg720Q8HHIWry4roiHC/8847ylIHBxZZ6+V1ZF2vI0LKnqd+sOAgJcXTn1rcH330UV1nZkKaE88ffvhB2zBp0iSrXZYDYRUcHKzYq1XskSoZK0kitsJDhNSS5KFDxLNKlSqKo2qVjL0jScZoxYoV1iLyQLJgwQLrub0HmNrlkUECARDIXQQkkEQiVwcOHKg7zp6PntqTgJMLFy5YXw1hb9QGDHuZ1nNZa5QpzNTUVL0+y56Nnl5l8dJBMbJ22KNHD2v+9AcsFiR1sWerA1kkaEeSRMlakuSRJFOLMv2cVf3s4eqpYFnjsySZvpXpSgm0kiRRzXeT8uXLR2+88QaNGDGCWGjo1q1beto5qzrt5ZlVeYkslilafjDJKou+Lv1auXIlSeSwJJnCtjfJ6z8zZ87UgWjsretiwix9wJG9dSHYyF5SyAcCIOAzBO4MuJGAGlkTlXcFf/75Z2LvNNO+ypd7ZkkCnngalNijpcWLFxN7qyTBOTxNnFl2fU3WF6U+WY9jj0pfS/8lLmuHkiRaNbv6Zf1Q+iPrg7LuK3+y1jtq1Ci69957dR087ak/7+Y/ErBVsGBB4qlTLV6yLplVspenpTx7z5ZD/SnCbW+SNVd55Ul+NMHyYGRPWVn3lldqLMzkUx6w0o+BPfVIHgipvaSQDwRAwGcJbNiwQQuWiINEZ8prM5KSk5Pt6rP86o0EBokX+cwzz2gxlXpEBC1JxMwSEcrrmjowSNqTl/3F85LPnTt3WrLrYxFaEejs6pf7IsrLli2zluUpU+09tmvXTgc97dq1y3pP3ruVZE/feGpTe7uSX6KfX3zxRfrPf/6jHzjEe8sq5cTTErD0999/0/nz54mnrrOqKtvr0meefifx6MVjnjZtGn311VfZlpGbwkMidOVVlvSJ12b1jET6a3Yd2zsHjHwgAAIgYGQCsq4pwUSy7sgejV47k3OO3tRmS+BIuXLl9L2mTZsqFi1rd2QtToKH5DpPoSr2JhV7cyooKEhxdKhq2LChLle+fHk1ZcoUxVOciiNw9VqktMFTqIojRXXwUc+ePRW/f6nSB+bIup/UJcFJnTt31muAEsAia4mWxOKt2+HoVZ2Ppzf1GqHcP3z4cLb1s4AojjRWLJzq8ccfV2LDjRs3dNUS3CTrs9I2TzUrWVNlcVAcVaxkjTizJOuRwoK9Zc3h3//+t84ma7Ucpav4gUCfy5qzBPzkz59fs5VgLAn6yY6nrGPydLj6xz/+oYOiZK3VEuAj3NjrVcJLgq0sib1z1ahRI90OT+cq/lEGfS4BW/zqk84mAVH8nqkOvOLXVXQblvLpP9lrVfxQoIOYJJDMksaMGaPHVNaB3333Xctluz79JJddiotMIAACIODjBFjYSNb3sntnMCsE4uHJFKt4WRwsY5NN1j7llRhZs2SB023YZEh3wsFC+pUOFol0V//nQWZVvyWjrOmKZ5vZDxfITwNKefFyxTOW+uXPmb5a2svpMyee4hWKZ5udd5tTG0a4DyE1wijABhAAAZ8mkF5IfbqjubRzWCPNpQOPboMACHiGgPyIg6zF8XumJD94juR7BOCR+t6YokcgAAIgAAIeJACP1IOw0RQIgAAIgIDvEYCQ+t6YokcgAAIgAAIeJAAh9SBsNAUCIAACIOB7BCCkvjem6BEIgAAIgIAHCUBIPQgbTYEACIAACPgeAQip740pegQCIAACIOBBAhBSD8JGUyAAAiAAAr5HAELqe2OKHoEACIAACHiQAITUg7DRFAiAAAiAgO8R8LqQyo8syxY6sg2Qo0k21LVsS2RvWfnBaHu2D7K3vrS0NLp586bew042+c0qyd4Aklf+ckqO5M2pruzuO8Mvu/ruvCd7KMqWS0ggAAIg4MsEvCakc+bM0bvKd+nSRW+Ae//99xNvk0Pff/+93bx5uyHirY/s3oj1zJkzVKZMGXriiSfsbiOnjLKJr+zSLnbITvVZJdmrUDaflZ0XZHeJK1euZJp1xowZeucG2di2VatWtGjRokzzZXZRNvN15IHEUX6ZtZndtd69e5OMMxIIgIDxCKxevdrpfUCN1xvvWuQVIZUvWPnx5uHDh9PGjRv1ZqwiArxPnt7g9uuvv7aLimyaK9sCiWdlT5Itg2S7HlduGySb6vJefTk2P3fuXJId2UXIxYOVDWjvTOKtLlmyRAut/MD17t276dlnn70zW5bnXbt2JfEC7U2O8rO3Xsn3448/6rF1Zrd5R9pBXhAAAccJyHcQ7+NJo0ePdrwwSmQg4PEfrecNXIk3mKWHHnqIfvrpJxuDZG862aH81q1bekd43mDV5v6dJzIFKh7Ynfv23Zkv/bnsyefqPfhEIIcOHap3aecNgNM3Z3PMm93Stm3btFiKaMqu97whrjWP7PQu09xjx47VDweyr6G9Sbxh3mCXIiMjiTcvtquYM/zsqVjGRB6UeINeOnXqFG3evNmeYsgDAiDgQQLysBsWFka8IbkHW/XNpvJ6ulsyFSqpY8eOGZoWgeNd0WnVqlVabHgnc+Ld6enixYskU78irCJGvOO9XudctmwZxcbG0qhRo7QAS4X79u0jESQRY94lXXto8g+Gd7SnP/74Q//JFzzvhk6//PKLrk/qkCez9evXE+9Erz1jmf61iNz27dt1nWLHvffeS7yjO7Vo0SKD/fZckCnb/v3708iRI2nhwoW6L5ZyMq0r10RI70zZ2cC7xVufLD/88EPiXe3prbfeInkwmTp1qmYkzPbu3Uu8ez29/fbb9Oeff1J6fhcuXNAsxFsXkX///fdpy5YttHPnTm2KTMFn95CQ3t7JkyfTv/71L/rhhx9o165d6W/hGARAwAAE5LtAvuvkexJCevcD4vGp3aNHj2qrQ0JCMrXe4oWGh4fr+w0aNCCZ6hWPb9OmTfTtt99Sv3799BSpeDtyTwROktTdvHlzvWY5ZMgQPXU8fvx4qlmzJh07dkx/Ll26VAuI5BfvVzxAETARNxFyEV+ZIp03b55k0Z6hiP6hQ4e0KElwk+QRsXY29e3bV3trIjiWdODAAS3SIoJ3Jpm6zs6Gxo0bU926dXWxRx55hDp06EDBwcF6iliEcfr06XqKWKZ95cFk5syZGfgJi9q1a+vpZ+EhgVOyu/2CBQtIxkCmpO1JERER+uFF1rsdWb+2p27kAQEQcA0B+f9ZHtrlYRrJBQR4es+jib+UFZutZs+enWm7HJSj77NHY71fokQJxcKg+MtdsUeqWFD1PfbcdN7ffvtNn/NapT7noCJ9zuuLioVEcWSwtS4WHVW1alXrOXuiusyUKVP0NV470OdPP/20NQ+LsVq5cqU+P336tL7Pa6PW+3faYb1xxwGLr+IHAn2VHwZ0PbxGrM+ff/55dfLkSX0s/WVBtSmdkw0Wbjy1a1OOp9J1O7wurViQ1cSJExWLnc6Tmd28lqvz8/qJYmFVHHVrU19OJ3369FFXr17V2djD13WxZ5xTMdwHARDwMIHHHntMde/e3cOt+mZzHp/alalR8e5kLS+zZAlOEe8ofRKPSyJeO3XqlP6yzXGTJk30uXimsk544sQJ4n8s2vuzyZjJicWjk+AlSTItaknt27cnFn4dXVyyZEl9WaZE7ibJ1Kd4ipMmTdKesrySU6VKlSyrvFsb6tWrp6OBBw0alGUbcqNXr1508OBB+uKLL0jy1qhRI9v86W/KWuj58+f1VLtct0QmR0dH61mC9HlxDAIgAAK+QsDjU7svvPCCZpfZlIKs3/38889UuHBhLYDpIcvaYk5JXueQaddZs2YRP2npgCYRQHtSnjyZo5B1VJmmlOljiTCWqWVXpOrVq+vpWqmTvTgaMGBAltU6asOePXtIQtvTJ3v4Sf6kpCS9jizT4Z9//rleJ01fT1bHUm7+/Pl6ylvWZeWPvXyd3fJwlFVZXAcBEAABMxPIXD3c2CNZ6/vggw+0t/jZZ59ZW5J1QFnXlIhPnn60O/LUWgEfSGCMeEE8LUvyWopEBkuY990kEXbxFnv06KE9W/HWJDnzow48qUHyZ0kW75CnQqlly5aWyxk+7bGhVKlSupys4UrAlSPvk1oalCAj8Ugff/xx/eqNzB7IA4msI+eUJJhL1n7TP5CITeLhSyATEgiAgHEISKzI8ePHaf/+/fpNAuNYZlJLvDVjzRGdir+oFQcHqZ49eyr2gBRPzSoWDRuTOHBGcZStknVDjui13uOFcr3WyZG1iqdlFQfGKFm/5OlfvS7Hw2H95AAcxcKiZP1O1lr5BxEUe4CKfyxAccSakjr4xxIUR7Epjg5W7L0pFgH1zjvvKPaSVbNmzVTZsmW1na+//rriVztUYGCgYsFR6e3g6VPFAVBWG9MfSB/5hxsUTw3rPvNrOPp269at1Zo1a/Qxe+mKf4RB91dskmN+rzRHG6QwT2NrnhUqVFBPPvmk4l9w0v0RNlJXtWrVlKwHW1J6uyUPv9eq2EvW/MQOSTyNrlmUL19ecZSxpajNp6y3tmvXTvH7uTb18wON4ocPXZ49esU/tGFTDicgAAIg4CsEPP4e6Z3PG/ITgbKGxgKgvZc77ztyLlO7so4q647iCcXExNDy5ctJPF95Z1U8VGeTZc3UkXdWnW0rq3L22CCv8sjUOBIIgAAIgIBnCHg82OjObsmv63AU7Z2XnToX8ZSpRQmQsawJyisy8gpIUFCQU3VaCnlTQB2xASJqoYVPEAABEPAMAa97pK7spgS1TJgwgfg1EhKBtqwTvvbaa/pHFFzZFuoCARAAARAAASHgU0KKIQUBEAABEAABTxPweNSupzuI9kAABEAABEDAnQQgpO6ki7pBAARAAAR8ngCE1OeHGB0EARAAARBwJwEIqTvpom4QAAEQAAGfJwAh9fkhRgdBAARAAATcSQBC6k66qBsEQAAEQMDnCUBIfX6I0UEQAAEQAAF3EoCQupMu6gYBEAABEPB5AhBSDw2x/NrSihUrPNQamgEBEAABEPAUAfyykYdIv/zyyzR37lySLdN4BxoPtYpmQAAEPEHgwIFDFBl5Qf+utyfac7YN2SqxZcsmFBJS2qYK2U/4p60b+NrtbR5tMhjoJCAgHz3S9nEDWUTk9R+tNxQNNxrz3nvvUZcuXSCibmSMqkHAWwT27z9MFcq34k0z/L1lgl3txsZep4MHj/Cew21t8svvlKckXaL69arYXDfiycbNv7JZEFIjjo1bbeK9QfW0rmwZx/udEu9J6tb2UDkIgIBnCcj2jcHBpSgwsJBnG3awtYiIBL3VZGbF8uULoPLly2Z2y1DXSpYobih7xBiskXpgSGRLN9mJRrxSmdpFAgEQAAEQ8B0CEFIPjKXsZfrggw96oCU0AQIgAAIg4GkCEFJPE0d7IAACIAACPkUAQupTw4nOgAAIgAAIeJoAhNTTxNEeCIAACICATxHA6y9uHs49e/bQ9evXSQKOJFk+3dwsqgcBEAABEPAQAQipm0EvXryYduzYQTVr1qSmTZtSpUqV3NwiqgcBEAABEPAkAQipm2mPHTuWtm3bRoULF+ZfFGlp+F8+cTMOVA8CIAACPkcAQurmIQ0MDORfETHWr3C4ucuoHgRAAARyFQEEG+Wq4UZnQQAEQAAEXE0AQupqoqgPBEAABEAgVxGAkOaq4UZnQQAEQAAEXE0AQupqoqgPBEAABEAgVxGAkOaq4UZnQQAEQAAEXE0AQupqoqgPBEAABFxA4NatWBfU4t0qEhISvWuAh1rH6y8eAo1mQAAEQMAeAomJCTR23ACqU7spHfx9Bw14YwyFhJSzp6hh8ny3diudPXuBt49Moj17D9HSxZMoICDAMPa52hB4pK4mivpAAARA4C4IzJo9hhrUb0lduvSmfz4zgEV14F3U5p2i74yYTA+0akSD336FTp85TytXbfaOIR5qFULqIdBoBgRAAATsIbBnzyYKDa2ss1aqFEb79m2mtLQ0e4oaJs/6tdOpbt3qlJqaSleu/E3Vq1UyjG3uMARC6g6qqBMEQAAEnCQQE3OJgoJK6NKFChWhpKREirt108navFOscuVytGnzLurQsQ8NH/YaNWxYyzuGeKhVrJF6CDSaAQEQAAF7CKSmplCePLY+Thp7dmZLHR5vQ2VCStFzLwzmqeqa1Lx5A7N1wW57bUfL7mLICAIgAAIg4A4CQUHFKTbuhq46PuEWi6o/FWTP1CwpPj6BFi5ap80VT7RVy4Y0cfIcs5jvlJ1u8UijoqIoJibGKYM8WShv3rxUo0aNTJs8duxYpteNdrF06dJUqlSpDGYJfxkHM6SwsDDy9/c3g6mwEQTcTqBhw9Z06eI5orpEF6MjqV695hSQ1zwRr9HRV2jIsAnUvdtjOlJXvOuwsCpu5+bNBtwipFOmzGOBaunNftnVdnj4IXrhBT+qXr26Tf7ff/+d1u1dTdV5sdzo6eia4/TRiDEZzJw7bRrVKRNi+JDz+IQEunb1KrVs3TpDH3ABBHIjgRd7DaEJEwdRcHApWrtuLg0c+ImpMMj6aP9+z9LiJT/w+m4y7d13mDas/8ZUfXDUWLcIaUhIKNWt08JRWzyfXylKSUnJ0K5cq920NtW6LyzDPaNduHAsOlOTgoOCqFmdOpQ/X75M7xvl4n9OnaabJotINAo72OGbBOSd0Y8/mk/h4UdoxPDplC9fftN1dPg7fej69Zt08mQk/bp3GffB2N9DdwvYLUJ6t0ahPAiAAAjkZgIinrVrNzE1gqCgIj4frWsZIAQbWUjgEwRAAARAAAScIAAhdQIaioAACIAACICAhQCE1EICnyAAAiAAAiDgBAEIqRPQUAQEQAAEQAAELAQgpBYS+AQBEAABEAABJwhASJ2AhiIgAAIgAAIgYCEAIbWQwCcIgAAIgAAIOEEAQuoENBQBARAAARAAAQsBCKmFBD5BAARAAARAwAkCEFInoKEICIAACIAACFgIQEgtJPAJAiAAAiAAAk4QgJA6AQ1FQAAEQAAEQMBCAEJqIYFPEAABEAABEHCCAITUCWgoAgIgAAIgAAIWAqbZRu3WrVgqWLCwxW5TfiYnJxt+o+3swMbFx1OhwMDssuAeCORKAmm8p258fBwlJiYYuv8pKcmkeB/mzJLsw3z16rXMbhnqWmpqqqHsEWMML6TyD3PsuAFUp3ZTOvj7DhrwxhiSjW/NlJZMX067t+yhGnWrU//3+prJdG3r6XPnaOK8eRSYvwD9evQPmjRkKDWsVdN0/YDBIOAuArVqVaXTp3eQv7+/u5pwSb1xcXHUtNlDGeoqXbo07bsVQL/sOJXhntEu+PkHGc0k4wvprNljqEH9ltS584tUo0YDFtWBNGniKsOBzM6gVu1b0K3YWxR7Iza7bIa9N3XxEhr26qtULiSEZq1cRf1Gj6Z9SxYb1l4YBgKeJtCmTUtq08bTrbquvUCeaere4wXXVZjLajL8GumePZsoNLSyHpZKlcJo377NJNMoZkrlKoeSn5+fmUy2sTUxOYmOhofra2FVKtPJyEib+zgBARAAgdxMwPBTuzExlygoqIQeo0KFilBSUiLF3bpJRQobz7331X9IU4YPt3Zt65691LNzJ+s5DkAABEAgtxMwvJCmpqZQnjy2jnOaARebc8M/pOMnT9KOg7/R8smTc0N30UcQAAEQsIuA4YU0KKg4xcbd0J2JT7jFoupPBdkzRfIsgegrV2jszFm0gkX0zgcbz1qC1kAABEDAWAQML6QNG7amSxfPEdUluhgdSfXqNaeAvAHGoujj1tzkSL9xs2fTlyNHUJFChehTFtRhr77i471G90DAfgJzFs6mG8nXKSCfsb+bkpOSqUW9VtSkYRObzl29epVmfDGVypYsZXPdiCeXr/9Ng0feXm4ygo2GF9IXew2hCRMHUXBwKVq7bi4NHPiJEbg5ZMO8yfNpw/JNlJSYpP9H6zfyNYfKeztzn/c/oNVbt9LXy5brQK8WDRpASL09KGjfUARSKJladW3Os2XGfs/6TPhZunYh47uisbGxVKN8JWrTqJmhuGZmzMa9OzK77NVrhhdSeWf044/mU3j4ERoxfDrly5ffq8CcabzXWz1J/syaFo0fZ1bTYTcIgAAIuJ2A4YVUCIh41q5tOxXhdjJoAARAAARAAATsIGAbDmtHAWQBARAAARAAARC4TQBCepsFjkAABEAABEDAYQIQUoeRoQAIgAAIgAAI3CYAIb3NAkcgAAIgAAIg4DABCKnDyFAABEAABEAABG4TgJDeZoEjEAABEAABEHCYAITUYWQoAAIgAAIgAAK3CUBIb7PAEQiAAAiAAAg4TABC6jAyFAABEAABEACB2wQgpLdZ4AgEQAAEQAAEHCYAIXUYGQqAAAiAAAiAwG0CENLbLHAEAiAAAiAAAg4TgJA6jAwFQAAEfIXAhg0b6PDhw77SHfTDSwSsQnrjxg2aM2cOJSYmeskUNAsCIAACniOQlpZGPXr0oMGDB3uuUQdaSk5OdiC3cbPGXPvbuMa5yDLrNmqrV6+ml19+mYoXL05PPvnkXVUfHx9D23csvKs6PFH40qUL9Gj7XhmaKlmyJK1atIKijkVnuGe0C36p1mchW9Py5qUVu3aTf54s7tvm9trZxcuXqeMzz3itfTScewnk4f831qxZQyEhIYaCEH3uIq1bsJ7WLVpPk5eOp6o1qxjKPkeMOXziGI2dPoUWfT7dkWKmy2sV0m7dulGBAgWoffv2d92JYcPevOs6vFlB5cqV6ZMRY71pwl23/VLfvnddByoAAV8mcODAAdq/fz8VK1aMatWqZZiuFilWhLq+8hRt/e4nSk1JNYxdjhqSkpKiRTSZP309Wd2V5cuXU3h4OJ08edLX+4z+gQAIgACVKVOGtmzZQlOnTjUUjUKFC1LxUsUNZZMzxsxbtYye7dyF/PycKW2uMlYhLV26NI0aNYpOnDhhrh7AWhAAARBwgkC5cuWoTp06TpREkZwInIw4Q4H589M9ZcrmlNUn7luFtFWrVj7RIXQCBEAABEDAewSUUrRk3Rr6J3ujuSVZhTS3dBj9BAEQAAEQcB+B77duonNR52nyrOm0YPVyOh0ZQV8vmue+Bg1QszXYyJW2rF61hFKSjf8aTVzcLera7XkqUqSITfevXr1KS+cvomJBQTbXjXhSOKgode7yVAbTtmz6ka5ciuL1CWMvUKSmplKbRx6j0NByGfqACyCQmwmwY0fi3ZktdWrbnuRP0ra9uygy6gL1eTbj2xFm61d29lqFdMSIERQQEEBffvkldejQgQoWLJhduWzvxd68TJ2eaJptHiPc3L3nCEVGRmaI2Dt16hRVK1mWqlaoaAQzs7Vh9fYtRJkIacTpcHqidT2SEH8jp8gL0XSSg9wgpEYeJd+zbdasWdSuXTtKSEig/LyWZ6R08fxFWjxtGUX8FUHTP/6GOj/fkR7q1MZIJtplyx9/nqBZSxfSib/+qz1SXxZTq5B+8cUXJH+uSHny+HFIeVFXVOXWOoKKFs6y/rz+/hRUxPh9yOpLQAS0UMFA/pLIl2UfjXAj6uJl8lPG9pqNwAk2uI6A/BDDu+++S/L6y8aNG6lfv36uq9wFNYWEhtC/Phqg/1xQndeqqFujJs2f/JXX2vdkw1Yh9WSjaAsEQAAEvEVAHjLl/dFdu3bpXzWqUqWKt0xBuz5CAELqIwOJboAACNhPIDQ0lLp3725/AeQEgWwIGHsBLRvDcQsEQAAEQAAEjEAAQmqEUYANIAACIAACpiUAITXt0MFwEAABEAABIxCAkBphFGADCIAACICAaQlASE07dDAcBEAABEDACAQgpEYYBdgAAiAAAiBgWgIQUtMOHQwHARAAARAwAgEIqRFGATaAAAiAAAiYlgCE1LRDB8NBAARAAASMQABCaoRRgA0gAAIgAAKmJQAhNe3QwXAQAAEQAAEjEICQGmEUYAMIgAAIgIBpCZhGSOPjE0wL2WJ4SkoK3YyNtZya6lM24LakpKRkyyE+QQAEQCDXEzD87i9RUZdp6rRFVL5cGdqwcSeNHN6XGjWqbcqBGzdjKlW8J5See+ppU9l/6Oif1OGZflSrelWKi4+nsaPeotYtGpmqDzAWBNxJIO5mHB3eeZjyBxZwZzN3Xfe50+epceWmGerJmzcvhZ85RfnzBmS4Z7QLZ86fM5pJZHgh/fqbpSTeUJ/XetDVq9dozKdf08rlrtmA3JOjcTz8T1r54zoa9EpfTzbrkrZkI+RB/V6gxg1qU8umDSggwPj/s7mk46gEBOwk8NI/X6Ho6Gg7c3svW+OyRFWrVs1ggGwr90zvlyghwfgzf/XbPpDBfm9fMLyQvjmgJwtpmuZ0IeoShYVV8TYzh9uXB4E1mzdQx4cfdbisUQrcV7cm1bi3Evn5+RnFJNgBAoYhEBQURPJn5lSxYkUzm+9V2w2/RhocHEQxMddp8NDxdJqnJd4b9bpXgTnT+II1K+j5p7qaVoREPKfPXUa79x+mXv1Hkkz1IoEACIAACPyPgOGFVMysXr0SDRvSmwID89NHY6abauzO8ny+CFGFe8qZyu70xtarVY0WTPuEunZqS71f6EqDRo1PfxvHIAACIJCrCRheSL9bu5UuX46hEiWK6XXSseNm8Tx+omkGbez0KXTp6hWaNHMa7Tt0kDZu/5m27NpuGvvF0K9mL6Wz56K0zeXuCaGDh4+byn4YCwIgAALuJGD4NdKPP/maPh0ziB5+6H727IgqVw6l/PnzuZOJS+v+6sOx1voioy5Qo7r1qW3L1tZrZjj4eeev9MSj/7M5ggW1Zo0qZjAbNoKAxwhERUXR2TNnPNbe3TRUv0EDnt0LtKlCKUVHjhymeI7KN3oqWrQo1aplrDc3jC+kH/6L9u07Qtevx9Lkz+fRhx8MNOVa47L1a2nv7wfodGQEVQwtRw80ud/o/16t9r3dvxd98+1Kat6kPk2bs5Q+/3io9R4OQAAEiDasWkU1iweTfx5/Q+OIT0yg7TEx1P6JJ2zsjIiIoL+O7KJKoSVtrhvx5IctR1hIPzaUaYYX0nZtW7A32owOH/6TVq+Yoqd4DUXQTmO6P9GZ5M+MqQW/8tKofi06eiKcVs6ZRAUL2j7NmrFPsBkEXElA3sOsxlGvhe7w9FzZhivq+osF81IWr68FFshP9WpVd0Uzbq3jzPkYt9bvTOWGF1LplL+/PzVsWMuZ/qGMiwjIdHojfo8UCQRAAARAwJaA4YONbM3FGQiAAAiAAAgYiwCE1FjjAWtAAARAAARMRgBCarIBg7kgAAIgAALGIgAhNdZ4wBoQAAEQAAGTEYCQmmzAYC4IgAAIgICxCEBIjTUesAYEQAAEQMBkBCCkJhswmAsCIAACIGAsAhBSY40HrAEBEAABEDAZAQipyQYM5oIACIAACBiLAITUWOMBa0AABEAABExGAEJqsgGDuSAAAiAAAsYiACE11njAGhAAARAAAZMRgJCabMBgLgiAAAiAgLEIQEiNNR6wBgRAAARAwGQE3LKNWnJyCp05c97wKBKTkrK0MTExkSIunMvyvlFupKamZmqKXD8ffYny5DH2s1JCYhKpAJVpH3ARBHIzgbj4eMPvb5rV+CQnJ1PA/+97Kt9Ffn5+hv8uyqov9lx3i5DWCGtE4adu2NO+V/OkpART1apVM9hQp04d2ng2gi6mJWS4Z7QLzR98IFOTGt//AJ3hBwE/v7RM7xvlYmJiAD3w4H1GMQd2gIDXCZw+d44mzptHgfkL0K9H/6BJQ4ZSw1o1vW6XIwaMnvA1rfnhJypTuiQLal5a8s04KlqksCNVmCqvW4S0eYvMv9zNQiaQd7l/6umuZjE3UzvrN2hI8ocEAiBgLgJTFy+hYa++SuVCQmjWylXUb/Ro2rdksak6UTCwAI1//99UvlwZql0jo7Niqs7YYayx5/3s6ACygAAIgIAvEUhMTqKj4eG6S2FVKtPJyEjTdU+80LZtmlHxYkVNZ7szBrvFI3XGEJQBARAAARAgmjJ8uBXD1j17qWfnTtZzsxzExt2icVPmUMXy99CPW3bSjEnvUsGCgWYx32E7IaQOI0MBEAABEHA/geMnT9KOg7/R8smT3d+Yi1sY/MZLVOj/hXP3r4dp3tK11O+lHi5uxTjVYWrXOGMBS0AABEBAE4i+coXGzpxFK1hE/Q0eeX/nkMXHJ9CHHGyk1P+i8SuElqGDh0/cmc2nziGkPjWc6AwIgIDZCdyMi6Nxs2fTlyNHUFCRIiTBR2ZKSfzqy54Dh/UrL2L32XNRFFa9spm64LCtEFKHkaEACIAACLiPQJ/3P6BpS5ZSmTYPUmDDRrR59273NeaGmoOKFqFO7dvQjHnLaeaCVXQ28gL1e7G7G1oyTpVYIzXOWMASEAABEKBF48eZnsLb/V+k6EtX6ErMNXrluX9YvVPTdyyLDkBIswCDyyAAAiAAAs4TkB9jkL/ckDC1mxtGGX0EARAAARBwGwEIqdvQomIQAAEQAIHcQABCmhtGGX0EARAAARBwGwEIqdvQomIQAAEQAIHcQABCmhtGGX0EARAAARBwGwEIqdvQomIQAAEQAIHcQABCmhtGGX0EARAAARBwGwEIqdvQomIQAAEQAIHcQABCmhtGGX0EARAAARBwGwEIqdvQomIQAAEQAIHcQABCmhtGGX0EARAAARBwGwEIqdvQomIQAAEQAIHcQABCmhtGGX0EARAAARBwGwHs/uI2tKgYBEAgNxFQSlFaWpqhuyw2ZpeMbn92tnvzHoTUm/TRNgiAgE8QKBgUREt37qKAgABD9+fva9eoY7duGWwMDg6mE6ei6fylXzLcM9qFhKRUo5lEfvyEkv0jiuFMhkEgAAIgAAIgYBwCWCM1zljAEhAAARAAARMSgJCacNBgMgiAAAiAgHEI/B9VPpTXDWCWTwAAAABJRU5ErkJggg=="
    }
   },
   "cell_type": "markdown",
   "id": "2c294c56",
   "metadata": {},
   "source": [
    "## Einsum Syntax with Transpose\n",
    "\n",
    "Let's look at transpose to make the basic syntax clear. \n",
    "\n",
    "![image.png](attachment:image.png)\n",
    "\n",
    "The first argument to [einsum](https://pytorch.org/docs/stable/generated/torch.einsum.html) is an _equation_ string describing the operation. And the second argument are the operands, the tensors on which to perform the operation. \n",
    "\n",
    "Consider _transpose_ operation $a'$ given by $torch.einsum('ij->ji', a)$. The equation for transposing matrix $a$ is $ij->ji$. The left hand side (LHS) of the equation labels each dimension of each operand with a subscript. i.e. the LHS tells us $i$ specifies the row dimension and $j$ the column. The right hand side (RHS) specifies which subscripts are part of the output and in what order. RHS tells us we want the subscript order to change; we want $j$ rows and $i$ columns.\n",
    "\n",
    "An alternate interpretation of the equation is: input tensor $a$ has shape $(i,j)$, and we want a result tensor of shape $(j,i)$. \n",
    "\n",
    "More generally, LHS is a comma-separated string of operand subscripts. RHS is a single string of subscripts specifying the output dimensions and their order. Note the actual alphabets for subscripts are irrelevant. This same equation could be $bs->sb$, as long as we are consistent on LHS and RHS.\n",
    "\n",
    "And, here is the code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1f59c24c-bb27-4790-9119-75aae891f231",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0, 1, 2],\n",
       "        [3, 4, 5]])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn.functional as F\n",
    "import math\n",
    "\n",
    "a = torch.arange(6).reshape(2, 3)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "da18153e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0, 3],\n",
       "        [1, 4],\n",
       "        [2, 5]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.einsum('ij->ji', a)"
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAakAAACQCAYAAACyLpSKAAAMQGlDQ1BJQ0MgUHJvZmlsZQAASImVVwdYU8kWnluSkEBoAQSkhN4EESkBpITQQu8INkISIJQQA0HFjiwquBZULGBDV0UUrIDYETuLYu8LKgrKuliwK29SQNd95XvzfXPnv/+c+c+ZM3PLAKB2kiMSZaPqAOQI88UxQX708UnJdFIPIAASUAY6gMDh5omYUVFhAJah9u/l3U2ASNtr9lKtf/b/16LB4+dxAUCiIE7l5XFzID4IAF7FFYnzASBKebNp+SIphhVoiWGAEC+S4nQ5rpLiVDneK7OJi2FB3AqAkgqHI04HQPUK5OkF3HSoodoPsaOQJxACoEaH2DsnJ5cHcQrE1tBGBLFUn5H6g0763zRThzU5nPRhLJ+LrCj5C/JE2ZwZ/2c6/nfJyZYM+bCEVSVDHBwjnTPM2+2s3FApVoG4T5gaEQmxJsQfBDyZPcQoJUMSHC+3Rw24eSyYM7jKAHXkcfxDITaAOFCYHRGm4FPTBIFsiOEOQacL8tlxEOtCvIifFxCrsNkszo1R+EIb0sQspoI/zxHL/Ep9PZRkxTMV+q8z+GyFPqZamBGXCDEFYvMCQUIExKoQO+RlxYYqbMYVZrAihmzEkhhp/OYQx/CFQX5yfawgTRwYo7Avzckbmi+2OUPAjlDg/fkZccHy/GCtXI4sfjgX7ApfyIwf0uHnjQ8bmguP7x8gnzvWwxfGxyp0Pojy/WLkY3GKKDtKYY+b8rODpLwpxM55BbGKsXhCPtyQcn08TZQfFSePEy/M5IREyePBl4MwwAL+gA4ksKaCXJAJBO19jX3wTt4TCDhADNIBH9grmKERibIeIbzGgkLwJ0R8kDc8zk/WywcFkP86zMqv9iBN1lsgG5EFnkKcA0JBNryXyEYJh70lgCeQEfzDOwdWLow3G1Zp/7/nh9jvDBMyYQpGMuSRrjZkSQwg+hODiYFEG1wf98Y98TB49YXVCWfg7kPz+G5PeEroIDwi3CB0Eu5MERSJf4oyHHRC/UBFLlJ/zAVuCTVdcD/cC6pDZVwH1wf2uDP0w8R9oGcXyLIUcUuzQv9J+28z+GE1FHZkRzJKHkH2JVv/PFLVVtVlWEWa6x/zI481dTjfrOGen/2zfsg+D7ahP1tii7AD2DnsFHYBO4o1Ajp2AmvC2rBjUjy8u57IdteQtxhZPFlQR/APf0MrK81knmOtY6/jF3lfPn+69B0NWLmiGWJBekY+nQm/CHw6W8h1GEV3cnRyBkD6fZG/vt5Ey74biE7bd27BHwB4nRgcHDzynQs5AcA+N/j4H/7OWTPgp0MZgPOHuRJxgZzDpRcCfEuowSdNDxgBM2AN5+MEXIEn8AUBIAREgjiQBCbD6DPgPheDaWAWmA9KQBlYDlaD9WAT2Ap2gj1gP2gER8EpcBZcAlfADXAP7p5u8AL0g3fgM4IgJISK0BA9xBixQOwQJ4SBeCMBSBgSgyQhKUg6IkQkyCxkAVKGlCPrkS1IDbIPOYycQi4gHcgdpAvpRV4jn1AMVUG1UEPUEh2NMlAmGorGoZPQdHQqWogWo0vRtWg1uhttQE+hl9AbaCf6Ah3AAKaM6WAmmD3GwFhYJJaMpWFibA5WilVg1Vgd1gzX+RrWifVhH3EiTsPpuD3cwcF4PM7Fp+Jz8CX4enwn3oC34tfwLrwf/0agEgwIdgQPApswnpBOmEYoIVQQthMOEc7AZ6mb8I5IJOoQrYhu8FlMImYSZxKXEDcQ64kniR3Ex8QBEomkR7IjeZEiSRxSPqmEtI60m3SCdJXUTfqgpKxkrOSkFKiUrCRUKlKqUNqldFzpqtIzpc9kdbIF2YMcSeaRZ5CXkbeRm8mXyd3kzxQNihXFixJHyaTMp6yl1FHOUO5T3igrK5squytHKwuU5ymvVd6rfF65S/mjiqaKrQpLZaKKRGWpyg6Vkyp3VN5QqVRLqi81mZpPXUqtoZ6mPqR+UKWpOqiyVXmqc1UrVRtUr6q+VCOrWagx1SarFapVqB1Qu6zWp05Wt1RnqXPU56hXqh9Wv6U+oEHTGKMRqZGjsURjl8YFjR5NkqalZoAmT7NYc6vmac3HNIxmRmPRuLQFtG20M7RuLaKWlRZbK1OrTGuPVrtWv7amtrN2gvZ07UrtY9qdOpiOpQ5bJ1tnmc5+nZs6n0YYjmCO4I9YPKJuxNUR73VH6vrq8nVLdet1b+h+0qPrBehl6a3Qa9R7oI/r2+pH60/T36h/Rr9vpNZIz5HckaUj94+8a4Aa2BrEGMw02GrQZjBgaGQYZCgyXGd42rDPSMfI1yjTaJXRcaNeY5qxt7HAeJXxCePndG06k55NX0tvpfebGJgEm0hMtpi0m3w2tTKNNy0yrTd9YEYxY5ilma0yazHrNzc2DzefZV5rfteCbMGwyLBYY3HO4r2llWWi5ULLRsseK10rtlWhVa3VfWuqtY/1VOtq6+s2RBuGTZbNBpsrtqiti22GbaXtZTvUztVOYLfBrmMUYZT7KOGo6lG37FXsmfYF9rX2XQ46DmEORQ6NDi9Hm49OHr1i9LnR3xxdHLMdtzneG6M5JmRM0ZjmMa+dbJ24TpVO18dSxwaOnTu2aewrZztnvvNG59suNJdwl4UuLS5fXd1cxa51rr1u5m4pblVutxhajCjGEsZ5d4K7n/tc96PuHz1cPfI99nv85WnvmeW5y7NnnNU4/rht4x57mXpxvLZ4dXrTvVO8N3t3+pj4cHyqfR75mvnyfLf7PmPaMDOZu5kv/Rz9xH6H/N6zPFizWSf9Mf8g/1L/9gDNgPiA9QEPA00D0wNrA/uDXIJmBp0MJgSHBq8IvsU2ZHPZNez+ELeQ2SGtoSqhsaHrQx+F2YaJw5rD0fCQ8JXh9yMsIoQRjZEgkh25MvJBlFXU1Kgj0cToqOjK6KcxY2JmxZyLpcVOid0V+y7OL25Z3L1463hJfEuCWsLEhJqE94n+ieWJneNHj589/lKSfpIgqSmZlJyQvD15YELAhNUTuie6TCyZeHOS1aTpky5M1p+cPfnYFLUpnCkHUggpiSm7Ur5wIjnVnIFUdmpVaj+XxV3DfcHz5a3i9fK9+OX8Z2leaeVpPele6SvTezN8Mioy+gQswXrBq8zgzE2Z77Mis3ZkDWYnZtfnKOWk5BwWagqzhK25RrnTcztEdqISUedUj6mrp/aLQ8Xb85C8SXlN+VrwR75NYi35RdJV4F1QWfBhWsK0A9M1pgunt82wnbF4xrPCwMLfZuIzuTNbZpnMmj+razZz9pY5yJzUOS1zzeYWz+2eFzRv53zK/Kz5vxc5FpUXvV2QuKC52LB4XvHjX4J+qS1RLRGX3FrouXDTInyRYFH74rGL1y3+VsorvVjmWFZR9mUJd8nFX8f8uvbXwaVpS9uXuS7buJy4XLj85gqfFTvLNcoLyx+vDF/ZsIq+qnTV29VTVl+ocK7YtIayRrKmc23Y2qZ15uuWr/uyPmP9jUq/yvoqg6rFVe838DZc3ei7sW6T4aayTZ82Czbf3hK0paHasrpiK3Frwdan2xK2nfuN8VvNdv3tZdu/7hDu6NwZs7O1xq2mZpfBrmW1aK2ktnf3xN1X9vjvaaqzr9tSr1Nfthfslex9vi9l3839oftbDjAO1B20OFh1iHaotAFpmNHQ35jR2NmU1NRxOORwS7Nn86EjDkd2HDU5WnlM+9iy45TjxccHTxSeGDgpOtl3Kv3U45YpLfdOjz99vTW6tf1M6JnzZwPPnj7HPHfivNf5oxc8Lhy+yLjYeMn1UkObS9uh311+P9Tu2t5w2e1y0xX3K80d4zqOX/W5euqa/7Wz19nXL92IuNFxM/7m7VsTb3Xe5t3uuZN959Xdgruf7827T7hf+kD9QcVDg4fVf9j8Ud/p2nmsy7+r7VHso3uPuY9fPMl78qW7+Cn1acUz42c1PU49R3sDe688n/C8+4Xoxee+kj81/qx6af3y4F++f7X1j+/vfiV+Nfh6yRu9NzveOr9tGYgaePgu593n96Uf9D7s/Mj4eO5T4qdnn6d9IX1Z+9Xma/O30G/3B3MGB0UcMUf2K4DBiqalAfB6BwDUJABo8HxGmSA//8kKIj+zyhD4T1h+RpQVVwDq4P97dB/8u7kFwN5t8PgF9dUmAhBFBSDOHaBjxw7XobOa7FwpLUR4Dtgc9TU1JxX8myI/c/4Q988tkKo6g5/bfwGaJXw+1TmUeAAAADhlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAAqACAAQAAAABAAABqaADAAQAAAABAAAAkAAAAACr4bEJAAAaRklEQVR4Ae2dCXQVRb7GvxCyLyRACCEsAVnCIgFk3zkyCgIiMjICI8jDQUDJgPNYHA/KENnFgTfosAVZFGQRlchDRCCA7LywmAENRNYsBBKWkH25r6tm+krMDYSQe2/1zVfnJN1dXV31r9+/k69r6S4nkxbA4DAEDhw4gG7dujlMfVgREiCBik2gUsWuPjBr1ix06dIFLi4u8kfsz5w50y5YTp48iZycnDKXvX79eowcObLM1/NCEiABElCNgBNbUkB2djY8PDykb8S+m5ubXfzUoEED7N+/H7Vr137k8tPT0xEWFoakpCRkZWU98vW8gARIgARUJFDhW1KqOOXMmTO4ePFimc2ZO3cupkyZIgX39u3bZc6HF5IACZCASgQqq2SMSrbs27cP27Ztw71792T33/bt23H69Gn06NED/fr1k62tqKgo7N27FxkZGRg2bBi+/fZbZGZmyu7DIUOGyOro+Yg0QkRCQkLwzjvvyHybNm2K8ePHY8WKFeYuxoiICPj5+WHSpEmoWbNmqZCcPXsWlStXRs+ePWV60ZoSeTCQAAmQgNEJsCVVggeDg4MRFxeH5cuX44033oC7u7uckDB48GCsWbNGXlW/fn0cPXpUplmwYAGGDx+Oxo0bY8SIEfJHJNLzWbZsGRISEuDk5IQmTZpgyZIl2LFjh8ynbdu2ePLJJ+X+008/jeeeew7+/v7yuDS/5s+fj6lTp6JWrVoyuRApBhIgARJwBAIUqRK82LBhQ7Rv316eFS0U0VLq06ePPN61a5fctmjRAi1btpT7ixcvhjgWgta1a1esW7cOogvv/nxEQiFSQ4cOldfov1q3bm1uNXXu3Fm21ko7LrZx40YMGDAAnp6e8PX1hZeXlxyX0vPmlgRIgASMTIDdfaXwnt7KETMARRCTKx4URJfg7t27ER0dbRaxB6Uv6znRFbly5UrZetuzZ4/MRnT7sSVVVqK8jgRIQDUCFKlSeKRSpUdrcObm5spcq1evbjF3IS4PCocPH0ZycjIGDRr0oGQQ3Xyi21B0H+ohNjaWIqXD4JYESMDwBB7tv6/hq2u9Cvz8888yczFxYvPmzahRowaeeeYZGRcYGCi3ujh9/fXX8jgvL09uxa+AgAC5L2bmiXGuh70vdfDgQdmiu1+gRAZiDCwxMVHmxV8kQAIkYHgC4j2pihxmzJhh0saBTK6uriatO0/uv//++6ZPPvnEpE2MMGljQ6ZmzZqZtJl9po4dO8p0mqCY3n77bYlt7Nix4osdJu0rDyZtRp8pKCjIpI1JmWJiYsxYU1JS5HlNUEzazECTNsFC5ivKnDhxokx37tw5kzZ+Zapbt65p4MCBprt375qv/+1OeHi4qWrVqqYOHTqYtFmD5tOzZ882aYJo0gTS9O6775rjuUMCJEACRiXAl3kf8zFj3LhxWLp0KS5cuCBn14kxIX3s6rdZi2nooutQvDiclpYmZwyKWYP3dyeK1pa3t/dvL+UxCZAACVRIAhyTKke361+tKClLMfNOD1pLSN8tsqVAFcHBAxIggQpOgC2px7gBFi1ahE2bNsmWU0FBAbZs2WKeSv4Y2fJSEiABEiCB/xCgSPFWIAESIAESUJYAZ/cp6xoaRgIkQAIkQJHiPUACJEACJKAsAYqUsq6hYSRAAiRAAhQp3gMkQAIkQALKEqBIKesaGkYCJEACJECR4j1AAiRAAiSgLAGKlLKuoWEkQAIkQAIUKd4DJEACJEACyhKgSNnQNWJ5ebEEPQMJkAAJkEDpCPCLE6Xj9NipCgsL5ZLw2pfL8d133z12fsyABEiABCoCAX5g1kZeFl86/+qrr6CvLWWjYlkMCZAACRiaAEXKRu47ceIEjh07Bj8/P2jrU9moVBZDAiRAAsYmwO4+G/nv2rVrGDVqFK5fv44zZ87YqFQWQwIkQALGJsCJEzbyX+3atdGiRQsblcZiSIAESMAxCFCkHMOPrAUJkAAJOCQBjkk5pFtZKRIgAWsROH78OH755Rdz9k5OTnLmrhhrDg4ONseX587JkycRExMD0SPz7LPPlmfWyufFlpTyLqKBJEACKhFwc3OTk6Befvll+d6js7OzFJBWrVrhtddeg1ilu7yDi4sL5s+fj1WrVpmzXr9+PU6dOmU+dtQdipQNPBsZGYkrV64gOzsb4gZnIAESMC6Bli1bom3btrIC4r3HwYMHY+rUqXJilPhb3759e7lXToxnN2jQoEi+a9euRVpaWpE4Rzxgd5+VvSpe4n333XchpqDv3LkT48aNs3KJzJ4ESMAeBMRDqAi+vr7m4hMSErBlyxakpKSgW7du6NOnjzx36NAhKWbu7u6oUaMGXn/9dSxfvlymGzhwIIKCgrB69WpkZGRgypQp8PDwMOeZnp6O6dOny/8n9erVk7OFJ06caD7vaDsUKSt7VLzEK96POnjwICZPnlzsacjKxTN7EiABKxIQL+ifP38ecXFx2LZtmxSanj17yhJv3rwp34mcNm2afDgVAiVeRendu7dsfYnrhOCIeCFS1atXl1shPKLVJARu0aJFGD9+fBGREl1/7du3l2WEhYWhU6dOVqyh/bNmd58NfCAGU4cMGUKBsgFrFkECtiQgBOmll16SX5K5ceMGxCQKPXz66aeyJSTO5+fny5bUmjVrIMawbt26hQEDBmDZsmVYunSpvKRHjx76pfDy8jJ3KZoj/7MjWl/66yyhoaFo3br1b5M41DFFyqHcycqQAAnYkoD4goxo+URERMjWzsqVK83Fixl5QpCOHj0qe1K6du0qu+nq1KmDzz//HD4+Pvj4448xaNAg5OTkmK/Td0wmk777wK1onYluQUcNFClH9SzrRQIkYFMC/v7+cnxIiMv+/ftlt15ubi46d+6M4cOHy5aTWAVh7969ECsiiO5BITCi1SS6C8WkKjE8kJeXJ+0WU85FsCRWojUlgshfjF1dunRJHjviL+cZWnDEirFOJEACJGANAmIGn/hJTEyU40aitSTGhoTA7Nq1Cz/99BOio6Mxd+5cuLq6YsGCBXKK+rp16zBs2DApLKIFde7cOYgJFKI1NmbMGClSYhxKiJcYxxZdgrGxsbh8+TLEBCwx5TwpKQn169eX41A//PCDTCsE609/+pM1qqpEnvx2nxJuoBEkQAKOQCArK0t+n7Nu3bpStPQ6iXh9hp4QHDF2JVpBlStXll2CejqxFV1/4rwQP/06sW8piBmFeqvK0nlHiKNIOYIXWQcSIAEScFACHJNyUMeyWiRAAiTgCAQoUo7gRdaBBEiABByUAEXKQR3LapEACZCAIxCgSDmCF1kHEiABEnBQAhQpB3Usq0UCJEACjkCAIuUIXmQdSIAESMBBCVCkHNSxrBYJkIDaBOLj4+UX0tW20v7WUaTs7wNaQAIkUAEJzJo1S354WnxZgqFkAlZ5mff48RhERe1DQEBgySUrcEa8+e3t7YTRo4cXs+aDJQvgWe3XNVyKJVAkIuFyAl4fNg7iDff7g/gm2NqPPkLdWrXuj1ZyPy0zE+HaonEMJFCRCIjPHf3444/o379/Rar2I9fVKutJ3b59F61aPat9HbjxIxtkywuyszO1m+Rri0U6e1RCr5e7WTynUuSPx2ORmppaTKTE01mXZk3RQVvRU/Wwft8+1U2kfSRQrgTu3r0ru/rE18s7duwo15Iq1wIcKDN29zmQM1kVEiABYxAQH54V3+h777335EOmMay2j5UUKftwZ6kkQAIVmID4KKxYMJHh4QQoUg9nxBQkQAIkQAJ2IkCRshN4FksCJEACJPBwAhSphzNiChIgARIgATsRsMrsPjvVhcWSAAmQgPIEDh8+jDt37shVe4WxYhIFQ8kEKFIls+EZEiABEih3Ahs2bMCBAwfQtGlTtG/fHiEhIeVehiNlSJFyJG+yLiRAAsoTmDdvHqKjo7UPCXijS5cucql45Y22o4EUKTvCZ9EkQAIVj4CHhwf69u1b8Spexhpz4kQZwfEyEiABEiAB6xOgSFmfMUsgARIgARIoIwGKVBnB8TISIAESIAHrE6BIWZ8xSyABEiABEigjAYpUGcHxMhIgARIgAesToEhZnzFLIAESIAESKCMBw4hUXl4u8vPzylhNNS7LyzO2/QUFBcjWlhdgIAESIAFbETDEe1Lr1i2EyWRCWtoNNG/eFr/73Uu24lMu5SRfu46oT7cj6rPt+PumBXiiaYNyydeWmWze+R1izp5FQsp15GpiGxkRAS/tfQ8GEiABErAmAeVbUnFxp3HiRDRGjPhvTJgwGytWvI+MjHRrMin3vH38fDB49Avw9PJAQX5Buedv7QxFC+rrPXswZ9JErJ0zB3fu3cOS9eutXSzzJwESIAEoL1JHjuxCUFCIdJWzszNq1AhGTMx+Q7nOy9sTVQOqGsrm+43NyMrC5cREZGpbEZo2aID4q1fvT8J9EiABErAKAeVFSnTx+flVM1fex8cPN28mmY+5Y30Cvto3xg6sWwtPrXuvsLAQe48ewysDnrd+wSyBBEigwhNQXqQKCvK1DzD+aqaTk5P8R1nhPWcnAAvXrMHzvXqh21Nt7GQBiyUBEqhIBH79769oratU8Udm5q9jUBkZd1Glyq8tK0XNdkizorQvN4tJE3978w2kZ2Q4ZB1ZKRIgAbUIKD+776mnemDjxo/M1K5fT0Dr1l3Nx0ba0SYoylmKRrJZt/Xw6dNyHOqdMWMgptKv2LIFb40cqZ/mlgQeSGDDhq3Izlb/FYysrAy88spL8PHxKVKf1NRUrN/6GXyr+BaJV/GgimcVvNB/UDHTdu3YgdSkJOWXBhETtXr06YPg2rVlHZQXqVatuuKbb9bh+++3aFPQU9Cr1wuoVi2wmANUjriecB0bPt6EKxeuYOn7K/D8K/3Ra0APlU0uYluatorooAnhSM/MxDuL/0d2ty6cPLlIGh6QwIMIJCenaw+X/R6URIlzsbGHcVWbFNSsWbMi9sTHxyMwrBrqNapbJF7Fg93r9uMFFBepK+fPo2/LJ1GpktodaNeSkxGv2WoYkRJjUNOnL0d8/L8gBKt69Zoq3hcPtCkwOBATZ02QPw9MqOjJqlWqIHn/PkWto1lGICD+MYpJT6oHb++SW0rOlSvD16/k86rUzd3dzaIpwgfi3UY3xZerT75xs0hrT/mWlE77iSea67vckgAJkAAJVBACarf7KogTWE0SIAESIAHLBChSlrkwlgRIgARIQAECFCkFnEATSIAESIAELBOgSFnmwlgSIAESIAEFCFCkFHACTSABEiABErBMgCJlmQtjSYAESIAEFCBAkVLACTSBBEiABEjAMgGKlGUujCUBEiABElCAAEVKASfQBBIgARIgAcsEKFKWuTCWBEiABEhAAQIUKQWcQBNIgARIgAQsE6BIWebCWBIgARIgAQUIUKQUcAJNIAESIAESsEyAImWZC2NJgAQUJ5CdnaW4hSWbJxb200NurvqLQeq26tuUtDS5Srd+bM1tkaU6YmNjcfnyZfTr93iLk2lLQOHs2SO4cyfZmrY/dt7Z2dnIyrpnMZ+0lDTE7D9l8ZxKked/PI/mvVoVM8nFxQUnf45D7n1/DMUSKRIh1o9hIIHSErh5MwmbNv0TgYG1cfjwTowe/Vc0bfpUaS9XIt2gVkPg6eUBsUZVh17tEB7xhhJ2ldaI5Zs3Y/mmzfDX1poT/+9FmPfWW3i2S5d/H5Tj7yIiNW3aNOzZswfp6elwdnYuczE9e3ZFnTpBZb7elhfWrGkZavhrE7WVgNNsaUqZymoX1AmNGzcudm1oaCicteXdCwsLi51TLSLsucd7KFKtPo5sj8lkktUTi5HaK3zxxQrtvi7A4MFjtAfhVKxaNRcLFmy2lzllKrdT747o9HQHtOrUEn7V1F8Mslgltdvg2+XL4KKJbE5uLuaujMTvOnUqlqw8IoqI1MKFC+XSyY8jUMKoyprhTZo0KQ/77JZHQEAAxI+RQ6NGjYxsPm1XkIAQpzlz5sgHoxdffLHICqq2Mnfo0AlSpER5N24kISQk1FZFl1s5AUHV0a7HU7i/26/cMrdBRn8c0B8hwcGypDkrVmLWn8Ottiy9WaRSU1MRFRWFjIwM9O7d2wbVZBEkUHoC+fn5+OCDD0p/AVNajUCu9uQ8adIkzJw5E++99x6EWNky+Pr6a8MScVi9egESEy9q98UXtiy+XMo6se//ZAsq6UoSfKr44NW3XimXfG2ViS5Q/7pwAQVaq7ZerVpWK9osUh4eHrh06RI++ugjeeNZrURmTAJlICCe4NkyLAM4K1xy69YtZGZmom3btnjiiSesUMLDs6xXrzFefXUKZs8eh8jI2Rg7dsbDL1IoxYebFsgxKWFS54Ce6D/8OVQPrKaQhaUz5cM1azB97NjSJS5jKrNIeXp6ok2bNmXMhpeRgHUJiC7owYMHW7cQ5v5QAmKMc8mSJdi9ezfCwsIemt4aCaKjt2lld4K/f4DWihuDiROfx6hR0+Dm5m6N4so9z5OHTuHqL9fw/B/7y7yD6gTibMw5dO/btdzLsmaGYixq58FDiIyIsGYxMItUeZaSnZWFU6dO2qW/+lHrUa9eCGoGFZ/kcfHiRVxPVnt2oqiraGG079ChGGsxwH38+DFDTJwICKhhtyfyR71fKnr6SpUqITw83K4YVq2agwkTZqNdu17yvg8Org9XVze72vQohV/4Vzyys3LkJWJMKiXxBuo3CXmULJRIe16bCe7pbv0HA6uI1JEjP8DdJRUentavwON4S9wgWzYfwJvhU4pls+3zzWjdILRYvGoRp87+CFc3N7Ru3bqIaeJ1gtuXz6B61SpF4lU82PjtNvx1xhwVTaNNChIYPz4CsbFHce/eHXz22WKMG/e3Yg9pCpptNum5oX3x4bTF2LllF47sPooRE/+IOg1qm88bZee6No/B19vb6uaaRSo+Pl7r242EeL9m3rx5mDp1apkLF0/3VbV/jo0bh5Q5D1tcmJmZhbM/pVosytXFFc0bNbF4TqVI0eQuKXh6eqBls+LT00tKb6/4M+cT7VU0yzUggY4de8tW1Pnzp7VJE1vg52essRwvb09MX/I2zsdeQJsurRAQZMxZxD20MckvFv3d6neQWaTEAOjBgwetXiALIAESIIHHJSDGKENDjT2G3qhFw8fFYNfrxatG1pzVp1eOn0XSSXBLAiRAAiSgHAGKlHIuoUEkQAIkQAI6AYqUToJbEiABEiAB5QhQpJRzCQ0iARIgARLQCVCkdBLckgAJkAAJKEeAIqWcS2gQCZAACZCAToAipZPglgRIgARIQDkCFCnlXEKDSIAESIAEdAIUKZ0EtyRAAiRAAsoRoEgp5xIaRAIkQAIkoBOgSOkkuCUBEiABElCOAEVKOZfQIBIgARIgAZ0ARUonwS0JkAAJkIByBChSyrmEBpEACZAACegEzEt16BGqbnNycrW1ripDrAxq5JB2+xaq+vkbrgp5eXlyrTFhuFgsUqwZZnRfGM4JBjZY3D+JiZeUr0FeXsnrs+Vm5yDhUoLydcjPz7doo/i7TUxJUf7vNjs3B2JlcT0YQqRmzV6KWrVq4PCRU2jerCH+HD5Ct99QW/GHOvD1kdjxyQZ4e3oZyvaZC5fhy//djZo1qsuHhY3LF8DXx/qrchoKEo0tkUD79qG4c+eXEs+rciIgIB9ibb3fhhYtWuDq91eQc6/wt6eUO+7erqdFm9p164bL167BqfBXAbCY0M6ROdqCs93vW2lceZGKi7uE5Ss34/Ivu9Gt61N4stVAvDb69/Dy8rQzykcvfvUXn+NqYkKRp4RHz8U+V3h4uOODGX9BneAgNG9S/I/YPlaxVKMQ6N69s1FMtWinp6cnBj//e4vnjBIZ1qYNxI/RgvJ9Z2IJ+u93RkquiYkpqFMnyJACFXcxXmt5+MLPt4rR7hFpr4u2Cmfv7h21rkpfQ9pPo0mABIxJQHmREljr1auFRYvX4i9T5uOrL/5hONKFhYXYsuMb/KHfQMPZrhuckZGJeUs+wZ4fjmH4uGnIzMzST3FLAiRAAlYjoHx3n6i5q6srJrw5HO7urhj5X28jevcaQ7WmNkRtxdABg6zmRFtkPPnNUfDy9JBFHTp2Cqs3bsP4UX+wRdEsgwRIoAITUL4lde5cPA4dOglnZ2e8PuYPiI+/im+2RxvGZVnZ2Vi/bSu+/G47Poz8J+7cu4t/rIlEQnKSceqQlY2IhUvNY2l1gmsi5sxZw9hPQ0mABIxLQPmWVNQ3e3HpciI6d24tKVeq5ITQJg0MQ9zD3R3bI9eb7V2ydhUmjBgNH2/jzIzL1WYlHj5xWk47FxW5ci0JoY2M4wMzfO6QAAkYjoDyLalxY4fKKc9ffvU93gyPQN8+3REWFmo40Dm5uYj4x0KIllXEkoVIvXXLMHWo4uuD/s/0xLK1m7Hy0624dC0R418dYhj7aSgJkIBxCSjfkvLx8cLiv/8VFy5cRovmDdGoUYghabtp42rTJ/xF/hixApPfeBXJKTdxM+02Rg8fZG5VGbEutJkESMA4BJQXKR1lw4b19F1u7URAvMgrfhhIgARIwFYElO/usxUIlkMCJEACJKAeAYqUej6hRSRAAiRAAv8hQJHirUACJEACJKAsAYqUsq6hYSRAAiRAAhQp3gMkQAIkQALKEqBIKesaGkYCJEACJECR4j1AAiRAAiSgLAGKlLKuoWEkQAIkQAIUKd4DJEACJEACyhKgSCnrGhpGAiRAAiRAkeI9QAIkQAIkoCwBipSyrqFhJEACJEACFCneAyRAAiRAAsoSsNpX0E0mEwoLC5WtuDBM2PigoLr9D7JdP+cIddDrwi0JkEDFI2AVkQoICETU9q3w949Tmmh+fgFqBNa2aGMlD1ds3v+dxXMqRd64cQNjn+lWzKSaNWsiautGnLuUUuycahGFTi6qmUR7SIAEFCHgpLUmHtycUMRQmkECJEACJFDxCHBMquL5nDUmARIgAcMQoEgZxlU0lARIgAQqHoH/B19p0d70BRbkAAAAAElFTkSuQmCC"
    }
   },
   "cell_type": "markdown",
   "id": "d6510c29-efc8-4881-9023-656aa13883ee",
   "metadata": {},
   "source": [
    "## Rowsum\n",
    "\n",
    "More interesting applications of einsum are when we want to sum along one or more dimensions. Let's look at rowsum, wherein we want to sum all rows of a matrix.\n",
    "\n",
    "In the einsum notation, any subscript that is absent in the RHS is summed out. Let's consider rowsum, with the equation $ij->j$. Note $i$ is not on the RHS. This tells us to sum across dimension $i$. Since the RHS has $j$, it tells us we want output a tensor with shape $(j)$\n",
    "\n",
    "Another way to interpret the einsum notation is to think of tensor operations as compressions. e.g., summing all rows can be thought of as compressing the tensor to a single row. \n",
    "\n",
    "![image.png](attachment:image.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4af6009e-f188-411e-a483-7d410fe639ea",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([3, 5, 7])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.einsum('ij->j', a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4c0ba615-557a-4fc4-9f32-e243c9b56cd8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([3, 5, 7])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.sum(a, dim=0)"
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAALEAAAErCAYAAACRo2JUAAAMQGlDQ1BJQ0MgUHJvZmlsZQAASImVVwdYU8kWnluSkEBoAQSkhN4EESkBpITQQu8INkISIJQQA0HFjiwquBZULGBDV0UUrIDYETuLYu8LKgrKuliwK29SQNd95XvzfXPnv/+c+c+ZM3PLAKB2kiMSZaPqAOQI88UxQX708UnJdFIPIAASUAY6gMDh5omYUVFhAJah9u/l3U2ASNtr9lKtf/b/16LB4+dxAUCiIE7l5XFzID4IAF7FFYnzASBKebNp+SIphhVoiWGAEC+S4nQ5rpLiVDneK7OJi2FB3AqAkgqHI04HQPUK5OkF3HSoodoPsaOQJxACoEaH2DsnJ5cHcQrE1tBGBLFUn5H6g0763zRThzU5nPRhLJ+LrCj5C/JE2ZwZ/2c6/nfJyZYM+bCEVSVDHBwjnTPM2+2s3FApVoG4T5gaEQmxJsQfBDyZPcQoJUMSHC+3Rw24eSyYM7jKAHXkcfxDITaAOFCYHRGm4FPTBIFsiOEOQacL8tlxEOtCvIifFxCrsNkszo1R+EIb0sQspoI/zxHL/Ep9PZRkxTMV+q8z+GyFPqZamBGXCDEFYvMCQUIExKoQO+RlxYYqbMYVZrAihmzEkhhp/OYQx/CFQX5yfawgTRwYo7Avzckbmi+2OUPAjlDg/fkZccHy/GCtXI4sfjgX7ApfyIwf0uHnjQ8bmguP7x8gnzvWwxfGxyp0Pojy/WLkY3GKKDtKYY+b8rODpLwpxM55BbGKsXhCPtyQcn08TZQfFSePEy/M5IREyePBl4MwwAL+gA4ksKaCXJAJBO19jX3wTt4TCDhADNIBH9grmKERibIeIbzGgkLwJ0R8kDc8zk/WywcFkP86zMqv9iBN1lsgG5EFnkKcA0JBNryXyEYJh70lgCeQEfzDOwdWLow3G1Zp/7/nh9jvDBMyYQpGMuSRrjZkSQwg+hODiYFEG1wf98Y98TB49YXVCWfg7kPz+G5PeEroIDwi3CB0Eu5MERSJf4oyHHRC/UBFLlJ/zAVuCTVdcD/cC6pDZVwH1wf2uDP0w8R9oGcXyLIUcUuzQv9J+28z+GE1FHZkRzJKHkH2JVv/PFLVVtVlWEWa6x/zI481dTjfrOGen/2zfsg+D7ahP1tii7AD2DnsFHYBO4o1Ajp2AmvC2rBjUjy8u57IdteQtxhZPFlQR/APf0MrK81knmOtY6/jF3lfPn+69B0NWLmiGWJBekY+nQm/CHw6W8h1GEV3cnRyBkD6fZG/vt5Ey74biE7bd27BHwB4nRgcHDzynQs5AcA+N/j4H/7OWTPgp0MZgPOHuRJxgZzDpRcCfEuowSdNDxgBM2AN5+MEXIEn8AUBIAREgjiQBCbD6DPgPheDaWAWmA9KQBlYDlaD9WAT2Ap2gj1gP2gER8EpcBZcAlfADXAP7p5u8AL0g3fgM4IgJISK0BA9xBixQOwQJ4SBeCMBSBgSgyQhKUg6IkQkyCxkAVKGlCPrkS1IDbIPOYycQi4gHcgdpAvpRV4jn1AMVUG1UEPUEh2NMlAmGorGoZPQdHQqWogWo0vRtWg1uhttQE+hl9AbaCf6Ah3AAKaM6WAmmD3GwFhYJJaMpWFibA5WilVg1Vgd1gzX+RrWifVhH3EiTsPpuD3cwcF4PM7Fp+Jz8CX4enwn3oC34tfwLrwf/0agEgwIdgQPApswnpBOmEYoIVQQthMOEc7AZ6mb8I5IJOoQrYhu8FlMImYSZxKXEDcQ64kniR3Ex8QBEomkR7IjeZEiSRxSPqmEtI60m3SCdJXUTfqgpKxkrOSkFKiUrCRUKlKqUNqldFzpqtIzpc9kdbIF2YMcSeaRZ5CXkbeRm8mXyd3kzxQNihXFixJHyaTMp6yl1FHOUO5T3igrK5squytHKwuU5ymvVd6rfF65S/mjiqaKrQpLZaKKRGWpyg6Vkyp3VN5QqVRLqi81mZpPXUqtoZ6mPqR+UKWpOqiyVXmqc1UrVRtUr6q+VCOrWagx1SarFapVqB1Qu6zWp05Wt1RnqXPU56hXqh9Wv6U+oEHTGKMRqZGjsURjl8YFjR5NkqalZoAmT7NYc6vmac3HNIxmRmPRuLQFtG20M7RuLaKWlRZbK1OrTGuPVrtWv7amtrN2gvZ07UrtY9qdOpiOpQ5bJ1tnmc5+nZs6n0YYjmCO4I9YPKJuxNUR73VH6vrq8nVLdet1b+h+0qPrBehl6a3Qa9R7oI/r2+pH60/T36h/Rr9vpNZIz5HckaUj94+8a4Aa2BrEGMw02GrQZjBgaGQYZCgyXGd42rDPSMfI1yjTaJXRcaNeY5qxt7HAeJXxCePndG06k55NX0tvpfebGJgEm0hMtpi0m3w2tTKNNy0yrTd9YEYxY5ilma0yazHrNzc2DzefZV5rfteCbMGwyLBYY3HO4r2llWWi5ULLRsseK10rtlWhVa3VfWuqtY/1VOtq6+s2RBuGTZbNBpsrtqiti22GbaXtZTvUztVOYLfBrmMUYZT7KOGo6lG37FXsmfYF9rX2XQ46DmEORQ6NDi9Hm49OHr1i9LnR3xxdHLMdtzneG6M5JmRM0ZjmMa+dbJ24TpVO18dSxwaOnTu2aewrZztnvvNG59suNJdwl4UuLS5fXd1cxa51rr1u5m4pblVutxhajCjGEsZ5d4K7n/tc96PuHz1cPfI99nv85WnvmeW5y7NnnNU4/rht4x57mXpxvLZ4dXrTvVO8N3t3+pj4cHyqfR75mvnyfLf7PmPaMDOZu5kv/Rz9xH6H/N6zPFizWSf9Mf8g/1L/9gDNgPiA9QEPA00D0wNrA/uDXIJmBp0MJgSHBq8IvsU2ZHPZNez+ELeQ2SGtoSqhsaHrQx+F2YaJw5rD0fCQ8JXh9yMsIoQRjZEgkh25MvJBlFXU1Kgj0cToqOjK6KcxY2JmxZyLpcVOid0V+y7OL25Z3L1463hJfEuCWsLEhJqE94n+ieWJneNHj589/lKSfpIgqSmZlJyQvD15YELAhNUTuie6TCyZeHOS1aTpky5M1p+cPfnYFLUpnCkHUggpiSm7Ur5wIjnVnIFUdmpVaj+XxV3DfcHz5a3i9fK9+OX8Z2leaeVpPele6SvTezN8Mioy+gQswXrBq8zgzE2Z77Mis3ZkDWYnZtfnKOWk5BwWagqzhK25RrnTcztEdqISUedUj6mrp/aLQ8Xb85C8SXlN+VrwR75NYi35RdJV4F1QWfBhWsK0A9M1pgunt82wnbF4xrPCwMLfZuIzuTNbZpnMmj+razZz9pY5yJzUOS1zzeYWz+2eFzRv53zK/Kz5vxc5FpUXvV2QuKC52LB4XvHjX4J+qS1RLRGX3FrouXDTInyRYFH74rGL1y3+VsorvVjmWFZR9mUJd8nFX8f8uvbXwaVpS9uXuS7buJy4XLj85gqfFTvLNcoLyx+vDF/ZsIq+qnTV29VTVl+ocK7YtIayRrKmc23Y2qZ15uuWr/uyPmP9jUq/yvoqg6rFVe838DZc3ei7sW6T4aayTZ82Czbf3hK0paHasrpiK3Frwdan2xK2nfuN8VvNdv3tZdu/7hDu6NwZs7O1xq2mZpfBrmW1aK2ktnf3xN1X9vjvaaqzr9tSr1Nfthfslex9vi9l3839oftbDjAO1B20OFh1iHaotAFpmNHQ35jR2NmU1NRxOORwS7Nn86EjDkd2HDU5WnlM+9iy45TjxccHTxSeGDgpOtl3Kv3U45YpLfdOjz99vTW6tf1M6JnzZwPPnj7HPHfivNf5oxc8Lhy+yLjYeMn1UkObS9uh311+P9Tu2t5w2e1y0xX3K80d4zqOX/W5euqa/7Wz19nXL92IuNFxM/7m7VsTb3Xe5t3uuZN959Xdgruf7827T7hf+kD9QcVDg4fVf9j8Ud/p2nmsy7+r7VHso3uPuY9fPMl78qW7+Cn1acUz42c1PU49R3sDe688n/C8+4Xoxee+kj81/qx6af3y4F++f7X1j+/vfiV+Nfh6yRu9NzveOr9tGYgaePgu593n96Uf9D7s/Mj4eO5T4qdnn6d9IX1Z+9Xma/O30G/3B3MGB0UcMUf2K4DBiqalAfB6BwDUJABo8HxGmSA//8kKIj+zyhD4T1h+RpQVVwDq4P97dB/8u7kFwN5t8PgF9dUmAhBFBSDOHaBjxw7XobOa7FwpLUR4Dtgc9TU1JxX8myI/c/4Q988tkKo6g5/bfwGaJXw+1TmUeAAAADhlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAAqACAAQAAAABAAAAsaADAAQAAAABAAABKwAAAAAZuTurAAAbcElEQVR4Ae2dCXQUxb7GvxCWhCyEfQnKKiFRZBGEpyCoyHVBkQuiwFHk6gHBC+LxyKo8NIoIIih4BENkkbAjQm6ugFcEEUFEEGSLENlCSCAEQkgIgTCvq3wzl2EmCZPunq6OX5+TpLu661/f/P5faqp7unoCHNoCLn9ZAps3b0anTp1s/frL2Vq9BeLfffdd3HvvvahQoYL8Eetvv/22BUqAXbt24fLly6Vue9GiRRgwYECp66tSMYA9se+pyM/PR3BwsKwo1itVquR7EANqNG7cGN9//z3q16/vc7ScnBy0bNkSp06dwqVLl3yur1IF9sQqZcMHLXv27MGRI0d8qOF+6KRJkzBy5EiIf8Lz58+777TZVnmb6VVW7qZNm7BmzRpcvHhRDi+SkpKwe/dudO7cGY899pjsrRMTE/Hdd98hNzcX/fr1w9q1a5GXlyeHJ3369JGvzRlHHCNM1rBhQ4wbN07GjY6OxtChQxEXF+cawsTGxiIiIgKvvvoq6tSpc1N89u/fj/Lly6NLly7yeNEbixi2XcRwgotvBLS3X3EyLH+0nkxWPnTokKN79+6yrFevXo6EhATH8uXL5fbs2bPlMb/99pvjnnvukWWasR1ie+bMmQ5tOOJ49tlnPeJoQwXHtWvXHHPnzpV1RHyx7Ny50/HII4/IsqVLlzo2btzocOqQB5TwSxsHO7R/Ekd2draM8e2335ZQQ+3dUFuemuq8mVgo1U7wpClmzJghhWvjTrndu3dv1wt56aWXZNnhw4ddZQ8++KAs03puWeaMI0wsFmFQ8U/jNLEoGzhwoCw7ceKE2LzpZcmSJY4VK1a4jg8JCXEsXLjQtW3HFQ4nTHgPbdGihYwqrmCIRYw7i1vEkEPrDaH1qLjzzjuLO1TXPjHUmTNnDpo1a4YNGzbIWGJYIYYTdl5oYhOyV66cb+fLBQUFUkWNGjW8qhHmK27ZunUr0tPT0bNnz+IOw+TJk6ENXxAVFeU6bu/evbY3sW+0XS+dK3oJJCcnyxDixE4bO6NWrVro1q2bLKtdu7b86zTv6tWr5faVK1fkX/GrZs2acl1cWfjpp59KvF68ZcsW+Y5wvYFFgMjISKSlpclYtv1lxzGQlZonTJggT84qVqzo0IYLcv2dd96RJ1+NGjWSJ2kxMTEOMb7t0KGDQxynGc4xZswYKds5JtY+JXNoVyQcdevWdXTs2FGerDlf1+nTpx1iv2Y4hzgB7N+/v4wrYo0YMUIeduDAAUfTpk0dt956q6NHjx6OCxcuOKt7/B0+fLijWrVqjvbt28sTOucBEydOdGj/MA7tH8gxfvx4Z7Ht/vLDDj93P0OGDMGsWbOgndihXr168lKXc+x8oxRxmU0MTcQHK1lZWQgKCpI/1w9XRG8dGhp6Y9W/1DbHxBam2/mpX1EStCsHrl1aT+pav37lr25gwYI98fWOMHl9+vTpWLZsmbznorCwENqlrpv+gMJkabYOTxPbOn0ULwjw6gR9YHsCNLHtU8gXQBPTA7YnQBPbPoV8ATQxPWB7AjSx7VPIF0AT0wO2J0AT2z6FfAE0scIeENOXxBQnLsUT4Cd2xfOxbK82LQlVq1aFducZ1q9fb5kOOzTMG4AUzZK4U+2rr76C895iRWUqIYsmViINniJ27NiB7du3y1nI2v3JngewxEWAwwkXCrVWUlNToU0GRUZGBsQzJrgUTYAndkWzsXSPeKrPHXfcYakGuzROE9slU9RZJAGauEg03GEXAjSxXTJFnUUSoImLRGPdjvj4eBw/flxOsbfqiZvWvXrfW+YlNt+ZmVpDfMihTZ+HuMS2bt06iNnRXIonwEtsxfOxZO/JkychHnbStm1biGcQcymeAE1cPB/utQEBjoltkCRKLJ4ATVw8H+61AQGa2AZJosTiCdDExfPhXhsQoIltkCRKLJ4ATVw8H+61AQGa2AZJosTiCdDExfPhXhsQoIltkCRKLJ4ATVw8H+61AQGa2AZJosTiCdDExfPhXhsQUOZWTO1rZW2ACxDfnVG9enUPrdpXzEL71iOPchULxJ1xgYGBHtLEPcyXL1/2KFetQHxPifatUy5ZSph43759SEj4D5o2VX9i5JEjKxEbO9oF0LkyZ2EcqjWpgoqVKjqLlPx7Of8y0jLS0LljZzd94vbP5XGfoUXjJm7lKm7s/iMFr8e+45KmhInFf39MTAdER9/lEqbqSlbWH16lhYaHoNV9dyIoOMjrflUKjyQfRWHGVQ854osemzdsiPtatfTYp1rB+Ru+ZphjYtUyRD0+E6CJfUbGCqoRoIlVywj1+EyAJvYZGSuoRoAmVi0j1OMzAZrYZ2SsoBoBmli1jFCPzwRoYp+RsYJqBGhi1TJCPT4ToIl9RsYKqhGgiVXLCPX4TIAm9hkZK6hGgCZWLSPU4zMBmthnZKygGgGaWLWMUI/PBMq0iS9dyvUZiEoVxD2+dl4KCwuR74eZIkrcFG90og4e3IWVKz9DVFRLpKTsx8iRHyEgIMDoZkyLl56agcSFSUhMSMK0ZVPQJLqxaW2ZFXj5uvXYuX8/Tp7OQIH2zxgfG4uQ4GBTmiuTPfGbbw7AK69MQu/eLyEoqDLWrl1sCjyzgoZFhKHXC0+ickgwCq8WmtWMaXFFD7x6wwa89+oILHjvPWRfvIiZixaZ1l6ZM3Fa2lGcO3cGoaFVJLSGDaOwdau9vuA7JLQyqtWsZlrSzQ6ce+kSjqWlIU/7K5ZobWJqyokTpjVb5kyclXVa+z7k/85GDguLwNmz6aYBZGBPAuHabOTNXyxAZW34IL5I57uftuPZx5/wPNCgkjJn4mvXCrXxr/vLEiC5WENg6vz5eOL++9HprjamCXDPtmnN+C9weHg15OZecDUo1qtU+W/P7NrBFdMJJG7cKE/q3vrny8jJNe9KUZkzcYMGzVChQkXtISD5MkkZGSfQpk0n0xNmRgMOB+AQv2y4bN29W46Dxw0aBHGpMG7FCtNeRZkzsbiUNmzYRHz66Xhs2/YfJCfvxhNPPG8aQDMCZ5zMwPRxM3D88HHMeicO3yVuMqMZ02JmaU9D6jlsOMZ99DFC2rZDePsOCKpYybT2yuR14q5de2u9733IysrAlCnLTYNnVuDakbUx4t1h8sesNsyMW61KFaR/779/vDJpYpGgatVqyR8zk8XYahAoc8MJNbBShT8J0MT+pM22TCFAE5uClUH9SYAm9idttmUKAZrYFKwM6k8CNLE/abMtUwjQxKZgZVB/EqCJ/UmbbZlCgCY2BSuD+pMATexP2mzLFAI0sSlYGdSfBGhif9JmW6YQoIlNwcqg/iSgy8R79+5FUlKSP/WyLRLwIKDrVszRo0djgzY1Oycnx+vXrHq0VkSB+KrZPXsWa7OUjxRxhDrFhYV/zuC9UVFAYQB++HKbNqtEF9Ibwxq+nXHqNB67r7tH3JCQEOw7dhzZl/6cEeNxgEIFp86dc1MToE1/KfX8l+TkZJzQpmJ37drVLSg3SMCfBEo9nDh79iwSExOxZcsWf+plWyTgQaDUJg7Wnilw9OhRTJgwwSMoC0jAnwRKbeLKlStr89jMe5aAPyGwLXsTKLWJ7f2yqb4sEVDiVFpc3Vi9ainCwkKVZxuiPeOt60OPeOj84fvNSDuRinLl1O4XxMP+Ot3fGfUiI91eQ35+PhJXrUD5CoFu5SpuBAZWwBM9e7ukldrEKSkpiI+P1y4pVcD777+PUaNGuYL6unLo0CE0bRyKxo3r+1rV78cvWbbJq4kP7tmLe5q1UN7EZ85m4sD+Ax4mTk9PR3BAHtpENfU7U18bXLtph1uVUpu4SZMmhl6ZENdXa9VS/3FTISGV3QA6N0QPHB4ahuCgIGeRkn/PX8jG1SLeLcoHBqJWDfWfxhkeHubGVu33Pjep3CAB7wRoYu9cWGojAjSxjZJFqd4J0MTeubDURgRoYhsli1K9E6CJvXNhqY0I0MQ2ShaleidAE3vnwlIbEaCJbZQsSvVOgCb2zoWlNiJAE9soWZTqnQBN7J0LS21EgCa2UbIo1TsBmtg7F5baiECZNnF+/mUbpcK71Kzz7tPTvR+lXqn4AkbnIm7EN/OriUt9P7FToIp/V6/5FseOpWnfKlqArdt+xdLFH8qb91XUWpwmYYQegwfg67mLEVo5pLhDldv39tTZWPXvb1GnVg35LI6ln01BuEkzd8qkiceMm4aEBZPRunUMWrf9O1Z++Q2eefpR5RJdkqB5K5fgRNpJW341bnBwED6Y8BpuiayL26OalPRSde0vkyZOWjMLt9xSB+JtLDPzHJrd1lAXJCsq/34kReu5whERXsWK5nW3WaF8eXS9rwPOnDV/OFQmx8SNGtXH+m+24NHugzF29CDt0QIxupPizwBi/Lji63/h6cd6+LNZQ9vKzc3D+zPnYsMP29F/yGjk5Xl//JcRjZbJnliAefSRzqhTuyb6P/c6WrWMxv/8TysjePklxuLEL9H38Z5+acusRl7/50CEVA6W4X/c/ivmLV2DoQOfNqW5MtcTX9IeiJewKFHCEj1wx3vbYOq0uabAMyPoJW3q/KI1X2LV+iR8GP8psi9ewIz58TiZfsqM5kyJKXIQO3WWayx/S2Qd7Nyz35S2RNAy1xOnp2di5OgP0Oeph+UVCTELuXnzxqYBNDqwmC2dFL/IFXbmgs8x7LkXEBaq/jM5nKILtKsqW3fsRkBAgCw6nnoKzW8zLwdlzsRiPPzykH5YvOTfKCi4gm0/7cbapDgnX9v8vVxQgMmzZ0D0zLEzp2LU4OGoXrWqLfRX0abUd+/WBbMXLEdguUAcTU2TVyrMEl/mTCxAjR0zGNnZOUhJOYGfty1DxYoVzeJnWtxKmuY3h70mf0xrxMTAr7/8PNJPZyIz6zxe6N/T1Sub0WSZNLEAVaVKmO2uSpiRYCtjig86xI/ZS5k7sTMbGOOrR4AmVi8nVOQjAZrYR2A8XD0CNLF6OaEiHwnQxD4C4+HqEaCJ1csJFflIgCb2ERgPV48ATaxeTqjIRwI0sY/AeLh6BGhi9XJCRT4SoIl9BMbD1SNAE6uXEyrykQBN7CMwHq4eAZpYvZxQkY8ElLkVUzwjIi3ttI/y/X+4mEHtbRHl57LP43zOn7MZvB2jQpmYdeFweL+/+srVq0hLVz8HVzWd1y9KmDgqKgpJ/9qHcxcyr9em5HrzaO9fyt6yXRukaF+L65ySo6R4TdSlgnzcf0d7D3l169bFnqDq2HPsgsc+1Qoiajd0kxTg0Ba3Em6QgM0IcExss4RRricBmtiTCUtsRoAmtlnCKNeTAE3syYQlNiNAE9ssYZTrSYAm9mTCEpsRoIltljDK9SRAE3syUabkl19+QUZGhjJ6VBVCE6uaGU3X4sWLcfDgQYUVqiGNJlYjD1ShgwBNrAMeq6pBgCZWIw9UoYMATawDHquqQYAmViMPVKGDAE2sAx6rqkGAJlYjD1ShgwBNrAMeq6pBgCZWIw9UoYMATawDHquqQYAmViMPVKGDAE2sAx6rqkGAJlYjD1ShgwBNrAMeq6pBgCZWIw9UoYMATawDHquqQYAmViMPVKGDAE2sAx6rqkGAJlYjD1ShgwBNrAMeq6pBgCZWIw9UoYMATawDHquqQYAmViMPVKGDAE2sAx6rqkGAJlYjD1ShgwBNrAMeq6pBgCZWIw9UoYMATawDHquqQYAmViMPVKGDAE2sAx6rqkGAJlYjD1ShgwBNrAMeq6pBgCZWIw9UoYMATawDHquqQYAmViMPVKGDAE2sAx6rqkGAJlYjD1ShgwBNrAMeq6pBgCZWIw9UoYMATawDnllVr1275hb6xm23ndwATaygCQoLCxEbG4srV64gNTUVcXFxCqpUR1J5daRQiZNAhQoVULduXcyZMwdffPEFfv31V+cu/vVCgD2xFygqFA0YMADlypXD008/jVtvvVUFScpqCHBoi7Lq/uLCFixYgC5dutDEJfiAJi4BEHerT4DDCfVzRIUlEKCJSwDE3eoToInVzxEVlkCAJi4BEHerT4AmVj9HVFgCAZq4BEDcrT4Bmlj9HFFhCQRo4hIAcbf6BHjvhEE5+uyz+bh4ERD3Pai8FBQUoHPnlmjbtrXKMn3SRhP7hKvog69eDUD79j0RFBRc9EEK7Dl6NBnZ2ZkKKDFOAocTxrFkJIsI0MQWgWezxhGgiY1jyUgWEaCJLQLPZo0jQBMbx5KRLCJAE1sEns0aR4AmNo4lI1lEgCa2CDybNY4ATWwcS0ayiABNbBF4NmscAZrYOJaMZBEBmtgi8GzWOAI0sXEsGckiAjSxReDZrHEEaGLjWDKSRQRoYovA30yz+fmXbuawv/wxvCleQQtkZp7CsmWfonbt+ti6dR1eeGEsoqPvUlCpGpLYE6uRBzcVK1fG4dq1QvTqNQgxMW3x+eeT3PZzw50Ae2J3Hkps9e07TJpYiDlz5hQaNmyuhC5VRbAnVjAz4eFVtXlwWZg+fRTS0o5g0KA3FVSpjiSaWJ1cuClp0KAZnn9+JCpVCkZ8/ES3fdxwJ0ATu/NQYmvjxjU4d+4MIiKq4+9/H4R58ybj8uV8JbSpKIImVjArn3/+Hg4f3iuVBQQEIDKyESpWrKSgUjUk8cROjTy4qRg6NBZ79/6kPYwlGwkJH2HIkLcgzMzFOwGa2DsXS0s7dOiKdu3ux6FDu/HBByvksMJSQYo3ThMrmqDAwEA0b95GUXVqyeKYWK18UE0pCNDEpYDGKmoRoInVygfVlIIATVwKaKyiFgGaWK18UE0pCNDEpYDGKmoRoInVygfVlIIATVwKaKyiFgGaWK18UE0pCNDEpYDGKmoRoInVygfVlIIATVwKaKyiFgGXia9du6bN63L/sUKq0JCTk4NTp07h6tWrVkhgmzYjIE186NAhdOzYEVWrVoW4eyomJkZuR0dHa1PFozF06FAcO3bMLy9t8eLFaNCgAerVq4c9e/a42hTm3r59u2ubKyTgJCBvxbztttvw448/omfPnvjqq68wZcoUPP744/KYn3/+Gffeey9Wr16NXbt2oVatWs66pvzt378/jh49ijfeeMMt/s6dOzFixAip022Hzo1Vq1bh9ttvR7NmzXRFysnJ1m5k36LNwKioK47ZlU+c+AOdOul7rWZr9DV+ifcTt2vXDi1atIAw0TfffANhMrMXb18tK/6JjF7EsOWpp57CM888g4ULF+oKP3jws8jIyNAVwz+VI9G0aVP/NOWnVko0cWFhoRyfiukxN/ZW69evx7p167RnI5xBq1attGk0QxAcHIzc3Fx8/PHHOHjwIEJCQuTP2LFj5fBgzZo1cv/IkSO15yk0xLhx47RpOBddw5YbX7fYN2rUKMyePRvVq1eX6+Ld4LXXXrvxUJ+3w8LCkJiYqN18rv+5DhEREdoMjAifNbCCfgKuE7vrQ23ZsgVLly6VRnzggQfgcDgwf/58bcpMO9dh48ePR48ePfC3v/0NkydPlmZ49NFH5f7BgwdLo86bNw+jR4+WsUQvFRkZid9//10a8uTJk3LeWFRUFGbOnImvv/7aFfv6FdEr9+nTR75NCxOLNoQmI5Zt27Zh9+7d8l3GiHiMYQ0BryYWUipVqqTNuD0MMSauXbu2m3HElYOJEyfioYceQrdu3VCnTh158rdx40Y5bt67dy+WLFmCGTNm4OzZs5g1a5Y8RryN3X333a5XKnr3vn37ura9rQgdnTt3Rrly5WRPJ9ZbtzbmG+KF7oSEBO25Z8u8Nc0ymxDwOpwQJ3LixO7JJ5+UxomNjcW0adO0SYsfyJe1b98+iGFGcnIyXn75ZVl2/vx5eYKUl5eHMWPG4MUXX8Qrr7wi9911111F9rRWchLDmfr161spgW0bQMCria+P26hRI7kphhjOxTmGbNOmDT755BNnsRx2iKGH6IXFcEFc8di8eTOmTp2KuLg4iHHxjYsY8/q6fPjhh3juuedQo0YNX6uadvyptDQcP+6fy5B6X0SrVq1RKShIbxhl6pdo4po1a0qx+/fvlx+GiLFr5cqV5Vu8uFqRnp4uhwr5+fno3r27HGaIk7Vq1arh4Ycflj/iem9oaKiMI4YmYnGa13nV4cqVK7K8qF/iZO7cuXNyd1JSEsS4W6Vl7ddfonXLenLYo5KuG7Xk5eVjy5YcPPBgtxt32XY7QOs5HeLDjoEDByIlJUUaRVw3FkMAcWImhg39+vXD8uXL5Qcg4srDypUr5fViMZTYsWOHvGSTmZkpjxNXKFq2bIm6detCnP0Haf/x2dnZMpYwtriS0atXL5w+fVrWE2f0K1askL24+FBFnDyKk8bU1FSIkz5x3Vq8G8THx2P48OHycp84mXzrrbd0Qxcnr+IqiRhvz507F126dCl1zIULZqNXz7u1qzNq93DJyUeQkRmE+zobc3JcamAGVpQmvpl4Yswrel1hKHGydf0irrcKwzqXgoICeTXhwoULstcuX96zwxf/DMI84pJcVlaWNLswvCgrahGf2okeX7wTqLbQxNZlxNNdRWgp7jro9QYW1Z2fWoWHhxcRDfLasXOn6KFvZhEGV9HAN6Odx5hHoOhuz7w2GZkEDCVAExuKk8GsIEATW0GdbRpKgCY2FCeDWUGAJraCOts0lABNbChOBrOCAE1sBXW2aSgBmthQnAxmBQGa2ArqbNNQAjSxoTgZzAoCNLEV1NmmoQRoYkNxMpgVBGhiK6izTUMJ0MSG4tQfrKjJAbm5efqDl9EINLEiiU1NTcek9+PQpl0vHDiQ4lL1xx8n8NLQCXhj/Ee4p1Nf/PLLPtc+rvxJgCZWxAkREWF44R+9tPusK2vPoCt0qfp45kK8MfYlTJs6Bs8/1xMvDn7TtY8rfxKgiRVxQmhoCGrW9JwccPlyAfb8lixVxkQ30R6jcFwRxerIuOmZHepI/msp+fST/3W94PX/2YLnBzzp2ubKnwRoYps4Yd++Q9j0/Q6s/nKmTRT7TyZN7D/WpW4pPf0M3n1vNtas+qTYibSlbsDmFTkmViyB4uEz4se55OTkYuKkzzD707dQpUoYPp7xhXMX//4/AZpYESucPJmBcW9Olydu70ychcR/fSeV/ePFcZjxSQKq1miPgAoxWLv+B0UUqyPjpp87oY5kNZXwuRPW5YU9sXXs2bJBBGhig0AyjHUEaGLr2LNlgwjQxAaBZBjrCNDE1rFnywYRoIkNAskw1hGgia1jz5YNIkATGwSSYawjQBNbx54tG0SAJjYIJMNYR4Amto49WzaIAE1sEEiGsY4ATWwde7ZsEAGa2CCQDGMdAc7sMJC9uJldfE2Zysv1N9yrrNMXbTSxL7SKOTYoOBQLF21EYKDab25ZWefxRI8+xbwS++3iTfH2yxkV30BA7W7jBrHcJAFvBGhib1RYZisC/wdEzv9BQLyhrgAAAABJRU5ErkJggg=="
    }
   },
   "cell_type": "markdown",
   "id": "ff3bab9e-1328-48ea-9cc2-065123225e52",
   "metadata": {},
   "source": [
    "## Colsum\n",
    "\n",
    "Similarly, colsum $ij->i$ is summing all cols, or compressing all columns to a single column. We sum along dimension $j$ (missing on the RHS) to get a vector of length $i$.\n",
    "\n",
    "![image.png](attachment:image.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0a559a5d-06a8-445b-9192-3d3448ff2af5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 3, 12])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.einsum('ij->i', a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3136f761-8a2b-4bdd-913c-532dbccdcaf3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 3, 12])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.sum(a, dim=1)"
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAARgAAAEqCAYAAADOPE8qAAAMQGlDQ1BJQ0MgUHJvZmlsZQAASImVVwdYU8kWnluSkEBoAQSkhN4EESkBpITQQu8INkISIJQQA0HFjiwquBZULGBDV0UUrIDYETuLYu8LKgrKuliwK29SQNd95XvzfXPnv/+c+c+ZM3PLAKB2kiMSZaPqAOQI88UxQX708UnJdFIPIAASUAY6gMDh5omYUVFhAJah9u/l3U2ASNtr9lKtf/b/16LB4+dxAUCiIE7l5XFzID4IAF7FFYnzASBKebNp+SIphhVoiWGAEC+S4nQ5rpLiVDneK7OJi2FB3AqAkgqHI04HQPUK5OkF3HSoodoPsaOQJxACoEaH2DsnJ5cHcQrE1tBGBLFUn5H6g0763zRThzU5nPRhLJ+LrCj5C/JE2ZwZ/2c6/nfJyZYM+bCEVSVDHBwjnTPM2+2s3FApVoG4T5gaEQmxJsQfBDyZPcQoJUMSHC+3Rw24eSyYM7jKAHXkcfxDITaAOFCYHRGm4FPTBIFsiOEOQacL8tlxEOtCvIifFxCrsNkszo1R+EIb0sQspoI/zxHL/Ep9PZRkxTMV+q8z+GyFPqZamBGXCDEFYvMCQUIExKoQO+RlxYYqbMYVZrAihmzEkhhp/OYQx/CFQX5yfawgTRwYo7Avzckbmi+2OUPAjlDg/fkZccHy/GCtXI4sfjgX7ApfyIwf0uHnjQ8bmguP7x8gnzvWwxfGxyp0Pojy/WLkY3GKKDtKYY+b8rODpLwpxM55BbGKsXhCPtyQcn08TZQfFSePEy/M5IREyePBl4MwwAL+gA4ksKaCXJAJBO19jX3wTt4TCDhADNIBH9grmKERibIeIbzGgkLwJ0R8kDc8zk/WywcFkP86zMqv9iBN1lsgG5EFnkKcA0JBNryXyEYJh70lgCeQEfzDOwdWLow3G1Zp/7/nh9jvDBMyYQpGMuSRrjZkSQwg+hODiYFEG1wf98Y98TB49YXVCWfg7kPz+G5PeEroIDwi3CB0Eu5MERSJf4oyHHRC/UBFLlJ/zAVuCTVdcD/cC6pDZVwH1wf2uDP0w8R9oGcXyLIUcUuzQv9J+28z+GE1FHZkRzJKHkH2JVv/PFLVVtVlWEWa6x/zI481dTjfrOGen/2zfsg+D7ahP1tii7AD2DnsFHYBO4o1Ajp2AmvC2rBjUjy8u57IdteQtxhZPFlQR/APf0MrK81knmOtY6/jF3lfPn+69B0NWLmiGWJBekY+nQm/CHw6W8h1GEV3cnRyBkD6fZG/vt5Ey74biE7bd27BHwB4nRgcHDzynQs5AcA+N/j4H/7OWTPgp0MZgPOHuRJxgZzDpRcCfEuowSdNDxgBM2AN5+MEXIEn8AUBIAREgjiQBCbD6DPgPheDaWAWmA9KQBlYDlaD9WAT2Ap2gj1gP2gER8EpcBZcAlfADXAP7p5u8AL0g3fgM4IgJISK0BA9xBixQOwQJ4SBeCMBSBgSgyQhKUg6IkQkyCxkAVKGlCPrkS1IDbIPOYycQi4gHcgdpAvpRV4jn1AMVUG1UEPUEh2NMlAmGorGoZPQdHQqWogWo0vRtWg1uhttQE+hl9AbaCf6Ah3AAKaM6WAmmD3GwFhYJJaMpWFibA5WilVg1Vgd1gzX+RrWifVhH3EiTsPpuD3cwcF4PM7Fp+Jz8CX4enwn3oC34tfwLrwf/0agEgwIdgQPApswnpBOmEYoIVQQthMOEc7AZ6mb8I5IJOoQrYhu8FlMImYSZxKXEDcQ64kniR3Ex8QBEomkR7IjeZEiSRxSPqmEtI60m3SCdJXUTfqgpKxkrOSkFKiUrCRUKlKqUNqldFzpqtIzpc9kdbIF2YMcSeaRZ5CXkbeRm8mXyd3kzxQNihXFixJHyaTMp6yl1FHOUO5T3igrK5squytHKwuU5ymvVd6rfF65S/mjiqaKrQpLZaKKRGWpyg6Vkyp3VN5QqVRLqi81mZpPXUqtoZ6mPqR+UKWpOqiyVXmqc1UrVRtUr6q+VCOrWagx1SarFapVqB1Qu6zWp05Wt1RnqXPU56hXqh9Wv6U+oEHTGKMRqZGjsURjl8YFjR5NkqalZoAmT7NYc6vmac3HNIxmRmPRuLQFtG20M7RuLaKWlRZbK1OrTGuPVrtWv7amtrN2gvZ07UrtY9qdOpiOpQ5bJ1tnmc5+nZs6n0YYjmCO4I9YPKJuxNUR73VH6vrq8nVLdet1b+h+0qPrBehl6a3Qa9R7oI/r2+pH60/T36h/Rr9vpNZIz5HckaUj94+8a4Aa2BrEGMw02GrQZjBgaGQYZCgyXGd42rDPSMfI1yjTaJXRcaNeY5qxt7HAeJXxCePndG06k55NX0tvpfebGJgEm0hMtpi0m3w2tTKNNy0yrTd9YEYxY5ilma0yazHrNzc2DzefZV5rfteCbMGwyLBYY3HO4r2llWWi5ULLRsseK10rtlWhVa3VfWuqtY/1VOtq6+s2RBuGTZbNBpsrtqiti22GbaXtZTvUztVOYLfBrmMUYZT7KOGo6lG37FXsmfYF9rX2XQ46DmEORQ6NDi9Hm49OHr1i9LnR3xxdHLMdtzneG6M5JmRM0ZjmMa+dbJ24TpVO18dSxwaOnTu2aewrZztnvvNG59suNJdwl4UuLS5fXd1cxa51rr1u5m4pblVutxhajCjGEsZ5d4K7n/tc96PuHz1cPfI99nv85WnvmeW5y7NnnNU4/rht4x57mXpxvLZ4dXrTvVO8N3t3+pj4cHyqfR75mvnyfLf7PmPaMDOZu5kv/Rz9xH6H/N6zPFizWSf9Mf8g/1L/9gDNgPiA9QEPA00D0wNrA/uDXIJmBp0MJgSHBq8IvsU2ZHPZNez+ELeQ2SGtoSqhsaHrQx+F2YaJw5rD0fCQ8JXh9yMsIoQRjZEgkh25MvJBlFXU1Kgj0cToqOjK6KcxY2JmxZyLpcVOid0V+y7OL25Z3L1463hJfEuCWsLEhJqE94n+ieWJneNHj589/lKSfpIgqSmZlJyQvD15YELAhNUTuie6TCyZeHOS1aTpky5M1p+cPfnYFLUpnCkHUggpiSm7Ur5wIjnVnIFUdmpVaj+XxV3DfcHz5a3i9fK9+OX8Z2leaeVpPele6SvTezN8Mioy+gQswXrBq8zgzE2Z77Mis3ZkDWYnZtfnKOWk5BwWagqzhK25RrnTcztEdqISUedUj6mrp/aLQ8Xb85C8SXlN+VrwR75NYi35RdJV4F1QWfBhWsK0A9M1pgunt82wnbF4xrPCwMLfZuIzuTNbZpnMmj+razZz9pY5yJzUOS1zzeYWz+2eFzRv53zK/Kz5vxc5FpUXvV2QuKC52LB4XvHjX4J+qS1RLRGX3FrouXDTInyRYFH74rGL1y3+VsorvVjmWFZR9mUJd8nFX8f8uvbXwaVpS9uXuS7buJy4XLj85gqfFTvLNcoLyx+vDF/ZsIq+qnTV29VTVl+ocK7YtIayRrKmc23Y2qZ15uuWr/uyPmP9jUq/yvoqg6rFVe838DZc3ei7sW6T4aayTZ82Czbf3hK0paHasrpiK3Frwdan2xK2nfuN8VvNdv3tZdu/7hDu6NwZs7O1xq2mZpfBrmW1aK2ktnf3xN1X9vjvaaqzr9tSr1Nfthfslex9vi9l3839oftbDjAO1B20OFh1iHaotAFpmNHQ35jR2NmU1NRxOORwS7Nn86EjDkd2HDU5WnlM+9iy45TjxccHTxSeGDgpOtl3Kv3U45YpLfdOjz99vTW6tf1M6JnzZwPPnj7HPHfivNf5oxc8Lhy+yLjYeMn1UkObS9uh311+P9Tu2t5w2e1y0xX3K80d4zqOX/W5euqa/7Wz19nXL92IuNFxM/7m7VsTb3Xe5t3uuZN959Xdgruf7827T7hf+kD9QcVDg4fVf9j8Ud/p2nmsy7+r7VHso3uPuY9fPMl78qW7+Cn1acUz42c1PU49R3sDe688n/C8+4Xoxee+kj81/qx6af3y4F++f7X1j+/vfiV+Nfh6yRu9NzveOr9tGYgaePgu593n96Uf9D7s/Mj4eO5T4qdnn6d9IX1Z+9Xma/O30G/3B3MGB0UcMUf2K4DBiqalAfB6BwDUJABo8HxGmSA//8kKIj+zyhD4T1h+RpQVVwDq4P97dB/8u7kFwN5t8PgF9dUmAhBFBSDOHaBjxw7XobOa7FwpLUR4Dtgc9TU1JxX8myI/c/4Q988tkKo6g5/bfwGaJXw+1TmUeAAAADhlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAAqACAAQAAAABAAABGKADAAQAAAABAAABKgAAAAA+Etk1AAAgWklEQVR4Ae2dB3xVRdrGnzQSklBCSEIIJfRIDURAkLYCoqAiKoIodVdxG7ZdFVD0A2HRXWApriKKUgXyQcAWBFHWTyCIoSosvYSQkNADpJLzzRw3NzcNknDOPeU+8/vd3Dntfd/5vzdP5sw9mfFQRIEbl969e+PMmTM4cuQIatWqhaZNm2LGjBm46667XEolPz8fP/30Ezp16lRpvw8//DBatGiBt99+u9I2eCEJaEnAU0tjVrS1adMmTJgwAXl5eaqobNmyxeXiIrnt3LkTzz//fKURxsfH4+uvv0ZKSkqlbfBCEtCagNsLjNZAK2tv3bp1lb0U2dnZWL58OUaMGEGBqTRFXqgHAW89jNrB5ty5c9XbpoiICAwePBgrVqzA1atXIW9DOnbsqDZx6tSpOHv2LCIjIxEWFoaEhATUrFkTjz76KKKjo9VzCux4e3urt1579+7FkiVLVFvDhw9H27Zt8corr2D+/PkIDg5W66GhoXjppZfKjXHWrFlq7+err76C7IGxkIBZCLAHU0Ym2rRpg2XLluH999/HG2+8gV69eiE5ORldunTBuXPn1KvatWunHpdiIG9NnnnmGRw7dkw9Z9WqVeo5BXZkD0OW2rVr4+LFi+p1ctzHx8cHjz/+OKpUqaIKTP/+/XHPPfeo55bnx6lTp5Ceno6YmBjUrVuXPZjyQOM5riMgB3ndvSxevFgOdCsPPfRQERStW7dW96empqr7Fy5cqG6vXLnScZ7odSjt27d3bAvxUTw8PJTmzZs79kk7derUcWyLHoxqZ9GiRY59AQEBihAvx3Z5K2PHjlXOnz+vnv7ll1+qdrOyssp7Oc8jAV0J8BbpFloeEhKi3v7I02RvQxbxC6y+l/ZD3uYIQcG+ffvU3kR4eHhpp2myb+PGjWqv6vXXX1ftFfSshCCiYcOGmvigERK4HQIUmFvQ8/LyusUZJQ/LQVd5nRyPKa3IsZyblZkzZ6oDtvJ2qqySk5OjjuXIwWFPz1/vdNPS0iBvzeTtGgWmLHLc70oCHIPRgLb8xb506ZJqadu2bTh8+DAee+wxVK1aVd0nbo9w7do1h6e4uDi1npub69gnB3bl2Iws4lbHca3jhGKViRMn4tlnn3WIizwse1uylyWf62EhAVMQ0PUGzALGe/Toodxxxx2Kr6+vIn5Bla5duypCJJTRo0crogeiyLGRQYMGKd98840SFRWlnifHV8Ttido6OQZTo0YNpU+fPuoYTlBQkCIGe5XLly87Wr9+/XpFDMAq/fr1Uzp37qwMGTJEHSsRPRRlw4YN6nkffvih4u/vrx6fNGmS49riFTGoq/Tt21cR30opkydPdhyWYz/SrhgsVsSAr/LFF184jrFCAkYR8JCOTaF0Fg1C3sY0aNBAfVDuypUrqF69eqktkZiF6KjHZT0jI0PtpQhhc5wvn+aV4ztCaBz7WCEBKxPgGIyG2StLXKQL8c1SkTGZ0sZn5FgKxUXDhNCU4QQ4BnMbKRg4cKD6vz/VqlXD7373u9uwxEtJwJ4EeItkz7yyVSRgCgLswZgiDQyCBOxJgAJjz7yyVSRgCgIUGFOkgUGQgD0JUGDsmVe2igRMQYACY4o0MAgSsCcBCow988pWkYApCFBgTJEGBkEC9iRAgbFnXtkqEjAFAQqMi9KQmZmJTz75RP1/JBe5pBsSMJwABcZFKdi8eTPEf2ir87W4yCXdkIDhBPivAi5KgVwWJTY2FgMGDCjzP65dFArdkIDLCLAH4yLUcqY5OSH4nj17XOSRbkjAeAKcrsFFOWjSpAmefvppyLlgunfv7iKvdEMCxhJgD8ZF/MVMdkXmg3GRW7ohAUMJUGAMxU/nJGBvAhQYe+eXrSMBQwlQYAzFT+ckYG8CFBid8ytWXYRcn1oWOaG3XCKWhQTchQC/RdI50/v378e4ceNUcZELssnnYFhIwF0I8EE7F2Q6MTERcpF6sXYS5AThLCTgLgQoMO6SabaTBAwgwDEYA6DTJQm4CwEKjLtkmu0kAQMIUGAMgE6XJOAuBCgw7pJptpMEDCBAgTEAOl2SgLsQoMC4S6bZThIwgAAFxgDodEkC7kKAAuMumWY7ScAAAhQYA6DTJQm4CwEKjLtkmu0kAQMIUGAMgE6XJOAuBCgw7pJptpMEDCCgy3QNly9fRlpamgHNqbjLxo0bw8vLq8SF8r+f5fQKZi+BgYEIDw8vEaaceyYpKanEfjPuiIiIgL+/vxlDY0y3SUAXgZk/f4lY+6exmFzJ9zbD0/fy7OwstGx5Fj17diviKDk5GQvXLECzdk2L7DfjxuGdR/DmS1NKhLb+889w9eQphAYHlzhmph25ebnY6eeHIcNHmCksxqIRAV0EJjCwOtq06Q4/v6oahamPmRMnDiIv71wJ47m5uWjcshE69IouccxsO65fzCw1JG9PL3Rp0xoRoaGlHjfLzvSLF/FjaqpZwmEcGhPgGIzGQGmOBEigkAAFppAFayRAAhoToMBoDJTmSIAECglQYApZsEYCJKAxAQqMxkBpjgRIoJAABaaQBWskQAIaE6DAaAyU5kiABAoJUGAKWbBGAiSgMQEKjMZAaY4ESKCQAAWmkAVrJEACGhOgwGgMlOZIgAQKCVBgClmwRgIkoDEBCozGQGmOBEigkAAFppAFayRAAhoToMBoDJTmSIAECglYRmCuX79aGLVFa3KeGSuX65mZUBTFyk1g7C4moMuEU1q2Qc46984749CqVUfs2vUD/vSnqQgLq6elC91t7fj3T/g+fgt+/G4HVm5fqrs/PRxM+Ods+Hh7Y9vu3RjQsyeeG/6UHm5o02YETC8wCxf+De3adcVDD41CixbRqtjMmLHGUmmIbBGJ4LBgbFr7raXiLgj2+8RE1AsLwx+eGIpzYga6hn3vxQNCZJo0qF9wCt9JoFQCpr9F2rZtAyIiGqnBR0ZGYfv2b5Cfn19qY8y6M6RObfj5+5k1vFvGdSXjKg6dPKmeVzsoCCHiddQiE4rfsnE8QVcCpu/BXLiQhho1fp24OiCgGnJysnH9egYCA2voCobGCwk80Ksn5EuWX44cgbdYheE3nToWnsAaCZRBwPQ9mBs38uDpWTTMGzdulNEc7taTgByk/us/ZmDR36bBx8dHT1e0bRMCRX9zTdioGjVq4dq1K2pkWVnXhdh4QfZkWFxP4JVZs/D675/FXW3bQn6jxEICtyJgeoHp0KEHzp5NVtuRmpqEtm27wNubfz1vlVitj89ctAhD778fXdq1w/Z9+/DTL79o7YL2bEjA9GMwI0f+FTNmvIigoNr4/PNFGDdumuXSsHXjNny25AukJadh5vjZeGTMw4hs1tAy7Vi9cSMmzp6j3qoWDLAf3/C1ZeJnoMYRML3AyGde3nprCQ4f3osJE94z/WqRpaWya98ukC+rlkf79oV8sZBARQmYXmBkg+QStPJBOxYSIAFrETD9GIy1cDJaEiABZwIUGGcarJMACWhKgAKjKU4aIwEScCZAgXGmwToJkICmBCgwmuKkMRIgAWcCFBhnGqyTAAloSoACoylOGiMBEnAmQIFxpsE6CZCApgQoMJripDESIAFnAhQYZxqskwAJaEqAAqMpThojARJwJkCBcabBOgmQgKYEKDCa4qQxEiABZwJFBCYuLg6HDh1yPs46CZAACVSagGO6hoyMDAwePBhDhw7F0qVLK21QXujpeQM//rjW9DPPpaWliOVQfp3M2rnBAQEBOLb3BK5fNP+0kOfPXHAO3VH39ffH+p8SEVTD3JOjZ1y9ivCmTR1xs2IvAh5ipT7HUn3x8fGIiopCo0a/LhNir6ayNSRAAq4m4OjBJCQkYM+ePWJJkOsUGFdngf5IwKYEHGMwderUwbJly7Bq1SqbNpXNIgEScDUBh8BERkaiXj1rrfnsalj0RwIkUDECDoGp2GU8mwRIgARuTcAxBnPrU8t/xrat/4fUlKQSKzKW34JrzpQrRHbp2gvhdSOKOMzKysLa2NXiWzBd8BTxdbsb3j7eePixR0uYOfif/diV+CN8fX1LHDPTDvkdQ/OoVmjdpp2ZwmIsGhFw/AatXLkS+/fvV0Vh8+bN6NWrV6VdHD70M+7t08r0AnP27HkcPHighMCkpqbC63oOWjcx/7dp3+1MKDVPh4XAxDQNQYB/1VKPm2Vnbm4eEvbupsCYJSEax+EQmCFDhkC+tChyLekaNaqhalU/LczpZuPixStliqBc4D2kVrBuvrUyXL1a6cvoenh4oIpYPzq0di2tXOliJ/38Ba5zrQtZcxjlGIw58sAoSMCWBCgwtkwrG0UC5iBAgTFHHhgFCdiSAAXGlmllo0jAHAQoMObIA6MgAVsSoMDYMq1sFAmYgwAFxhx5YBQkYEsCFBhbppWNIgFzEKDAmCMPjIIEbEmAAmPLtLJRJGAOAhQYc+SBUZCALQlQYGyZVjaKBMxBgAJjjjwwChKwJQEKjC3TykaRgDkIWEZgMjOzzEHsNqO4cOnibVow5nI5OVdBycnJLajynQRuSsD0ApOSko7XJs3G4iXrMOjRPyMx8ZebNsjMB/cc+AV/enO8mUMsM7aYPkNw94Dh6DlwFKbO+qDM83iABJwJOCacct5ppvr8BSsh/3qOfWYIzp+/hGnT52N17BwzhViuWPLy8vD2/LnIFe9WLL17dMY93Trjrpi2CK5V04pNYMwGEDC9wDz35+FCYPJVNGdS0sTCcI0NwHT7LhetWYVhDz2CxeLdiqVOaAh6dIlx5MKKbWDMridgeoEJCqoh1ss+gbf//iGOH09G3Grr9V6OnjqBqn6+qBsW7voMa+Tx+22JCBa5SEpORfXqgXhh7HCNLNOMnQmYfgxGwm/ePBKvvvy0mOPXF29Ne99S+ZCz5q/4fC2eePARS8VdPNgVH7yDMcMG4Y2//h5vz12Is+nni5/CbRIoQcD0ArPus01IT7+A4OCa6jjM2+98hKys7BINMeuOL77dgNMpyZi18H0sXRuL40mnMH/5IrOGW2pcW3fsxtr4bx3H6tcNw659BxzbrJBAWQRMf4s09W/zMX3ai7jnN3dBTJQv1s2OEGv9VCmrPabb/2DvfpAvWTYnbEFSyhmMHTbSdHHeLKD9B48hU6wVJYsccD+Tmo7mTSLVbf4ggZsRMH0PZuqU57F9+17Erf0Gk996D1P+Z5wQGqE0Fiv7xPpLH61ahgNHDlmuBzN00H04fOwkVn+xEeMmTMe4p59E44ZcZthiH0FDwjV9D6Zvn66i99IZe/YcRNz/zlVvlQwhdZtO27S4A0tm/us2rRhzeWCAP+ZMG4+f/3MEXTtGIzwsxJhA6NVyBEwvMJKol1gErUOHlpaDa7eAW0c1tVuT2B6dCZj+Fknn9tM8CZCAjgQoMDrCpWkScHcCFBh3/wSw/SSgIwEKjI5waZoE3J0ABcbdPwFsPwnoSIACoyNcmiYBdydAgXH3TwDbTwI6EqDA6AiXpknA3QlQYNz9E8D2k4COBCgwOsKlaRJwdwIUGHf/BLD9JKAjAQqMjnBpmgTcnQAFxt0/AWw/CehIgAKjI1yaJgF3J6DLdA1y1jO5xIinp7n1Kzs7B4riW+pnQC4vkpJ+ttRjZtqZm1v6MihyLuDsnBwx+1yamcItEYvknJ//66oRJQ5yh+UJ6CIwrVp3wO59p00/85xcLbJHjy4lkhgeHo4qtWvgaEZ6iWNm2xEaWfrMcne0boc9O3egShVzTy+am5uLVm2jzYaV8WhEwEP8pVM0skUzJEACJFCEgLnvYYqEyg0SIAGrEaDAWC1jjJcELESAAmOhZDFUErAaAQqM1TLGeEnAQgQoMBZKFkMlAasRoMBYLWOMlwQsRIACY6FkMVQSsBoBCowLM/bll1+60BtdkYDxBPignQtzEBMTg8TERBd6pCsSMJYAezDG8qd3ErA1AQqMrdPLxpGAsQQoMMbyp3cSsDUBCoyt08vGkYCxBCgwxvKndxKwNQEKjK3Ty8aRgLEEKDDG8qd3ErA1AQqMrdPLxpGAsQQoMMbyp3cSsDUBCoyt08vGkYCxBCgwxvKndxKwNQEKjK3Ty8aRgLEEKDDG8qd3ErA1AQqMrdPLxpGAsQQoMMbyp3cSsDUBCoyt08vGkYCxBCgwxvKndxKwNQEKjK3Ty8aRgLEEKDDG8qd3ErA1AQqMrdPLxpGAsQQoMMbyp3cSsDUBCoyt08vGkYCxBCgwxvKndxKwNQEKjK3Ty8aRgLEEKDDG8qd3ErA1AQqMrdPLxpGAsQQoMMbyp3cSsDUBCoyt08vGkYCxBCgwxvKndxKwNQEKjK3Ty8aRgLEEKDDG8qd3ErA1AQqMrdPLxpGAsQS8jXVP7yRAAjcjoCgK5Mu5eHh4QL5cXXJzc3H16lXI99DQ0HK5Zw+mXJh4EgkYQ+C9995D165d4eXlBW9vb3Tr1g2dO3dGs2bNEBMTg9mzZyM7O9slwT3zzDOoXbu2Go+zw127dpUZg4dQx6Ly6Hwl65oSkB+IxMRETW3SmP0J5OXlwcfHB76+vsjKynI0+LXXXsPUqVPxxBNPYPny5Y79elaio6ORkZGBo0ePOtw0btwY33//PerVq+fYV1BhD6aABN9JwGIExowZo0a8atUqZGZmuiR6KXTOt2d79+7F8ePHy/TNMZgy0fAACZibQFJSkhpggwYNULVqVUew8pZp4cKFam9ZCkL//v3x4IMPqsf379+PDz74ABcuXIC8LjAwEK+++irmzp2LI0eOqLdhM2bMgBSOJUuWqGMuw4cPL3FbJI0tWLAAkydPVu1OmTIFNWvWxAsvvIA6deqo+9Qf8haJxTUEOnTo4BpH9GIrAmJQVQ5jKGIMRomNjVWWLl2qiF9oJSwsTOnUqZNy8OBBR3tFT0YRtzFK+/btlVOnTilbt25VqlWrprz77ruKPCbrYsxEyc/PV221bt1avfa7775TgoODFSEO6nZycrLy29/+VvW7aNEih/0777xTadKkibq9c+dO5f7771fPWblypbJ582ZF3MI5zpUV3iIVai1rJGBqAjdu3FDHYmSQP/zwA86ePQshMOqAb0Hgsueye/dutSdRv359dOnSBffddx9mzZoF2eOR4yeTJk3Cp59+qg4Wy3EcWXr16oXw8PACM6hbt666z7GjlIoQMUdvRQ5E9+zZUx0ncj6Vt0jONHSqi78W8PQs1PLi2zq5pVmbEahSpQoGDhyoturJJ59Ey5YtMW/ePDz11FOqWMgDe/bsUY+vWLECCQkJav3SpUuoVauWKkQjR47E4sWL8fnnn6ufyeeeew5DhgxRz9PjR+GnXg/rtKkS2LJlC9atW6fW5X2rvNdlIYHbJRAZGamakJ+vghIVFaVWxe0NxG2R+tqwYYPa40lPT4cUJvkNkOzBFPRsyvo8ymdeKlK2bduGuLi4IpdQYIrg0Geje/fukINgly9fRnx8PJo3b66PI1p1KwIhISFqe3/++Wf1ffz48apoVK9eHWKcxsFCDtjKZ2dOnjyJ0aNHQ4y1YOjQoarIyG+EAgIC1HPl4Oy1a9cc1xWIhXywrqxSEIPsJW3fvr3k8zBFRmS4oRsB0YNRRDIVcX+smw8ath8BcQuk3H333Yq4PVL8/PwU8aCdUjDoKgd3xQN3irj9VsRtjjJgwAAVgLhNUs+TA8CDBg1SevfurYjnrxS5v1GjRkqfPn0U8c2QIr5dUt58800HtPXr1yti7EXp16+fIgRJtSmERREP1ymiF6SMHTtWCQoKUsQ3T8qwYcPU6w4cOKA0bdpUEd9IKeL2Tbly5YrDnqzwQbuypFmH/fKhqIkTJ+pgmSbdmcDp06fVx/eFeBTBIB/Qk70P56+w5bb86vrixYsQYlHkfLkhNEHtactekKzLQWF5vXzI72ZF3k7Jr7yLFwpMcSLcJgES0IwAx2A0Q0lDJEACxQlQYIoT4TYJkIBmBCgwmqGkIRIggeIEKDDFiXCbBEhAMwIUGM1Q0hAJkEBxAhSY4kS4TQIkoBkBCoxmKGmIBEigOAEKTHEi3CYBEtCMAP+buhSU586dw5w5HyMkJKKUo+badelSKl5//UVzBcVoSOC/BCgwpXwU5GPPDRq0Qfv2fUo5aq5d27evNVdAjIYEnAjwFskJBqskQALaEqDAaMuT1kiABJwIUGCcYLBKAiSgLQEKjLY8aY0ESMCJAAXGCQarJEAC2hKgwGjLk9ZIgAScCFBgnGCwSgIkoC0BCoy2PGmNBEjAiQAFxgkGqyRAAtoSoMBoy5PWSIAEnAhQYJxgsEoCJKAtAQqMtjxpjQRIwIkABcYJBqskQALaEqDAaMuT1kiABJwIUGCcYLBKAiSgLQEKjLY8b2otOzvrpsd5kATsRoATTrkgo5s3f4bU1JPIycnG3r0JmD79U3h7+7jAM12QgLEE2INxAf958yYiOrobRoz4C86cOYFNm9a4wCtdkIDxBNiDcUEOZs/+DGFh9XHjxg1cunQODRs2c4FXuiAB4wmwB+OCHERENEJCwkY899wDGDPmVURFdXCBV7ogAeMJsAfjohx063Y/atcOw2uvjUDz5u3Qtm0XF3mmGxIwjgB7MDqzz8rKRHz8ctWL7LlER9+NpUtn6eyV5knAHATYg9E5D+fPp2L27FfRt+9g9ZsjDw9PREZG6eyV5u1EYN26r3DhQobpm5SXl4eBA/siNDTUESsFxoFCn4ocf3n88d9j/foVyMvLwb592zFv3pf6OKNVWxI4dCgJd975CDw8PEzdvuTkYzh69BgFxtVZGjNmPK5evYzTp49iyZIE+PhUcXUI9GdhAp6enggMrAH5bubi5SX7KzlFQmQPpggO/TbkB4TfHunHl5bNScDckmhOZoyKBEignAQoMOUExdNIgAQqToACU3FmvIIESKCcBCgw5QTF00iABCpOgAJTcWa8ggRIoJwEKDDlBMXTSIAEKk6AAlNxZryCBEignAQoMOUExdNIgAQqToACU3FmvIIESKCcBCgw5QTF00iABCpOgAJTcWa8ggRIoJwEKDDlBMXTSIAEKk6AAlNxZryCBEignAQoMOUExdNIwOoE8vJySzRBTkRfUHJzi061ULD/dt5tNV3DlStXsHr1agwbNgy+vr6V5uLl5YXjxw/Ay8v887YkJx+vdDt5oXsQOHv2tJi2dZl4faquydWo0R1qww8e3I0//vF+NGnSEpmZ18Wk9NMRE9NTUyi2Epi4uDgxa/8Y1KpVS0zdN7DSoOrXr4+nnx6E7OzsSttw1YX33tvCVa7oRwMCsscg/4C5slSrVlP8PoyBXADwxo08h2tFycfw4S+iVauO6lzReiwGaCuBGTx4MPz8/NCvXz8HxMpWIiMjK3spryOBMgksXLhQnZlu5MiRYo5m1/z6+fsHQr5KK3IStMhI+UdKn+k4XdPC0lqmw77Y2FgkJSWJeUGPClVupYMH65qcM2cOrl+/bt0G2CTy/Px8LF++HFOnTsXEiRMxevRoA6fC9EBs7Hu4774nxGqjqzFq1Mto0SJaU9K2Ehg5m/moUaPE1JRRFJhiH5PGjRtb4pavWNi228zKylKFvmHDhmjatKmB4gI0a9YW06YtQ5UqvqhZMxj/+MeLWLDgW02Z20pgunXrpikcOxl74IEH7NQcy7bl448/hnz17KntYGplgKxa9S66dr1PLGXcXCxtXA8HDiRWxsxNr7GVwNy0pRU4qCgK9u7ZDfnXxuylevUauKNlS7OHyfj+S0DeEhlV5OdavgrKjh3foXv3X//wpKScQuPG2n+OKDAFtJ3eT548iZPHdyCyYeECUk6HTVVdHx8vBGayqWJiMOYikJaWLMZa3hfjk0fw0UfT0L//U+jR4wGMGPEXrFmzQF3GODb2X3j55X9qHritBEYOmvn4+IiFzeYJiP3FyLl/pYFVreonwJv/K+Cjx9Ir3UZe6B4EQkMjxPMuU9SXc4vlMsYtW96JI0f2YcaMNeIb2Mr/vjjbda7b6kle+U1JTk6O+L5/822JizMg1knAzgTkAK8UGT3ERXKzlcDY+YPAtpGAFQlQYKyYNcZMAhYhQIGxSKIYJglYkQAFxopZY8wkYBECFBiLJIphkoAVCVBgrJg1xkwCFiFAgbFIohgmCViRAAXGilljzCRgEQIUGIskimGSgBUJUGCsmDXGTAIWIUCBsUiiGCYJWJEABcaKWWPMJGARAhQYiySKYZKAFQlQYKyYNcZMAhYhYKv5YMzIPDc3V52jxjk2uXRFbm6e+Bd5X+fdrJNAqQTkUiMpKSfh4aHPzP+lOq3EzuxsOQNk0T4LBaYSIMtzyb//vQPxX3+P7zb/iO1bVzouWbnqK/yU+AtOn04Vc9fkYvEn0xEQoP1EPw6HrFieQM+enZCWZv4F9gICstCmzT1FeFNgiuDQbqNFi0gxkXIw1q7b5DAqey5r4r7Byk9nqvv63DsGs+cswYTxYx3nsEICxQl07hxTfJdltov2ZywTtvkDrVMnRMyq51ck0GvXMnHiZLJYtiJT3d+yZRMcOXqqyDncIAE7EaDAuDCb1asHqrdL/v5VIRfg2vRtAkaNeNiFEdAVCbiWAAXGtbwd3t75+0d4eGBvMbt7R8c+VkjAbgQoMAZkdN1nm9QB3qlTnkdGxjUDIqBLEnANAQqMazg7vGzdukssE3EKk17/g/iqOhfvz1/hOMYKCdiNAAVGp4xu/GYrXps0B8nJaRg/cSYOHz6BCxcuYcBDz+KV8TPg7dsaVfzb8VkYnfjTrDkI8GtqnfLQt09XyFfxcvHc9uK7uE0CtiXAHoxtU8uGkYDxBCgwxueAEZCAbQlQYGybWjaMBIwnQIExPgeMgARsS4ACY9vUsmEkYDwBCozxOWAEJGBbAhQY26aWDSMB4wlQYIzPASMgAdsSoMDYNrVsGAkYT4ACY3wOGAEJ2JYABca2qWXDSMB4AhQY43PACEjAtgQoMLZNLRtGAsYToMAYnwNGQAK2JcDpGm6SWjlvLgsJkEDlCVBgSmEXFBSEn39JwqmkC6UcNdeu7Owb5gqI0ZCAEwEPRRSnbVZJgARIQDMCHIPRDCUNkQAJFCdAgSlOhNskQAKaEfh/yZf2BbpSZyUAAAAASUVORK5CYII="
    }
   },
   "cell_type": "markdown",
   "id": "4bea4019",
   "metadata": {},
   "source": [
    "## Sum\n",
    "\n",
    "If we want to sum all elements of the tensor to a single scalar, the RHS has no subscripts $ij->$. That means we compress or sum along both rows and columns.\n",
    "\n",
    "![image.png](attachment:image.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6dd6a7b7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(15)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.einsum('ij->', a)"
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbYAAAEQCAYAAADcTFGDAAAKp2lDQ1BJQ0MgUHJvZmlsZQAASImVlwdUk9kSgO//pzdaQihSQm+CdAJICT303myEJEAoMQZCtyGLK7CiiIiAuiKLIgquBZC1IIrYFsXeF2QRUNbFgg2V9wOHsLvvvPfOm5w598v8c2fm/udOzgQACo0jEqXCcgCkCTPEod5ujOiYWAZuBGAADpAAHWA43HQRKzjYHyAyt/5d3t8F0PR6y3Q61r8//68iz+OncwGAghGO56Vz0xA+jug4VyTOAABVj9h1sjJE03wJYZoYKRDhx9OcOMvj0xw/w2j0jE94qDvCygDgyRyOOBEAsi5iZ2RyE5E4ZA+EzYU8gRBh5DtwTktbyUMYyQsMER8RwtPxmfF/iZP4t5jx0pgcTqKUZ88yI3gPQboolZPzf76O/y1pqZK5HPqIkpPEPqHIitQF3U9Z6SdlYXxg0BwLeDP+M5wk8YmYY266e+wc8zgeftK9qYH+c5wg8GJL42Sww+eYn+4ZNsfilaHSXAlid9Ycc8TzeSUpEVJ7Ep8tjZ+bFB41x5mCyMA5Tk8J85v3cZfaxZJQaf18obfbfF4v6dnT0v9yXgFbujcjKdxHenbOfP18IWs+Znq0tDYe38Nz3idC6i/KcJPmEqUGS/35qd5Se3pmmHRvBnIh5/cGS99hMsc3eI6BB/AE/siHASKAJbBD1AqEAM8Mfvb0HQXuK0U5YkFiUgaDhXQZn8EWcs0WMizNLa0BmO7Z2Svx9v5ML0J0/LytsACAxVMIXJ+3BSB6dBtyffLmbXpIXXJIT3T/zpWIM2dt0+2E/BYQgSygARWgAXSAITBFarMFjsAVqdgXBIFwEAOWAy5IAmlADLJAPlgPikAJ2AK2g2qwB+wDB8BhcBS0gVPgHLgIroIb4A54BPrBEHgJxsF7MAlBEA6iQFRIBdKE9CATyBJiQs6QJ+QPhUIxUByUCAkhCZQPbYBKoHKoGtoLNUI/Qyehc9BlqA96AA1Ao9Ab6DOMgskwDVaH9eFFMBNmwX5wOLwMToRXwblwIbwZroLr4ENwK3wOvgrfgfvhl/AECqBIKDpKC2WKYqLcUUGoWFQCSoxagypGVaLqUM2oDlQP6haqHzWG+oTGoqloBtoU7Yj2QUeguehV6DXoUnQ1+gC6FX0BfQs9gB5Hf8NQMGoYE4wDho2JxiRisjBFmEpMA+YEphtzBzOEeY/FYulYA6wd1gcbg03G5mFLsbuwLdhObB92EDuBw+FUcCY4J1wQjoPLwBXhduIO4c7ibuKGcB/xJLwm3hLvhY/FC/EF+Er8QfwZ/E38MH6SIEfQIzgQggg8Qg6hjFBP6CBcJwwRJonyRAOiEzGcmExcT6wiNhO7iY+Jb0kkkjbJnhRCEpDWkapIR0iXSAOkT2QFsjHZnbyULCFvJu8nd5IfkN9SKBR9iisllpJB2UxppJynPKV8lKHKmMmwZXgya2VqZFplbsq8kiXI6smyZJfL5spWyh6TvS47JkeQ05dzl+PIrZGrkTspd09uQp4qbyEfJJ8mXyp/UP6y/IgCTkFfwVOBp1CosE/hvMIgFUXVobpTudQN1HpqN3WIhqUZ0Ni0ZFoJ7TCtlzauqKBorRipmK1Yo3hasZ+OouvT2fRUehn9KP0u/bOSuhJLia+0SalZ6abSB+UFyq7KfOVi5RblO8qfVRgqniopKltV2lSeqKJVjVVDVLNUd6t2q44toC1wXMBdULzg6IKHarCasVqoWp7aPrVrahPqGure6iL1nern1cc06BquGskaFRpnNEY1qZrOmgLNCs2zmi8YigwWI5VRxbjAGNdS0/LRkmjt1erVmtQ20I7QLtBu0X6iQ9Rh6iToVOh06YzrauoG6ObrNuk+1CPoMfWS9Hbo9eh90DfQj9LfqN+mP2KgbMA2yDVoMnhsSDF0MVxlWGd42whrxDRKMdpldMMYNrYxTjKuMb5uApvYmghMdpn0LcQstF8oXFi38J4p2ZRlmmnaZDpgRjfzNyswazN7tUh3UeyirYt6Fn0ztzFPNa83f2ShYOFrUWDRYfHG0tiSa1ljeduKYuVltdaq3eq1tYk133q39X0bqk2AzUabLpuvtna2Yttm21E7Xbs4u1q7e0waM5hZyrxkj7F3s19rf8r+k4OtQ4bDUYc/HU0dUxwPOo4sNljMX1y/eNBJ24njtNep35nhHOf8o3O/i5YLx6XO5ZmrjivPtcF1mGXESmYdYr1yM3cTu51w++Du4L7avdMD5eHtUezR66ngGeFZ7fnUS9sr0avJa9zbxjvPu9MH4+Pns9XnHludzWU3ssd97XxX+17wI/uF+VX7PfM39hf7dwTAAb4B2wIeB+oFCgPbgkAQO2hb0JNgg+BVwb+EYEOCQ2pCnodahOaH9oRRw1aEHQx7H+4WXhb+KMIwQhLRFSkbuTSyMfJDlEdUeVR/9KLo1dFXY1RjBDHtsbjYyNiG2Iklnku2LxlaarO0aOndZQbLspddXq66PHX56RWyKzgrjsVh4qLiDsZ94QRx6jgT8ez42vhxrjt3B/clz5VXwRvlO/HL+cMJTgnlCSOJTonbEkeTXJIqk8YE7oJqwetkn+Q9yR9SglL2p0ylRqW2pOHT4tJOChWEKcILKzVWZq/sE5mIikT9qxxWbV81LvYTN6RD6cvS2zNoyHB0TWIo+U4ykOmcWZP5MSsy61i2fLYw+1qOcc6mnOFcr9yf8tB53LyufK389fkDq1mr966B1sSv6Vqrs7Zw7dA673UH1hPXp6z/tcC8oLzg3YaoDR2F6oXrCge/8/6uqUimSFx0b6Pjxj3fo78XfN+7yWrTzk3finnFV0rMSypLvpRyS6/8YPFD1Q9TmxM295bZlu3egt0i3HJ3q8vWA+Xy5bnlg9sCtrVWMCqKK95tX7H9cqV15Z4dxB2SHf1V/lXtO3V3btn5pTqp+k6NW01LrVrtptoPu3i7bu523d28R31PyZ7PPwp+vL/Xe29rnX5d5T7svsx9z+sj63t+Yv7U2KDaUNLwdb9wf/+B0AMXGu0aGw+qHSxrgpskTaOHlh66cdjjcHuzafPeFnpLyRFwRHLkxc9xP9896ne06xjzWPNxveO1J6gniluh1pzW8baktv72mPa+k74nuzocO078YvbL/lNap2pOK54uO0M8U3hm6mzu2YlOUefYucRzg10ruh6djz5/+0LIhd5uv+5LF70unu9h9Zy95HTp1GWHyyevMK+0XbW92nrN5tqJX21+PdFr29t63e56+w37Gx19i/vO3HS5ee6Wx62Lt9m3r94JvNN3N+Lu/XtL7/Xf590feZD64PXDzIeTj9Y9xjwufiL3pPKp2tO634x+a+m37T894DFw7VnYs0eD3MGXv6f//mWo8DnleeWw5nDjiOXIqVGv0RsvlrwYeil6OTlW9If8H7WvDF8d/9P1z2vj0eNDr8Wvp96UvlV5u/+d9buuieCJp+/T3k9+KP6o8vHAJ+anns9Rn4cns77gvlR9Nfra8c3v2+OptKkpEUfMmRkFUIjCCQkAvNkPACUGAOoNAIhLZmfqGYFm/wfMEPhPPDt3z4gtAIc7AQh2BcCjc5b1ELOs66wt3BXAVlZSnZt/Z2b1aZE7BIBbDsvL339Au20d+IfMzvF/qfufK5BG/dv6L6CkBpneI4q0AAAAOGVYSWZNTQAqAAAACAABh2kABAAAAAEAAAAaAAAAAAACoAIABAAAAAEAAAG2oAMABAAAAAEAAAEQAAAAAACIeJYAADShSURBVHgB7Z0HfBTFF8dfQkkChBBCTVCDQem9F8GCHRBEsYCCKGLF9ldURES6ikrvSJHeQRSpAaRIr1IDhAAJoSSkQer+543ekeT2jssld7e795vPJ7m92SlvvjO7b96b2VsvRQRCAAEQAAEQAAGDEPA2SDvQDBAAARAAARCQBKDYdD4QUlJS6OrVqzpvBcQHARAAgYIjAMX2H8tHHnmEqlevTkWKFKHy5ctTy5YtaefOnQVH2s6SsrKyaNeuXXal3rRpE9WrV4969uxpV3okAgEQAAFPIADF9l8vb9iwgb788kvKyMigZs2a0bZt2+SnqwfBvn376MMPP7Sr2oceeojq1q1LrVu3tis9EoEACICAJxCAYtNYL69YscJuiXjfz+bNm+nBBx+0Ow8SggAIgIDRCRQ2egPz274xY8bQ6dOnKSQkhJ5//nmaP38+JSUlUceOHalx48ay+CFDhtDly5cpNDRUujHZhVmqVCnq3LmzdBVyIlM5hQsXppEjR9KhQ4do9uzZsqxXXnmF6tSpQ3379qVJkyZRUFCQPC5Xrhx98sknVpvwzz//UFpaGiUkJNCPP/5IvXr1In9/f6vpcQIEQAAEPIEALLY79HLt2rVpzpw5NHHiRBowYIC0ji5evEjNmzc3b9pgdyCfZyUUHR1Nb775Jp05c0amWbhwoazBVM7cuXPl9zJlylBcXJzMx4qT1/a6dOlCRYsWlYrtqaeeoocfftimdOHh4ZScnCyVY5s2bYj/EEAABEDA4wnwc2wI/xKYNWsWP9OndOjQIQeSWrVqyfiYmBgZP336dPl9wYIF5nTCylLq169v/i52KipeXl7K/fffb47jcipUqGD+Liw2Wc7MmTPNccWLF1eE0jR/t3Xw3HPPKYMHD5ZJxKYTxdfXV4mMjLSVBedAAARAwPAEYLHZObUpW7asdDNycrauONy6dUt+qv1jd6JQZHTy5ElpxamlyW8cr689/vjjshih0KQ8Qrnlt1jkBwEQAAFdE4Bis7P7ChUqZGfK28lSU1OJ8/F6m1rgtTpbgdfNrD2jxvHshhRWoixi9erVVK1aNeJ1OQQQAAEQ8GQCUGwF2PuxsbEUHx8vS9yxYwedOnWKhLuQ/Pz8ZJxwQ0plZKpy2bJl8jA9Pd0UJRUTr71xYGVlymtO8N9BYGCgTCvcndJSmzBhAn333Xe5k+E7CIAACHgeAcM7W+1soHgWTBEPaCs+Pj6KcDsqLVq0UIRyUl577TVFWFwKr3116tRJWb9+vSIsI5mO18/WrVsna+A1toCAAKVt27ZyjU4oHkVsIlFu3LhhlmDNmjVKcHCwItyHStOmTZUXXnhBrrGJjSTK2rVrZbqpU6cqxYoVk+e//vprc161g0GDBiliJ6UidmsqM2bMUEuCOBAAARDwOAJe3GLPU+cF32Le5Xj33XcTP2DN2+9LliypWgnjFspOnufjxMREaZUJhWpOz78+wut3QsGZ46wdiA0txOt/jrhKrZWJeBAAARDQMwE8x+aE3rOm1Lgqdh1mX3PLfmwSxdvb2y6lxunZvYkAAiAAAiBwmwDW2G6zcPjomWeeoapVq8qHo9944w2Hy0FGEAABEACB/BOAKzL/DFECCIAACICAhgjAYtNQZ0AUEAABEACB/BOAYss/Q5QAAiAAAiCgIQJQbBrqDIgCAiAAAiCQfwJQbPlniBJAAARAAAQ0RACKTUOdAVFAAARAAATyTwCKLf8MUQIIgAAIgICGCECxaagzIAoIgAAIgED+CUCx5Z+hXSVERETQ4sWL7UqLRCAAAiAAAo4TgGJznF2ecg4ZMkS+Idv0y/15yozEIAACIAACdhPAL4/YjSp/CflFoIcPH6Z27drlryDkBgEQAAEQsEkAFptNPAVzkn/tn92Q/Mv/1l4cWjA1oRQQAAEQAAEoNheMgaJFixK/TXvAgAF07do1F9SIKkAABEDAcwlAsbmg7319fenBBx90QU2oAgRAAARAAIoNYwAEQAAEQMBQBKDYDNWdaAwIgAAIgAAUG8YACIAACICAoQgUNlRrNNiYHTt20I0bN4g3kHAwfWpQVIgEAiAAAoYgAMXm5G6cN28ebd26lapXr05NmjSh0NBQJ9eI4kEABEDAswngAW0n9//NmzcpPDycSpQoQS1btiRvb3h/nYwcxYMACHg4ASg2Dx8AaD4IgAAIGI0AzAej9SjaAwIgAAIeTgCKzcMHAJoPAiAAAkYjAMVmtB5Fe0AABEDAwwlAsXn4AEDzQQAEQMBoBKDYjNajaA8IgAAIeDgBKDYPHwBoPgiAAAgYjQAUm9F6FO0BARAAAQ8nAMXm4QMAzQcBEAABoxGAYjNaj6I9IAACIODhBKDYPHwAoPkgAAIgYDQCUGxG61G0BwRAAAQ8nAAUm4cPADQfBEAABIxGAIrNaD2K9oAACICAhxNwyvvYoqKiKC4uTvNofXx8qGrVqhZyZmVl0ZEjRyzitRgRHBxMZcqUsRDtypUrFB0dbRGvxYjatWuTl5eXFkWDTCAAAjok4BTFNmH8XPFizQc0j+P4iXDq2bMwhYWF5ZB15/btdGTDBqoWdm+OeK19URSFfl+yhD4fONBCtNFTR1HNNtU0rzBuJt+klF0p1KxpM4s2IAIEQAAEHCHgFMUWHHwP1azZ1BF5XJonI+MmsXVmEYTCaFijOtWsUsXilJYiMjIy6NS166oiBZUtTbWb1NT8i01PHDpJyi1FtQ2IBAEQAAFHCGCNzRFqyAMCIAACIKBZAlBsmu0aCAYCIAACIOAIASg2R6ghDwiAAAiAgGYJQLFptmsgGAiAAAiAgCMEoNgcoYY8IAACIAACmiUAxabZroFgIAACIAACjhCAYnOEGvKAAAiAAAholgAUm2a7BoKBAAiAAAg4QgCKzRFqyAMCIAACIKBZAlBsmu0aCAYCIAACIOAIASg2R6ghDwiAAAiAgGYJQLFptmsgGAiAAAiAgCMEoNgcoYY8IAACIAACmiUAxabZroFgIAACIAACjhCAYnOEGvKAAAiAAAholoBuFFtKSpJmIdorWHp6ur1JNZkuRbwUFAEEQAAEtE7AKS8aLchGp6beohHfvU+1ajahffu30vvvDaXy5SsVZBVOL2vKosW04e+dVOu+++ir3r2dXl9BVxB15gJN+34G+fj50KG/D9NXoz+nmg1rFHQ1KA8EQAAECoSA5i22adOHUr26LenZZ3vRSy++L5RcnwJpuCsLebRFc2papw5lZma6stoCq+vXMXPprX696IsfP6VO3TtQ/14DC6xsFAQCIAACBU1A84ptx461FBJSWbY7NLQa/f33OsrKyipoDk4tLzQkhLy8vJxahzMLT0tNoxOHTsoqwmqE0fnT551ZHcoGARAAgXwR0Lwr8vr1WAoICJKNLF7cn9LSUik5JZH8SwTkq+HIbD+BAeO/Mifetm47dezxjPk7DkAABEBAawQ0r9gyMzPI2zunYZmlU5ee1jo/r/KcOnqa9mzeS2OX/5zXrEgPAiAAAi4joHnFFhBQmpKSEySQm7dShJIrRMWE5YbgWgJXYq7SpKFTadyKURYTDddKgtpAAARAwDaBnKaQ7bRuOdugQWuKvXxB1n05Jorq1GlORQoXcYssnlppcmIyTR42lQZO7E/+Af40e/RcT0WBdoMACOiAgOYVW4/un9GfaxfS3r2b6ZcZw6lPn2E6wJpTxNG/zqEla9fR6s1baNiUqTlP6uBbv9cH0Jyx86lp6QeoRqF69Nef23QgNUQEARDwVAKad0XyM2tDBs+mU6cOUb8vJ1LRoj6666s+3boS/+k1/LzwB72KDrlBAAQ8kIDmFRv3CSuzmjUbe2D3oMkgAAIgAAJ5JaB5V2ReG4T0IAACIAACnk0Ais2z+x+tBwEQAAHDEYBiM1yXokEgAAIg4NkEoNg8u//RehAAARAwHAEoNsN1KRoEAiAAAp5NAIrNs/sfrQcBEAABwxGAYjNcl6JBIAACIODZBKDYPLv/0XoQAAEQMBwBKDbDdSkaBAIgAAKeTQCKzbP7H60HARAAAcMRgGIzXJeiQSAAAiDg2QSg2Dy7/9F6EAABEDAcASg2w3UpGgQCIAACnk0Ais2z+x+tBwEQAAHDEXDKa2sSk2IofPMszcO6fv0ytWtfx0LOMuXK0aLfV9OBC/++udsigUYisrIU8vbzU5WmUFYRWj8rnMjLS/W8ViJjoy9TzxcaaEUcyAECIGAAAl6KCAZoB5oAAiAAAiAAApIAXJEYCCAAAiAAAoYiAMVmqO5EY0AABEAABKDYMAZAAARAAAQMRQCKzVDdicaAAAiAAAhAsWEMgAAIgAAIGIoAFJuhuhONAQEQAAEQgGLDGAABEAABEDAUASg2Q3UnGgMCIAACIADFhjEAAiAAAiBgKAJQbIbqTjQGBEAABEAAig1jAARAAARAwFAEoNgM1Z1oDAiAAAiAABQbxgAIgAAIgIChCECxGao70RgQAAEQAAEoNowBEAABEAABQxGAYjNUd6IxIAACIAACUGwYAyAAAiAAAoYiAMVmqO5EY0AABEAABKDYMAZAAARAAAQMRQCKzVDdicaAAAiAAAhAsWEMgAAIgAAIGIoAFJuhuhONAQEQAAEQgGLDGAABEAABEDAUASg2Q3UnGgMCIAACIADFhjEAAiAAAiBgKAJQbIbqTjQGBEAABEAAig1jAARAAARAwFAEoNgM1Z1oDAiAAAiAABQbxgAIgAAIgIChCECxGao70RgQAAEQAAEoNowBEAABEAABQxGAYjNUd6IxIAACIAACUGwYAyAAAiAAAoYiAMVmqO5EY0AABEAABAo7A8Gon4ZSxYqlnVF0gZYZFRVD3V59m8qXL5+j3GP/HKU/fl9IlSpVzBGvtS+KolBCQgb16v2+hWgTxv1EpUv7kpeXl8U5LUWkpqZS02ZP0P1Vq2pCrC1bttOmTXspKKisJuSxJkRWVhaVDixC3V553iLJz8OHUbkS/hbxWouIio6mrm+/La6zSjlEO3fuHE2eN5HurnxXjngtfkmITaLP+vTVjGiDBv1IgYHBmpHHmiBXYi/Qe+93p7JlnXOdOUWxsVLr8lwba23STHz45n1CMSRYKLb4+Dh6rG1DqlXrPs3IqiZIRkYGzZ4TrnaKihb1ouc7tyZvb20b5YcOnaC4uOuqbXBHZGJiMjWo/ySFhFR2R/V215mYdIMiItaqpg/w9aVnW7ZQPaelyJ1HjlB8fLyFYrt+/To1aFuXajasoSVxVWVZM229ary7IoNK30XNmj3rrurtrnf//vWUlJTkNMWm7bue3ZiQEARAAARAAAT+JQDFhpEAAiAAAiBgKAJQbIbqTjQGBEAABEAAig1jAARAAARAwFAEoNgM1Z1oDAiAAAiAABQbxgAIgAAIgIChCECxGao70RgQAAEQAAEoNowBEAABEAABQxGAYjNUd6IxIAACIAACUGwYAyAAAiAAAoYiAMVmqO5EY0AABEAABKDYMAZAAARAAAQMRQCKzVDdicaAAAiAAAhAsWEMgAAIgAAIGIoAFJuhuhONAQEQAAEQ0I1iS0pK1n1vpaen67oNyckpupbfXcKnp6dReoa++17vY9fU9/HX4k2H+LwDgexjNjMzk/jltnoJmldst26l0htv9qe581bTK9370oULMXpha5Zz4qT51Pn5PjR02GRznJ4OzpyJorfe+Ya++noUtXjgJdq796iexHerrLNmjxRjdzSNG9uP1q5b5FZZHKn8wuXL9P0vv1Czl7vS8TNnHSlCM3n+2XeMvnitv2bk0bogkycNpM7P1aTeb7WlPh+0o5SUJK2LbJbPKW/QNpdeAAdDhk2ili3q02s9nqX69arTe30G0/KlYwugZNcV8fhjrcTbYlPoRoJ+BkZ2OqPH/kpfffmWeNNxBZo8ZSG90bs/7d+zNHsSHKsQOHnyIO3ZE06jR60invG++FJ9MZafoOLF/VVSazOqlL8/9ejYkVZtCqeMzAxtCmmHVPy2+THfjKeMdP22wY5mFmgSX99i9PFHP1CFCndTWFjNAi3b2YVp3mL7c+1fdG/luySHatXupXXrt+vKJGbBK1euRF5eXs7uS6eVn5qaRocOn5Dl16geRqdPn3daXUYqeMfOdRRc8R7ZpEKFClH5cpVo777NumpiiWLFqGxgoK5kVhN20eQl9GyPjrq+DtXa5cy4woWLUNOmj1LJgNLOrMYpZWveYouNvU5BQaVk4/39ixPfZBMTkykgQD+zXqf0nAsLnTBugLm2teu3UY/uHc3fcWCdQNz1WAooFWROUMI/gK5e1Z8r3dwAnR6cOxVJPn4+VOGu8jptgXvETrmZRDNmjKDg4FDa+tdq+vrrqeQnrDg9BM0rNnYheHvntHbYrYPgegJHj56izVv20AqduYJdT+rfGjOF687b67ZThK12RUcL8O7iVpD1KopCK2atoj7fvktH9mBtOC9se3T/jPz8isssBw5uo5UrZ9ALXd7JSxFuS3v7qnObCLYrLl06gBIS/t0RmZJyk9ilw5YbgmsJxMRcIV7vXLlsnOwD19auz9rYhZOckmgWPjk5UXga9OfWMTdAhwfrlm6gS5HRNGnoVFo8bRmdPx1Fs0bN0WFLXCvyrVs3afLkb4knBhx4ne3Ysb2uFSIftWlesbVp3ZguXPzXfRMVFUPNm9elIkWK5KPJyJpXAuz6HTp8Mk2aMFC6gEePmZ3XIjwyfcOGbejy5QvmtvNx/foPmL/r6YBvcP/d4/QkNj3WuS2NmDWE3urXix4Xx3eFVaJXP+iqqza4Q9j0jDQ6eGiHeU0yOjpS7BWo7g5RHKpT84qt76dv0PwFv9Pmzbtp2IjJNGLoJw411J2Zfvp5Ji1cvIZW/baJBg+Z4E5RHKq75xv9aMy4ORRYpil5FalBa8SGHoQ7E6hfr5XcAblu/WJasHA8PfxQR7FerK91nouxsTRg7DiKiIqi4VOn0urNW+7ccA2mOHbgOM0dv4BOHYmAxWZH//iXCKA2rdvT4sWTaOmyqRR9KZK66MQNyc3T/BobbzGfM/t7OnTohLQYfHyK2tEt2kry0Yfdif/0GhYt+FmvortVbl5T+7r/FIqIOCoeVWlFZcpUcKs8jlQeUq4cDXzvXfnnSH6t5KlerxqNXYZxnJf+6N79U7nZKT7+KnXq+LrZestLGe5Kq3nFxmBYmTVuXNtdjFAvCOSLgN6eAcpXY5HZUAR4MqbHCZnmXZGGGiVoDAiAAAiAgNMJQLE5HTEqAAEQAAEQcCUBKDZX0kZdIAACIAACTicAxeZ0xKgABEAABEDAlQSg2FxJG3WBAAiAAAg4nQAUm9MRowIQAAEQAAFXEoBicyVt1AUCIAACIOB0AlBsTkeMCkAABEAABFxJAIrNlbRRFwiAAAiAgNMJQLE5HTEqAAEQAAEQcCUBKDZX0kZdIAACIAACTicAxeZ0xKgABEAABEDAlQSg2FxJG3WBAAiAAAg4nQAUm9MRowIQAAEQAAFXEjC/tiYhIYGWLFlCL7/8snhNjE++ZDh//hKFb96XrzJckfnQoZMUGtbSoip+Q/f2HYfo6rVEi3NaisjMzKQbN9RljItLoM1b9mv+HUrHj0dQk2ZPaQirQkeObBPvoYrUkEyWoiQnJ5J3oZuWJ0TM5avXaMuBg6rntBR5NCKCnm7azEIkvv8c2XiUUhNTLc5pLSLuWrymRLp46RwdPBiuKZnUhDl77h8qXLiG2im7444cOUKRkZH09NNPW+TxEq98Vzh25syZ1KNHD1q+fDk988wzFgnzEhETE0OJieo33LyU4+y0rMBCQ0MtqmEkp0+ftojXYkSZMmUoMDDQQrS4uDhxc75qEa/FiCpVqjhFAWdkZIiLxzx3s6vpaWlp8mKxK7GbE1WsWJFKlChhIcWVK1coPl5bN1wLIUWEt7c3hYWFqZ2iU6dOqcZrLbJ06dLirehBBS4WT1oLFSqU53Kjo6MpKSkpz/lcnaFo0aJ0zz335Kvadu3a0caNG6Wuyc3KrNhSUlJo1apVUqn5+vrmq0JkBgEtEDhw4ADNnTuXPv74Y6pQQX9vr9YCQ8jgHgI8ue7fvz81bdqU2rdv7x4hNF7riRMnKCoqitq2bWshqXk6u2jRIpkoQrgHatasaZEQESDgTgJ//vkn7dq1K88i8Mz/3nvvpd69e1O/fv3E24DL5LkMZACB/BBgo2HkyJF5LoI9Dj179pSWzaBBg+jJJ5/McxlGzXDt2jVpiCUnJ9tWbOXKlZOuyGrVqkGxGXU06LhdISEhVLdu3Ty3gJVhcHCwzFuqVKk850cGEMgvAXaHOzJ2L1y4QOwar1OnDlWtWjW/Yhgqv5+fH507d47GjRtHAwYMsGib2WJr1aqVxUlEgIBWCNSqVYv4Ly9hz5491LlzZ+rWrVue19ryUg/SgoAtArye1KFDB1tJLM6lp6fT2LFjaf/+/dLjYJHAwyOKFStGDRo0sErBrNispnDgxLF/jordetpfvC5cuAg1bNTYYuMCD6rdu3bKxW0Hmu/SLMHBlehulUXYc2fPUkzMJZfK4khlvJbQpGlzhxbK71Rfo0aNiP/yEnjT08GDhy3GRF7KcFXasLDKqmuHJ8XawzUdbBzy8vISO2KbWVxnWVlZtG3HNot4V3HNSz3ly5WnKmFV8pLFrrS8se2jjz6yK232RIcPH6GEBO1v3OOdrw0b1nfadeYUxbZn93qqVyc0O29NHu/YeZgCSgXSfffdl0M+qdSUi1Tc1y9HvNa+sFJYMH8dfdr3GwvR1vyxhFo0c85uQ4vK8hGRlJQiJhHe1Kx5i3yUUnBZw8O3U3xcABX10fYGqkyx/rJ79wr68MPeFo3fuGI5NVLZ7WuR0M0Re48dpxIlS1pY4rzp52z6KQoILOlmCe9c/ep5q2joV8PvnNBFKdavO0DBIXnzbLhItBzVHBWP1AQFBVLlypVzxBfUF7Ni44V1niWw+fvUU08Rm3qOBtbGtWvf72h2l+W7dl19WyzPJEsULyYuuJwKz2WC2VkRLy7v3R+lmrpIkcJSft5SreVw6NAJupnmpRkRmVepUmUoJMQ5F1xBNTQx6QbdSlV/JMVHuL5q5ZqsFVS9BVlOUqr159SKFfej+2tr+/pjFmd2RRYkknyXxffe+6rUznc5zi4gKfFyvqrgTY7Tpk2TOmvEiBHUt2/fHOWZFdvo0aOJ/xBAAARAAARAQMsE+PnHbdu2WRVR29N5q2LjBAiAAAiAAAioE4BiU+eCWBAAARAAAZ0SgGLTacdBbBAAARAAAXUCUGzqXBALAiAAAiCgUwJQbDrtOIgNAiAAAiCgTgCKTZ0LYkEABEAABHRKAIpNpx0HsUEABEAABNQJQLGpc0EsCIAACICATglAsem04yA2CIAACICAOgEoNnUuiAUBEAABENApASg2nXYcxAYBEAABEFAnAMWmzgWxIAACIAACOiUAxabTjoPYIAACIAAC6gSg2NS5IBYEQAAEQECnBHSj2G7evKVTxLfF5venJSSovwPudiptHmVmZpoFS0tLMx/j4M4E0tPTKD0j/c4JNZ7iWny8xiW0FC893ZJ7Wlo6qcVb5vbcGFvjNT7+mubBaF6xRUdfoa++HkWzZq+gTp3fp717j2oeqjUBBw4aT4uX/GnttGbjDxw4RhUrtaaHHulOTZp3oR07DmpWVq0JNmv2SJo7bzSNG9uP1q5bpDXx7JaHFcHjb/ampJQUu/O4M+HuzXto5BejqNsDr+UQY/oPM2n2qDn0c78x9MdC/V2LORrjhC+XL1+gX2aMoK5dG9HZs8csajh2fB99801Pi3itRZhfNKo1wUzyTJqygNha6P3mC3TtWjwNHT6JlizS3wtRDx8+KW5wv1G/L3qbmqabz6wshf738WvUuHFtatWygXxrrW6Ed6OgJ08epD17wmn0qFVyDL/4Un1q2eIJKl7c341SOVb1lMVLKPLSJVIUxbECXJwrtGooBZUPog3LN5prPn7wBO0K300Tfxsr+6NTvS7U+slWVNy/uDmNpx/4+5eijs/0pPDwFZSRmZEDB3ucJk0cSBk68D5o3mL74P1X6JOP/p11XYqOpWrV7s0BWw9fWDEvXLyGnu30qB7EVZWxQYMaVK3qveTl5aV6HpGWBHbsXEfBFe+RJwoVKkTly1Wivfs2WybUeMzxM2cpwN+fSgeU1Likt8UrW6EM+RbzvR0hjrat3UHBocEyTvZHpXK0e/PeHGk8/UuxYiUoMLCsKoaly6ZQhw7dSdwEVM9rKVLzii0wMICuX79Bn/b9XpjGF2lA/3e0xM8uWaZNX0Kvv9ZZt0qBx/H4CfNo2/Z91O3VvsSuSYQ7E4i7HksBpYLMCUv4B9DVqzHm73o4yMrKogVr/qCu7Z7Wg7g2Zbx+5TqVCgowpylZyp+uiKUOhDsTOH/+FPn6+FH58nfdObEGUmhesTGj++8Ppc8/60V+fj40eOhEDWCzX4SzZy9IhRYaGmJ/Jo2lrFOnKs2bM5Ke6/y4cAl3oQ8+HqYxCbUpTqZw5Xh73b7E2NpVhKLQU5i9chW90r69nkS2KmtGRiZ5e9/uD7Y8WHEj2CbA7udVv82i9u2FtaaTkK2XtSnxipUb6IqYaQUFlZLrbCO+m0a3bqVqU1gVqb75dizFxFylIUIhs8Wz6rdw+mPNFpWU2o0aO24ORUZekgLeVamCrjfwuJJyyYDSlJySaK4yOTmRAkScXsLNW7do1sqVtPDPtTR86jSKT0yiH2fOpKgYfVmdJt6lhPcnOTHZ9JWSE5KFBVfK/B0H6gQ2blxK0dGRNH36MFq+fBpFRZ2WG6LUU2sjVvObR4YMm0TDh35MDz/UTLp2K1cOIR+fotqgZ4cUM38Zbk4Vef4SNW1Sh558orU5Tg8HGzf9Te3bPSRF5TbUqBGmB7HdLmPDhm1owYJxZjl4x1n9+g+Yv2v9wM/XlzZMn2YW8+dZs+ijV1+lkiVKmOP0dND4wUY0Z+w8s8gxFy5TwwcamL/j4DYBuUnov41CjzzSmfiPw06xbhwdfZ5efqnP7cQaPNK8xTZk0If099+HaNny9fTt4Ak0aGAfXa5V/TpnJW39ay/NmfsbbQr/W4NDwbpIn/3vdZo0ZSGx9Tziu6k05ud+1hPjjJlA/Xpix53YAblu/WJasHC8mJx1FJ6H8ubzejlIFc8t9h89hlKEBdd/zFi6GhenedG3r9tBo/uPpdiLsfTjl6Po3KlIatiqPpUoWZzWLFpLc8fNp7adHqYyYuckwm0CsbEXady4r6RVNnXaUNqyZZX55IkTB2jhogl0OuIILDYzFQcPHm3bQtwQmtJBsVV32eIx0iXpYFFuzdatawfiPz2GlmKLf6NGtYgfWVi+dCwVK+anx2a4XGZeU/u6/xSKiDhKrOTKlKngchkKokKfokVpUJ/35V9BlOeKMlo82pz4j2bmrG3QlG/o1NHT0lLjnZMIOQmUKxdC7747WP7lPENUtWo9+nHk0tzRmvyueVckU+OtubzdHMF9BNj9y8oNIe8EwsJq5j0TcjiNwH01qzitbBSsDQKad0VqAxOkAAEQAAEQ0AsBKDa99BTkBAEQAAEQsIsAFJtdmJAIBEAABEBALwSg2PTSU5ATBEAABEDALgJQbHZhQiIQAAEQAAG9EIBi00tPQU4QAAEQAAG7CECx2YUJiUAABEAABPRCAIpNLz0FOUEABEAABOwiAMVmFyYkAgEQAAEQ0AsBKDa99BTkBAEQAAEQsIsAFJtdmJAIBEAABEBALwSg2PTSU5ATBEAABEDALgJQbHZhQiIQAAEQAAG9EHDKr/vzS+oyMzM1z0C+TM+KlHpogy35uVn82vs7pbHSfJdFa1E+PfS96FibfYTrzyYew57UxdgV9J193TtFsd24kU4zZ2/U/OC5cSOR6jV82kLO4OAQWjB/He3eG2lxTksRPDiKlwhUFcnXryTN+nWT6jktRV65cpW6duulGZEqVChLv/22kiLOBGhGJjVB0sTLP6tXr6x2ijIKF6G5W/9SPaelyGvx8fTGU5bXX0hICC2bvpSiD8dqSVxVWYoX0dbbxFNTr9Ff22apyqqlyJSURCpdWrwvz0nBS9wcbU/9nFQxigUBEAABEAABZxDAGpszqKJMEAABEAABtxGAYnMbelQMAiAAAiDgDAJQbM6gijJBAARAAATcRgCKzW3oUTEIgAAIgIAzCECxOYMqygQBEAABEHAbASg2t6FHxSAAAiAAAs4gAMXmDKooEwRAAARAwG0EoNjchh4VgwAIgAAIOIMAFJszqKJMEAABEAABtxGAYnMbelQMAiAAAiDgDAJQbM6gijJBAARAAATcRgCKzW3oUTEIgAAIgIAzCECxOYMqygQBEAABEHAbASg2t6FHxSAAAiAAAs4gAMXmDKoqZUZERNDixYtVziAKBEAABECgIAlAsRUkTRtlDRkyhLp06UJxcXE2UuEUCIAACIBAfgngRaP5JWhn/sjISDp8+DC1a9fOzhxIBgIgAAIg4AgBWGyOUMtjnoSEBOmG3LdvH129ejWPuZEcBEAABEAgLwSg2PJCy8G0RYsWpdTUVBowYABdu3bNwVKQDQRAAARAwB4CUGz2UMpnGl9fX3rwwQfzWQqygwAIgAAI2EMAis0eSkgDAiAAAiCgGwJQbLrpKggKAiAAAiBgDwEoNnsoIQ0IgAAIgIBuCBTWjaQ6FXTHjh1048YN4g0kHEyfOm0OxAYBEAABzROAYnNyF82bN4+2bt1K1atXpyZNmlBoaKiTa0TxIAACIODZBPCAtpP7/+bNmxQeHk4lSpSgli1bkrc3vL9ORo7iQQAEPJwAFJuHDwA0HwRAAASMRgDmg9F6FO0BARAAAQ8nAMXm4QMAzQcBEAABoxGAYjNaj6I9IAACIODhBKDYPHwAoPkgAAIgYDQCUGxG61G0BwRAAAQ8nAAUm4cPADQfBEAABIxGAIrNQD26a9cu+XocAzUJTQEBEACBPBPw6OfY+Oeu/ve//9Hx48cpMTGRwsLCKCQkhNavX59nkPnNEB0dTWlpaXTPPfc4VFRsbCxVrVqV1q1bR40aNXKoDGRynAA/iL969WoqVKiQLKR06dJUq1Yt+XA+P5SvKApVqFCBWrRoQXv37qXz58/LOO6ru+++26JifjntkiVL6OWXXyYfHx+L86YILuvAgQOyjEcffdQUne9PHk+///67HJNvvvmmzfIyMzOJr6Vjx47JiRX/EAG3fdu2bTZf17R//37il+9WqlSJHn/8cZt1GPnkyZMnJYeLFy/S/fffT0899RRt3LiR2rZtS15eXlabvnz5crp8+bJMx/cuV4elS5dSenq6uVoe+8HBwVS3bl0qXry4Ob4gD+xts0dbbM2bN5cXH1+I3EH9+vVzi1Ljjv/555+JB4qjYdCgQRQfH0+sIBFcTyArK4v474UXXqAPP/xQ/sJM4cKF5YTp2WefpW+//Zb4O4ciRYrQ5MmTpUKyduNatmwZ9ezZk9asWWOzMVzW0KFDadasWVbTffbZZ1bPWTvBsv7xxx/0ySefWEsi41k5sXLmNpcpU0b+dNwvv/wiFRuPaVuBZf/uu+9o+vTptpIZ9lxSUhL16NFDTnZ4ItG0aVM6ceIEPfzww1LRZ1caahCSk5PprbfeIu4DdwQ/Pz8aOXIkdevWTU6+eGLO4/Cuu+4i/ilBZwS1NquObzGT9PjQvn17RXSCMnv2bLewEANYqVmzpvLjjz86VL+YLSs//PCDIqwEZdKkSQ6VgUwFQ6Bjx45yLJ06dcpcYJUqVRTx49eKeHu6OU5YKIqwdMzfcx+IC1iZP3++IizB3Kcsvj/00EOKuLlYxHPE9evXlXLlyqmeu1Pk6NGjFfFTcFaTxcXFKeImprRu3VrJyMjIkU68LV555plncsSpfXniiSeULl26qJ0yfJyYuCjCGlfOnj2bo62HDx9WxIRHSU1NzRGf+4uw6uVYW7RoUe5TLvveqVMnObazVyi8Toqw2BS+rxV0yN1ma+MbP4KsMq0QNyA5k+QZlbg45Uyb3UyVK1cmnn2zuX3u3DkaP348cRqeYZ05c4bEACVxE6M33niDAgICyFQOzzLYBcVupTlz5tCePXuI48aNG0cHDx6kTz/9lI4ePUqrVq2imJgYOVvjMu0JbCVwOTzrnTFjBiw2e6A5Mc1rr71G7C7hvhg8eLAcJ+xW5Nns3Llz6b333qPdu3dTw4YNzb8b+tdff0nXE7tyXnzxRekSFzcrioqKooiICBKTHpmf3VPipidn9uw+55DdTXjo0CFau3at/LFtoWzkTJ4tKa5fTJpkOezyY0tgwYIF9M8//0gXZvfu3Yln3xx4PK9YsUKOXXaX2go8lllGnrWbXLCm9CwXexE4sKuS3apsWbCr//nnn6fy5cubkpo/2Yply4WvuYoVK0qGfJ3wjJy9Eb/++qt0zb366qvSuyFu/MSyX7p0iX777TdpOfKb6tk9Z0r70ksvybRly5aldu3ayd9sZXnEJFa6TvlarlatmstdoVeuXJHXLLPI/cPo7MYVCsM8PpgJj4cLFy5IN5+YCJjPmeGJg+3bt0t3uK+vL4nJDPXu3Vt6Bqwx5TV5zsP1sZXFFjrfv/gex8sxR44coQ4dOtB9992XvZo7HnO/8G/jZv9dXLUxrtYPderUkfdIHodfffWVvFa2bNki3fjc19kDjye18S3TFLRG1WN5uS02ni0PHz5czoaEm1L58ssvFZ5FiYtAEX5v2USefQuFJNOINQJFuIyUDRs2KOJX/BWx1qUIxadkL+f999+X+cRgkmkEfDkbFzcGZcSIEbKct99+WxE/mGwxg7PFdMKECbJeTvPYY48pYjDbSo5zTibAs1Rx05aWDFtk33zzjSJuGIpQHEqDBg1k7e+++64i1lXksbgBS4tq06ZNsh/9/f0VsdaiiPUtOSZMs3Eed6+88ooi1qTkOBQuKKV///6yDLbYhCKQY1asscp8nE7c8BW2CrhMoUwVoSRlerakeKyIyZjy0UcfyWM+IZSPItyJyp9//inHr5iM2bTYevXqJeviMW0rCGWtCBetIpSTItztCl8vQjHLLNktNq6frwvhzlLEjU0RNy35XdyYFfHqJ0UoS0XctBUxcVTETVcRCkkRblDJgWXgvEKx50jL1xSnFTdn5YsvvpB1sndDuAAVMSlUhMtMEZMNW+I75RzLyfLyPcRWYKs4KChIepOY3+uvv64IBS2t/ezWC1t9Yg1XEXsFZL8LBSGLtcWUvQrMUKzPybHD9y+W6ZFHHpFjduDAgbLu3NZ4dnnZYuM87G0SExmZt379+opQOuZk1sa4Wj+Y+pnLZG8Fj1Eur02bNrK87G22Nr45obcoACEXgWLFislZI0fzTHTIkCFyVsOzCf6lfp7x8uYAcVHKnOJGZbayeAbBfvKpU6dS9nJMVTRu3FjOikzfeeG8du3a8ivPjEQHWszgTGlzf169elVafCbrjmefWGPLTcm133ltitcc2JLhjTx///23HCc8C+aNEjxLZkvINAtma5vXqHjWfu+991JgYKC0blq1amUWnK09ttbYChMXuUwrrl25bmdKxLN0tmweeOABGbVz505p9bDlw+8A5Dq4fJaLPQM8E+f1PR47bOWxp4DXAXkDklB6chOTaXyb6sj9ydYXB954pRb4OuFyFy9eLNce2YvBlghbkGrrajz2TYE3H7DMplCyZEnZ9lu3bkkrla1YfhVUZGQkCQVPfF1xHuadPS1b0JyWrTJmwoGtU5aJrWOhOOQGMlM9rvo0sWMWaoHZceC1KubL3h7mxxYeW6e519X4PsVtEZN0EssRNHHiRJnfFlO2znh8sNeHxw6PLw7smeK+Z6bsdTp9+rSMt/aP10pZvieffFJuiOIxxrKagrUxrtYPpr4z5WVZ1DZX8XmWPff4NuWDYjORsPLJZropcAeK2Yt0rZjicn+aBgcrQGcHVrh8AbBi5T92U0GxOZv6ncvnmykH7hPTeDDF8Q1ArMPJ83xDYfch37h4ByH/8UaQ3LsbWTHxppSffvqJ+vTpIzeffPDBB7IM0z++ebOiMm1GYcWXO/CuRa6P62VXOtfHL8Fltx3fUMR6hXT55M5n7btJTh53uQO72t555x3pSuJrJvuNjuvim9+dglobOI9pIshtZUXM1yUHdn1x27IHU9rs57gv+EbO18rHH38sJ67Z87jiuEaNGtIty+4+tcDuRm4/7zbN7tYzccztJmZXoliTJWGdyyUSnkCwSzB3UGOae+zwBJ6DrbGUvVxOx65ldq/zLnOecJs2wtka4/b2g5rM2es3HfP4NgUoNhMJK5+51w6sJDNH8+yag3AfmONyH/C6ga0gXCa2TstzvE7DFhpbhjwj4j/hXoFiuyM55yfgGwXPdll58HoQB7aM+FEOtmB41s2Bb7bC7UNshbDC69q1K4UKy43XanMHno3zTFy4/4h3TLK1Yk9gS840JtkiY0uQ43jNievjP16jYuUjXJok3FPmYrlOWzcV3sXH1qlwpcv1KnNGcSBc+fTcc8/JdxCy4mEvBgdeO2GFwvlyB36sgZmYrBW2cDnYkiF3GfZ85zXtZs2aEa/dmP7syVfQaXhtkq/jKVOm5CiaZWJLhBXGg2LNkCccrCw48KMB7C1i6yh7EK5suYN25cqVci2OrVdO6yqmJlnY48CB9w5w2Lx5s9Uxbq0fTOu9PCa579k6tTYGco9vWan4h80jJhL5+DQt5HMRM2fOlCWJ9RD5yTcQvlj5gubApj0PXA58AZtuMvydbyT8LAsPSFuBZ/i8MM8L8tkDuzd44ZxnSVwngvsIsALimwxPPjjwTUqsj0hlx7NqU5ghNpnwtn624tjlwjcxVmBiXVdaImPHjpXPNbHL2uRm5LysAHnzBitBVkZcPrvX2BXHioTHIT8HxTN33uTUuXNnqbh4xs8bA9hS4Xys9FgRsxXFFiErYt5wwZ4KHqc8CeMb0Pfff28S2fzJkz7ehDFs2DCpINl64vJ5IxRvQzc9mybW7Ojzzz+X1iIrOD7Hri7TBhYuh1mxe5QZsby8qSElJUXWxRtueLxzm7htzIYf0eEbHqfheLZgeDMCK32WPXtafqyHb7TcVpaX03F72FXMGyvYynBHYCucFRg/ZsSbQ5gfWx3cH2L9SYrEVg27W9mC4/O88YzHB9832OpkHmL3qrTk2UvEXgIeX7xZjcvh+4A1prxJJ/vY4bK5PLbg2fpiRibeJgssOyfeIMTPULLS4fsdu4TFOrLsd+4H3rzC7s4ZVsY497laP/C4YYuWxwkHvq/xJGfatGmyH01t5ush9/g2yyeE8tggLh6FF8h5sZ+33YqBo4iZtVy854V+3qLN25nHjBmjiMEn0wmoMo+YTcjFfgFSEZ2gPP3004oYDIq4OSliBpaDad++feWivLi5KE2aNFFEh8gFV/EwptwSy5sMRAfJDQa8qM+LuNaCuHlJOXkLt7jxmJPxpgFhJUiZxUxZEbNi8zkcuIeAsMRyVMz9LG6uOeJMX3g8WdvezRsjhMKQC/K8mM7feYFfWH6m7Hf8VHtsgOPEJMgiL8vB8fzJmzhYbnsCpxUTK6tJhRKyei77CebGdTMTMYmzeJQge1pHjsWEUmbjsrUSmLHY9WiTtS1+nJ/7jNkxt9zB2Uxz1ydc0Rb3oNxj/E79wP3DbeJxym1XG6umenOPb4/+5RGzdnfwgBf02ZXEM1aePfHskf3hakF0qpz98oyW04mBKBexTQ/tch4x+OSjBdnj1MpCnGcRYDcUr3vw4wNsfbCVwWttbMWwlYMAAiCQkwBckTl5OPRNzBqkyW9NqXGhrKxYqXHg3ZJqgd0LCCCQmwD/ogfvTNu6dat0NfJaHa9dlCpVKndSfAcBEBAEYLE5OAx4dxn7kHnHmjCRSTwvIzcAOFgcsoEACIAACBQQASi2AgKJYkAABEAABLRBAFvntNEPkAIEQAAEQKCACECxFRBIFAMCIAACIKANAv8H2lWywkBVrD8AAAAASUVORK5CYII="
    }
   },
   "cell_type": "markdown",
   "id": "9cddf1f4",
   "metadata": {},
   "source": [
    "## Matrix vector multiplication\n",
    "\n",
    "More often we want to do things with multiple tensors. Let's consider a matrix vector multiplication with equation $ij,j->i$. Given input $a$ of shape $(i,j)$ and $b$ of shape $j$, we want output of shape $(i)$.\n",
    "\n",
    "The index missing in $ij, j -> i$ on the RHS is $j$. This means we sum across $j$. \n",
    "\n",
    "Let's think of this in terms on tensor compressions to visualize better:\n",
    "<ol>\n",
    "<li>Weight all columns of $a$ with elements of $b$, each column $i$ of $a$ is multiplied by $b[i]$.</li>\n",
    "<li>Sum/compress all columns of intermediate result.</li>\n",
    "</ol>\n",
    "\n",
    "The advantage of thinking in this way is it provides a useful abstraction for tensors of higher dimensions. And einsum provides a straightforward way to translate this thinking explicitly into code.\n",
    "\n",
    "![image.png](attachment:image.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "cb6f9f0b-3854-4fc3-84d3-928596390265",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0, 1, 2])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = torch.arange(3)\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "14ee5923",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0,  1,  4],\n",
       "        [ 0,  4, 10]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a*b.view(1, -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "831a600e-dc3c-415c-9dbf-3f1c77143dea",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 5, 14])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.einsum('ij,j->i', a, b)"
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbgAAAEPCAYAAAAwBdF+AAAMQGlDQ1BJQ0MgUHJvZmlsZQAASImVVwdYU8kWnluSkEBoAQSkhN4EESkBpITQQu8INkISIJQQA0HFjiwquBZULGBDV0UUrIDYETuLYu8LKgrKuliwK29SQNd95XvzfXPnv/+c+c+ZM3PLAKB2kiMSZaPqAOQI88UxQX708UnJdFIPIAASUAY6gMDh5omYUVFhAJah9u/l3U2ASNtr9lKtf/b/16LB4+dxAUCiIE7l5XFzID4IAF7FFYnzASBKebNp+SIphhVoiWGAEC+S4nQ5rpLiVDneK7OJi2FB3AqAkgqHI04HQPUK5OkF3HSoodoPsaOQJxACoEaH2DsnJ5cHcQrE1tBGBLFUn5H6g0763zRThzU5nPRhLJ+LrCj5C/JE2ZwZ/2c6/nfJyZYM+bCEVSVDHBwjnTPM2+2s3FApVoG4T5gaEQmxJsQfBDyZPcQoJUMSHC+3Rw24eSyYM7jKAHXkcfxDITaAOFCYHRGm4FPTBIFsiOEOQacL8tlxEOtCvIifFxCrsNkszo1R+EIb0sQspoI/zxHL/Ep9PZRkxTMV+q8z+GyFPqZamBGXCDEFYvMCQUIExKoQO+RlxYYqbMYVZrAihmzEkhhp/OYQx/CFQX5yfawgTRwYo7Avzckbmi+2OUPAjlDg/fkZccHy/GCtXI4sfjgX7ApfyIwf0uHnjQ8bmguP7x8gnzvWwxfGxyp0Pojy/WLkY3GKKDtKYY+b8rODpLwpxM55BbGKsXhCPtyQcn08TZQfFSePEy/M5IREyePBl4MwwAL+gA4ksKaCXJAJBO19jX3wTt4TCDhADNIBH9grmKERibIeIbzGgkLwJ0R8kDc8zk/WywcFkP86zMqv9iBN1lsgG5EFnkKcA0JBNryXyEYJh70lgCeQEfzDOwdWLow3G1Zp/7/nh9jvDBMyYQpGMuSRrjZkSQwg+hODiYFEG1wf98Y98TB49YXVCWfg7kPz+G5PeEroIDwi3CB0Eu5MERSJf4oyHHRC/UBFLlJ/zAVuCTVdcD/cC6pDZVwH1wf2uDP0w8R9oGcXyLIUcUuzQv9J+28z+GE1FHZkRzJKHkH2JVv/PFLVVtVlWEWa6x/zI481dTjfrOGen/2zfsg+D7ahP1tii7AD2DnsFHYBO4o1Ajp2AmvC2rBjUjy8u57IdteQtxhZPFlQR/APf0MrK81knmOtY6/jF3lfPn+69B0NWLmiGWJBekY+nQm/CHw6W8h1GEV3cnRyBkD6fZG/vt5Ey74biE7bd27BHwB4nRgcHDzynQs5AcA+N/j4H/7OWTPgp0MZgPOHuRJxgZzDpRcCfEuowSdNDxgBM2AN5+MEXIEn8AUBIAREgjiQBCbD6DPgPheDaWAWmA9KQBlYDlaD9WAT2Ap2gj1gP2gER8EpcBZcAlfADXAP7p5u8AL0g3fgM4IgJISK0BA9xBixQOwQJ4SBeCMBSBgSgyQhKUg6IkQkyCxkAVKGlCPrkS1IDbIPOYycQi4gHcgdpAvpRV4jn1AMVUG1UEPUEh2NMlAmGorGoZPQdHQqWogWo0vRtWg1uhttQE+hl9AbaCf6Ah3AAKaM6WAmmD3GwFhYJJaMpWFibA5WilVg1Vgd1gzX+RrWifVhH3EiTsPpuD3cwcF4PM7Fp+Jz8CX4enwn3oC34tfwLrwf/0agEgwIdgQPApswnpBOmEYoIVQQthMOEc7AZ6mb8I5IJOoQrYhu8FlMImYSZxKXEDcQ64kniR3Ex8QBEomkR7IjeZEiSRxSPqmEtI60m3SCdJXUTfqgpKxkrOSkFKiUrCRUKlKqUNqldFzpqtIzpc9kdbIF2YMcSeaRZ5CXkbeRm8mXyd3kzxQNihXFixJHyaTMp6yl1FHOUO5T3igrK5squytHKwuU5ymvVd6rfF65S/mjiqaKrQpLZaKKRGWpyg6Vkyp3VN5QqVRLqi81mZpPXUqtoZ6mPqR+UKWpOqiyVXmqc1UrVRtUr6q+VCOrWagx1SarFapVqB1Qu6zWp05Wt1RnqXPU56hXqh9Wv6U+oEHTGKMRqZGjsURjl8YFjR5NkqalZoAmT7NYc6vmac3HNIxmRmPRuLQFtG20M7RuLaKWlRZbK1OrTGuPVrtWv7amtrN2gvZ07UrtY9qdOpiOpQ5bJ1tnmc5+nZs6n0YYjmCO4I9YPKJuxNUR73VH6vrq8nVLdet1b+h+0qPrBehl6a3Qa9R7oI/r2+pH60/T36h/Rr9vpNZIz5HckaUj94+8a4Aa2BrEGMw02GrQZjBgaGQYZCgyXGd42rDPSMfI1yjTaJXRcaNeY5qxt7HAeJXxCePndG06k55NX0tvpfebGJgEm0hMtpi0m3w2tTKNNy0yrTd9YEYxY5ilma0yazHrNzc2DzefZV5rfteCbMGwyLBYY3HO4r2llWWi5ULLRsseK10rtlWhVa3VfWuqtY/1VOtq6+s2RBuGTZbNBpsrtqiti22GbaXtZTvUztVOYLfBrmMUYZT7KOGo6lG37FXsmfYF9rX2XQ46DmEORQ6NDi9Hm49OHr1i9LnR3xxdHLMdtzneG6M5JmRM0ZjmMa+dbJ24TpVO18dSxwaOnTu2aewrZztnvvNG59suNJdwl4UuLS5fXd1cxa51rr1u5m4pblVutxhajCjGEsZ5d4K7n/tc96PuHz1cPfI99nv85WnvmeW5y7NnnNU4/rht4x57mXpxvLZ4dXrTvVO8N3t3+pj4cHyqfR75mvnyfLf7PmPaMDOZu5kv/Rz9xH6H/N6zPFizWSf9Mf8g/1L/9gDNgPiA9QEPA00D0wNrA/uDXIJmBp0MJgSHBq8IvsU2ZHPZNez+ELeQ2SGtoSqhsaHrQx+F2YaJw5rD0fCQ8JXh9yMsIoQRjZEgkh25MvJBlFXU1Kgj0cToqOjK6KcxY2JmxZyLpcVOid0V+y7OL25Z3L1463hJfEuCWsLEhJqE94n+ieWJneNHj589/lKSfpIgqSmZlJyQvD15YELAhNUTuie6TCyZeHOS1aTpky5M1p+cPfnYFLUpnCkHUggpiSm7Ur5wIjnVnIFUdmpVaj+XxV3DfcHz5a3i9fK9+OX8Z2leaeVpPele6SvTezN8Mioy+gQswXrBq8zgzE2Z77Mis3ZkDWYnZtfnKOWk5BwWagqzhK25RrnTcztEdqISUedUj6mrp/aLQ8Xb85C8SXlN+VrwR75NYi35RdJV4F1QWfBhWsK0A9M1pgunt82wnbF4xrPCwMLfZuIzuTNbZpnMmj+razZz9pY5yJzUOS1zzeYWz+2eFzRv53zK/Kz5vxc5FpUXvV2QuKC52LB4XvHjX4J+qS1RLRGX3FrouXDTInyRYFH74rGL1y3+VsorvVjmWFZR9mUJd8nFX8f8uvbXwaVpS9uXuS7buJy4XLj85gqfFTvLNcoLyx+vDF/ZsIq+qnTV29VTVl+ocK7YtIayRrKmc23Y2qZ15uuWr/uyPmP9jUq/yvoqg6rFVe838DZc3ei7sW6T4aayTZ82Czbf3hK0paHasrpiK3Frwdan2xK2nfuN8VvNdv3tZdu/7hDu6NwZs7O1xq2mZpfBrmW1aK2ktnf3xN1X9vjvaaqzr9tSr1Nfthfslex9vi9l3839oftbDjAO1B20OFh1iHaotAFpmNHQ35jR2NmU1NRxOORwS7Nn86EjDkd2HDU5WnlM+9iy45TjxccHTxSeGDgpOtl3Kv3U45YpLfdOjz99vTW6tf1M6JnzZwPPnj7HPHfivNf5oxc8Lhy+yLjYeMn1UkObS9uh311+P9Tu2t5w2e1y0xX3K80d4zqOX/W5euqa/7Wz19nXL92IuNFxM/7m7VsTb3Xe5t3uuZN959Xdgruf7827T7hf+kD9QcVDg4fVf9j8Ud/p2nmsy7+r7VHso3uPuY9fPMl78qW7+Cn1acUz42c1PU49R3sDe688n/C8+4Xoxee+kj81/qx6af3y4F++f7X1j+/vfiV+Nfh6yRu9NzveOr9tGYgaePgu593n96Uf9D7s/Mj4eO5T4qdnn6d9IX1Z+9Xma/O30G/3B3MGB0UcMUf2K4DBiqalAfB6BwDUJABo8HxGmSA//8kKIj+zyhD4T1h+RpQVVwDq4P97dB/8u7kFwN5t8PgF9dUmAhBFBSDOHaBjxw7XobOa7FwpLUR4Dtgc9TU1JxX8myI/c/4Q988tkKo6g5/bfwGaJXw+1TmUeAAAADhlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAAqACAAQAAAABAAABuKADAAQAAAABAAABDwAAAACjHIBTAAAxhUlEQVR4Ae2dCXgURfrGv4QEEk65I5fcCHKIFwgih64iIAiCIIgKeCy4XnggoIt44bIqrAoInoiINyirAoIgyCGnEA45QghHCCZcIQmBEPpfb/13YoaZSSaZq6f7redJZrq6uuqrX/X0W/VVdXeEoYIwkAAJkAAJkIDFCERarD62rA76KH/88Yds3749r/779+/P+84vJEACJGBHArYWuNWrV0v79u2lcuXKUrJkSWnatKnceOONITkPDh8+LElJSUUqOzMzUx566CHp3r27fPrpp7J06VK58847Zd68efL0008XKS8mJgESIAGrEYigi1KkZ8+eMn/+fJk1a5bcddddIWnjUaNGSVxcnDz++ONelR8fHy8DBgyQ++67z+mYQ4cOSa1atWTy5Mny6KOPepUXE5EACZCAFQnYegRnlgY9d+6cfP/9916bk5GRIb1793YRN2RQs2ZNufzyy6Vjx45e58eEJEACJGBFAlFWrJSvdTp69KhMnDhRICS9evWSqKgoLUD16tWTPn36SI0aNWTfvn0ydepUnaZLly6yd+9eSUxMlIYNG2rhqVChgjjygSuxXbt2MnDgQJk9e7asX79eEDdlyhTZvHmzPPXUU7Jt2zY9ikxJSZGbb75ZkKen8OSTT0qZMmU8jtA6deokLVu29HQ440mABEjAFgQ4gnPTzLGxsVKpUiUtYC+88IIsWbJEhgwZIhMmTJB77rlHH1G+fHn9OW3aNO0ibNWqlfTr10/ef/99adOmjZ5Pc+QDIVuzZo1O37hxY1m4cKG8++67kpubq92St9xyi9536aWXSrdu3aR+/fp6290/jPY++eQTLaKRke6bb9KkSeJpn7s8GUcCJEACViTg/gppxZoWoU6lS5fWIzccUqJECXn55ZelefPmelS0bNkyycnJ0QLYtWtXnSsWejhGXY899pjs3LlT3nvvPcmfj6P4q6++Wo/yHNuYL2vRooXebNSokXYt1q1b17Hb5XPLli169HfTTTe57GMECZAACZDAXwQocH+xcPsNwuYI0dHRghEURl6ewvXXX693QQgDERzL/+EKdRe+++47gZuTgQRIgATsToACV8gZgBFcUcLZs2d1ctx64Clg/q2gMHr0aI+7cVtDRESEJCcnu6SB8H7zzTfa7emykxEkQAIkYDMCFDg/NDhusnaEmTNn6q+DBw/Wn1WrVtXzYViwgoCFJ8uXL9ff4epEQBqE48ePC5b579q1S2+7+4e0WEQyY8YMp90QtyeeeEKee+45p3hukAAJkIBdCdj6Pjjc6I0ViQkJCXLixAnB3BeW2eN+uFtvvVW2bt0q1atX1zdNw+2HebVjx44J5tEgUr/88ovccMMN0qxZM8EKS6RJTU3VIoP70xzhmWee0YtPsHT/wIEDUqpUKVmxYoVgwQlWT2JBSN++fWXBggXSoUMHwT1xBa2ihFjecccdcvLkSRk0aJBg1Ih8cA8dbGEgARIgARIQsbXA+XoC/Pzzz1rgsLoSTw7JysqSsmXLus0Wc3dwTeL2AaTDiAurLHELgiNkZ2fr7fxxjn3uPjHaw+ixSZMm+uZud2kYRwIkQAJ2JfDX1dWuBPxQbzwLEqMwT+KGIiBaEDcErK50F2JiYtxFe4zDaBN/DCRAAiRAAq4EOIJzZeJVDJbrP/jgg/oZlufPn5fhw4frG7m9OpiJSIAESIAEAk6AAhdwxCyABEiABEggFAS4ijIU1FkmCZAACZBAwAlQ4AKOmAWQAAmQAAmEggAFLhTUWSYJkAAJkEDACVDgAo6YBZAACZAACYSCAAUuFNRZJgmQAAmQQMAJUOACjpgFkAAJkAAJhIIABS4U1FkmCZAACZBAwAlQ4AKO+P8LwPMuv/rqqyCVxmJIgARIgAQocEE6B/DSVDwgGW8MYCABEiABEgg8AT7JJPCMdQlJSUkSHx8vPXr0CFKJLIYESIAE7E2AI7ggtH96erp2T27cuFHS0tKCUCKLIAESIAESoMAF4RwoWbKknDlzRsaNG6dfeBqEIlkECZAACdieAAUuCKcAXoODt3AzkAAJkAAJBI8ABS54rFkSCZAACZBAEAlQ4IIIm0WRAAmQAAkEjwAFLnisWRIJkAAJkEAQCUQFsSxbFrV69Wo5efKkfvM3AGDBCQMJkAAJkEDgCVDgAsx4zpw5smLFCmnatKlcc801Urdu3QCXyOxJgARIgARAgDd6B/g8OH36tCxbtkzKli0r7du3l8hIeoUDjJzZkwAJkIAmQIHjiUACJEACJGBJAhxOWLJZWSkSIAESIAEKHM8BEiABEiABSxII+SKTXr16CZ70ERERYUnArJR/CRw6dEjmzZsnlStX9m/GzI0ESMByBEIucGXKlJFJkyZJ6dKlLQeXFfI/gbfeektSU1MpcP5HyxxJwHIEQiJwePBwSkqKhpmZmSkHDhyQ2NhYy8FlhfxDICoqSmrUqOGfzJgLCZCAbQiEROAefvhhqVSpkuDCVa9ePfnmm2/oorTNKVf0im7fvl2GDRsmHTt2LPrBPIIESMC2BEIicOfOnZMhQ4awV27b065oFf/hhx8kMTGRAlc0bExNArYnEDSBg6h9+eWXGnhCQoLMnz9fj+Js3wIE4JFA3759ObL3SIc7SIAECiMQNIF74YUXJC4uTi8mGTBggJQqVYpP9SisdWy8H3OzWHw0cuRIG1Ng1UmABHwhEDSBy8jIkCuvvFKaNWvmi7081iYE4uPj5YsvvrBJbVlNEiCBQBAIuMBNnDhR0BtftWqVfqp+1apVA1EP5mkRApybtUhDshokYAICARW4qVOnCh423LlzZ+nUqRPnU0zQ4GY2ITs7W8aOHSsffvihmc2kbSRAAmFCIKACl5ubK61atdKuyTDhQTNDSGD//v1Srly5EFrAokmABKxEICACN2/ef9XTJtJk06atyi2ZJYYRbUpmhmFI69bNpUoV58c+HTt6VLZs3BgWI85o9QLVdtdf78J34+8b5Xj6MZd4M0bUrVVPGtRvYEbTaBMJkEAYEwiIwC1dukm6dLlDevW61tRozpzJljVrNkiPHjc52blx3TqpI4bEqpWeZg8/b1jvVuAWr1oo193ezuzmS+65XPl56RIKnOlbigaSQPgRCIjAVatWQ2rVMn+PPDl5n5w54zq6jFZPWIktESn1a9c2fYv+kXzYrY0XVaoodRqY3/6M9Aw5UDbZbR0YSQIkQAK+EODrcnyhx2NJgARIgARMS4ACZ9qmoWEkQAIkQAK+EKDA+UKPx5IACZAACZiWAAXOtE1Dw0iABEiABHwhQIHzhR6PJQESIAESMC0BCpxpm4aGkQAJkAAJ+EKAAucLPR5LAiRAAiRgWgIUONM2DQ0jARIgARLwhQAFzhd6PJYESIAESMC0BChwpm0aGkYCJEACJOALAQqcL/R4LAmQAAmQgGkJUOBM2zQ0jARIgARIwBcCFDhf6PFYEiABEiAB0xKgwJm2aWgYCZAACZCALwQocL7Q47EkQAIkQAKmJRCQ98GZtrY0zJIEZs+eLadOnZI+ffpItWrVLFlHVooESKDoBDiCKzozHmEyAvv375fhw4dLQkKCySyjOSRAAqEkQIELJX2W7RcCvXv39ks+zIQESMBaBOiitFZ72r422dnZMn36dImIiJDu3btLgwYNbM+EAEjArgQ4grNry1u03klJSfL2229LXFycVKpUyaK1ZLVIgAS8IcARnDeUmCYsCCxcuFA+/PBDWblypdSqVSssbKaRJEACgSPAEVzg2DLnIBNYtWqVJCcnC1ZVMpAACZAABY7ngGUIvPHGGzJhwgQZO3asLF261DL1YkVIgASKR4ACVzxuPMpEBJ555hmJjo6W0aNHy2233SaRkZHSv39/ef/9901kJU0hARIINgHOwQWbOMvzO4F58+Y55Xn27FmnbW6QAAnYkwBHcPZsd9aaBEiABCxPgAJn+SZmBUmABEjAngQocPZsd9aaBEiABCxPgAJn+SYO/woahiE5OTly5swZwXcGEiABEvCGABeZeEOJaUJKAML22Wefyfr16+XIkSPy0EMPhdQeFk4CJBAeBChw4dFOtrYyNjZW7r//fv1naxCsPAmQQJEI0EVZJFxMHEwCmZmZesTGZf/BpM6ySMA6BDiCs05bWq4mv/32m7z00kuSkZEhkydPtlz9WCESIIHAEqDABZYvc/eBQNeuXWXixIk+5MBDSYAE7EyALko7t74J656VlSU7d+4UfDKQAAmQgC8EOILzhR6P9TuBlJQUmTVrluDFpa+88orf82eGJEAC9iFAgbNPW4dFTRs3biz/+c9/wsJWGkkCJGBuArYXuN27d0vp0s4Y4CKr36ypuVvuf9ZlZWXKihUrXGw9fvyYS5xZIw4dStZ1OHjwoGDlJAMJkAAJ+IOA85XdHzmGWR47duxQr1pxfjoG4m4JE4FLT0+XlO3bXainpBxxiTNrRGJiopSPqqDNwzvdGEiABEjAHwRsL3Bt214rI0YMc2L50bvvOm2beaNsmTIy4sknXUwcPWGUS5xZIxo1aiRPuqmDWe2lXSRAAuFBICACt2/fDlm8eI7pCaSmHpYrr6zvYmfFypXll02/yzrlqjR72KJcrO7CybST8uPsRRIREeFut2nisjIyJdIoYRp7aAgJkIB1CESoh9c6++esUzfWhARIgARIwMYEeB+cjRufVScBEiABKxOgwFm5dVk3EiABErAxAQqcjRufVScBEiABKxOgwFm5dVk3EiABErAxAQqcjRufVScBEiABKxOgwFm5dVk3EiABErAxAQqcjRufVScBEiABKxOgwFm5dVk3EiABErAxAQqcjRufVScBEiABKxOgwFm5dVk3EiABErAxAQqcjRufVScBEiABKxOgwFm5dVk3EiABErAxAQqcjRufVScBEiABKxOgwFm5dVk3EiABErAxAQqcjRufVScBEiABKxOgwFm5dVk3EiABErAxAQqcjRufVScBEiABKxOgwFm5dVk3EiABErAxAQqcjRufVScBEiABKxOgwFm5dVk3EiABErAxAQqcjRufVScBEiABKxOgwFm5dVk3EiABErAxgSgb151VJwGfCPz555/yxBNPSExMjE/58GB7EMjNzZXBgwdL586dTVHhjz/+WDZv3iwRERGmsKcgIxo3biwPPPBAQUnc7oswVHC7h5EkQAIFEpg0aZKUKlVKOnbsWGA67iQBEMjOzpbJkyfLrFmzTAGkSZMmMm3aNClRooQp7PFkBCRq4MCBkpyc7CmJx3iO4Dyi4Q4SKJxAnTp1BH8MJFAYgaSkJImNjS0sWdD216xZU66++mopV65c0MosTkEY+Rb3N0aBKw5xHmNbAsePH5e7775bypcvLydOnJDly5fL7NmzbcuDFS+YwPnz5yUuLk5eeumlghMGae97770n8+fPl6ioKImPj5dBgwZJdHR0kEovWjGRkZEyc+ZM7SUp2pF/pabA/cWC30igUAJ79+6Vtm3byj333FNoWiYggXPnzul5WrOQWLZsmYwZM0YuvvhiGTBggLz66qtStmxZs5jnZAd+Y6mpqVKrVi2n+KJsUOCKQotpSUARgEunQoUKZEEChRI4efKklCxZstB0wUqAOWMsioLLD5+1a9c2rYuycuXKPmOhwPmMkBnYgUCHDh3kkksuEcwHoFf+22+/2aHarGMxCWC+7ccffyzm0f49bO7cuTJu3DgpXbq0wAOxdu1aKVOmjOzYsUO6dOliqkUm+/btkzVr1kjdunX9AoEC5xeMzMTqBC6//HJ55ZVXrF5N1s9PBB588EE/5eR7NhCy8ePHS+/evWXYsGHyyCOPSKtWrbS4ffvtt6YawT311FMCkaPA+d7uzIEECiQA9xLudUPAgpI9e/YUmJ477U0Aruvq1aubBkJCQoJgkUtaWppeYr97927BOY3RJUZzWVlZ+pwO9RwcFpM0aNAgINw4ggsIVmZqBQL9+vWT9u3bC36ANWrUkO+++84K1WIdAkRg9erVMmXKFKlWrVqASvA+219//VX69u2rR2nbtm2TdevWycqVK2XTpk2SmZkpFStWlMTERO2VCPUqyiVLlsicOXO0rd7X0LuUFDjvODGVDQlUqVJFRowYwSeV2LDti1Pl1157TTIyMkwhcAcOHJCnn35aRo4cqUWsadOmbl2UH3zwQchdlFOnTtVuyeIwL+wYClxhhLjfVgTwtIlDhw7pOp86dUr3crHyzIwBriV3owW4VXGhDYeAkfGFjzqDWw3zMOEQsELSl2Xs/q4jzl2cw0eOHNGuSbgpjx49KikpKYLv6enpAvHDuXP69Gm96CRULkos2sL9eIEMAcn9rf9MVL7o8oG02y95JyUdlDsG3K9Xx+XPECfAZ59OV/HFv/8if36B/H7kSLo8/OjTLkV8/tnHkn36mHpygrmfk3j27Flp2OgKaXvtdS51CEXEjBkz5ezZi9TNpTHSocNA9ay+Y8oMcz6rLzHxdxk9eoQLpi9Ur/zy+vVc4s0WkaNWo26PiZUeffo4mbZh43rZc2qXxMSas2OR39g98Qly5y13mUbkxo9/S1q0uFat9q2h3JBx8sMPW9W1+Go131ZCf7/iitvUvNsZ1XHbKj16jFAPKkhU1Qn++Z2VlamurxvVvXj98uP0+/eACFyZMtFyR1/zP59v/fqtcuzYMReBw4KCDu2bqxt6W/kduL8z/PiTn9xnaeRK717t1RM3zHkTp8PovXsPSOL+U47NkH/m5JxTF4h2UqXKxSG3pTADsrLS3CapXb2adL7iCrf7zBR5WN3Eu/7I/y/iyW8XhC+ufjVpdFnD/NGm/I4n+ebk5JjGtoYNm0u7dt1NY48nQw4c2KNGmIG/1Yavy/HUAownARIgARIIawIUuLBuPhpPAiRAAiTgiQAFzhMZxpMACZAACYQ1AQpcWDcfjScBEiABEvBEgALniQzjSYAESIAEwpoABS6sm4/GkwAJkAAJeCJAgfNEhvEkQAIkQAJhTYACF9bNR+NJgARIgAQ8EaDAeSLDeBIgARIggbAmQIEL6+aj8SRAAiRAAp4IUOA8kWE8CZAACZBAWBOgwIV189F4EiABEiABTwQocJ7IMJ4ESIAESCCsCVDgwrr5aDwJkAAJkIAnAhQ4T2QYTwIkQAIkENYEKHBh3Xw0ngRIgARIwBMBCpwnMownARIgARIIawIUuLBuPhpPAiRAAiTgiQAFzhMZxpMACZAACYQ1AQpcWDcfjScBEiABEvBEgALniQzjSYAESIAEwpoABS6sm4/GkwAJkAAJeCJAgfNEhvEkQAIkQAJhTYACF9bNR+NJgARIgAQ8EaDAeSLDeBIgARIggbAmQIEL6+aj8SRAAiRAAp4IUOA8kWE8CZAACZBAWBOgwIV189F4EiABEiABTwQocJ7IMJ4ESIAESCCsCVDgwrr5aDwJkAAJkIAnAhQ4T2QYTwIkQAIkENYEKHBh3Xw0ngRIgARIwBMBCpwnMownARIgARIIawIUuLBuPhpPAiRAAiTgiQAFzhMZxpMACZAACYQ1gaiwtp7GkwAJkAAJhCWBHTt2yAcffCCrVq2S6OhoXYf4+HiJiIiQvXv3yr59++Tzzz+XKlWqSGxsbLHqSIErFjYeRAIkQAIk4AuBDh2ulxtu6CAnTpyQUqVKSb9+/WTz5s3SqVMn6dChg3z77bfSvXt3qV27tvz000/FKoouymJh40EkQAIkQAJmJ8ARnNlbiPaRAAmQgAUJbNmyWXJyMmX79u3aRblgwQJJTEyUtWvXSmZmZrHdkvlRUeDy0+B3EiABEiCBoBBo2bKVdlEeOHBAuyi7du0qv/76q1xzzTXaRblp0yaf7cgTuPT0dPn6669l4MCBujBfck5NPS7LftnoSxZBOXbbtt3S6YZmLmWVLFlSftmwQ7LP5LrsM1tESkqaW5MyM7Nk+YpNUrZsGbf7zRJ54MBhqVGruVnMkchIeu1N0xg0pMgEcnPPFfmYUB2QnHxIsNAkJSVFcM3Fd8zHeRtmz54tp06dkj59+ki1atXcHpYncHPnzpWhQ4dKpUqVpFevXm4Text53wOPyrFjx7xNHrJ0NetcI40aNXIpv0mTJhIVdbecP3/eZZ/ZIoa0vMmtSf36361PHLc7TRRZs45I3bp1A2LRuXPnVDvmneJelVGzZk2v0jERCQSSgGEY+vpTokSJIhVTocJFRUofysSnT5+WkydPCj5zc3P197i4OElNTfXKrP3798uYMWOkVatWhQscVrDExMTIzTff7FXmBSWqXLmy4C+cQ4MGDcLZfClXrpz+C+tK+Gj8wYMHZfLkyfLII49I/fr1fcyNh5NA8Ahgqfwbb7whF198sdx5553irdB5my54NfFcUoMGDaVt27aCWwOwihLf8Zv1dmDRu3dvLXCeSxDJ695++eWXAl9oQkKCXHbZZQUdw30kEHQCa9asKdZSYbg/MCIfPHiwPP/881KnjhoyMpBAkAm8/vrrkpaWJhs3bpQXX3zRq9JzcnJk2rRpOv24ceP09JFXB4ZJoj17dkvp0lFad+rVq+eT1dnZ2TJ9+nR9Dx1uLXAMUPIEDj7Me++9Vy699FIKnE+oeXAgCFStWlW7Ioqa9+7du8VxrCc/fVHzZHoSKCoBDBrQ2cL5CJeaNwHzURkZGXrBRfPm5pmn9sZ2b9JgfQCmxKA5WD3pS0hKSpK3335bXn75ZZ2nI688gbvuuusccfwkAdMRQI/M0Svz1jhcTNq0aSMzZszQ7ndvj2M6EvA3AawQxEUY7riePXsWmj3m4N555x1ZvHixtGzZstD04ZggLq6G9q5goYgvArdw4UL58MMPZeXKlVKrVi0nFHkC5xTr40aCGnqmpv7pYy6BPxwnUZu27VxWzsEHvG7tGsF+s4cqVapKw0aNXcxMOXxY9u3b6xJvxogWLVpJmbJl/W4aFhC5W0RUUEHnz5u/zQuyn/usQQBzcMOHDy9yZc6ePVvkY8L9ADzqKzk5WbCqctSoUU7VCYjA/bRorrS/tolTQWbc2Lhpu8TElpHWrVs7mbdt2zbJSN8p1apWcoo348bXX/0oo0a/5GLaksX/lWaXViryKkKXjAIccfp0tixfflRu6VZ4rzbApujsMcldseIVwSiKZZCA3wlkZWX5PU+zZ4jFOLhJ/JlnntH30HXu3DnP5DyBGzt2rL6bHH7Mbt26qcm/0nmJivqlZMloadHCdVRR1HwCnf7MGc+9nTKlY8OiDps2J7nFhOXxDRrUkfLl/T8ycltgMSP37j0gp7LMc7+h6jgzkEDYEsDIzy4BgoaHNI8ePVomTZqkV1T2799fJkyYIMOGDdMY8gTuzTffFPwxkAAJkAAJkIDZCcybN8/JRHfuWT62wQkRN0iABEiABKxCgAJnlZZkPUiABEiABJwIUOCccHCDBEiABEjAKgQocFZpSdaDBEiABEjAiQAFzgkHN0iABEiABKxCgAJnlZZkPUiABEiABJwIUOCccHCDBEiABEjAKgQocFZpSdaDBEiABEjAiQAFzgkHN0iABEiABKxCgAJnlZZkPUiABEiABJwIUOCccHCDBEiABEjAKgQocFZpSdaDBEiABEjAiQAFzgkHN0iABEiABKxCgAJnlZZkPUiABEiABJwIUOCccHCDBEiABEjAKgQocFZpSdaDBEiABEjAiQAFzgkHN0iABEiABKxCgAJnlZZkPUiABEiABJwIUOCccHCDBEiABEjAKgQocFZpSdaDBEiABEjAiQAFzgkHN0iABEiABKxCgAJnlZZkPUiABEiABJwIUOCccHCDBEiABEjAKgQocFZpSdaDBEiABEjAiQAFzgkHN0iABEiABKxCgAJnlZZkPUiABEiABJwIUOCccHCDBEiABEjAKgSirFIR1oMESIAEzEjg/PnzcuzYMcnJyRF8Z/COgGEYcuLECSlbtqzk5uZ6d9AFqShwFwDhJgmQAAn4k0BaWpo8++yzkp6eLrfffrs/s7Z0XllZWfLaa69pgStdunSx6kqBKxY2HkQCJEAC3hGIi4uTWbNmeZeYqfIIYOQ2efJkueSSS/LiivqFAldUYkxPAiRAAoUQgCty7969cu7cOf1XSHLu/h8BcEtNTdXszp496zMXCpzPCJkBCZAACTgTOHPmjEybNk3PHXXr1s15J7c8EgC3xYsXy759+6RChQpSs2ZNj2m92UGB84YS05AACZBAEQhgzuidd94pwhFMCgLgdt9998mAAQP8AoS3CfgFIzMhARKwOwGs9Nu6dats2LBBDh48aHccXtcfqyUPHz6sucXHx0tERITXxxaWkCO4wghxPwmQAAl4QQC3AaxZs0YqVaokY8aM8eIIJgEBCBxckkuXLpV69epJ//79/QaGAuc3lMyIBEjAzgTgXnvggQekQYMGdsZQ5LpHRkZKp06dZPz48UU+trADKHCFEeJ+EjApgbS0ozJ37lwX644fPeoSZ9aIrdu2ymnVg88f9iTukatub5U/yrTfs7IyZdGiRVKtWjX5/fffZejQoaa11WyGrVu3Vt0bmCZbtmyRunXrBsQ8ClxAsDJTEgg8gYyMDClRorJLQRmnMlzizBpx4vgJVYcSTubhhuhwCVjKDvvxN3LkSKldu3a4mB5yO/GUEnC76qqr/Lao5MJKBUzgivtolQsNDOQ2fL+eAvaFQx082Y/4cKhDQW1QUN24T6RcuXIyaNAgFxT/2bXLJc6sEY0bN3apQ2R0+Kx9K1UqRm7qdYvUr1/frIhNa1fr1q1d2t7fxgZE4CIiY2XmrJ/9bavf80tNTZMhwzq55IsnD3z0w9eyY2eqyz7zRUS7NalsuYoy5/PlUrJkSbf7zRJ58mS63Pi3HmYxRypXrihLlnwmMTExprHJnSHoGPz55z53u2RfSopMnzfP7T4zRR5To7cmV17pYlLtmrXl+x/my+/LtrjsM1vEocRk6XNtP9OYtWvXBsnIOG4aezwZkpqaIj17dvK022/xEeqH4nkY47dimBEJkAAJkAAJBJdA+PgCgsuFpZEACZAACYQ5AQpcmDcgzScBEiABEnBPgALnngtjSYAESIAEwpwABS7MG5DmkwAJkAAJuCdAgXPPhbEkQAIkQAJhToACF+YNSPNJgARIgATcE6DAuefCWBIgARIggTAnQIEL8wak+SRAAiRAAu4JUODcc2EsCZAACZBAmBOgwIV5A9J8EiABEiAB9wQocO65MJYESIAESCDMCVDgwrwBaT4JkAAJkIB7AhQ491wYSwIkQAIkEOYEKHBh3oA0nwRIgARIwD0BCpx7Ln6PTUhIkK+++srv+TJDEggGgQULFsjmzZuDURTLIAG/EeD74PyGsuCMhg4dKh999JEcPXpUKlasWHBi7iUBExE4f/68PmfbtGkjixYtMpFlNIUECiZAgSuYj9/2JiUlSXx8vPToYZ63V/utcszI8gSWLl0q1atXl2bNmlm+rqygdQjQRRmEtkxPT9fuyY0bN0paWloQSmQRJOA/AuvXr5e1a9fKihUr/JcpcyKBIBCgwAUBcsmSJeXMmTMybtw47aIMQpEsggT8RiAuLk4WL14sU6ZM8VuezIgEgkGAAhcEyjExMdKpU6cglMQiSMD/BGrVqiXNmzf3f8bMkQQCTIACF2DAzJ4ESIAESCA0BChwoeHOUkmABEiABAJMgAIXYMDMngRIgARIIDQEokJTrH1KXb16tZw8eVKw0ATB8WkfAqxpOBN4//335W9/+5tkZ2dLqVKlwrkqtN2GBChwAW70OXPm6OXVTZs2lWuuuUbq1q0b4BKZPQn4hwBu8P7nP/8puE1g4cKFMnz4cP9kzFxIIEgEeKN3gEGfPn1ali1bJmXLlpX27dtLZCS9wgFGzuz9SODQoUOycuVKueqqq6R+/fp+zJlZkUDgCVDgAs+YJZAACZAACYSAAIcTIYDOIkmABEiABAJPgAIXeMYsgQRIgARIIAQEKHAhgM4iSYAESIAEAk+AAhd4xiyBBEiABEggBAQocCGAziJJgARIgAQCT4ACF3jGLIEESIAESCAEBChwIYAeqCLxzi68loeBBEiABEhAxPb3wd1www2SnJwse/bskUqVKknDhg3l9ddfl7Zt2wb1/MBTI/DECDztpDjhzz//lCZNmshPP/2kb8otTh48xncCuLH/+++/lxIlSujMcE7hVTO42R83+RuGIXi/Wrt27WTDhg2yf/9+HYcbqevUqeNiAF6W+/XXX8vAgQMLfFQW8vr99991Hni0lr8CzqsffvhBzp49Kw888IBfsj127JjMnz9fsrKybP10FHRG586dK7iZvkWLFvpt6WjvYLw1fcGCBXLxxRdLq1at/NKmmzZtErzQGa9Wuvnmm/2Spz8ysf0IbsmSJTJmzBg5d+6cFjU8tSHY4oaGxMnx2GOPFbtNX3zxRTlx4oQcPny42HnwQN8JoKOCv/79++v2hKhFRUXJqVOnpE+fPvLCCy/obZQUHR0tM2bM0MIUERHhtnBcAIcOHSq4IBUUkNcrr7wiH3/8scdkTz/9tMd9nnbA9h9//FGeeOIJT0mKHI+6/vzzzzJixIgiH2ulA2655RaJj4+Xq6++Wj/O79JLLw3KW9Md5+dTTz3lN5w4/yZOnCgffPCB3/L0R0a2Fzh/QPRHHt9++22xs1mzZo1+xiVGCxS4YmP0y4FlypSRO+64Q7p3765HZ+glV6hQQe69917tHdixY4f+RGEtW7bUo7fnn39eateu7bb8fv36yWeffVZorxh5XXLJJW7zQOTx48dl5syZHvd72oFz6rrrrvO0u1jxFStWlI4dOxbrWKsctG3bNv0ItPHjx8v1118v6KDeeuutQakeOl3z5s2TyZMn+608eCnM+Cg3PmzZQxO/9dZb2m1Zs2ZNcVxkMjIy5LbbbtM9Lhz28ssvy5EjR7S4VK9eXSA0F110kdx+++1y+eWX65wd+aAnDNfnli1bZNasWYK8Bg8erC9yo0aNkunTp0vlypUF36tVq+Z1jxm9sSlTpuie00cffUSB89CewY4eMmSIvoigTV566SXZt2+fwP0EV9+nn34q//jHP2TdunVy5ZVX5j2f9Ndff9UjG7g3BwwYIA0aNJAvv/xSDhw4IAkJCXLZZZfp4zH6Qc+/TZs28scff+iq5Xcf4hxbtGiRdnfj4gn3EbwDKP+NN97Q+cCNlJOTI59//rls375duzbvueceiY2N1fnh/ESn6+jRo1qoC+OHMuAeh/sR4oXRCQLqiHi4ZuE6deeC/+WXXwRv3airHkSOeoMZ3HY33nijtuuTTz4RjPruvvtu+eabb/Q8M2zF1MJ///tf7ZLv1KmT5ogRMX6TyAdekZSUFOnVq5fOOzc3V//20MmoUaOGYMQUKnea41zo3bu33HfffdpljREQ3jyCTurs2bP1NeLZZ5/VDJcvX65d26g3eK1atUq7vtExwggbUyvwECxevFi2bt0qPXv2lEaNGrltNkyFYL4e1ypP7lC0F7wHaFe8AQU8kR9c1jgnDx48qN2b6My5e74u2gFpwR6dPLRpZmamwIuAsr21Hywc7X/nnXfq9q9atar06NFDP98X7Yu84SFBPK7VuGbnBVUR2wfl1jEUEEOdFHksli5daijBMerVq2coITJUgxjq5DLUxcdITU3V6dQ8gqGG5vrYf//734a6sBhqrsSIiYkx1IVDp3Hko+Zd9Lb64RrDhg3Tx6getaFeQ2Ko+RlDXVgM9cYB/V39MPPsKOzLtGnTDOVm1cluuukm48EHHyzsEO4PAgElHobq9BjqAmSoC6uhRmmGuhDpdr7iiiu0BQ899JCxa9cu/V39iA3VsTFwvqA9y5UrZ+BcUfNf+lxRFxWdTl309fmIc0RdpI2///3vxnPPPaf3de7c2VAXE+PVV181lKjo45BOCYGh3Jw6TyU4hhJLnV5dfAycM3v37jUef/xx/R071JyfUaVKFUO9QcBQwmyo+UJDPSxcH+Pun5pz1HnD7i+++MJQFzxDXdyMr776StcfdUQZjRs3Nt59912dhXoNj7YPG7t37zaU2BjqoqX3KXHW+1THzVAXfEOJt/5NKSEw1MVbp1Vzlrre6mKp0+IYBNiO37ISWc1RCbuhOgp632uvvWaokbShOoWGesuHoToZOj4U/3BOqBGbthX2KgE31MVZXw8cdUa8mtPV7Fq3bq3rBFsdvFAvtDW4I61aT6DPMTUq1NcuNe3itmqqw2TgPFLzfm73IxLXqGuvvVa3vxI3o0uXLobyAuh8VQfdUNMhOg3aDHVB6Nq1q6EET393tAOuraqzZKAdYCPOi6LY72CBa6p6m4VufyW0xujRo3U5qpNn4PxTHUdDTQsYqiOg4x3/0LOyfXAncICiht26UVQvQTNS/mW97RAvREIEcfI5Qlpamj5Z8WN2BOTjEDjE4QRBY0PgHEG5tvQJ5dj25hNCiwucI+DHm1+kHfH8DA0BNW+l21nNnxlqRKONGDRokI777bff9AXOYRkuJt26dTMSExP1n1pwYrz55puG6unr9BA4tShBi4dDJCA8+Ts0EDh0yHABR8cJ59jUqVN1EWPHjtXnqqM8tbhF5/X222/r8tBZQ3rVYzbUwgNDvfnCkdRQ84YFChzOOVxcEWAjRBkB9qhet/6OfxBZ2IeQX+CwjbQOgUMesAUCh4BOHLYhZghq1GOo3rq+qGEbvx3l4sNXA78/pFXzS3rb8VtDPPKDUOMiDC4Q71AGtBOEWY1qDHR6YLdaD6BNctQZAoeAzkhHJdqOkL+t0ZnCsegAIDg6RWp070ju8gnB8SRwEC8IyjvvvKOPQ+dk8+bNmpkazeUJGs5rlKtGhDpdfoFztAOurQjowCEtBA6hKPY7WDjaHx0DHI+gRu6Gco0a6Cyi0wKhyx84B6eoFxQw7IX7EQETqQh4+aOnADcj/NGq1xpwdyFcpHAzqcbVf3BbcQ7OU8sEPx5uSgS0D1yFCI44rIqEuxsBbma4FeFmwSIn/GHByIWrIeEqwuKVSZMmySOPPKLPx0cffVTn4fgHNybcefhDUD92x668T7joUB7KhesU5cE1BldQ+fLltZsRKz29DXBj4XeCABvhnkTeSsR1fo584BKDSwnnbHECVhoioG5YMez4PcJFhvLyh/xpEY/9YA8XGX4jI0eO1FMM+Y8J5ne4HLHaFm38r3/9S6+ohTsP7jt3wV07XtjWmIdFKKjt3eV9YRzc3rjGOdpUdUr0VArcyHjtl8MlibllBKwELiz4w35Hm+Zvb0ztqNGldtvCham8YU6mUOCccLhuOJZ7u+7xHIPlvzgOP2h3AfMbBQXMk6geUEFJtF8e8wjvvfeenoPDPJwawVHgCqQW3J24AGGFHEQE80cI+DFiMQgu9JgvQMAPFrer4KIC4VOjPD1nlJiYqPfn/4fFIvhR33///XqOBC/S9SaoHrmev0NarOTEwhHE4SKG8vCHOS2sJla9Y1FupLxsUaa7C5QjAeqEuTaHcGHFJzpbEPWdO3c6kunvmLNxCFPeDvUFc38oGwG3PCAUVKZOUMR/WDWIFdIQF8dfEbPwW3LMTymXqdN9q1hij3lHBMdcKJiAAzoR/ubhqTJqNKkXRqHj4whqCkbUaEl3fhzXJnTisQjJMd/qSItPvP0d57XjnMB8KIK/64DfC9Y24Hx78skndbvqgv73j4tM8tMo5necrFiiD0FDLwcXB/yQHScpesNJSUl5uWPyFsHR+PiOhSW4kCCgZ6dcT/q7u3/o6WPVFe6Pyh8wuYoJdvRWHb2s/Pv5PfgEIETfffedXtSA0tG7VvMbWvTUPFueQZgox+0AGNXhfjgs1oCQ4RYWCIJyJYpyYer7jDp06JB3XF21MAMLEiCGOO+Qv5r70qMnHIeVk7hoYjGDcsvpBVAQMPS+sSABIxkch8UvEGSM4DBChCCjRwxvBMQACwQgELjQXRiUK1WPjtR8nl5Jh84dylLuKZ0PxBPnIzp2SIsFM8pFqW+XUK5TPZICEzUPKA8//LDOHrYjLUYPqAO2wQIvDcbFHvfQIR6jBywewW8KCw+wuAFplVtLH6vcbHnHYqEX7MeiF/xmcUEMVQAjMEF7Y1UpBAH34uIN6ghY/IIFIGoKQm/jtw2RADd0TPK3NRaNoM4YgWPREuro4IVFOUUNOBaM0CZ33XWX7niBrVqDoK9juLZhBI1ycY7CHsdiJdQL5zsWueB4nAdYUIL2QsDiKoxUvbV/woQJTu2vXPmi3KX6fMViPXge8BvDAhjkidGwU1CKauugepl6cYc6wbRfH/MaSqQM5c4wlGBp/z58/upioCe3kQ7za5jER8AcnLpY6ElbzEWok1VPimNy1BHgq8aCAHXSGpgUxWSoagQ9ke+YHFcjMaN06dJ6vzrJHYe6fGLiXp1cekGCuvDk7Yc9arRgwEfet29fPZeSt5NfQkoA82H5AyblL5wrcOzHwgDMQbkLmIPAuaYu8HrhAbYx94XFCd4Gx5xO/vSIU52i/FH6O+xAPD6xUMCxmMAl4f8iVIfNbb08xV+YD/JHOQj4/XjicOFx3m7DDgTVQfT2kIClwznhsAdz/JizchdgK9oAbaREwm07uTvOUxyuM6qzrefusVCnsIAy3QVP8RemRT3RjjivURd8+jOAIc4bd+c1yrH9k0yc1L4YG2q1me5xo3eFpb/oAbsLirXubWA/vmMUhhEeem6OgJEX3FRK6BxR/CSBPAJwDWGeBbcdoCeLUQjm4jCaR4+ZgQQKIoDrC24rwLmiVsjqp8j482bvgsoO1T4KnI/k8wucj1nxcBIolABcPStWrNBzrZjLUyt4Pc71FpoZE9iOAO4vxNwaHg1nxhuz/d0gFDgfiMKXjF41fPvwAWPBBwMJkAAJkIA5CFDgzNEOtIIESIAESMDPBHibgJ+BMjsSIAESIAFzEKDAmaMdaAUJkAAJkICfCfwfVciOB7SelYEAAAAASUVORK5CYII="
    }
   },
   "cell_type": "markdown",
   "id": "05c5a5fc-4731-4975-b35b-f89645ca6508",
   "metadata": {},
   "source": [
    "## Matrix-Matrix multiplication\n",
    "\n",
    "Again, the index missing in $ij, jk -> ik$ on the RHS is $j$. This means we sum across $j$. \n",
    "\n",
    "To help visualize using tensor compressions, do $ij,j->i$ as we did in matrix vector multiplication, $k$ times for each column $k$ of $C$. \n",
    "\n",
    "![image.png](attachment:image.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "79ef0d15-fec3-4cba-8422-d7b5ee902dde",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([3, 4])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = torch.arange(12).reshape(3, 4)\n",
    "c.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c622c50c-5fce-40f5-9ca6-f4526c9c6e7c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[20, 23, 26, 29],\n",
       "        [56, 68, 80, 92]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.einsum('ij, jk -> ik', [a, c])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28ebf46c-1211-4b9e-a7e6-ae806010e63b",
   "metadata": {},
   "source": [
    "## Dot product"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "2e04ea53-29a8-4c81-abf1-732f7119a38f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(5)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.einsum('j,j->',[b,b])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bc4356d-a32d-4e33-bbf1-5244cccebe89",
   "metadata": {},
   "source": [
    "## softmax($w^T M_t$)\n",
    "\n",
    "It is not obvious from reading this single line below what probs shape is. One has to rely either on comments or look at the input tensor dimensions, determine what the output shape is, and then remember it. Instead with einsum, it is clear the distribution is across j output nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "f69e917e-74c7-482b-8e08-bc79f8747dc5",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/q5/_q4871x926x75g_ycdd6p9k80000gn/T/ipykernel_39324/3521217325.py:3: UserWarning: The use of `x.T` on tensors of dimension other than 2 to reverse their shape is deprecated and it will throw an error in a future release. Consider `x.mT` to transpose batches of matrices or `x.permute(*torch.arange(x.ndim - 1, -1, -1))` to reverse the dimensions of a tensor. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/native/TensorShape.cpp:3679.)\n",
      "  probs = F.softmax(w.T.matmul(M_t), dim=0)\n"
     ]
    }
   ],
   "source": [
    "w = torch.randn(5)\n",
    "M_t = torch.randn(5, 7)\n",
    "probs = F.softmax(w.T.matmul(M_t), dim=0) \n",
    "\n",
    "#einsum implementation\n",
    "probs_e = F.softmax(torch.einsum(\"i,ij->j\",[w,M_t]), dim=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73936b93-d5d0-417e-9997-7fbeaff9d401",
   "metadata": {},
   "source": [
    "## $W h$ -  matrix vector multiplication for a batch of vectors\n",
    "\n",
    "Caveat: The goal here is to show equivalence of outputs from einsum, Linear and matmul. In practice, one would rather use efficient library implementations like Linear, especially when used as part of a larger network. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "cb1eeb51-21c5-4e96-bc51-ec297a1e19ab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([5, 3])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = 2\n",
    "ip_dim = 3\n",
    "op_dim = 5\n",
    "\n",
    "h = torch.randn(batch_size, ip_dim) #a batch of 3-dimensional vectors\n",
    "model = torch.nn.Linear(ip_dim, op_dim, bias = False)\n",
    "w = model.weight\n",
    "w.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "8dfe5df4-b3ba-4f9e-8166-2fad1a0f6efa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.2521, -0.3848,  0.0324, -0.7824, -0.0392],\n",
       "        [-0.1701, -0.0202, -0.1171, -0.0762, -0.3855]], grad_fn=<MmBackward0>)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_output = model(h)\n",
    "batch_output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "2b41cfd0-620e-44a4-899e-61807802eb9f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.2521, -0.3848,  0.0324, -0.7824, -0.0392],\n",
       "        [-0.1701, -0.0202, -0.1171, -0.0762, -0.3855]], grad_fn=<MmBackward0>)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.matmul(h, w.T)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "31f1b7cc-c0dd-4c7a-9626-2bb42ca16f75",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.2521, -0.3848,  0.0324, -0.7824, -0.0392],\n",
       "        [-0.1701, -0.0202, -0.1171, -0.0762, -0.3855]],\n",
       "       grad_fn=<ViewBackward0>)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.einsum(\"ij,jk->ik\", [h, w.T])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4216bb43",
   "metadata": {},
   "source": [
    "# Self-attention using einsum\n",
    "\n",
    "In the rest of this post, I implement self-attention as one would traditionally using transpose, matmul vs using einsum. For clarity, I focus on the fundamental operations and leave out masking as in decoder setup. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4128a79-b4ed-43b8-b149-fe0c812c0ab9",
   "metadata": {},
   "source": [
    "## Linear projection for a batch of a sequence of vectors\n",
    "\n",
    "Taking the above one step further. What if each \"example\" in our dataset is a sequence of items, and each item is a vector. e.g., In transformer-based NLP models, a sentence is a sequence of tokens (roughly) corresponding to subwords. Each token has a learned embedding. In self-attention mechanism, we first do a linear projection of the sequence of query, key and value token embeddings using $W_q$, $W_k$ and $W_v$ respectively. In this setting, ip_dim is the embedding dimension, and op_dim the head dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "5618684b-070b-4d26-8ff3-6f92fc48a704",
   "metadata": {},
   "outputs": [],
   "source": [
    "seq_length = 6\n",
    "sequence_data = torch.randn(batch_size, seq_length, ip_dim)\n",
    "model(sequence_data);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "c0c6f79e-7407-4619-ad24-1a51eceda1ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "Q = torch.einsum('ilj,jk->ilk',[sequence_data, w.T]) #output shape [batch size, seq length, op_dim] i.e., [2, 6, 5]\n",
    "Q;"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6a17de6-d1de-42fc-a2b4-d70b44f17215",
   "metadata": {},
   "source": [
    "## Now consider the self-attention mechanism across multiple attention heads\n",
    "\n",
    "We want to process the same sequence with multiple attention heads, each first doing a linear projection with corresponding weights. But why do n_head multiplications, when we could do one giant multiplication and use GPUs more efficiently? So, we do a _single_ linear projection into n_head*head_dim, i.e. $4*5=20$. Then, we view results spit across n_heads (4). This allows us to compute scores using softmax independently for each head."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "8acc17a7-c1e0-4ebe-a9d9-7a03bbc61695",
   "metadata": {},
   "outputs": [],
   "source": [
    "n_heads= 4\n",
    "head_dim = op_dim\n",
    "wq = torch.nn.Linear(ip_dim, n_heads*op_dim, bias = False)\n",
    "\n",
    "wk = torch.nn.Linear(ip_dim, n_heads*op_dim, bias = False)\n",
    "wv = torch.nn.Linear(ip_dim, n_heads*op_dim, bias = False)\n",
    "wo = torch.nn.Linear(n_heads*head_dim, ip_dim, bias = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "93118677-3c71-49d8-b2fb-b95398045be8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 6, 20])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xq = wq(sequence_data)\n",
    "xq.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "4a99adb5-0b62-4a7d-8349-40e191db27af",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 6, 4, 5])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xq = xq.view(batch_size, seq_length, n_heads, head_dim)\n",
    "xq.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5b1b8c3-34ae-4aa6-899a-6a35b763fbf8",
   "metadata": {},
   "source": [
    "Similarly, compute keys and values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "71dcd965-5a2f-4ae9-bc97-0ecb87c08b30",
   "metadata": {},
   "outputs": [],
   "source": [
    "keys = wk(sequence_data)\n",
    "values = wv(sequence_data)\n",
    "\n",
    "keys = keys.view(batch_size, seq_length, n_heads, head_dim)\n",
    "values = values.view(batch_size, seq_length, n_heads, head_dim)\n",
    "\n",
    "# Rearrange tensors so we compute one set of scores per attention-head\n",
    "xq = xq.transpose(1, 2)  #(bs, n_heads, seqlen, head_dim)\n",
    "keys = keys.transpose(1, 2)  #(bs, n_heads, seqlen, head_dim)\n",
    "values = values.transpose(1, 2)  #(bs, n_heads, seqlen, head_dim)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "850a8e8f-51a0-4d21-886e-360986d3e3a3",
   "metadata": {},
   "source": [
    "Compute scores for all pairs of tokens in the sequence. For matmul between xq and keys to produce a seq_len*seq_len set of scores, we need to transpose keys so it has shape #(bs, n_heads, head_dim, seq_len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "0d578892-ac9a-496a-a604-795cc2e1fae8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 4, 6, 6])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = torch.matmul(xq, keys.transpose(2, 3)) / math.sqrt(head_dim)\n",
    "scores = F.softmax(scores.float(), dim=-1).type_as(xq)\n",
    "scores.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa51995d-9664-46c4-9a67-c7b9ac0e499c",
   "metadata": {},
   "source": [
    "Compute attention-weighted vector of the input sequence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "c94c39b5-3bf1-4458-ac23-2bd6f03cfece",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 4, 6, 5])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output = torch.matmul(scores, values)  # (bs, n_heads, seqlen, head_dim)\n",
    "output.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "838e5727-3a94-4a1a-aa4d-8d1f9251ac1c",
   "metadata": {},
   "source": [
    "Concatenate outputs from multiple heads so we get an output tensor of shape (batch_size, seq_length, n_heads*head_dim) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "c5b14144-418b-426c-88f6-68a5683a9e8f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 6, 20])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output = output.transpose(1, 2).contiguous().view(batch_size, seq_length, -1)\n",
    "output.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "caeae9c1-1e35-44dc-80fe-aa8e40c4212f",
   "metadata": {},
   "source": [
    "And one final linear projection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "3c652637-f03a-41f0-b325-702da97e628d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 6, 3])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wo(output).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79a9917c-e002-4b71-947c-b1cdbcfacebf",
   "metadata": {},
   "source": [
    "## Einsum implementation of multi-head self-attention\n",
    "\n",
    "The index that is missing in the output string is the one that is being compressed or summed over. So note in output_e computation below bhlj, bhjd -> bhld. Here we want to compress/sum over j. Perhaps it's easier to use the ellipsis notation, which brings the focus to lj, jd and now we can think again in 2 dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "b7cd2dd5-c9af-4979-9999-777c915dd6f2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scores shape: torch.Size([2, 4, 6, 6])\n"
     ]
    }
   ],
   "source": [
    "xqq = wq(sequence_data).view(batch_size, seq_length, n_heads, head_dim)\n",
    "kk  = wk(sequence_data).view(batch_size, seq_length, n_heads, head_dim)\n",
    "vv  = wv(sequence_data).view(batch_size, seq_length, n_heads, head_dim)\n",
    "\n",
    "xqq = torch.einsum('blhd->bhld',[xqq])\n",
    "kk  = torch.einsum('blhd->bhld',[kk])\n",
    "vv  = torch.einsum('blhd->bhld', [vv])\n",
    "\n",
    "scores_e = torch.einsum(\"bhid, bhjd -> bhij\", [xqq, kk]) / math.sqrt(head_dim)\n",
    "scores_e = F.softmax(scores_e.float(), dim =-1).type_as(xq)\n",
    "\n",
    "print(f\"Scores shape: {scores_e.shape}\")\n",
    "output_e = torch.einsum(\"bhlj,bhjd->bhld\", scores_e, vv) \n",
    "output_e = torch.einsum(\"bhld->blhd\", [output_e]).contiguous().view(batch_size, seq_length, -1)\n",
    "output == output_e;"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f133da69-5a7c-44c6-b040-b9f853aae44e",
   "metadata": {},
   "source": [
    "It is easier to see the crux of the operation with the ellipsis notation. We ignore the batch and head dimensions, and see we are summing across the j dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "819d0d50-64eb-47bf-a48d-36512be1cde4",
   "metadata": {},
   "outputs": [],
   "source": [
    "output_ee = torch.einsum(\"...lj,...jd->...ld\", scores_e, vv) \n",
    "output_ee = torch.einsum(\"bhld->blhd\", [output_ee]).contiguous().view(batch_size, seq_length, -1)\n",
    "output == output_ee;"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94abb317-22b9-4bda-b78b-936b3da3aa6a",
   "metadata": {},
   "source": [
    "In conclusion, einsum is a nifty tool to implement and visualize almost any tensor operation. An even more versatile tool is einops, that I plan to cover in a subsequent post."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}