Matrices.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyspark\n",
    "\n",
    "try: \n",
    "    type(sc)\n",
    "except NameError:\n",
    "    sc = pyspark.SparkContext('local[*]')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matrices\n",
    "\n",
    "### Representacion\n",
    "\n",
    "Vamos a estar operando con matrices dispersas con la siguiente representacion en un archivo distribuido donde cada registro es de la forma: (fila,columna,valor).\n",
    "\n",
    "Por lo tanto este tipo de representacion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = [\n",
    "(1,2,4),\n",
    "(1,5,3),\n",
    "(2,1,3),\n",
    "(3,2,2),\n",
    "(4,4,-1),\n",
    "(5,1,1),\n",
    "(5,5,2)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1, 2, 4), (1, 5, 3), (2, 1, 3), (3, 2, 2), (4, 4, -1), (5, 1, 1), (5, 5, 2)]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Representa la siguiente matriz dispersa\n",
    "\n",
    "```\n",
    "    0  4  0  0  3\n",
    "    3  0  0  0  0\n",
    "    0  2  0  0  0\n",
    "    0  0  0  -1 0\n",
    "    1  0  0  0  2\n",
    "```\n",
    "\n",
    "Notar que **la representacion que hemos hecho esta indexada de 1 en adelante en vez de 0, como se hace en la mayoria de los lenguajes de programacion. Esto es algo a tener en cuenta en las subsiguientes operaciones**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1, 2, 4), (1, 5, 3), (2, 1, 3), (3, 2, 2), (4, 4, -1), (5, 1, 1), (5, 5, 2)]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matrixRDD = sc.parallelize(data,8);\n",
    "matrixRDD.take(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiplicacion matriz vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Buscamos realizar la siguiente operacion:\n",
    "    \n",
    "Representa la siguiente matriz dispersa\n",
    "\n",
    "```\n",
    "    0  4  0  0  3     1\n",
    "    3  0  0  0  0     2\n",
    "    0  2  0  0  0  *  3 \n",
    "    0  0  0  -1 0     4\n",
    "    1  0  0  0  2     5\n",
    "```\n",
    "\n",
    "Para poder realizar la operacion con las matrices deberia multiplicar cada fila de la matriz por el vector.\n",
    "En el caso de la matriz dispersa esto es equivalente a **multiplicar cada elemento de la matriz dispersa por el elemento que le corresponde en el vector y acumular por filas.**\n",
    "\n",
    "Para poder realizar la operacion tenemos que llevar la matriz a una representacion de (fila,columna,valor) en (fila,(columna,valor))\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#definimos el vector que estaremos multiplicando\n",
    "vector = [1, 2, 3, 4, 5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrixPerRowRDD = \\\n",
    "  matrixRDD.map(lambda x: (x[0],(x[1],x[2])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1, (2, 4)),\n",
       " (1, (5, 3)),\n",
       " (2, (1, 3)),\n",
       " (3, (2, 2)),\n",
       " (4, (4, -1)),\n",
       " (5, (1, 1)),\n",
       " (5, (5, 2))]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matrixPerRowRDD.take(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Una vez que la llevamos al formato que necesitamos podemos multiplicar cada elemento de la matriz por el elemento del vector que le corresponde.\n",
    "\n",
    "**Cada elemento de la matriz nos indica el numero de columna (y ese numero de columna menos 1 nos indica el indice del vector por el que tenemos que multiplicar).**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "partialResultRDD = \\\n",
    "  matrixPerRowRDD.map(\n",
    "        lambda x: (x[0], vector[x[1][0]-1] * x[1][1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1, 8), (1, 15), (2, 3), (3, 4), (4, -4), (5, 1), (5, 10)]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "partialResultRDD.take(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "todos los valores parciales por fila que tenemos que agregar fila para obtener el total "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1, 23), (2, 3), (3, 4), (4, -4), (5, 11)]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resultPerRow = partialResultRDD\\\n",
    "   .reduceByKey(lambda x,y: x+y)\n",
    "resultPerRow.take(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Notese que el resultado esta expresado como (fila, valor) para llevarlo a una representacion de vector podemos hacer lo siguiente."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[23, 3, 4, -4, 11]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = resultPerRow.map(lambda x: x[1])\n",
    "result.take(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiplicacion de Matrices\n",
    "\n",
    "Suponiendo que las dimensiones son compatibles, haremos la multiplicacion de dos matrices dispersas definidas nuevamente como (fila, columna, valor)\n",
    "\n",
    "\n",
    "```\n",
    "  1  2  x  5  6  =  19  22\n",
    "  3  4     7  8     43  50\n",
    "```\n",
    "\n",
    "Tenemos que notar que en el caso de las matrices dispersas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# matriz 1\n",
    "m1 = [(1,1,1),\n",
    "(1,2,2),\n",
    "(2,1,3),\n",
    "(2,2,4)]\n",
    "\n",
    "# matriz 2\n",
    "m2 = [(1,1,5),\n",
    "(1,2,6),\n",
    "(2,1,7),\n",
    "(2,2,8)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1, 1, 1), (1, 2, 2), (2, 1, 3), (2, 2, 4)]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1, 1, 5), (1, 2, 6), (2, 1, 7), (2, 2, 8)]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "m1RDD = sc.parallelize(m1,8)\n",
    "m2RDD = sc.parallelize(m2,8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Para realizar el producto entre dos matrices debemos notar que **los elementos de la primer columna de la matriz 1 (1 y 3) siempre se multiplican unicamente por los elementos de la primera fila de la matriz 2 (5 y 6) por lo tanto la estrategia es realizar un join en donde la columna de la matriz 1 coincida con la fila de la matriz 2.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1, (1, 1)), (2, (1, 2)), (1, (2, 3)), (2, (2, 4))]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# llevamos a matriz 1 a una representacion \n",
    "# por columna\n",
    "\n",
    "r1 = m1RDD.map(lambda x: (x[1],(x[0],x[2])))\n",
    "r1.take(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# llevamos a matriz 2 a una representacion \n",
    "# por fila"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1, (1, 5)), (1, (2, 6)), (2, (1, 7)), (2, (2, 8))]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r2 = m2RDD.map(lambda x: (x[0],(x[1],x[2])))\n",
    "r2.take(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "luego usando join juntamos los datos que necesitamos procesar en conjunto."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1, ((1, 1), (1, 5))),\n",
       " (1, ((1, 1), (2, 6))),\n",
       " (1, ((2, 3), (1, 5))),\n",
       " (1, ((2, 3), (2, 6))),\n",
       " (2, ((1, 2), (1, 7))),\n",
       " (2, ((1, 2), (2, 8))),\n",
       " (2, ((2, 4), (1, 7))),\n",
       " (2, ((2, 4), (2, 8)))]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rj = r1.join(r2)\n",
    "rj.take(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tenemos que multiplicar los valores y el resultado hay que acumularlo en el numero de fila y columna indicado en los registros. Por ejemplo para el registro ``(2, ((2, 4), (1, 7)))`` tenemos que multiplicar ``4 * 7`` y ese resultado acumularlo para la fila 2, columna 1 obteniendo ``((2,1), 28)``.\n",
    "\n",
    "La idea entonces es acumular por fila y columan para luego acumular usando un reduce para obtener el resultado final"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[((1, 1), 5),\n",
       " ((1, 2), 6),\n",
       " ((2, 1), 15),\n",
       " ((2, 2), 18),\n",
       " ((1, 1), 14),\n",
       " ((1, 2), 16),\n",
       " ((2, 1), 28),\n",
       " ((2, 2), 32)]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rj2 = rj.map(lambda x:((x[1][0][0], x[1][1][0]), x[1][0][1] * x[1][1][1]))\n",
    "rj2.take(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[((2, 2), 50), ((1, 2), 22), ((1, 1), 19), ((2, 1), 43)]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = rj2.reduceByKey(lambda x,y: x+y)\n",
    "result.take(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tener en cuenta que el realizar la operacion los resultados quedan en la representacion **((fila, columna), valor)**. Si quisieramos llevarlo a la representacion original de matriz dispera podriamos hacer la siguiente transformacion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "final = result.map(lambda x: (x[0][0], x[0][1], x[1]))\n",
    "final.take(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15rc1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}