Apache SystemML.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 4\n",
    "## Understaning scaling of linear algebra operations on Apache Spark using Apache SystemML\n",
    "\n",
    "In this assignment we want you to understand how to scale linear algebra operations from a single machine to multiple machines, memory and CPU cores using Apache SystemML. Therefore we want you to understand how to migrate from a numpy program to a SystemML DML program. Don't worry. We will give you a lot of hints. Finally, you won't need this knowledge anyways if you are sticking to Keras only, but once you go beyond that point you'll be happy to see what's going on behind the scenes.\n",
    "\n",
    "So the first thing we need to ensure is that we are on the latest version of SystemML, which is 1.2.0:\n",
    "\n",
    "The steps are:\n",
    "- pip install\n",
    "- start execution at the cell with the version - check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Markdown, display\n",
    "def printmd(string):\n",
    "    display(Markdown('# <span style=\"color:red\">'+string+'</span>'))\n",
    "\n",
    "\n",
    "if ('sc' in locals() or 'sc' in globals()):\n",
    "    printmd('<<<<<!!!!! It seems that you are running in a IBM Watson Studio Apache Spark Notebook. Please run it in an IBM Watson Studio Default Runtime (without Apache Spark) !!!!!>>>>>')\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting pyspark==2.4.5\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/9a/5a/271c416c1c2185b6cb0151b29a91fff6fcaed80173c8584ff6d20e46b465/pyspark-2.4.5.tar.gz (217.8MB)\n",
      "\u001b[K     |████████████████████████████████| 217.8MB 146kB/s  eta 0:00:01    |███████                         | 48.3MB 36.0MB/s eta 0:00:05\n",
      "\u001b[?25hCollecting py4j==0.10.7 (from pyspark==2.4.5)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e3/53/c737818eb9a7dc32a7cd4f1396e787bd94200c3997c72c1dbe028587bd76/py4j-0.10.7-py2.py3-none-any.whl (197kB)\n",
      "\u001b[K     |████████████████████████████████| 204kB 33.8MB/s eta 0:00:01\n",
      "\u001b[?25hBuilding wheels for collected packages: pyspark\n",
      "  Building wheel for pyspark (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Stored in directory: /home/dsxuser/.cache/pip/wheels/bf/db/04/61d66a5939364e756eb1c1be4ec5bdce6e04047fc7929a3c3c\n",
      "Successfully built pyspark\n",
      "Installing collected packages: py4j, pyspark\n",
      "Successfully installed py4j-0.10.7 pyspark-2.4.5\n"
     ]
    }
   ],
   "source": [
    "!pip install pyspark==2.4.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting systemml\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/b1/94/62104cb8c526b462cd501c7319926fb81ac9a5668574a0b3407658a506ab/systemml-1.2.0.tar.gz (9.7MB)\n",
      "\u001b[K     |████████████████████████████████| 9.7MB 9.7MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: numpy>=1.8.2 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from systemml) (1.19.4)\n",
      "Requirement already satisfied: scipy>=0.15.1 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from systemml) (1.4.1)\n",
      "Requirement already satisfied: pandas in /opt/conda/envs/Python36/lib/python3.6/site-packages (from systemml) (0.24.1)\n",
      "Requirement already satisfied: scikit-learn in /opt/conda/envs/Python36/lib/python3.6/site-packages (from systemml) (0.20.3)\n",
      "Requirement already satisfied: Pillow>=2.0.0 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from systemml) (5.4.1)\n",
      "Requirement already satisfied: python-dateutil>=2.5.0 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from pandas->systemml) (2.7.5)\n",
      "Requirement already satisfied: pytz>=2011k in /opt/conda/envs/Python36/lib/python3.6/site-packages (from pandas->systemml) (2018.9)\n",
      "Requirement already satisfied: six>=1.5 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from python-dateutil>=2.5.0->pandas->systemml) (1.12.0)\n",
      "Building wheels for collected packages: systemml\n",
      "  Building wheel for systemml (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Stored in directory: /home/dsxuser/.cache/pip/wheels/cf/07/79/b3ed6f12afe06b2ab55d60dcfd62e66240f5d8c6088a518177\n",
      "Successfully built systemml\n",
      "Installing collected packages: systemml\n",
      "Successfully installed systemml-1.2.0\n"
     ]
    }
   ],
   "source": [
    "!pip install systemml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark import SparkContext, SparkConf\n",
    "from pyspark.sql import SQLContext, SparkSession\n",
    "from pyspark.sql.types import StructType, StructField, DoubleType, IntegerType, StringType\n",
    "sc = SparkContext.getOrCreate(SparkConf().setMaster(\"local[*]\"))\n",
    "from pyspark.sql import SparkSession\n",
    "spark = SparkSession \\\n",
    "    .builder \\\n",
    "    .getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p /home/dsxuser/work/systemml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.2.0\n"
     ]
    }
   ],
   "source": [
    "from systemml import MLContext, dml\n",
    "import numpy as np\n",
    "import time\n",
    "ml = MLContext(spark)\n",
    "ml.setConfigProperty(\"sysml.localtmpdir\", \"mkdir /home/dsxuser/work/systemml\")\n",
    "print(ml.version())\n",
    "    \n",
    "if not ml.version() == '1.2.0':\n",
    "    raise ValueError('please upgrade to SystemML 1.2.0, or restart your Kernel (Kernel->Restart & Clear Output)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations, if you see version 1.2.0, please continue with the notebook..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use an MLContext to interface with Apache SystemML. Note that we passed a SparkSession object as parameter so SystemML now knows how to talk to the Apache Spark cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we create some large random matrices to have numpy and SystemML crunch on it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "u = np.random.rand(1000,10000)\n",
    "s = np.random.rand(10000,1000)\n",
    "w = np.random.rand(1000,1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we implement a short one-liner to define a very simple linear algebra operation\n",
    "\n",
    "In case you are unfamiliar with matrxi-matrix multiplication: https://en.wikipedia.org/wiki/Matrix_multiplication\n",
    "\n",
    "sum(U' * (W . (U * S)))\n",
    "\n",
    "\n",
    "| Legend        |            |   \n",
    "| ------------- |-------------| \n",
    "| '      | transpose of a matrix | \n",
    "| * | matrix-matrix multiplication      |  \n",
    "| . | scalar multiplication      |   \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "27.17450475692749\n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "res = np.sum(u.T.dot(w * u.dot(s)))\n",
    "print (time.time()-start)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see this executes perfectly fine. Note that this is even a very efficient execution because numpy uses a C/C++ backend which is known for it's performance. But what happens if U, S or W get such big that the available main memory cannot cope with it? Let's give it a try:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#u = np.random.rand(10000,100000)\n",
    "#s = np.random.rand(100000,10000)\n",
    "#w = np.random.rand(10000,10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After a short while you should see a memory error. This is because the operating system process was not able to allocate enough memory for storing the numpy array on the heap. Now it's time to re-implement the very same operations as DML in SystemML, and this is your task. Just replace all ###your_code_goes_here sections with proper code, please consider the following table which contains all DML syntax you need:\n",
    "\n",
    "| Syntax        |            |   \n",
    "| ------------- |-------------| \n",
    "| t(M)      | transpose of a matrix, where M is the matrix | \n",
    "| %*% | matrix-matrix multiplication      |  \n",
    "| * | scalar multiplication      |   \n",
    "\n",
    "## Task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to show you the advantage of SystemML over numpy we've blown up the sizes of the matrices. Unfortunately, on a 1-2 worker Spark cluster it takes quite some time to complete. Therefore we've stripped down the example to smaller matrices below, but we've kept the code, just in case you are curious to check it out. But you might want to use some more workers which you easily can configure in the environment settings of the project within Watson Studio. Just be aware that you're currently limited to free 50 capacity unit hours per month wich are consumed by the additional workers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "script = \"\"\"\n",
    "U = rand(rows=1000,cols=10000)\n",
    "S = rand(rows=10000,cols=1000)\n",
    "W = rand(rows=1000,cols=1000)\n",
    "res = sum( t(U) %*% (W * (U %*% S)))\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get consistent results we switch from a random matrix initialization to something deterministic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ANTLR Tool version 4.5.3 used for code generation does not match the current runtime version 4.7ANTLR Runtime version 4.5.3 used for parser compilation does not match the current runtime version 4.7ANTLR Tool version 4.5.3 used for code generation does not match the current runtime version 4.7ANTLR Runtime version 4.5.3 used for parser compilation does not match the current runtime version 4.7\n",
      "SystemML Statistics:\n",
      "Total execution time:\t\t10.513 sec.\n",
      "Number of executed Spark inst:\t0.\n",
      "\n",
      "\n",
      "6252865381309.084\n"
     ]
    }
   ],
   "source": [
    "prog = dml(script).output('res')\n",
    "res = ml.execute(prog).get('res')\n",
    "print(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If everything runs fine you should get *6252492444241.075* as result (or something in that bullpark). Feel free to submit your DML script to the grader now!\n",
    "\n",
    "### Submission"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-11-21 06:02:34--  https://raw.githubusercontent.com/romeokienzler/developerWorks/master/coursera/ai/rklib.py\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 2289 (2.2K) [text/plain]\n",
      "Saving to: ‘rklib.py’\n",
      "\n",
      "100%[======================================>] 2,289       --.-K/s   in 0s      \n",
      "\n",
      "2020-11-21 06:02:34 (21.2 MB/s) - ‘rklib.py’ saved [2289/2289]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!rm -f rklib.py\n",
    "!wget https://raw.githubusercontent.com/romeokienzler/developerWorks/master/coursera/ai/rklib.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from rklib import submit\n",
    "key = \"esRk7vn-Eeej-BLTuYzd0g\"\n",
    "\n",
    "\n",
    "email = \"turki32235@gmail.com\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submission successful, please check on the coursera grader page for the status\n",
      "-------------------------\n",
      "{\"elements\":[{\"itemId\":\"P1p3F\",\"id\":\"tE4j0qhMEeecqgpT6QjMdA~P1p3F~RAg6tiu_EeuCNxIlakc3BQ\",\"courseId\":\"tE4j0qhMEeecqgpT6QjMdA\"}],\"paging\":{},\"linked\":{}}\n",
      "-------------------------\n"
     ]
    }
   ],
   "source": [
    "part = \"fUxc8\"\n",
    "token = \"nQNm0YAQHWcn1nId\" #you can obtain it from the grader page on Coursera (have a look here if you need more information on how to obtain the token https://youtu.be/GcDo0Rwe06U?t=276)\n",
    "submit(email, token, key, part, [part], script)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}