Assignment 2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Warmup Assignment 2\n",
    "Below you see some ApacheSpark code written in Python. You don't have to change code now, the only thing we want you to do is to make sure that you have a proper Apache Spark Notebook environment available for this course\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is designed to run in a IBM Watson Studio default runtime (NOT the Watson Studio Apache Spark Runtime as the default runtime with 1 vCPU is free of charge). Therefore, we install Apache Spark in local mode for test purposes only. Please don't use it in production.\n",
    "\n",
    "In case you are facing issues, please read the following two documents first:\n",
    "\n",
    "https://github.com/IBM/skillsnetwork/wiki/Environment-Setup\n",
    "\n",
    "https://github.com/IBM/skillsnetwork/wiki/FAQ\n",
    "\n",
    "Then, please feel free to ask:\n",
    "\n",
    "https://coursera.org/learn/machine-learning-big-data-apache-spark/discussions/all\n",
    "\n",
    "Please make sure to follow the guidelines before asking a question:\n",
    "\n",
    "https://github.com/IBM/skillsnetwork/wiki/FAQ#im-feeling-lost-and-confused-please-help-me\n",
    "\n",
    "\n",
    "If running outside Watson Studio, this should work as well. In case you are running in an Apache Spark context outside Watson Studio, please remove the Apache Spark setup in the first notebook cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "# <span style=\"color:red\"><<<<<!!!!! It seems that you are running in a IBM Watson Studio Apache Spark Notebook. Please run it in an IBM Watson Studio Default Runtime (without Apache Spark) !!!!!>>>>></span>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import Markdown, display\n",
    "def printmd(string):\n",
    "    display(Markdown('# <span style=\"color:red\">'+string+'</span>'))\n",
    "\n",
    "\n",
    "if ('sc' in locals() or 'sc' in globals()):\n",
    "    printmd('<<<<<!!!!! It seems that you are running in a IBM Watson Studio Apache Spark Notebook. Please run it in an IBM Watson Studio Default Runtime (without Apache Spark) !!!!!>>>>>')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting pyspark==3.3.0\n",
      "  Downloading pyspark-3.3.0.tar.gz (281.3 MB)\n",
      "\u001b[K     |████████████████████████████████| 281.3 MB 55 kB/s s eta 0:00:01MB 10.6 MB/s eta 0:00:24         | 47.3 MB 10.6 MB/s eta 0:00:23         | 51.2 MB 10.6 MB/s eta 0:00:22     |█████████████                   | 113.8 MB 96.8 MB/s eta 0:00:02�█████████▍                 | 126.6 MB 96.8 MB/s eta 0:00:02�█████████▉                 | 130.2 MB 62.5 MB/s eta 0:00:03████████████▌         | 197.7 MB 84.1 MB/s eta 0:00:01     |███████████████████████▎        | 204.6 MB 84.1 MB/s eta 0:00:01�██████████▍        | 206.0 MB 84.1 MB/s eta 0:00:01�██████████▊        | 208.7 MB 84.1 MB/s eta 0:00:01     |█████████████████████████       | 219.8 MB 84.1 MB/s eta 0:00:01��███████▏      | 221.6 MB 84.1 MB/s eta 0:00:01MB/s eta 0:00:01��█████████████▍    | 241.1 MB 43.5 MB/s eta 0:00:01��█████████████▋    | 243.1 MB 43.5 MB/s eta 0:00:01��███████████▌| 276.8 MB 71.2 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: py4j==0.10.9.5 in /opt/ibm/conda/miniconda3.9/lib/python3.9/site-packages (from pyspark==3.3.0) (0.10.9.5)\n",
      "Building wheels for collected packages: pyspark\n",
      "  Building wheel for pyspark (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for pyspark: filename=pyspark-3.3.0-py2.py3-none-any.whl size=281764027 sha256=75271762a28e2a23b9fb8f5cad467ea48d893a61aa147cc59753d4be944f1609\n",
      "  Stored in directory: /home/spark/shared/.cache/pip/wheels/05/75/73/81f84d174299abca38dd6a06a5b98b08ae25fce50ab8986fa1\n",
      "Successfully built pyspark\n",
      "Installing collected packages: pyspark\n",
      "  Attempting uninstall: pyspark\n",
      "    Found existing installation: pyspark 3.3.1\n",
      "    Uninstalling pyspark-3.3.1:\n",
      "\u001b[31mERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/opt/ibm/conda/miniconda3.9/bin/__pycache__/find_spark_home.cpython-39.pyc'\n",
      "Consider using the `--user` option or check the permissions.\n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install pyspark==3.3.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    from pyspark import SparkContext, SparkConf\n",
    "    from pyspark.sql import SparkSession\n",
    "except ImportError as e:\n",
    "    printmd('<<<<<!!!!! Please restart your kernel after installing Apache Spark !!!!!>>>>>')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc = SparkContext.getOrCreate(SparkConf().setMaster(\"local[*]\"))\n",
    "\n",
    "spark = SparkSession \\\n",
    "    .builder \\\n",
    "    .getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def assignment1(sc):\n",
    "    rdd = sc.parallelize(list(range(100)))\n",
    "    return rdd.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100\n"
     ]
    }
   ],
   "source": [
    "print(assignment1(sc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2023-02-03 16:15:01--  https://raw.githubusercontent.com/IBM/coursera/master/rklib.py\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.109.133, 185.199.108.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 2540 (2.5K) [text/plain]\n",
      "Saving to: ‘rklib.py’\n",
      "\n",
      "rklib.py            100%[===================>]   2.48K  --.-KB/s    in 0s      \n",
      "\n",
      "2023-02-03 16:15:01 (18.6 MB/s) - ‘rklib.py’ saved [2540/2540]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!rm -f rklib.py\n",
    "!wget https://raw.githubusercontent.com/IBM/coursera/master/rklib.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please provide your email address and obtain a submission token on the grader’s submission page in coursera, then execute the cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submission successful, please check on the coursera grader page for the status\n",
      "-------------------------\n",
      "{\"elements\":[{\"itemId\":\"D277m\",\"id\":\"sUpST4RAEeawAApvKZgcCQ~D277m~6fm4J6PdEe2Y5goOxW8pcQ\",\"courseId\":\"sUpST4RAEeawAApvKZgcCQ\"}],\"paging\":{},\"linked\":{}}\n",
      "-------------------------\n"
     ]
    }
   ],
   "source": [
    "from rklib import submit\n",
    "import json\n",
    "\n",
    "key = \"R1eDmiHNEei9kxIYdin0mA\"\n",
    "part = \"fnFg7\"\n",
    "email = \"mdanasmondol43@gmail.com\"\n",
    "token = \"JllWARGl3MkqZnLP\" #you can obtain it from the grader page on Coursera (have a look here if you need more information https://youtu.be/GcDo0Rwe06U?t=276)\n",
    "\n",
    "\n",
    "submit(email, token, key, part, [part], json.dumps(assignment1(sc)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9 with Spark",
   "language": "python3",
   "name": "python39"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}