chapter_02_GettingStarted.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 2: Getting Started with Spark\n",
    "April 4, 2022   \n",
    "Simple Python examples of chapter 2 in the book *Learning Spark*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import required python spark libraries\n",
    "import findspark\n",
    "import pyspark\n",
    "from pyspark.sql import SparkSession\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Connect Jupyter Notebook with the Spark application and create Spark Context\n",
    "findspark.init()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#create a SparkSession\n",
    "spark = (SparkSession\n",
    "       .builder\n",
    "       .appName(\"chapter_02\")\n",
    "       .getOrCreate())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reading text from a file into a Spark DataFrame text:\n",
    "text = spark.read.text (\"README.md\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|value                                                                                                                                                                                                                                                                            |\n",
      "+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|# Spark                                                                                                                                                                                                                                                                          |\n",
      "|This is my repository of my Spark experiments while reading the book *Learning Spark* by Jules S. Damji, Brooke Wenig, Tathagata Das and Denny Lee ([available as pdf from Databricks](https://pages.databricks.com/rs/094-YMS-629/images/LearningSpark2.0.pdf)).                |\n",
      "|                                                                                                                                                                                                                                                                                 |\n",
      "|I only run the experiments in Python. The book contains code snippets in Python, Scala and Java. In the book, not all snippets and programs are available in all languages. I will try to  write all snippets in executable and tested Python code (mostly in Jupyter Notebooks).|\n",
      "|                                                                                                                                                                                                                                                                                 |\n",
      "|This repository also contains the end-to-end examples from the book's github repository, which are originally saved in a Databricks Notebook (.dbc file). I have converted the Python files into Jupyter Notebooks and saved them in the folder `BookExamples`.                  |\n",
      "|                                                                                                                                                                                                                                                                                 |\n",
      "|## Status                                                                                                                                                                                                                                                                        |\n",
      "|                                                                                                                                                                                                                                                                                 |\n",
      "|Chapter 2 is done. I am currently working on chapter 3.                                                                                                                                                                                                                          |\n",
      "+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Showing the content of the DataFrame \"text\":\n",
    "#    - showing ony the first 10 lines\n",
    "#    - do not truncate the lines, show them in their full length\n",
    "\n",
    "text.show(10, truncate = False)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}