11-intro-to-hadoop-04.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <center> Introduction to Hadoop MapReduce </center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Integrating Hadoop job into Palmetto workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With `hdp` module, access to Cypress Hadoop cluster can now be invoked within a Palmetto PBS script, allowing the integration of large-data processing components into a standard HPC workflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%writefile movieAnalyzer.pbs\n",
    "#!/bin/bash\n",
    "\n",
    "#PBS -N movieData\n",
    "#PBS -l select=1:ncpus=8:mem=8gb\n",
    "#PBS -l walltime=00:15:00\n",
    "#PBS -j oe\n",
    "\n",
    "# load hdp module and initilalize Keberos tokens\n",
    "module load hdp/0.1\n",
    "cypress-kinit\n",
    "klist\n",
    "\n",
    "# cd into directory containing the PBS script\n",
    "cd $PBS_O_WORKDIR\n",
    "\n",
    "# attempt to remove output directory\n",
    "hdfs dfs -rm -r intro-to-hadoop/output-movielens-03\n",
    "\n",
    "# submit Hadoop job to Cypress\n",
    "yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar \\\n",
    "    -input /repository/movielens/ratings.csv \\\n",
    "    -output intro-to-hadoop/output-movielens-03 \\\n",
    "    -file ./codes/avgRatingMapper02.py \\\n",
    "    -mapper avgRatingMapper02.py \\\n",
    "    -file ./codes/avgRatingReducer02.py \\\n",
    "    -reducer avgRatingReducer02.py \\\n",
    "    -file ./movielens/movies.csv\n",
    "\n",
    "# export output data back to Palmetto for further analysis\n",
    "hdfs dfs -get intro-to-hadoop/output-movielens-03/part-00000 ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "source": [
    "Open a terminal, ssh to login001 (DUO required), and submit this script\n",
    "\n",
    "```\n",
    "ssh login001\n",
    "cd ~/intro-to-hadoop-python\n",
    "qsub movieAnalyzer.pbs\n",
    "```\n",
    "\n",
    "View the final output when the job is finished"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!qstat -anu $USER"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!cat part-00000 2>/dev/null | head -n 20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## <center> Final Cleanup </center>\n",
    "\n",
    "Executing the cell below will clean up all HDFS output directories created as a result of previous MapReduce programs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!hdfs dfs -ls intro-to-hadoop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!hdfs dfs -rm -r intro-to-hadoop/\n",
    "!rm -Rf codes/\n",
    "!rm -Rf movielens/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2 (Anaconda)",
   "language": "python",
   "name": "anaconda_py2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}