Resume_scraping_hh_final_code-Compact-Part1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1\n",
    "\n",
    "This part of the program collects unique resume identification numbers from the online job portal HeadHunter.ru (hh.ru).  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize log file (only once)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def init_log_writer(log_path, col_names): \n",
    "    ''' Initialize log file.\n",
    "    Arguments:\n",
    "        log_path (str) -- Relative or absolute path to log file.\n",
    "        col_names (list) -- Column names to be exported with data.\n",
    "    \n",
    "    Returns: io Buffered Writer to tab-separated log file with column names.\n",
    "    '''\n",
    "    with open(log_path, 'wb') as log_writer:\n",
    "        header = \"\\t\".join(col_names) + '\\n'\n",
    "        log_writer.write(header.encode('utf-8'))\n",
    "    return log_writer\n",
    "\n",
    "# Initialize log file before running the main program. \n",
    "# Once log file is initialized, no need to repeat this code.\n",
    "log_path = resumes_path + \"resumes_log.txt\"\n",
    "col_names = [\"Request time\", \"Unexpected error\", \"Applicants added\"]\n",
    "log_file = init_log_writer(log_path, col_names) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a new directory for a given day\n",
    "\n",
    "Every directory contains csv files that were created during the same day. The name of a directory corresponds to date when these files were created.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import datetime as dt\n",
    "\n",
    "def directory(resumes_directory):\n",
    "    ''' Create a directory for a given day.\n",
    "    Argument:\n",
    "        resumes_directory (str) -- Relative or absolute path to a given directory.\n",
    "        \n",
    "    Returns: directory with a given name.  \n",
    "    '''\n",
    "    if not os.path.exists(resumes_directory):\n",
    "        os.makedirs(resumes_directory)\n",
    "    return resumes_directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Write information to log file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def log_writer(log_path, log_row):\n",
    "    '''Write information to log file.\n",
    "    Argument:\n",
    "        log_path(str) -- Relative or absolute path to log file.\n",
    "        log_row(list) -- Data to be added to log file.\n",
    "        \n",
    "    Returns: io Buffered Writer to tab-separated log file with relevant information.\n",
    "    '''\n",
    "    with open(log_path, 'ab') as log_file:\n",
    "        row = \"\\t\".join(log_row) + '\\n'\n",
    "        log_file.write(row.encode('utf-8'))\n",
    "    return log_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Write data to file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def data_writer(data_path, data_row):\n",
    "    '''Write data to file.\n",
    "    Arguments:\n",
    "        data_path (str) -- Relative or absolute path to data file.\n",
    "        data_row (list) -- Data to be added to data file.\n",
    "        \n",
    "    Returns: io Buffered Writer to data file.\n",
    "    '''\n",
    "    with open(data_path, \"wb\") as toWrite:\n",
    "        for item in data_row:\n",
    "            row = \",\".join(item) + '\\n'\n",
    "            toWrite.write(row.encode('utf-8'))\n",
    "    return toWrite"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scraping data from the website hh.ru\n",
    "\n",
    "The main goal of this program is to collect unique resume numbers for all the job seekers who have recently posted their resumes on the website. Every applicant may post more than one resume. All resume numbers from the same person are written on the same raw in csv data file.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scraping is finished!\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "from time import sleep\n",
    "import datetime as dt\n",
    "import sys\n",
    "import csv\n",
    "\n",
    "resumes_path = \"hh_data/hh_resumes/\" \n",
    "now=dt.datetime.today()\n",
    "final_time = now + dt.timedelta(days=7) # time to stop scraping\n",
    "\n",
    "while now < final_time: # scrape until final_time \n",
    "    \n",
    "    date_prefix = dt.datetime.now().strftime('%Y-%m-%d--%H-%M-%S') # exact time of scraping\n",
    "    log_row = [date_prefix] # the first element to be added to a log file\n",
    "    \n",
    "    # A new directory is created for every day of scraping:\n",
    "    date_prefix_dir = dt.datetime.now().strftime('%Y-%m-%d')\n",
    "    resumes_directory = resumes_path + date_prefix_dir\n",
    "    directory(resumes_directory) # create a directory if it did not exist before\n",
    "    \n",
    "    # URL to search resumes with specific parameters: \n",
    "    # age: from 16 to 60; \n",
    "    # area: Russia; citizenship: Russian; \n",
    "    # salary: find resumes that state desired salary; \n",
    "    # type of employment: full-time. \n",
    "    url = 'https://hh.ru/search/resume?age_to=60&items_on_page=100&order_by=publication_time& \\\n",
    "    citizenship=113&area=113&text=&pos=full_text&label=only_with_salary&label=only_with_age& \\\n",
    "    exp_period=all_time&logic=normal&clusters=true&age_from=16&employment=full&page='\n",
    "\n",
    "    resumes_list = [] # list of all resumes' id numbers for all applicants \n",
    "\n",
    "    # Collect resume numbers for every applicant using specific search url:  \n",
    "    last_page = 50 # checking resumes on 50 pages\n",
    "    for i in range(0,last_page):\n",
    "        url_page = url + str(i) # url for every page\n",
    "        headers = {'User-Agent': 'career-success (olga.boldareva@gmail.com)'} \n",
    "        try:\n",
    "            response = requests.get(url_page, headers=headers)\n",
    "            if i == (last_page-1):\n",
    "                log_row.append('No errors') # to be added to a log file if no errors occured \n",
    "        except:\n",
    "            print (\"Unexpected error:\", sys.exc_info()[0])\n",
    "            log_row.append(sys.exc_info()[0]) # error code to be added to a log file\n",
    "            break\n",
    "    \n",
    "        if response.status_code == 200: # 200 sends the right response \n",
    "            soup = BeautifulSoup(response.text, 'html.parser')\n",
    "            # Every class identifies a unique applicant with one or more resumes available. \n",
    "            resumes_summary = soup.find_all('td', class_='output__main-cell')\n",
    "            for item in resumes_summary:\n",
    "                res_one_appl = [] #list of all resumes from one applicant\n",
    "                res_temp=item.find_all('a')\n",
    "                for resume in res_temp:\n",
    "                    res_one_appl.append(resume.attrs[u'href'][8:]) # adding a resume number \n",
    "                # Adding all resumes (sorted by name) from one applicant:\n",
    "                resumes_list.append(sorted(res_one_appl))  \n",
    "        else:\n",
    "            print (response.status_code)\n",
    "    log_row.append(str(len(resumes_list))) # number of scraped resumes to be added to a log file\n",
    "    \n",
    "    data_path = resumes_directory + \"/resumes_\" + date_prefix +\".csv\"\n",
    "    data_writer(data_path, resumes_list) # write data to csv file\n",
    "\n",
    "    log_path = resumes_path + \"resumes_log.txt\"\n",
    "    log_writer(log_path, log_row) # save information to log file \n",
    "    \n",
    "    sleep(7200) # 2 hours to wait until the next request\n",
    "    now=dt.datetime.today()\n",
    "    \n",
    "else:  \n",
    "    print (\"Scraping is finished!\")       "
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:python3]",
   "language": "python",
   "name": "conda-env-python3-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}