Part_1_Amazon_Web_Scrapping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9dcca774",
   "metadata": {},
   "source": [
    "# Amazon Best Selling Books - Data Exploration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b57d9537",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "Amazon is one of the biggest e-commerce websites in the world that started off as from a garage in Seattle. But under Bezos' careful management, the company has thrived and has expanded from selling only books to being a store where you buy almost everything, both physical and digital and it's still the largest online store for all types of books.\n",
    "\n",
    "Amazon began operations in July 1995, advertising itself as \"Earth's Biggest Bookstore\" and leveraging significant book distributors and wholesalers to quickly fill its orders.\n",
    "\n",
    "# Aim of the Project\n",
    "\n",
    "The aim of this project is to analyze the best selling books on Amazon between 2009 and 2019. The dataset for this project was scrapped from the Amazon website and the insights we are looking forward to derive from this analysis are?\n",
    "\n",
    " - Distribution of Genre for the Unique Books - How many Fiction and Non Fiction Books were Best Sellers on Amazon between 2009 and 2019?\n",
    " \n",
    "\n",
    " - Distribution of Genre for the Unique Years - How many Fiction and Non Fiction Books were sold in each of the years?\n",
    " \n",
    "\n",
    " -  Top 10 Authors - Based on the number of times their names appeared in the best selling listings from 2009 to 2019\n",
    " \n",
    "\n",
    " - The Top 10 Books - Based on the number of times the book appeared as a best selling book from 2009 to 2019\n",
    "\n",
    "\n",
    " - Relationship between Reviews and Price - do low priced books get a better review and vice versa?\n",
    "\n",
    "\n",
    " - Genre performance based on reviews - Overall did Fiction books had more ratings than Non Fiction books?\n",
    "\n",
    "\n",
    " - Relationship between price and Year - Were the cost of the books increasing or decreasing with Years?\n",
    "\n",
    "\n",
    " - Top 20 Authors with highest User Ratings\n",
    "\n",
    "\n",
    " - Top 20 Authors with highest Reviews\n",
    "\n",
    "\n",
    " - Relationship between length of book title and and User Ratings - Did Short titled books get more ratings than long titled books?\n",
    "\n",
    "\n",
    " - Top 10 Best Selling Authors based on Genre and Number of Appearances\n",
    "\n",
    "\n",
    " - Top 10 Books based on User Ratings\n",
    "\n",
    "\n",
    " - Top 10 Books Based on Reviews\n",
    "\n",
    "\n",
    " - Top 10 Best Selling Books by Genre and Number of Reviews"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9c40d002",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load EDA packaes\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# load web scrapping packages\n",
    "import requests\n",
    "from selenium import webdriver\n",
    "from bs4 import BeautifulSoup\n",
    "from urllib.request import urlopen\n",
    "from time import sleep\n",
    "\n",
    "\n",
    "pd.set_option('display.max_rows',500)\n",
    "pd.set_option('display.max_columns',500)\n",
    "pd.set_option('display.width',1000)\n",
    "\n",
    "# to bypass warnings\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37e14b53",
   "metadata": {},
   "source": [
    "# Gathering the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "081f53ed",
   "metadata": {},
   "source": [
    "###### Let's get the urls and the second page for each year beginning from 2009 to 2021"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d514ecb2",
   "metadata": {},
   "outputs": [],
   "source": [
    "#since the first page and last page has the same pattern url for all the years \n",
    "urls =[]\n",
    "years = [str(i) for i in range(2009,2022)] #list of years between 2009-2022\n",
    "for year in years:\n",
    "    urls.append (f\"https://www.amazon.com/gp/bestsellers/{year}/books\")\n",
    "    urls.append(f\"https://www.amazon.com/gp/bestsellers/{year}/books/ref=zg_bsar_pg_2/ref=zg_bsar_pg_2?ie=UTF8&pg=2\")\n",
    "    \n",
    "#urls"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5a4a874",
   "metadata": {},
   "source": [
    "###### Let's use this function to get the details for each book in each year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ed5f6852",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_dir(book,year): \n",
    "    '''to get the details of each book for each year''' \n",
    "    \n",
    "    import numpy as np\n",
    "    '''to get the name of price'''\n",
    "\n",
    "    try:\n",
    "        price = book.find('span',class_=\"_cDEzb_p13n-sc-price_3mJ9Z\").text[1:]\n",
    "    except Exception as e:\n",
    "        price = np.nan\n",
    "    try:\n",
    "        ranks = book.find('span', class_='zg-bdg-text').text[1:]\n",
    "    except Exception as e:\n",
    "        ranks = np.nan\n",
    "    try:\n",
    "        title = book.find('div',class_=\"_cDEzb_p13n-sc-css-line-clamp-1_1Fn1y\").text\n",
    "    except Exception as e:\n",
    "        title = np.nan\n",
    "    try:\n",
    "        ratings= book.find('span',class_=\"a-icon-alt\").text[:3] \n",
    "    except Exception as e:\n",
    "        ratings = np.nan\n",
    "    try:\n",
    "        no_of_reviews  = book.find('span',class_=\"a-size-small\").text\n",
    "    except Exception as e:\n",
    "        no_of_reviews = np.nan\n",
    "    try:\n",
    "        author = book.find('a',class_=\"a-size-small a-link-child\").text\n",
    "    except Exception as e:\n",
    "        author = np.nan\n",
    "    try:\n",
    "        cover_type = book.find('span',class_=\"a-size-small a-color-secondary a-text-normal\").text\n",
    "    except Exception as e:\n",
    "        cover_type = np.nan\n",
    "    year = year\n",
    "    return [price,ranks,title,no_of_reviews,ratings,author,cover_type, year]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b77b7c7c",
   "metadata": {},
   "source": [
    "###### Let's get the year for the first and second page"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e6aa996a",
   "metadata": {},
   "outputs": [],
   "source": [
    "year = [(str(i),str(i)) for i in range(2009,2022)] #create list that contains the a set of each year\n",
    "years = [j for i in year for j in i] #get a list from the above line\n",
    "#years"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80405020",
   "metadata": {},
   "source": [
    "###### Let's get the books in every page(first and second) of every year from year 2009- 2021\n",
    "###### Note that this cell takes about 25 minutes to run, so you will have to exercise patience"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2034a839",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_year = [] #Empty list of the content on the first and second page for the all the years of interest\n",
    "from time import sleep \n",
    "for url in urls:  # loop through the urls created in cell 2\n",
    "    \n",
    "    website = url \n",
    "    \n",
    "    driver = webdriver.Chrome(\"C:/webDrivers/chromedriver.exe\") #to load the selenium webdriver \n",
    "    \n",
    "    driver.get(website)        # use selenium webdriver above to get the webpage\n",
    "    \n",
    "    sleep(30)                  #to make sure the website is fullly loaded before going to the next page\n",
    "    \n",
    "    the_soup = BeautifulSoup(driver.page_source, 'html.parser')           #to get the page content\n",
    "    \n",
    "    books = the_soup.find_all(id = 'gridItemRoot')                 #get every books on the page.\n",
    "    \n",
    "    all_year.append(books) #add the books to the the list above\n",
    "    \n",
    "    \n",
    "    \n",
    "    driver.quit()                      #to close the chrome windows tab after extracting the data\n",
    "    \n",
    "#"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e7134167",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(26, 26)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(all_year), len(years)  #to confirm you got all the years(first and second page)\n",
    "#should be the same"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbe9ff6f",
   "metadata": {},
   "source": [
    "###### use the code below to get index and year so that looping through the files will be easier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "246c8ce2",
   "metadata": {},
   "outputs": [],
   "source": [
    "year_index = (list(enumerate(years)))\n",
    "dc = year_index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca844cb5",
   "metadata": {},
   "source": [
    "###### use the code to ge the observation for all the books in the top 100 for every year with the period of 2009-2021"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2fc023f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [] #create an empty list for the observation for all the books in the top 100 for every year with the period of 2009-2022\n",
    "for i in dc:   #loop through the year index in the cell above\n",
    "    for books in all_year[i[0]]:             #loop through the books for all the years\n",
    "        for book in books:                   #loop through the books for on each page\n",
    "            data.append(get_dir(book,i[1]))  # get the details of each book and add to data(line 1)\n",
    "            \n",
    "#data #to print the data collected"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "181ffc1b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File written successfully\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# open file\n",
    "with open('Amazon.txt', 'w+') as f:\n",
    "     \n",
    "    # write elements of list\n",
    "    for items in data: \n",
    "        try:\n",
    "            f.write('%s\\n' %items)\n",
    "        except Exception as e:\n",
    "            f.write('%s\\n' 'nothing')\n",
    "     \n",
    "    print(\"File written successfully\")\n",
    " \n",
    " \n",
    "# close the file\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6463fc0",
   "metadata": {},
   "source": [
    "###### This cell is for converting the data extracted in list format to a dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d803bb18",
   "metadata": {},
   "outputs": [],
   "source": [
    "best_selling_books= pd.DataFrame(data, columns = [\n",
    "                         'price',\n",
    "                         'ranks',\n",
    "                         'title',\n",
    "                         'no_of_reviews',\n",
    "                         'ratings',\n",
    "                         'author',\n",
    "                       'cover_type',\n",
    "                          'year'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1990d51",
   "metadata": {},
   "source": [
    "###### save the data to csv file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e5cb7fc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "best_selling_books.to_csv('best_selling_books_2009-2021.csv')   #To save to csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "3a1ed394",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>ranks</th>\n",
       "      <th>title</th>\n",
       "      <th>no_of_reviews</th>\n",
       "      <th>ratings</th>\n",
       "      <th>author</th>\n",
       "      <th>cover_type</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>12.81</td>\n",
       "      <td>1</td>\n",
       "      <td>The Lost Symbol</td>\n",
       "      <td>16,129</td>\n",
       "      <td>4.4</td>\n",
       "      <td>Dan Brown</td>\n",
       "      <td>Hardcover</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10.43</td>\n",
       "      <td>2</td>\n",
       "      <td>The Shack: Where Tragedy Confronts Eternity</td>\n",
       "      <td>23,398</td>\n",
       "      <td>4.7</td>\n",
       "      <td>William P. Young</td>\n",
       "      <td>Paperback</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9.93</td>\n",
       "      <td>3</td>\n",
       "      <td>Liberty and Tyranny: A Conservative Manifesto</td>\n",
       "      <td>5,037</td>\n",
       "      <td>4.8</td>\n",
       "      <td>Mark R. Levin</td>\n",
       "      <td>Hardcover</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>14.30</td>\n",
       "      <td>4</td>\n",
       "      <td>Breaking Dawn (The Twilight Saga, Book 4)</td>\n",
       "      <td>16,923</td>\n",
       "      <td>4.7</td>\n",
       "      <td>Stephenie Meyer</td>\n",
       "      <td>Hardcover</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9.99</td>\n",
       "      <td>5</td>\n",
       "      <td>Going Rogue: An American Life</td>\n",
       "      <td>1,572</td>\n",
       "      <td>4.6</td>\n",
       "      <td>Sarah Palin</td>\n",
       "      <td>Hardcover</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1286</th>\n",
       "      <td>16.69</td>\n",
       "      <td>96</td>\n",
       "      <td>Will</td>\n",
       "      <td>Will Smith</td>\n",
       "      <td>4.8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Hardcover</td>\n",
       "      <td>2021</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1287</th>\n",
       "      <td>7.49</td>\n",
       "      <td>97</td>\n",
       "      <td>Think and Grow Rich: The Landmark Bestseller N...</td>\n",
       "      <td>83,367</td>\n",
       "      <td>4.7</td>\n",
       "      <td>Napoleon Hill</td>\n",
       "      <td>Paperback</td>\n",
       "      <td>2021</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1288</th>\n",
       "      <td>8.95</td>\n",
       "      <td>98</td>\n",
       "      <td>Dragons Love Tacos</td>\n",
       "      <td>15,771</td>\n",
       "      <td>4.8</td>\n",
       "      <td>Adam Rubin</td>\n",
       "      <td>Hardcover</td>\n",
       "      <td>2021</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1289</th>\n",
       "      <td>7.49</td>\n",
       "      <td>99</td>\n",
       "      <td>The Truth About COVID-19: Exposing The Great R...</td>\n",
       "      <td>Doctor Joseph Mercola</td>\n",
       "      <td>4.8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Hardcover</td>\n",
       "      <td>2021</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1290</th>\n",
       "      <td>13.29</td>\n",
       "      <td>100</td>\n",
       "      <td>First Little Readers Parent Pack: Guided Readi...</td>\n",
       "      <td>27,350</td>\n",
       "      <td>4.7</td>\n",
       "      <td>Deborah Schecter</td>\n",
       "      <td>Paperback</td>\n",
       "      <td>2021</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1291 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      price ranks                                              title          no_of_reviews ratings            author cover_type  year\n",
       "0     12.81     1                                    The Lost Symbol                 16,129     4.4         Dan Brown  Hardcover  2009\n",
       "1     10.43     2        The Shack: Where Tragedy Confronts Eternity                 23,398     4.7  William P. Young  Paperback  2009\n",
       "2      9.93     3      Liberty and Tyranny: A Conservative Manifesto                  5,037     4.8     Mark R. Levin  Hardcover  2009\n",
       "3     14.30     4          Breaking Dawn (The Twilight Saga, Book 4)                 16,923     4.7   Stephenie Meyer  Hardcover  2009\n",
       "4      9.99     5                      Going Rogue: An American Life                  1,572     4.6       Sarah Palin  Hardcover  2009\n",
       "...     ...   ...                                                ...                    ...     ...               ...        ...   ...\n",
       "1286  16.69    96                                               Will             Will Smith     4.8               NaN  Hardcover  2021\n",
       "1287   7.49    97  Think and Grow Rich: The Landmark Bestseller N...                 83,367     4.7     Napoleon Hill  Paperback  2021\n",
       "1288   8.95    98                                 Dragons Love Tacos                 15,771     4.8        Adam Rubin  Hardcover  2021\n",
       "1289   7.49    99  The Truth About COVID-19: Exposing The Great R...  Doctor Joseph Mercola     4.8               NaN  Hardcover  2021\n",
       "1290  13.29   100  First Little Readers Parent Pack: Guided Readi...                 27,350     4.7  Deborah Schecter  Paperback  2021\n",
       "\n",
       "[1291 rows x 8 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_selling_books"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee335960",
   "metadata": {},
   "source": [
    "###### This will be the end of this notebook.\n",
    "\n",
    "We will continue with the Analysis in Part Two Notebook. This is so that we won't keep scrapping the data from Amazon whenever we want to rerun the whole body of code for uniformity.\n",
    "\n",
    "We would also want to respect the ethics of web scrapping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b915e2a7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec3a2759",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:StarNPMS] *",
   "language": "python",
   "name": "conda-env-StarNPMS-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}