Notebooks/1_Flipkart_ETL.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4efa297b",
   "metadata": {},
   "source": [
    "# Create Mobiles DataSet From FlipKart With Full ETL Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "502aaaa5",
   "metadata": {},
   "source": [
    "# Extract"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc275735",
   "metadata": {},
   "source": [
    "* The Extract part is the first step of the pipeline. In this step, the data is extracted from  different source and cached. In this project, I want to extract data from the FLIPKART resource."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ec2138e",
   "metadata": {},
   "source": [
    "# Webscraping Mobile Data From Flipkart"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16d1b5ac",
   "metadata": {},
   "source": [
    "* Lets Now Scrape The Data From FLIPKART : https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2286887b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5c64819f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def log(message):\n",
    "    # set timestamp format\n",
    "    timestamp_format = '%Y-%h-%d-%H:%M:%S'\n",
    "    # get current timestamp\n",
    "    now = datetime.now()\n",
    "    timestamp = now.strftime(timestamp_format)\n",
    "    with open(\"logfile.txt\",\"a\") as f:\n",
    "        f.write(timestamp + ',' + message + '\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b6362f9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "424f2118",
   "metadata": {},
   "outputs": [],
   "source": [
    "# libraries for web scraping\n",
    "from urllib.request import urlopen\n",
    "from bs4 import BeautifulSoup as soup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f065d9db",
   "metadata": {},
   "source": [
    "# Scraping The Data From Page 1 Of FLIPKART"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c9f11a1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create log message\n",
    "log(\"Start Extract Part\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1caecc52",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create log message\n",
    "log(\"Start Webscraping Flipkart Data\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6049aa5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "url = urlopen('https://www.flipkart.com/search?q=iphones%20mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off')\n",
    "webdata = url.read()\n",
    "#print(webdata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ddf4bf73",
   "metadata": {},
   "outputs": [],
   "source": [
    "soup1 = soup(webdata)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe79bf28",
   "metadata": {},
   "source": [
    "# Scraping The Data From Page 2\n",
    "   We are taking only 2 Pages Data As For Our Convinience"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "26478415",
   "metadata": {},
   "outputs": [],
   "source": [
    "url = urlopen('https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=2')\n",
    "webdata_2 = url.read()\n",
    "#print(webdata_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "16326220",
   "metadata": {},
   "outputs": [],
   "source": [
    "soup2 = soup(webdata_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "4803cdbd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create log message\n",
    "log(\"Finished Webscraping Flipkart Data\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "8a04fe4c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://www.flipkart.com1\n",
      "https://www.flipkart.com2\n",
      "https://www.flipkart.com3\n",
      "https://www.flipkart.com4\n",
      "https://www.flipkart.com5\n",
      "https://www.flipkart.com6\n",
      "https://www.flipkart.com7\n",
      "https://www.flipkart.com8\n",
      "https://www.flipkart.com9\n",
      "https://www.flipkart.com10\n",
      "https://www.flipkart.com11\n",
      "https://www.flipkart.com12\n",
      "https://www.flipkart.com13\n",
      "https://www.flipkart.com14\n",
      "https://www.flipkart.com15\n",
      "https://www.flipkart.com16\n",
      "https://www.flipkart.com17\n",
      "https://www.flipkart.com18\n",
      "https://www.flipkart.com19\n",
      "https://www.flipkart.com20\n",
      "https://www.flipkart.com21\n",
      "https://www.flipkart.com22\n",
      "https://www.flipkart.com23\n",
      "https://www.flipkart.com24\n",
      "https://www.flipkart.com25\n",
      "https://www.flipkart.com26\n",
      "https://www.flipkart.com27\n",
      "https://www.flipkart.com28\n",
      "https://www.flipkart.com29\n",
      "https://www.flipkart.com30\n",
      "https://www.flipkart.com31\n",
      "https://www.flipkart.com32\n",
      "https://www.flipkart.com33\n",
      "https://www.flipkart.com34\n",
      "https://www.flipkart.com35\n"
     ]
    }
   ],
   "source": [
    "domain = 'https://www.flipkart.com'\n",
    "for i in range(1,36):\n",
    "    print(domain+str(i))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c388306",
   "metadata": {},
   "source": [
    "# Transform"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1dc1e06",
   "metadata": {},
   "source": [
    "* Transform The Page 1 and Page 2 Data Into One To Make Final Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "4f3cc4b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "#create log\n",
    "log(\"Start Getting The Final Data\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "b1f82b9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create log message\n",
    "log(\"Start Transform Part of Pipeline\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "0cdfd9bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "final = str(soup1) + str(soup2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "05613f97",
   "metadata": {},
   "outputs": [],
   "source": [
    "#soupdata = soup(final)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "631626b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "#type(soupdata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "66b8b312",
   "metadata": {},
   "outputs": [],
   "source": [
    "soupdata = soup(final,'html.parser')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "ebf2e89f",
   "metadata": {},
   "outputs": [],
   "source": [
    "containers = soupdata.findAll('div',{'class':'_2kHMtA'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "ba3db455",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'bs4.element.ResultSet'> 48\n"
     ]
    }
   ],
   "source": [
    "print(type(containers),len(containers))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "a68de165",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<div class=\"_2kHMtA\"><a class=\"_1fQZEK\" href=\"/apple-iphone-11-white-64-gb/p/itmfc6a7091eb20b?pid=MOBFWQ6BVWVEH3XE&amp;lid=LSTMOBFWQ6BVWVEH3XEMXQMLO&amp;marketplace=FLIPKART&amp;q=iphones+mobiles&amp;store=tyy%2F4io&amp;srno=s_1_1&amp;otracker=search&amp;otracker1=search&amp;fm=organic&amp;iid=91d9b365-6102-4147-8943-3bc00dde3747.MOBFWQ6BVWVEH3XE.SEARCH&amp;ppt=None&amp;ppn=None&amp;ssid=ypn50nzc0g0000001663432504777&amp;qH=65a678740bbb0b54\" rel=\"noopener noreferrer\" target=\"_blank\"><div class=\"MIXNux\"><div class=\"_2QcLo-\"><div><div class=\"CXW8mj\" style=\"height:200px;width:200px\"><img alt=\"APPLE iPhone 11 (White, 64 GB)\" class=\"_396cs4 _3exPp9\" src=\"https://rukminim1.flixcart.com/image/312/312/k2jbyq80pkrrdj/mobile-refurbished/k/y/d/iphone-11-256-u-mwm82hn-a-apple-0-original-imafkg25mhaztxns.jpeg?q=70\"/></div></div></div><div class=\"_3wLduG\"><div class=\"_3PzNI-\"><span class=\"f3A4_V\"><label class=\"_2iDkf8\"><input class=\"_30VH1S\" readonly=\"\" type=\"checkbox\"/><div class=\"_24_Dny\"></div></label></span><label class=\"_6Up2sF\"><span>Add to Compare</span></label></div></div><div class=\"_2hVSre _3nq8ih\"><div class=\"_36FSn5\"><svg class=\"_1l0elc\" height=\"16\" viewbox=\"0 0 20 16\" width=\"16\" xmlns=\"http://www.w3.org/2000/svg\"><path class=\"eX72wL\" d=\"M8.695 16.682C4.06 12.382 1 9.536 1 6.065 1 3.219 3.178 1 5.95 1c1.566 0 3.069.746 4.05 1.915C10.981 1.745 12.484 1 14.05 1 16.822 1 19 3.22 19 6.065c0 3.471-3.06 6.316-7.695 10.617L10 17.897l-1.305-1.215z\" fill=\"#2874F0\" fill-rule=\"evenodd\" opacity=\".9\" stroke=\"#FFF\"></path></svg></div></div></div><div class=\"_3pLy-c row\"><div class=\"col col-7-12\"><div class=\"_4rR01T\">APPLE iPhone 11 (White, 64 GB)</div><div class=\"gUuXy-\"><span class=\"_1lRcqv\" id=\"productRating_LSTMOBFWQ6BVWVEH3XEMXQMLO_MOBFWQ6BVWVEH3XE_\"><div class=\"_3LWZlK\">4.6<img class=\"_1wB99o\" src=\"data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIxMyIgaGVpZ2h0PSIxMiI+PHBhdGggZmlsbD0iI0ZGRiIgZD0iTTYuNSA5LjQzOWwtMy42NzQgMi4yMy45NC00LjI2LTMuMjEtMi44ODMgNC4yNTQtLjQwNEw2LjUuMTEybDEuNjkgNC4wMSA0LjI1NC40MDQtMy4yMSAyLjg4Mi45NCA0LjI2eiIvPjwvc3ZnPg==\"/></div></span><span class=\"_2_R_DZ\"><span><span>95,955 Ratings </span><span class=\"_13vcmD\">&amp;</span><span> 7,038 Reviews</span></span></span></div><div class=\"fMghEO\"><ul class=\"_1xgFaf\"><li class=\"rgWa7D\">64 GB ROM</li><li class=\"rgWa7D\">15.49 cm (6.1 inch) Liquid Retina HD Display</li><li class=\"rgWa7D\">12MP + 12MP | 12MP Front Camera</li><li class=\"rgWa7D\">A13 Bionic Chip Processor</li><li class=\"rgWa7D\">Brand Warranty of 1 Year</li></ul></div></div><div class=\"col col-5-12 nlI3QM\"><div class=\"_3tbKJL\"><div class=\"_25b18c\"><div class=\"_30jeq3 _1_WHN1\">₹43,900</div></div><div class=\"_3tcB5a p8ucoS\"><div><div class=\"_2Tpdn3\" style=\"color:#000000;font-size:12px;font-weight:400\">Free delivery</div></div></div></div><div class=\"_13J9qT\"><img height=\"21\" src=\"//static-assets-web.flixcart.com/fk-p-linchpin-web/fk-cp-zion/img/fa_62673a.png\"/></div><div class=\"_2ZdXDB\"><div class=\"_3xFhiH\"><div class=\"_2Tpdn3 _18hQoS\" style=\"color:#000000;font-size:14px;font-style:normal;font-weight:400\">Upto </div><div class=\"_2Tpdn3 _18hQoS\" style=\"color:#000000;font-size:14px;font-style:normal;font-weight:700\">₹19,000</div><div class=\"_2Tpdn3 _18hQoS\" style=\"color:#000000;font-size:14px;font-style:normal;font-weight:400\"> Off on Exchange</div></div></div><div class=\"_2ZdXDB\"><div class=\"_3xFhiH\"><div class=\"_2Tpdn3 _18hQoS\" style=\"color:#26A541;font-size:14px;font-weight:700\">Bank Offer</div></div></div></div></div></a></div>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "containers[0]          "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "23d0bbe1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "6765b6e9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['APPLE iPhone 11 (White, 64 GB)']\n",
      "['APPLE iPhone 11 (White, 128 GB)']\n",
      "['APPLE iPhone 11 (Green, 128 GB)']\n",
      "['APPLE iPhone 11 (Black, 128 GB)']\n",
      "['APPLE iPhone 11 (Black, 64 GB)']\n",
      "['APPLE iPhone 11 (Purple, 128 GB)']\n",
      "['APPLE iPhone 14 Pro (Space Black, 256 GB)']\n",
      "['APPLE iPhone 14 Pro (Silver, 256 GB)']\n",
      "['APPLE iPhone SE (Red, 64 GB)']\n",
      "['APPLE iPhone 11 (Red, 64 GB)']\n",
      "['APPLE iPhone 11 (Red, 128 GB)']\n",
      "['APPLE iPhone 13 (Midnight, 128 GB)']\n",
      "['APPLE iPhone 14 (Starlight, 128 GB)']\n",
      "['APPLE iPhone 11 (Green, 64 GB)']\n",
      "['APPLE iPhone SE (White, 64 GB)']\n",
      "['APPLE iPhone 14 Pro (Silver, 128 GB)']\n",
      "['APPLE iPhone 13 (Pink, 128 GB)']\n",
      "['APPLE iPhone SE (White, 128 GB)']\n",
      "['APPLE iPhone 14 (Purple, 128 GB)']\n",
      "['APPLE iPhone 11 (Purple, 64 GB)']\n",
      "['APPLE iPhone 13 ((PRODUCT)RED, 128 GB)']\n",
      "['APPLE iPhone 13 (Green, 128 GB)']\n",
      "['APPLE iPhone 12 (Green, 128 GB)']\n",
      "['APPLE iPhone 13 (Blue, 128 GB)']\n",
      "['APPLE iPhone 13 ((PRODUCT)RED, 128 GB)']\n",
      "['APPLE iPhone 13 (Green, 128 GB)']\n",
      "['APPLE iPhone 14 ((PRODUCT)RED, 128 GB)']\n",
      "['APPLE iPhone 12 (Green, 128 GB)']\n",
      "['APPLE iPhone 12 (Black, 128 GB)']\n",
      "['APPLE iPhone 13 Mini (Midnight, 128 GB)']\n",
      "['APPLE iPhone 13 Mini (Blue, 128 GB)']\n",
      "['APPLE iPhone 14 (Purple, 256 GB)']\n",
      "['APPLE iPhone 13 (Starlight, 128 GB)']\n",
      "['APPLE iPhone 14 Pro Max (Space Black, 256 GB)']\n",
      "['APPLE iPhone SE (Black, 128 GB)']\n",
      "['APPLE iPhone 13 Mini ((PRODUCT)RED, 128 GB)']\n",
      "['APPLE iPhone 11 (Yellow, 128 GB)']\n",
      "['APPLE iPhone 12 (White, 128 GB)']\n",
      "['APPLE iPhone 14 Plus (Blue, 128 GB)']\n",
      "['APPLE iPhone 14 Plus ((PRODUCT)RED, 128 GB)']\n",
      "['APPLE iPhone 14 ((PRODUCT)RED, 256 GB)']\n",
      "['APPLE iPhone 12 (Purple, 128 GB)']\n",
      "['APPLE iPhone 13 Mini (Starlight, 128 GB)']\n",
      "['APPLE iPhone 14 Plus (Starlight, 256 GB)']\n",
      "['APPLE iPhone 12 Mini (Blue, 64 GB)']\n",
      "['APPLE iPhone 14 Pro (Gold, 128 GB)']\n",
      "['APPLE iPhone 13 Mini (Pink, 128 GB)']\n",
      "['APPLE iPhone 13 (Starlight, 256 GB)']\n"
     ]
    }
   ],
   "source": [
    "for container in containers:\n",
    "    product = container.findAll('div',{'class':'_4rR01T'})\n",
    "    print(product[0].text.split('-'))\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "a82f3ed9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<a class=\"ge-49M _2Kfbh8\" href=\"/search?q=iphones+mobiles&amp;otracker=search&amp;otracker1=search&amp;marketplace=FLIPKART&amp;as-show=on&amp;as=off&amp;page=1\">1</a>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soupdata.find('nav',{'class':'yFHi8N'}).a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "dd42189b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=2'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "page = soupdata.findAll('a',{'class':'ge-49M'})\n",
    "domain = 'https://www.flipkart.com'\n",
    "domain + page[1].get('href')\n",
    "#page[0].get('href')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "0fdd8ef6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=2'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "page = soupdata.findAll('a',{'class':'ge-49M'})\n",
    "domain = 'https://www.flipkart.com'\n",
    "domain + page[1].get('href')\n",
    "#page[0].get('href')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "f3c32f33",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=1\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=2\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=3\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=4\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=5\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=6\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=7\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=8\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=9\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=10\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=11\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=12\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=13\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=14\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=15\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=16\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=17\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=18\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=19\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=20\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=21\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=22\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=23\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=24\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=25\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=26\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=27\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=28\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=29\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=30\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=31\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=32\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=33\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=34\n",
      "https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page=35\n"
     ]
    }
   ],
   "source": [
    "page = soupdata.findAll('a',{'class':'ge-49M'})\n",
    "domain = 'https://www.flipkart.com/search?q=iphones+mobiles&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page='\n",
    "#domain + page[1].get('href')\n",
    "\n",
    "for i in range(1,36):\n",
    "    print(domain+str(i))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "1a99d523",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create log message\n",
    "log(\"Finished Transform Part of Pipeline\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "bc74cfd1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08bf30eb",
   "metadata": {},
   "source": [
    "# Load\n",
    "* The last step of the pipeline is to load the dataset, I am loading the final dataset into a csv file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "1a12b93e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create log message\n",
    "log(\"Start Load Part of Pipeline\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "76e4e475",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "APPLE iPhone 11 (White 4.6 95955 7038 ₹43900 64 GB ROM 15.49 cm (6.1 inch) Liquid Retina HD Display 12MP + 12MP | 12MP Front Camera A13 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 11 (White 4.6 95955 7038 ₹48900 128 GB ROM 15.49 cm (6.1 inch) Liquid Retina HD Display 12MP + 12MP | 12MP Front Camera A13 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 11 (Green 4.6 95955 7038 ₹48900 128 GB ROM 15.49 cm (6.1 inch) Liquid Retina HD Display 12MP + 12MP | 12MP Front Camera A13 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 11 (Black 4.6 95955 7038 ₹48900 128 GB ROM 15.49 cm (6.1 inch) Liquid Retina HD Display 12MP + 12MP | 12MP Front Camera A13 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 11 (Black 4.6 95955 7038 ₹43900 64 GB ROM 15.49 cm (6.1 inch) Liquid Retina HD Display 12MP + 12MP | 12MP Front Camera A13 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 11 (Purple 4.6 95955 7038 ₹48900 128 GB ROM 15.49 cm (6.1 inch) Liquid Retina HD Display 12MP + 12MP | 12MP Front Camera A13 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 14 Pro (Space Black 0 0 0 ₹139900 256 GB ROM 15.49 cm (6.1 inch) Super Retina XDR Display 48MP + 12MP + 12MP + 12MP | 12MP Front Camera A16 Bionic Chip, 6 Core Processor Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 14 Pro (Silver 0 0 0 ₹139900 256 GB ROM 15.49 cm (6.1 inch) Super Retina XDR Display 48MP + 12MP + 12MP + 12MP | 12MP Front Camera A16 Bionic Chip, 6 Core Processor Processor\n",
      "\n",
      "\n",
      "APPLE iPhone SE (Red 4.5 152 589 ₹30499 64 GB ROM 11.94 cm (4.7 inch) Retina HD Display 12MP Rear Camera | 7MP Front Camera A13 Bionic Chip with 3rd Gen Neural Engine Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 11 (Red 4.6 95955 7038 ₹43900 64 GB ROM 15.49 cm (6.1 inch) Liquid Retina HD Display 12MP + 12MP | 12MP Front Camera A13 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 11 (Red 4.6 95955 7038 ₹48900 128 GB ROM 15.49 cm (6.1 inch) Liquid Retina HD Display 12MP + 12MP | 12MP Front Camera A13 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 13 (Midnight 4.6 12494 1016 ₹69900 128 GB ROM 15.49 cm (6.1 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A15 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 14 (Starlight 5 3 0 ₹79900 128 GB ROM 15.49 cm (6.1 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A15 Bionic Chip, 6 Core Processor Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 11 (Green 4.6 95955 7038 ₹43900 64 GB ROM 15.49 cm (6.1 inch) Liquid Retina HD Display 12MP + 12MP | 12MP Front Camera A13 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone SE (White 4.5 152 589 ₹30499 64 GB ROM 11.94 cm (4.7 inch) Retina HD Display 12MP Rear Camera | 7MP Front Camera A13 Bionic Chip with 3rd Gen Neural Engine Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 14 Pro (Silver 0 0 0 ₹129900 128 GB ROM 15.49 cm (6.1 inch) Super Retina XDR Display 48MP + 12MP + 12MP + 12MP | 12MP Front Camera A16 Bionic Chip, 6 Core Processor Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 13 (Pink 4.6 12494 1016 ₹69900 128 GB ROM 15.49 cm (6.1 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A15 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone SE (White 4.5 152 589 ₹35499 128 GB ROM 11.94 cm (4.7 inch) Retina HD Display 12MP Rear Camera | 7MP Front Camera A13 Bionic Chip with 3rd Gen Neural Engine Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 14 (Purple 5 3 0 ₹79900 128 GB ROM 15.49 cm (6.1 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A15 Bionic Chip, 6 Core Processor Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 11 (Purple 4.6 95955 7038 ₹43900 64 GB ROM 15.49 cm (6.1 inch) Liquid Retina HD Display 12MP + 12MP | 12MP Front Camera A13 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 13 ((PRODUCT)RED 4.6 12494 1016 ₹69900 128 GB ROM 15.49 cm (6.1 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A15 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 13 (Green 4.6 12494 1016 ₹69900 128 GB ROM 15.49 cm (6.1 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A15 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 12 (Green 4.6 181 662 ₹64900 128 GB ROM 15.49 cm (6.1 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A14 Bionic Chip with Next Generation Neural Engine Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 13 (Blue 4.6 12494 1016 ₹69900 128 GB ROM 15.49 cm (6.1 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A15 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 13 ((PRODUCT)RED 4.6 12528 1017 ₹69900 128 GB ROM 15.49 cm (6.1 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A15 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 13 (Green 4.6 12528 1017 ₹69900 128 GB ROM 15.49 cm (6.1 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A15 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 14 ((PRODUCT)RED 5 3 0 ₹79900 128 GB ROM 15.49 cm (6.1 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A15 Bionic Chip, 6 Core Processor Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 12 (Green 4.6 181 692 ₹64900 128 GB ROM 15.49 cm (6.1 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A14 Bionic Chip with Next Generation Neural Engine Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 12 (Black 4.6 181 692 ₹64900 128 GB ROM 15.49 cm (6.1 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A14 Bionic Chip with Next Generation Neural Engine Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 13 Mini (Midnight 4.5 2189 198 ₹64900 128 GB ROM 13.72 cm (5.4 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A15 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 13 Mini (Blue 4.5 2189 198 ₹64900 128 GB ROM 13.72 cm (5.4 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A15 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 14 (Purple 5 3 0 ₹89900 256 GB ROM 15.49 cm (6.1 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A15 Bionic Chip, 6 Core Processor Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 13 (Starlight 4.6 12528 1017 ₹69900 128 GB ROM 15.49 cm (6.1 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A15 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 14 Pro Max (Space Black 5 3 1 ₹149900 256 GB ROM 17.02 cm (6.7 inch) Super Retina XDR Display 48MP + 12MP + 12MP + 12MP | 12MP Front Camera A16 Bionic Chip, 6 Core Processor Processor\n",
      "\n",
      "\n",
      "APPLE iPhone SE (Black 4.5 152 601 ₹35499 128 GB ROM 11.94 cm (4.7 inch) Retina HD Display 12MP Rear Camera | 7MP Front Camera A13 Bionic Chip with 3rd Gen Neural Engine Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 13 Mini ((PRODUCT)RED 4.5 2189 198 ₹64900 128 GB ROM 13.72 cm (5.4 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A15 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 11 (Yellow 4.6 95996 7039 ₹48900 128 GB ROM 15.49 cm (6.1 inch) Liquid Retina HD Display 12MP + 12MP | 12MP Front Camera A13 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 12 (White 4.6 181 692 ₹64900 128 GB ROM 15.49 cm (6.1 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A14 Bionic Chip with Next Generation Neural Engine Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 14 Plus (Blue 0 0 0 ₹89900 128 GB ROM 17.02 cm (6.7 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A15 Bionic Chip, 6 Core Processor Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 14 Plus ((PRODUCT)RED 0 0 0 ₹89900 128 GB ROM 17.02 cm (6.7 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A15 Bionic Chip, 6 Core Processor Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 14 ((PRODUCT)RED 5 3 0 ₹89900 256 GB ROM 15.49 cm (6.1 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A15 Bionic Chip, 6 Core Processor Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 12 (Purple 4.6 181 692 ₹64900 128 GB ROM 15.49 cm (6.1 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A14 Bionic Chip with Next Generation Neural Engine Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 13 Mini (Starlight 4.5 2189 198 ₹64900 128 GB ROM 13.72 cm (5.4 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A15 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 14 Plus (Starlight 0 0 0 ₹99900 256 GB ROM 17.02 cm (6.7 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A15 Bionic Chip, 6 Core Processor Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 12 Mini (Blue 4.5 101 815 ₹55359 64 GB ROM 13.72 cm (5.4 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A14 Bionic Chip with Next Generation Neural Engine Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 14 Pro (Gold 5 4 1 ₹129900 128 GB ROM 15.49 cm (6.1 inch) Super Retina XDR Display 48MP + 12MP + 12MP + 12MP | 12MP Front Camera A16 Bionic Chip, 6 Core Processor Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 13 Mini (Pink 4.5 2189 198 ₹64900 128 GB ROM 13.72 cm (5.4 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A15 Bionic Chip Processor\n",
      "\n",
      "\n",
      "APPLE iPhone 13 (Starlight 4.6 12528 1017 ₹79900 256 GB ROM 15.49 cm (6.1 inch) Super Retina XDR Display 12MP + 12MP | 12MP Front Camera A15 Bionic Chip Processor\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "f = open('Mobiles_data.csv','wb')\n",
    "f.write('ProductName,Stars,Ratings,Reviews,Price,ROM,Display,Camera,Processor\\n'.encode())\n",
    "for container in containers:\n",
    "    #Finding Product Name \n",
    "    product = container.find('div',{'class':'_4rR01T'})\n",
    "    ProductName = product.text.split(',')[0]\n",
    "    \n",
    "    #Finding Stars\n",
    "    star = container.find('div',{'class','_3LWZlK'})\n",
    "    try:\n",
    "        Stars = star.text\n",
    "    except:\n",
    "        Stars = 0\n",
    "        \n",
    "    #Finding Rating & Reviews\n",
    "    Rating = container.find('span',{'class':'_2_R_DZ'})\n",
    "    try:\n",
    "        ratRev = re.findall('\\d+,?\\d*',Rating.text)\n",
    "        Ratings = ratRev[0].replace(',','')\n",
    "        Reviews = ratRev[1].replace(',','')\n",
    "    except:\n",
    "        Ratings = 0\n",
    "        Reviews = 0\n",
    "    #print(Ratings,Reviews) \n",
    "    \n",
    "    #PRICE\n",
    "    Price = container.find('div',{'class','_30jeq3 _1_WHN1'}).text.replace(',','')\n",
    "    #print(Price.text)\n",
    "    \n",
    "    #Information\n",
    "    info = container.findAll('li',{'class':'rgWa7D'})\n",
    "    ROM = info[0].text\n",
    "    Display = info[1].text\n",
    "    Camera = info[2].text\n",
    "    Processor = info[3].text\n",
    "    \n",
    "    #Image\n",
    "    Image = container.img\n",
    "    ImageURL = Image.get('src')\n",
    "    \n",
    "    print(ProductName,Stars,Ratings,Reviews,Price,ROM,Display,Camera,Processor)\n",
    "    f.write(f\"{ProductName},{Stars},{Ratings},{Reviews},{Price},{ROM},{Display},{Camera},{Processor}\\n\".encode())\n",
    "    print('\\n')\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "bf3f519b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create log message\n",
    "log(\"Finished Load Part of Pipeline\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "557e8232",
   "metadata": {},
   "outputs": [],
   "source": [
    "#re.findall('\\d+,*\\d+','95,970 Ratings & 7,038 Reviews'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "ba9126e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "#ratRev = re.findall('\\d+,?\\d*','95,970 Ratings & 7,038 Reviews')\n",
    "#print(ratRev[0],ratRev[1])\\\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "49eec5dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "#info = container.findAll('li',{'class':'rgWa7D'})\n",
    "#print(info[2].text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b764e99f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}