.ipynb_checkpoints/EDS Assignment I - Preprocessing Training Dataset-checkpoint.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Automatic URL classifying based on IAB taxonomy categories\n",
    "\n",
    "## Introduction\n",
    "IAB is Interactive Advertising Bureau that empowers the media and marketing industries to thrive in the digital economy. The IAB Technology Laboratory (IAB Tech Lab) is a nonprofit research and development consortium charged with producing and helping companies implement global industry technical standards and solutions for the digital media and advertising industries. \n",
    "\n",
    "So in this project I'll creating a classifier and engine to detect the content of the URL and label it automatically. \n",
    "\n",
    "\n",
    "## My day to day problem\n",
    "* I need to classified the URL manually by looking at the website and tag 1 to 1 inside sql database. URL/Website content changes everyday and this hampering my day to day time.\n",
    "* So many taxonomy out there.\n",
    "\n",
    "## Goal of my project\n",
    "* Be able to classified unknown website.\n",
    "* Reduce manually searching on website tagging\n",
    "* Using single taxonomy to clasified the website\n",
    "* Can be extend to DMP integration\n",
    "\n",
    "\n",
    "## Scope\n",
    "This is for EDS final project. The output would be classifying the URL to IAB taxonomy. Since IAB taxonomy having more than 1100+ categories, we will only focus on ecommerce website and below IAB categories:\n",
    "\n",
    "1. Others (600)\n",
    "2. Technology & Computing > Consumer Electronics > Cameras and Camcorders (621)\n",
    "3. Technology & Computing > Consumer Electronics > Home Entertainment Systems (622)\n",
    "4. Technology & Computing > Consumer Electronics > Smartphones (623)\n",
    "5. Technology & Computing > Consumer Electronics > Tablets and E-readers (624)\n",
    "6. Technology & Computing > Consumer Electronics > Wearable Technology (625)\n",
    "\n",
    "1st phase will targetted Lazada since 70% of our link going to Lazada compare to other ecommerce.\n",
    "\n",
    "## Task\n",
    "### To create training dataset\n",
    "1. Go to e-commerce website scrape it's content.\n",
    "2. Label the content based on scope IAB taxonomy\n",
    "3. Train the dataset using the define classifier\n",
    "\n",
    "### To test result:\n",
    "1. Extract unknown category data from Hadoop and create a list\n",
    "2. Feed the data into machine learning preprocessing\n",
    "3. Extracting the data and remove unwanted content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Go to e-commerce website scrape it's content\n",
    "\n",
    "* Using Web Scraper io chrome extension, we can provide the link and it will scrape the whole item URL and solve it pagination automatically\n",
    "\n",
    "![image1](images/webscraper_tools.JPG)\n",
    "\n",
    "* Export the links into a csv file\n",
    "\n",
    "![image2](images/extracted_data.JPG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess extracted links from lazada for training set\n",
    "* Since there is no existing training dataset, so I need to create the training dataset which scraping the website and categories the data.\n",
    "\n",
    "* I've been exploring so may scraper out there but most of the webscraping package doesn't support ajax/javascript, however selenium is different. It's using available browser engine like chrome, firefox and etc\n",
    "\n",
    "* For this project I'm using phantomjs since it can run in the background."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# import all the required package\n",
    "from selenium import webdriver\n",
    "from selenium.webdriver.support.ui import WebDriverWait\n",
    "from selenium.webdriver.support import expected_conditions as EC\n",
    "from selenium.webdriver.common.by import By\n",
    "from selenium.common.exceptions import TimeoutException\n",
    "from selenium.webdriver.common.action_chains import ActionChains\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "import re\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.set_option('display.max_colwidth', -1) #set this for remove truncated by jupyter/python\n",
    "raw_links = pd.read_csv(\"./webpage_scrap/lz_smartphone.csv\") #load the url list from scraping list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "clean_links len: 216\n",
      "clean_links2 len:  178\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'5    https://www.lazada.com.my/allwin-original-unlocked-50-touch-screen-dual-sim-dual-standbysmart-phone-white-11793727.html?ff=1&sc=EwI='"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_links = raw_links['linked-href'][raw_links['pagination'] != 100]\n",
    "print('clean_links len:',len(clean_links))\n",
    "clean_links2 = clean_links.drop_duplicates() #remove duplicate inside list\n",
    "clean_links2.dropna(inplace=True)\n",
    "print('clean_links2 len: ',len(clean_links2))\n",
    "clean_links2[4:5].to_string()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def gettingResult(pageSource):\n",
    "    pageSource = browser.page_source\n",
    "    bsObj = BeautifulSoup(pageSource,'lxml')  \n",
    "\n",
    "    try:\n",
    "        result = bsObj.find(id = 'prod_title').text.strip() + bsObj.find(class_='product-description__block').getText(' ').strip().replace(\"\\n\",\" \").replace(\"\\t\",\" \") + bsObj.find(class_='prd-attributesList ui-listBulleted js-short-description').getText(' ').strip()\n",
    "    except:\n",
    "        result = bsObj.find(class_ = 'product-description__title').text.strip() + bsObj.find(class_='product-description__block').getText(' ').strip().replace(\"\\n\",\" \").replace(\"\\t\",\" \") + bsObj.find(class_='prd-attributesList ui-listBulleted js-short-description').getText(' ').strip()\n",
    "\n",
    "    result2 = re.sub(' +',' ',result.lower().strip().replace(\")\",\" \").replace(\"(\",\" \").replace(\"\\\"\",\"\\'\").replace(\"\\n\",\" \").replace(\"/\",\"or\"))\n",
    "    print('=========== data scraped! ============')\n",
    "    return result2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "final = []\n",
    "count = 1\n",
    "ttl_url = len(clean_links2)\n",
    "csv_name = './scraping_data/result_smartphone.csv'\n",
    "# ttl_url = len(clean_links2) - 62"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "====== Getting web browser ready ======\n",
      "1/216 Brows to https://www.lazada.com.my/korean-style-solid-spring-and-elastic-waist-wide-leg-pants-40956105.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "2/216 Brows to https://www.lazada.com.my/summer-icy-sleeves-sun-protection-gloves-for-men-and-women-anti-uv-thin-long-ice-silk-sun-protection-sleeve-sleeves-driving-arm-sleeve-black-thumb-models-90562797.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "3/216 Brows to https://www.lazada.com.my/kitchen-three-layer-multi-function-storage-rack-dish-rack-82660746.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "4/216 Brows to https://www.lazada.com.my/men-backpack-mens-bag-kangaroo-business-vertical-version-messengerbag-70401280.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "5/216 Brows to https://www.lazada.com.my/yoga-exercise-shoes-2017-new-style-korean-style-breathable-casual-flat-mesh-shoes-couple-sports-shoes-women-running-shoes-pink-color-73933767.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "6/216 Brows to https://www.lazada.com.my/couple-shoes-2017-new-style-korean-style-fashion-wild-casual-shoeshong-kong-style-breathable-running-shoes-street-shooting-sportsshoes-female-summer-black-81108164.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "7/216 Brows to https://www.lazada.com.my/stainless-steel-drying-rack-folding-outdoor-balcony-multifunctionhangers-towel-clothes-hanging-towel-small-hanging-shoes-rack-42699491.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "8/216 Brows to https://www.lazada.com.my/mm-stitching-chiffon-female-slimming-effect-lanterns-culottes-pants-collapse-40880248.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "9/216 Brows to https://www.lazada.com.my/students-stationery-jingle-cats-pencil-cute-cartoon-machine-catwooden-pencil-school-supplies-hb-8003-69412584.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "10/216 Brows to https://www.lazada.com.my/long-50-wide-30-kitchen-floor-shelving-rack-storage-microwave-rackdoes-not-rust-steel-color-multi-purpose-seasoning-rack-shelf-82654518.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "11/216 Brows to https://www.lazada.com.my/200-short-sleeved-summer-t-shirt-plus-sized-women39s-70204988.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "12/216 Brows to https://www.lazada.com.my/mm-spring-and-summer-sleeveless-plus-sized-slimming-effect-shoulder-strap-dress-denim-dress-57251311.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "13/216 Brows to https://www.lazada.com.my/cupboard-plastic-kitchen-drain-water-dish-rack-with-lid-chopstickscutlery-storage-box-put-dish-rack-dripping-dishes-glove-rack-82585136.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "14/216 Brows to https://www.lazada.com.my/bathroom-hand-washing-room-double-layer-toilet-storage-rack-shelf-98460754.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "15/216 Brows to https://www.lazada.com.my/spring-and-autumn-and-autumn-new-style-korean-style-breathablecanvas-shoes-white-shoes-flat-shoes-a-pedal-to-help-low-casualshoes-female-shoes-a8-white-88045164.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "16/216 Brows to https://www.lazada.com.my/jacquard-table-skirt-conference-table-skirt-wedding-table-skirts-welcome-area-hotel-restaurant-table-skirts-sign-tables-dress-92228006.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "17/216 Brows to https://www.lazada.com.my/2016-new-arrival-women39s-handbag-shoulder-bag-backpack-schoolbag-korean-version-of-the-trend-of-fashion-leather-travel-backpack-multifunctional-40025711.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "18/216 Brows to https://www.lazada.com.my/screw-nut-combination-disassembly-tool-sets-baby-boy-children-aged3-4-6-removable-toy-puzzle-hands-39916992.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "19/216 Brows to https://www.lazada.com.my/3m-4565-hooded-red-strip-piece-breathable-protective-clothing-workclothes-dust-chemical-warfare-suits-anti-liquid-spray-splash-88021684.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "20/216 Brows to https://www.lazada.com.my/double-bed-quilt-pillowcase-three-sheets-four-sets-of-bed-linen-53615046.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "21/216 Brows to https://www.lazada.com.my/air2ipad4pro9ipad56-apple-ipad-tablet-computer-liner-bag-41321185.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "22/216 Brows to https://www.lazada.com.my/fuk-wing-full-of-diamond-earrings-female-s925-silver-needleearrings-japan-and-south-korea-sweet-temperament-korean-simpleearrings-44991395.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "23/216 Brows to https://www.lazada.com.my/liu-tao-recommended-wash-towel-beauty-salon-cotton-a-cleansingtowel-non-woven-cloth-beauty-facial-tissue-75030757.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "24/216 Brows to https://www.lazada.com.my/summer-new-style-korean-style-mens-short-sleeved-shirt-tidepopular-mens-slim-fit-spell-color-casual-shirt-thin-section-toptide-63289798.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "25/216 Brows to https://www.lazada.com.my/2017-autumn-dress-korean-style-new-style-digital-male-children39s-clothing-children39s-fleece-hooded-hoodie-coat-wt-0857-73960106.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "26/216 Brows to https://www.lazada.com.my/floor-towel-simple-free-punch-rack-bath-towel-rack-47355495.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "27/216 Brows to https://www.lazada.com.my/a4-sticker-blank-not-sticky-laser-adhesive-stickers-printing-paper-40357391.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "28/216 Brows to https://www.lazada.com.my/kitchen-refrigerator-rack-side-wall-rack-kitchen-storage-rackshelving-rack-seasoning-material-rack-finishing-frame-65827-82682085.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "29/216 Brows to https://www.lazada.com.my/dusto-korean-style-summer-sandals-pregnant-female-student-softbottom-flat-with-non-slip-mom-women-shoes-dw16x8810a-87880035.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "30/216 Brows to https://www.lazada.com.my/color-diana-2017-autumn-and-winter-new-korean-version-of-womensbutterfly-knot-slim-was-thin-big-yards-long-sleeved-dress-bottomingdress-81115914.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "31/216 Brows to https://www.lazada.com.my/qzm-magnetic-sketchpad-can-lift-whiteboard-small-blackboard-children39s-double-sided-stand-home-painting-frame-writing-board-45694300.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "32/216 Brows to https://www.lazada.com.my/color-diana-new-style-slim-fit-sleeve-slimming-effect-solid-colordress-47375996.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "33/216 Brows to https://www.lazada.com.my/college-british-style-small-leather-shoes-2017-spring-and-autumn-new-style-wild-korean-style-music-fu-shoes-in-thick-with-single-shoes-female-tide-wine-red-color-81134384.html?ff=1&sc=IYsE\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "34/216 Brows to https://www.lazada.com.my/black-female-thin-section-summer-pencil-pants-bottoming-pants-black-black-40040450.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "35/216 Brows to https://www.lazada.com.my/cat-bath-cage-supplies-two-door-pet-wash-cat-cage-cat-hairinjections-anti-catch-bite-cat-cage-free-shipping-87222652.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "36/216 Brows to https://www.lazada.com.my/teapot-old-bamboo-pot-large-capacity-ore-famous-suit-58214305.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "37/216 Brows to https://www.lazada.com.my/lauva-3-way-cat-tunnel-toy-for-cats-kittens-puppy-bunny-90873553.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "38/216 Brows to https://www.lazada.com.my/han-feng-chic-top-female-wild-retro-hoodie-female-hong-kong-stylelong-sleeved-t-shirt-female-ulzzang-social-female-clothes-158-white-thin-section-81099863.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "39/216 Brows to https://www.lazada.com.my/shuangqing-single-layer-kitchen-soap-rack-soap-dish-110928486.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "40/216 Brows to https://www.lazada.com.my/evening-dresses-2016-summer-long-party-dress-big-size-dress-dress-45772383.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "41/216 Brows to https://www.lazada.com.my/adult-flight-chess-large-game-chess-crawling-pad-41780205.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "42/216 Brows to https://www.lazada.com.my/flat-with-pointed-hollow-shoes-hole-shoes-beige-beige-51192136.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "43/216 Brows to https://www.lazada.com.my/2017-korean-style-spring-and-summer-new-style-pointed-flat-shoeswomen-shoes-flat-with-shallow-mouth-four-seasons-shoes-simple-spellcolor-shoes-gray-73932102.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "44/216 Brows to https://www.lazada.com.my/cat-cage-double-three-layer-cat-villa-large-two-layer-foldingthrough-body-to-send-cat-climbing-frame-breeding-cat-cage-87133598.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "45/216 Brows to https://www.lazada.com.my/color-diana-new-style-temperament-slim-fit-waist-knit-dress-47375834.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "46/216 Brows to https://www.lazada.com.my/2017-spring-and-autumn-canvas-shoes-women-shoes-wild-street-shooting-white-shoes-spring-korean-style-casual-flat-student-shoes-a76-blue-88045189.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "47/216 Brows to https://www.lazada.com.my/korean-style-mini-bucket-bag-handbag-black-black-51245527.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "48/216 Brows to https://www.lazada.com.my/versatile-autumn-new-style-a-pedal-i-women39s-shoes-6622-black-88045169.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "49/216 Brows to https://www.lazada.com.my/childrens-toys-desktop-educational-wooden-beaded-beads-around-theframe-52509596.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "50/216 Brows to https://www.lazada.com.my/304-stainless-steel-kitchen-drain-water-dish-rack-hanging-put-dishes-plate-supplies-chopsticks-storage-box-glove-rack-double-cabinet-82616822.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "51/216 Brows to https://www.lazada.com.my/indoor-simple-clothes-rack-42630732.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "52/216 Brows to https://www.lazada.com.my/as-hong-crystal-original-stone-natural-calcite-ore-ornamentsthrough-the-body-specimens-stone-optical-research-ore-74576207.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "53/216 Brows to https://www.lazada.com.my/southern-copper-cover-copper-the-pig-iron-pot-cast-iron-teapot-104424139.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "54/216 Brows to https://www.lazada.com.my/cartoon-children39s-kindergarten-rubber-rub-67525170.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "55/216 Brows to https://www.lazada.com.my/girl-is-foot-piano-newborn-children-0-1-years-old-toys-50865617.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "56/216 Brows to https://www.lazada.com.my/bf-korean-style-new-style-loose-and-plus-sized-wild-denim-jacket-45579639.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "57/216 Brows to https://www.lazada.com.my/color-diana-new-style-sleeve-knit-bottoming-skirt-dress-43313672.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "58/216 Brows to https://www.lazada.com.my/non-toxic-wooden-bead-toys-for-baby-taobao-51055187.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "59/216 Brows to https://www.lazada.com.my/color-diana-2017-autumn-and-winter-new-womens-fashion-koreanversion-of-slim-was-thin-big-yards-bottoming-fashion-knitted-dress-43588619.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "60/216 Brows to https://www.lazada.com.my/korean-style-mens-short-sleeved-collar-short-sleeve-t-shirt-poloshirt-62736110.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "61/216 Brows to https://www.lazada.com.my/children-over-every-family-toys-can-be-cut-fruit-and-vegetableshonestly-happy-to-see-the-boy-girls-kitchen-toys-set-75261623.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "62/216 Brows to https://www.lazada.com.my/summer-collar-short-sleeved-denim-skirt-denim-dress-57250708.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "63/216 Brows to https://www.lazada.com.my/enlighten-yi-zhi-early-childhood-toys-beaded-bracelet-101791570.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "64/216 Brows to https://www.lazada.com.my/montessori-intelligence-childrens-toys-early-childhood-beat-taiwanhammer-box-64696833.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "65/216 Brows to https://www.lazada.com.my/shoes-female-2017-spring-and-autumn-new-style-korean-style-round-flat-lace-casual-sports-shoes-shoes-college-white-shoes-81101064.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "66/216 Brows to https://www.lazada.com.my/bathroom-toilet-hand-washing-room-washed-taiwan-wall-hangers-bathroom-shelf-98934772.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "67/216 Brows to https://www.lazada.com.my/color-diana-slim-fit-gray-grid-bottoming-fashion-wild-dress-47356206.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "68/216 Brows to https://www.lazada.com.my/new-style-temperament-v-neck-slim-fit-korean-style-long-sleeveddress-70394399.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "69/216 Brows to https://www.lazada.com.my/door-after-hook-free-nail-bold-hanging-clothes-rack-seamless-storage-bathroom-bedroom-towel-rack-hanging-clothes-hook-44926197.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "70/216 Brows to https://www.lazada.com.my/2017-summer-new-style-plus-sized-loose-short-sleeved-t-shirt-femaleround-neck-striped-top-wild-bottoming-shirt-female-small-effort579-red-striped-73932750.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "71/216 Brows to https://www.lazada.com.my/children39s-3-6-12-a-month-newborn-children-music-toys-baby-toys-fitness-frame-baby-0-1-year-old-fitness-device-52517767.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "72/216 Brows to https://www.lazada.com.my/i-the-world-people-aberdeen-minecraft-building-blocks-people-cooliecave-scene-dolls-doll-hand-to-do-toys-59001308.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "73/216 Brows to https://www.lazada.com.my/ms-new-style-shoulder-tide-wild-chest-pack-67291575.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "74/216 Brows to https://www.lazada.com.my/milk-tea-barrel-insulation-barrel-not-stainless-steel-commercialmilk-bucket-herbal-tea-barrel-juice-bucket-drink-faucet-doublelarge-capacity-90941948.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "75/216 Brows to https://www.lazada.com.my/home-folding-hanging-scarf-the-28-circle-single-scarf-rack-43458683.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "76/216 Brows to https://www.lazada.com.my/baby-striped-hoodie-2017-autumn-and-dress-new-style-boy-children39s-clothing-children39s-long-sleeved-jacket-zip-wt-7759-73956190.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "77/216 Brows to https://www.lazada.com.my/female-slim-fit-lace-top-t-shirt-5304-black-83111893.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "78/216 Brows to https://www.lazada.com.my/5060-blank-painting-frame-acrylic-painting-paint-practice-paintingbox-sail-cotton-50-60cm-oil-painting-frame-60157062.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "79/216 Brows to https://www.lazada.com.my/black-thin-section-spring-and-summer-outer-wear-pants-bottomingpants-62760849.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "80/216 Brows to https://www.lazada.com.my/2017-new-style-pointed-flat-shoes-female-shallow-mouth-shoes-to-help-low-casual-shoes-scoop-shoes-spring-and-autumn-minimalist-four-seasons-shoes-deep-brown-83235806.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "81/216 Brows to https://www.lazada.com.my/female-shoes-new-style-heavy-bottomed-student-breathable-lace-whiteshoes-platinum-platinum-51217816.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "82/216 Brows to https://www.lazada.com.my/stylish-knot-casual-watch-fashion-watch-68163228.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "83/216 Brows to https://www.lazada.com.my/children39s-sd01-stationery-cartoon-christmas-rubber-sleeve-pencil-67536617.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "84/216 Brows to https://www.lazada.com.my/frameless-painting-hanging-clocks-restaurant-crystal-living-roombedroom-mediterranean-decorative-painting-cartoon-paintings-meterbox-wall-painting-60439671.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "85/216 Brows to https://www.lazada.com.my/cartoon-nap-pillow-quilt-car-pillow-quilt-dual-cushions-were-largeair-conditioning-was-pillow-blanket-52628612.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "86/216 Brows to https://www.lazada.com.my/shuangqing-punched-wall-hangers-bathroom-storage-rack-sucker-towel-rack-98936076.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "87/216 Brows to https://www.lazada.com.my/color-diana-new-style-slim-fit-korean-style-solid-color-short-sleeved-chiffon-shirt-47380298.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "88/216 Brows to https://www.lazada.com.my/european-leg-new-style-print-sleeveless-slimming-effectkorean-style-a-word-dress-white-44072015.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "89/216 Brows to https://www.lazada.com.my/hyhan-yu-male-girl-childrens-wooden-made-educational-baby-1-2-3year-old-to-fight-to-mouse-building-blocks-toys-large-64687299.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "90/216 Brows to https://www.lazada.com.my/benro-accessories-tripod-accessories-head-quick-release-plate-screwconnection-14-to-38-screw-cover-screw-cap-85425131.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "91/216 Brows to https://www.lazada.com.my/rubber-rub-cute-cartoon-students-children39s-2b-rubber-rub-art-special-rubber-eraser-exam-stationery-prizes-wholesale-72217040.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "92/216 Brows to https://www.lazada.com.my/round-pot-with-glove-rack-floor-multi-layer-seasoning-rack-kitchentable-storage-rack-dishes-storage-rack-with-a-pulley-82686197.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "93/216 Brows to https://www.lazada.com.my/2017-spring-and-autumn-korean-style-sexy-female-high-heeled-shoes-fine-with-pointed-suede-diamond-shoes-black-professional-shoes-shoes-black-general-paragraph-73950090.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "94/216 Brows to https://www.lazada.com.my/baseus-iphone87-plus-apple-clip-charger-portable-phone-case-96648181.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "95/216 Brows to https://www.lazada.com.my/korean-style-cotton-loose-horn-short-sleeved-top-women39s-striped-t-shirt-44938904.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "96/216 Brows to https://www.lazada.com.my/modern-geometric-living-room-coffee-table-full-shop-big-black-andwhite-gray-mats-70897091.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "97/216 Brows to https://www.lazada.com.my/childrens-sketchpad-baby-painting-board-wood-magnetic-stand-51129009.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "98/216 Brows to https://www.lazada.com.my/southern-iron-pot-pig-iron-pot-cast-iron-pot-kung-fu-tea-101863517.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "99/216 Brows to https://www.lazada.com.my/as-hong-crystal-original-stone-natural-calcite-ball-ornaments-homefeng-shui-crystal-ball-ornaments-yellow-crystal-ball-74875511.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "100/216 Brows to https://www.lazada.com.my/solid-family-plastic-put-dish-rack-dishes-drain-rack-cupboard-kitchen-shelf-plate-dishes-storage-box-supplies-2-82623878.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "101/216 Brows to https://www.lazada.com.my/korean-spring-new-style-hit-color-cardigan-jacket-white-40906618.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "102/216 Brows to https://www.lazada.com.my/color-diana-new-style-off-the-shoulder-lotus-leaf-sleeve-tideshort-sleeved-chiffon-shirt-47371244.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "103/216 Brows to https://www.lazada.com.my/cartoon-small-yellow-people-pencil-childrens-primary-schoolkindergarten-prizes-gift-school-supplies-stationery-wholesale-hbpencil-69410497.html?ff=1&sc=IYsE\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "104/216 Brows to https://www.lazada.com.my/hino-300-gogo-van-hino-tail-plate-truck-model-64617148.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "105/216 Brows to https://www.lazada.com.my/color-diana-2017-autumn-and-winter-new-korean-women-slim-wildbottoming-long-sleeved-shirt-fashion-shirt-chiffon-shirt-pinkshort-sleeve-43439119.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "106/216 Brows to https://www.lazada.com.my/byl-12-hanging-clothes-hanger-coat-stand-storage-popheko-96044111.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "107/216 Brows to https://www.lazada.com.my/lucky-0780-natural-gold-feng-shui-home-accessories-furnishings-39645522.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "108/216 Brows to https://www.lazada.com.my/childrens-multi-function-educational-nut-car-50943615.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "109/216 Brows to https://www.lazada.com.my/yongjun-athletic-flying-stack-of-cup-tournament-cup-104499818.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "110/216 Brows to https://www.lazada.com.my/fiat-hot-wheels-wind-fire-wheel-boutique-car-51055617.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "111/216 Brows to https://www.lazada.com.my/art-linen-plus-sized-loose-long-dress-dress-navy-blue-navy-blue-47369169.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "112/216 Brows to https://www.lazada.com.my/metal-lens-hood-52mm-standard-straight-metal-hood-for-slr-41605917.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "113/216 Brows to https://www.lazada.com.my/name-is-glass-cup-whiskey-wine-glass-wine-glasses-suit-retro-whitewine-brandy-glass-87884990.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "114/216 Brows to https://www.lazada.com.my/kitchen-door-back-stainless-steel-towel-rack-40881157.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "115/216 Brows to https://www.lazada.com.my/korean-style-printed-slim-fit-slimming-dress-chiffon-skirt-97801288.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "116/216 Brows to https://www.lazada.com.my/multi-functional-clothes-rack-does-not-rust-steel-s-typemulti-layer-pants-rack-pants-hanger-closet-storage-pants-hangingpants-folder-pants-hanging-clothes-rack-52046221.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "117/216 Brows to https://www.lazada.com.my/rr9018-waterproof-motorcycle-car-back-seat-bag-rear-box-bag-rear-tail-box-91315282.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "118/216 Brows to https://www.lazada.com.my/baby-wooden-trailer-early-childhood-puzzle-fight-inserted-small-train-64531008.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "119/216 Brows to https://www.lazada.com.my/factory-direct-free-shipping-blue-automatic-double-hole-electricpencil-sharpener-students-dedicated-childrens-brush-to-pen-volumepen-knife-73724371.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "120/216 Brows to https://www.lazada.com.my/small-small-yi-sheng-xiang-children39s-doctor-toys-41876967.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "121/216 Brows to https://www.lazada.com.my/home-creative-seamless-hanger-clothes-child-support-hanging-clothes-rack-drying-racks-home-stays-hanging-clothes-rack-clothing-62026770.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "122/216 Brows to https://www.lazada.com.my/korea-stationery-wooden-small-blackboard-drawer-pen-fashion-cutecreative-office-desktop-storage-to-send-chalk-eraser-42640915.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "123/216 Brows to https://www.lazada.com.my/multi-function-slr-decompression-camera-strap-can-be-hangingshoulder-back-photography-bag-with-fast-photo-hand-camera-fast-hangstrap-85414944.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "124/216 Brows to https://www.lazada.com.my/qiudong-plus-velvet-winter-style-children39s-shirt-green-64514273.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "125/216 Brows to https://www.lazada.com.my/micro-flaws-products-processing-beginner39s-yoga-mat-girl-children39s-dance-practice-mat-fitness-mat-yoga-mat-carpet-73726139.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "126/216 Brows to https://www.lazada.com.my/old-beijing-shoes-shoes-shoes-korean-style-wild-female-flat-studentcasual-lace-sports-shoes-women-running-shoes-916-black-88045205.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "127/216 Brows to https://www.lazada.com.my/spring-new-korean-suspenders-denim-overalls-feet-casual-trousers-pants-female-college-students-wind-was-thin-piece-pants-40876781.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "128/216 Brows to https://www.lazada.com.my/korean-style-waterproof-large-wash-bag-makeup-bag-mary-kay-largecapacity-portable-cosmetic-bag-storage-bag-makeup-bag-87112447.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "129/216 Brows to https://www.lazada.com.my/baby-home-plastic-wind-towel-air-dry-rack-shelf-63324178.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "130/216 Brows to https://www.lazada.com.my/2017-autumn-new-style-korean-style-thick-with-pointed-high-heeled-shoes-a-word-buckle-pearl-shoes-shallow-mouth-suede-shoes-female-beige-83242966.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "131/216 Brows to https://www.lazada.com.my/korean-style-female-summer-new-white-top-chiffon-shirt-57361749.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "132/216 Brows to https://www.lazada.com.my/english-color-word-card-english-flash-cards-large-card-wild-animalclass-28-zhang-english-children-teacher-teaching-41901266.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "133/216 Brows to https://www.lazada.com.my/korean-style-autumn-young-slim-fit-jacket-coat-solid-color-black-solid-color-black-73943203.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "134/216 Brows to https://www.lazada.com.my/original-sd-to-cf-card-sets-support-wireless-wifi-sd-card-typei-type-adapter-is-slr-camera-card-88016424.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "135/216 Brows to https://www.lazada.com.my/cat-cage-three-layer-double-four-layer-cat-villa-large-foldingencryption-cat-cage-small-cat-long-cat-pet-cage-87141296.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "136/216 Brows to https://www.lazada.com.my/pet-supplies-pet-nest-dogs-and-cats-house-nest-small-pet-nest-canbe-cleaning-quality-good-price-low-a-piece-free-shipping-87180011.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "137/216 Brows to https://www.lazada.com.my/adult-feather-velvet-health-care-single-to-shoot-2-care-cervicalpillow-62143149.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "138/216 Brows to https://www.lazada.com.my/cartoon-big-packs-children39s-rubber-rub-rubber-72223835.html?ff=1&sc=IYsE\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "139/216 Brows to https://www.lazada.com.my/storage-rack-long-75-wide-40-kitchen-shelf-floor-microwave-ovenstorage-rack-stainless-steel-color-kitchen-shelf-82663172.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "140/216 Brows to https://www.lazada.com.my/retro-female-long-sleeved-women-heattech-top-122-more-casual-96055217.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "141/216 Brows to https://www.lazada.com.my/simple-solid-color-bedroom-living-room-blackout-curtains-half-curtain-58168476.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "142/216 Brows to https://www.lazada.com.my/baby-stitching-t-shirt-2017-spring-korean-style-new-style-boy-children39s-clothing-children39s-casual-long-sleeved-shirt-bottoming-tx-7600-navy-blue-47350259.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "143/216 Brows to https://www.lazada.com.my/c-nex-adapter-ring-movie-lens-cctv-lens-to-sony-nex6nex5ra5000-41315018.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "144/216 Brows to https://www.lazada.com.my/week-notepad-business-work-day-notes-cute-message-stickers-70356407.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "145/216 Brows to https://www.lazada.com.my/fall-fashion-shoes-new-style-casual-shoes-the-new-d-2-black-51227265.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "146/216 Brows to https://www.lazada.com.my/8-hook-kitchen-cabinet-bottom-with-rack-free-nail-wardrobe-hangerskitchen-creative-storage-bearing-hook-shelf-rack-hook-82671467.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "147/216 Brows to https://www.lazada.com.my/art-linen-loose-casual-pants-trousers-white-white-43643815.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "148/216 Brows to https://www.lazada.com.my/kelly-korean-style-canvas-female-new-style-messenger-bag-theme-small-98341132.html?ff=1&sc=IYsE\n",
      "Loading took too much time!\n",
      "====== Getting web browser ready ======\n",
      "149/216 Brows to https://www.lazada.com.my/shelving-rack-microwave-oven-rack-storage-rack-shelf-kitchensupplies-room-90687507.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "150/216 Brows to https://www.lazada.com.my/multi-function-wrought-iron-pot-rack-multi-shelf-put-lid-the-rack-kitchen-cutting-board-rack-cutting-board-storage-rack-81115568.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "151/216 Brows to https://www.lazada.com.my/dell-mouse-notebook-desktop-computer-wired-mouse-usb-game-officemouse-ms111-87161168.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "152/216 Brows to https://www.lazada.com.my/yuan-benseo-basique-2017-spring-new-style-bottoming-shirt-menshigh-collar-hedging-long-sleeved-t-shirt-modal-tide-white-45213485.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "153/216 Brows to https://www.lazada.com.my/victory-victor-sword-full-carbon-light-single-badminton-racket-46263066.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "154/216 Brows to https://www.lazada.com.my/diana-korean-style-color-slim-fit-long-sleeved-small-suit-jacket-black-black-63309620.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "155/216 Brows to https://www.lazada.com.my/bright-chinese-gold-bride-wedding-toast-clothing-wedding-dress-63316816.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "156/216 Brows to https://www.lazada.com.my/kitchen-is-boil-pill-pot-meatballs-shrimp-balls-processing-spoon-meat-39677925.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "157/216 Brows to https://www.lazada.com.my/sports-shoes-female-korean-style-ulzzang-harajuku-wild-new-style-breathable-mesh-student-casual-running-travel-shoes-pink-color-73937928.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "158/216 Brows to https://www.lazada.com.my/kitchen-fresh-film-rack-cabinet-roll-holder-towel-rack-cloth-hookjapanese-style-bathroom-wardrobe-storage-rack-91092674.html?ff=1&sc=IYsE\n",
      "Loading took too much time!\n",
      "====== Getting web browser ready ======\n",
      "159/216 Brows to https://www.lazada.com.my/fuji-instax-diy-frame-stickerscartoon-frame-stickers-li-beat-waspaper-the-stickers-41651552.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "160/216 Brows to https://www.lazada.com.my/cotton-old-coarse-plaid-bed-alone-double-student-sheets-singlepiece-dormitory-1518-75325526.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "161/216 Brows to https://www.lazada.com.my/children39s-early-childhood-cognitive-knowledge-map-material-26-english-letters-card-baby-enlighten-yi-zhi-learning-toys-1-3-year-old-41820374.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "162/216 Brows to https://www.lazada.com.my/children-math-early-childhood-writing-board-box-52275014.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "163/216 Brows to https://www.lazada.com.my/stainless-steel-can-be-retractable-under-the-sink-shelf-microwaveoven-kitchen-shelf-storage-rack-shoe-rack-pot-rack-shelf-91011170.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "164/216 Brows to https://www.lazada.com.my/caught-cabinet-drawer-storage-cabinets-plastic-gap-narrow-cabinetfinishing-storage-cabinet-caught-storage-cabinet-gap-cabinet-82600016.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "165/216 Brows to https://www.lazada.com.my/female-tide-new-style-summer-student-rivet-shoulder-bag-70422883.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "166/216 Brows to https://www.lazada.com.my/handbag-2017-new-style-korean-style-tide-wild-shoulder-bag-fashion-handbag-flow-su-xiaofang-bag-messenger-bag-small-bag-black-45946923.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "167/216 Brows to https://www.lazada.com.my/teaching-with-thirteen-hole-early-childhood-shape-educational-toysbuilding-blocks-59116082.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "168/216 Brows to https://www.lazada.com.my/lr-old-beijing-breathable-mesh-shoes-men39s-casual-shoes-gray-gray-66689505.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "169/216 Brows to https://www.lazada.com.my/leather-snowy-moon-cake-hand-pressure-home-mooncake-mold-101842296.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "170/216 Brows to https://www.lazada.com.my/autumn-dress-female-2017-new-style-top-long-sleeved-t-shirt-femalegauze-letter-print-hollow-chic-bottoming-shirt-female-outer-wear158-white-81108709.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "171/216 Brows to https://www.lazada.com.my/us-life-large-explosion-bear-16-m-plush-toy-teddy-bear-doll-large-bear-hug-birthday-gift-64676804.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "172/216 Brows to https://www.lazada.com.my/foreign-trade-clothing-store-non-slip-straps-seamless-color-hanger-43153629.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "173/216 Brows to https://www.lazada.com.my/simple-in-the-flat-with-heavy-bottomed-sandals-white-white-51223802.html?ff=1&sc=IYsE\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "174/216 Brows to https://www.lazada.com.my/10-dress-wood-slip-hanger-seamless-wood-hangers-wooden-hotelhanger-retro-clothing-store-hanger-garment-support-62170180.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "175/216 Brows to https://www.lazada.com.my/spring-and-autumn-men-and-women-hotel-seven-points-long-sleeveddouble-sided-cotton-towel-material-long-plus-sized-robe-bathrobe-79577938.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "176/216 Brows to https://www.lazada.com.my/korean-style-spring-new-loose-white-shirt-40906501.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "177/216 Brows to https://www.lazada.com.my/applicable-new-style-fujifilm-xt10-x100-x100st-x10-x20-xa2-microsingle-camera-shoulder-87111753.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "178/216 Brows to https://www.lazada.com.my/english-enlighten-color-word-card-english-flash-cards-large-cardfruit-class-24-zhang-english-children-teacher-teaching-41905422.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "179/216 Brows to https://www.lazada.com.my/gelert-camping-quick-drying-outdoor-comes-with-storage-bag-39851616.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "180/216 Brows to https://www.lazada.com.my/bathroom-bathroom-shelf-desktop-makeup-products-storage-rack-wash-station-supplies-finishing-rack-drawer-storage-box-81104356.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "181/216 Brows to https://www.lazada.com.my/waterproof-environmentally-friendly-odorless-toilet-rack-bathroom-shelf-98933524.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "182/216 Brows to https://www.lazada.com.my/wooden-clock-wooden-pairing-toys-digital-shape-building-blocks-39917625.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "183/216 Brows to https://www.lazada.com.my/hot-wheels-small-sports-car-59011703.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "184/216 Brows to https://www.lazada.com.my/wind-tour-tent-outdoor-3-4-people-full-automatic-two-room-a-hall-2people-outdoor-camping-beach-tent-set-73795826.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "185/216 Brows to https://www.lazada.com.my/multi-layer-stainless-steel-home-s-shaped-pants-folder-pants-rack-62021778.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "186/216 Brows to https://www.lazada.com.my/teddy-bear-doll-plush-toy-birthday-gift-pillow-64662515.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "187/216 Brows to https://www.lazada.com.my/female-handbag-tide-oxford-cloth-rivet-nylon-canvas-shoulder-bag-45806230.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "188/216 Brows to https://www.lazada.com.my/blue-lily-storage-basket-iron-basket-egg-basket-potato-garliccontainer-ornaments-egg-hollow-container-basket-77479762.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "189/216 Brows to https://www.lazada.com.my/xd-adult-split-electric-car-outdoor-oxford-cloth-raincoat-black-39823086.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "190/216 Brows to https://www.lazada.com.my/color-diana-slim-fit-short-sleeved-chiffon-shirt-shirt-bottoming-shirt-sky-blue-sky-blue-47356519.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "191/216 Brows to https://www.lazada.com.my/creative-half-transparent-hanging-hanging-kitchen-storage-bag-58996778.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "192/216 Brows to https://www.lazada.com.my/zhu-yun-usb-charging-student-dormitory-portable-with-a-clip-fanbaby-car-small-fan-bed-dual-52039358.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "193/216 Brows to https://www.lazada.com.my/canon-eos-m-2-m3-eos-m10-g7x-g15-g16-camera-bag-leather-87122899.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "194/216 Brows to https://www.lazada.com.my/2017-autumn-dress-korean-ulzzang-harajuku-bf-wind-loose-stripedstitching-long-sleeved-top-t-shirt-student-hoodie-women-81138015.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "195/216 Brows to https://www.lazada.com.my/kitchen-can-be-superimposed-wrought-iron-shelf-wardrobe-storage-rack-layered-storage-rack-dormitory-cabinet-hanging-basket-finishing-rack-81099375.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "196/216 Brows to https://www.lazada.com.my/snow-australia-swimming-pool-electric-pump-pumping-suction-pumpinflatable-pump-52635905.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "197/216 Brows to https://www.lazada.com.my/spring-and-autumn-and-autumn-new-style-korean-style-pointedbutterfly-knot-gray-suede-shallow-mouth-in-with-thick-with-singleshoes-women-high-heeled-shoes-blue-nude-models-83109566.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "198/216 Brows to https://www.lazada.com.my/commuter-new-style-picture-bucket-bag-black-black-47377707.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "199/216 Brows to https://www.lazada.com.my/children39s-educational-toys-wooden-mini-small-around-the-bead-cute-enlighten-baby-toys-1-2-year-old-wood-base-41797278.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "200/216 Brows to https://www.lazada.com.my/2017-new-style-tide-sandals-female-summer-flat-korean-students-wildkorean-style-cute-retro-muffin-bottom-in-with-shoes-sky-bluecolor-67354225.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "201/216 Brows to https://www.lazada.com.my/stainless-steel-bathroom-paper-towel-holder-kitchen-towel-rack-98936959.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "202/216 Brows to https://www.lazada.com.my/storage-rack-long-100-wide-50-shelving-rack-living-room-floor-shelf4-layer-5-layer-6-layer-finishing-rack-not-stainless-steel-colorshelf-82584509.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "203/216 Brows to https://www.lazada.com.my/gongting-european-home-luxury-tea-cup-bone-china-coffee-cup-96699560.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "204/216 Brows to https://www.lazada.com.my/pearl-pointed-flat-shoes-shoes-silver-silver-73953039.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "205/216 Brows to https://www.lazada.com.my/7-6-summer-models-korean-style-fashion-wild-off-the-shoulderhedging-chiffon-shirt-womens-stylish-slim-fit-top-brown-57240745.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "206/216 Brows to https://www.lazada.com.my/korean-models-cute-cartoon-spider-man-pencil-spider-man-pencil-hbstudents-wooden-pencil-gift-69534595.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "207/216 Brows to https://www.lazada.com.my/children-learn-painting-tools-baby-graffiti-coloring-fill-colordrawing-painting-template-suit-children-educational-toys-50962415.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "208/216 Brows to https://www.lazada.com.my/autumn-cool-oblique-collar-long-sleeved-female-t-shirt-loose-highschool-students-striped-stitching-top-han-fan-wild-bottoming-shirttide-73953557.html?ff=1&sc=IYsE\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "209/216 Brows to https://www.lazada.com.my/european-style-wrought-iron-wedding-road-lead-stage-ornaments-candlestick-lantern-60173834.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "210/216 Brows to https://www.lazada.com.my/security-sanitation-traffic-rain-pants-reflective-raincoat-navyblue-suit-52002586.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "211/216 Brows to https://www.lazada.com.my/color-diana-2017-autumn-and-winter-new-korean-long-sleeved-knitslim-bottoming-waist-was-thin-package-hip-flounced-dress-81103149.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "212/216 Brows to https://www.lazada.com.my/shuangqing-wall-hangers-toothbrush-hanging-box-teeth-with-seat-toothbrush-rack-110927418.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "213/216 Brows to https://www.lazada.com.my/cartoon-cute-small-cat-kitchen-long-carpet-non-slip-bedroom-bed-front-floor-mats-absorbent-non-slip-foot-mat-77541223.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "214/216 Brows to https://www.lazada.com.my/storage-rack-long-60-wide-45-kitchen-floor-storage-rack-microwaveoven-rack-stainless-steel-color-3-layer-4-layer-finishing-rack-potrack-82611607.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "215/216 Brows to https://www.lazada.com.my/early-childhood-word-card-daily-behavior-27-zhang-yingwen-enlightenteacher-teaching-with-flash-card-waterproof-and-durable-childrens-41835432.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n",
      "====== Getting web browser ready ======\n",
      "216/216 Brows to https://www.lazada.com.my/lamaze-multi-function-music-bb-device-ring-paper-caterpillar-64579216.html?ff=1&sc=IYsE\n",
      "=========== data scraped! ============\n"
     ]
    }
   ],
   "source": [
    "for url in clean_links2:\n",
    "    print('====== Getting web browser ready ======')\n",
    "#     browser = webdriver.Chrome('./engine/chromedriver.exe')\n",
    "    browser = webdriver.PhantomJS('./engine/phantomjs.exe')\n",
    "    browser.get(url)\n",
    "    counter = str(count)+'/'+str(ttl_url)\n",
    "    print(counter,'Brows to',url)\n",
    "    delay = 3 # seconds\n",
    "    count +=1\n",
    "    \n",
    "    try:\n",
    "        link3 = browser.find_element_by_class_name(\"delivery-option-st__label\") #this for removing popup on oversea delivery\n",
    "        ActionChains(browser).move_to_element(link3).perform()\n",
    "        link3.click()\n",
    "        \n",
    "#         try:\n",
    "#             myElem = WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.CLASS_NAME, 'product-description__block-expand-button')))\n",
    "\n",
    "        try:\n",
    "            link = browser.find_element_by_class_name(\"product-description__block-expand-button\")\n",
    "            ActionChains(browser).move_to_element(link).perform()\n",
    "            link.click()\n",
    "\n",
    "            try:\n",
    "                link2 = browser.find_element_by_class_name(\"more-desc-button\")\n",
    "                ActionChains(browser).move_to_element(link2).perform()\n",
    "                link2.click()\n",
    "\n",
    "                final.append(gettingResult(browser.page_source))\n",
    "                browser.close()\n",
    "            except:\n",
    "                final.append(gettingResult(browser.page_source))\n",
    "                browser.close()\n",
    "\n",
    "        except:\n",
    "            try:\n",
    "                link2 = browser.find_element_by_class_name(\"more-desc-button\")\n",
    "                ActionChains(browser).move_to_element(link2).perform()\n",
    "                link2.click()\n",
    "\n",
    "                final.append(gettingResult(browser.page_source))\n",
    "                browser.close()\n",
    "\n",
    "            except:\n",
    "                final.append(gettingResult(browser.page_source))\n",
    "                browser.close()\n",
    "\n",
    "#         except TimeoutException:\n",
    "#             print (\"Loading took too much time!\")\n",
    "    except:\n",
    "        try:\n",
    "            myElem = WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.CLASS_NAME, 'product-description__block-expand-button')))\n",
    "            \n",
    "            try:\n",
    "                link = browser.find_element_by_class_name(\"product-description__block-expand-button\")\n",
    "                ActionChains(browser).move_to_element(link).perform()\n",
    "                link.click()\n",
    "                \n",
    "                try:\n",
    "                    link2 = browser.find_element_by_class_name(\"more-desc-button\")\n",
    "                    ActionChains(browser).move_to_element(link2).perform()\n",
    "                    link2.click()\n",
    "                    \n",
    "                    final.append(gettingResult(browser.page_source))\n",
    "                    browser.close()\n",
    "                except:\n",
    "                    final.append(gettingResult(browser.page_source))\n",
    "                    browser.close()\n",
    "\n",
    "            except:\n",
    "                try:\n",
    "                    link2 = browser.find_element_by_class_name(\"more-desc-button\")\n",
    "                    ActionChains(browser).move_to_element(link2).perform()\n",
    "                    link2.click()\n",
    "                    \n",
    "                    final.append(gettingResult(browser.page_source))\n",
    "                    browser.close()\n",
    "                    \n",
    "                except:\n",
    "                    final.append(gettingResult(browser.page_source))\n",
    "                    browser.close()\n",
    "                \n",
    "            \n",
    "#             final.append(gettingResult(browser.page_source))\n",
    "#             browser.close()\n",
    "        except TimeoutException:\n",
    "            print (\"Loading took too much time!\")\n",
    "            browser.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "215"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(final)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Push the list into csv file\n",
    "with open(csv_name, \"w\", encoding='utf-8') as output:\n",
    "    writer = csv.writer(output, lineterminator='\\n')\n",
    "    for val in final:\n",
    "        writer.writerow([val])  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}