Robust_extraction_of_web_data_with_Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Who am I?\n",
    "- Slawomir Tulski (Slaw)\n",
    "- currently: Big Data Engineer at WorldRemit\n",
    "- previously: Python Data Programmer at Import.io (web scraping start-up)\n",
    "- linkedin: https://www.linkedin.com/in/slawomir-tulski-091611116/\n",
    "- personal website: http://slawomirtulski.com/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### My goals for today\n",
    "- show how to tackle problem of web scraping in different ways than \"standard\" approach\n",
    "- present useful tips and tricks in web-scraping \n",
    "- avoid making tutorial on popular html parsing / scraping libraries\n",
    "\n",
    "### Plan\n",
    "- Quick intoduction to scraping\n",
    "- Stop crawling, investigate your target instead\n",
    "    + case study 1: getting all urls you need from website \n",
    "- Look for APIs, even if service does not provide (public) one\n",
    "    + case study 2: getting API KEY and using hidden API in store locator service\n",
    "    + case study 3: getting available airbnb properties in London\n",
    "    + case study 4: api json response embeded in html\n",
    "- Handling JavaScript with Selenium\n",
    "    + case study 5: handling infinite scroll\n",
    "- Keynotes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Some basic...\n",
    "\n",
    "### How your browser works?\n",
    "- World Wide Web operates on a client/server model\n",
    "- Web browser contacts a web server and requests information or resources\n",
    "- Server locates and then sends the information (html, images etc.) back to the web browser \n",
    "- Browser displays the results\n",
    "- Browser can execute JavaScript code to dynmically \"do things\" (sends requests, site appreance and bassicaly everyting)\n",
    "- 4 basic types of http requests (GET and POST - you'll use those most often while scraping, PUT, DELETE)\n",
    "\n",
    "### How to see what my browser is doing?\n",
    "- web browsers usually have some sort of \"Developers Toolkit\" (if not you should think about changing your browser)\n",
    "- there should be 'Network' tab which shows you what is being sent from/to your broweser/server\n",
    "- you can check exactly what type of request were sent, headers, parameters, cookies etc.\n",
    "- also you can find in your Developers Tools console to execute JavaScript\n",
    "\n",
    "### \"Standard\" scraping approach\n",
    "0. I want DATA!!!!\n",
    "1. don't scrape... find data somewhere else!\n",
    "2. don't scrape... they should provide an API!\n",
    "3. ok.. you're screwed. get HTML and parse it!\n",
    "4. you need a lot of data from different pages of one web service? - build crawler and \"catch them all\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using sitemaps instead of crawling whole website\n",
    "\n",
    "### what is web \"crawler\" ?\n",
    "* automate bot which recurse from strat page to all internal link it founds\n",
    "* theoretacaly, it will traverse through all urls on website\n",
    "\n",
    "### why it's not the best idea?\n",
    "* not precise (it's brute force... a lot of requests made and a lot of garbage scraped)\n",
    "* need to write more code and care about lot of things (what type of url it got, can I go there?)\n",
    "* assumes particual page layout and test whatever it encounter\n",
    "* easy to catch into trap (honeypots)\n",
    "\n",
    "### what to use instead?\n",
    "* very often there is sitemap of whole website already available!\n",
    "* very often sitemaps are hidden! if you can't see it on page, try **/sitemap.xml** [https://www.skipthedishes.com/]\n",
    "* also, information about sitmap can be found in **robots.txt** file [https://www.walmart.com/]\n",
    "* if there is no sitemap, try to follow a pattern **get categories -> get pages -> get listing -> get item**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# built-in\n",
    "import json\n",
    "import random\n",
    "import re\n",
    "import time\n",
    "# 3rd part\n",
    "from IPython.display import HTML\n",
    "import pandas as pd\n",
    "import requests\n",
    "from selenium import webdriver"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "getting properties from:  http://www.rightmove.co.uk/sitemap_propertydetails0.xml\n",
      "getting properties from:  http://www.rightmove.co.uk/sitemap_propertydetails1.xml\n",
      "getting properties from:  http://www.rightmove.co.uk/sitemap_propertydetails2.xml\n",
      "I've got 150000 of urls with properties.\n",
      "Some examples:\n",
      "\n",
      "- http://www.rightmove.co.uk/property-to-rent/property-50480715.html\n",
      "\n",
      "- http://www.rightmove.co.uk/property-to-rent/property-53775521.html\n",
      "\n",
      "- http://www.rightmove.co.uk/commercial-property-for-sale/property-64919567.html\n",
      "\n",
      "- http://www.rightmove.co.uk/property-to-rent/property-68904185.html\n",
      "\n",
      "- http://www.rightmove.co.uk/commercial-property-to-let/property-47279781.html\n",
      "\n",
      "- http://www.rightmove.co.uk/property-to-rent/property-61726357.html\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "Case Study 1: getting all links from sitemap \n",
    "\n",
    "You want to analyse housing market in UK. Data which interest you most are on http://www.rightmove.co.uk/.\n",
    "Unfortunately, there is no API available and you need get data from HTMLs. As a first step, before putting your hands\n",
    "on data, you need to know urls of all avaiable properties on website. Later, you will use those links to extract data.\n",
    "\n",
    "Find all urls to properties on rightmove.co.uk. Be as precise as possible. Do not built inefficient crawlers.\n",
    "\"\"\"\n",
    "main_sitemap_url = 'http://www.rightmove.co.uk/sitemap.xml'\n",
    "main_sitemap_text = requests.get(main_sitemap_url).text\n",
    "properties_sitemaps = re.findall(r'<loc>(http://www.rightmove.co.uk/sitemap_propertydetails\\d+.xml)</loc>', main_sitemap_text)\n",
    "limit_pages = 3\n",
    "all_properites = []\n",
    "for pmap_url in properties_sitemaps[:limit_pages]:\n",
    "    print('getting properties from: ', pmap_url)\n",
    "    pmap_text = requests.get(pmap_url).text\n",
    "    p_urls = re.findall(r'<loc>(http://www.rightmove.co.uk/[\\-a-z]+/property-\\d+.html)</loc>', pmap_text)\n",
    "    all_properites.extend(p_urls)\n",
    "print('I\\'ve got ' + str(len(all_properites)) + ' of urls with properties.\\nSome examples:')\n",
    "for url in all_properites[:6]:\n",
    "      print('\\n- '+url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Look for APIs - even if service does not provide (public) one\n",
    "\n",
    "### why APIs are better (I know... silly question)\n",
    "* web appearance can change frequently (which will brake scrapers dependant on html tags), but API stays same for longer time\n",
    "* often, responses from API contains very structured data (e.g. in JSON or XML format)\n",
    "\n",
    "### but there is no API available for website 'X' ;(\n",
    "* a lot of modern web services uses some kind of APIs internally [https://www.airbnb.co.uk/s/London/homes]\n",
    "* to find out if web service is using API track network in your developer’s tools. (I like Chrome’s tools, but Firefox, Opera etc. also has nice ones)\n",
    "* there are some treasures hidden in requests with type xhr, fetch, json etc.\n",
    "* often, you need to supply additional information with your request (like API keys or tokens)\n",
    "* API responses can be dynamically embeded in HTML [https://www.walmart.com/]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Got API KEY from main page:  41C97F66-D0FF-11DD-8143-EF6F37ABAA09\n",
      "Raw response from API:  {'response': {'collectioncount': 16, 'attributes': {'country': 'US', 'province': '', 'postalcode': '20004', 'city': 'WASHINGTON', 'radiusuom': 'mile', 'radius': '40', 'state': 'DC', 'address': '', 'centerpoint': '-77.0255,38.8957'}, 'activeobject': '', 'collection': [{'fri_open_time': '8:00 AM', 'giftcard': '1', 'icon': 'default', 'tvadv': None, 'cs': None, 'csurl': None, 'phone': '(202) 462-3146', 'email': 'truevalue17@truevalue.net', 'google_notes': None, 'tv': '1', 'corronado': None, 'thur_close_time': '- 7:30 PM', 'google_shared': None, 'uid': 1051076976, 'address2': None, '_distanceuom': 'mile', 'province': None, 'fri_close_time': '- 7:30 PM', 'yelp': None, 'sun_open_time': '10:00 AM', 'tvurl': 'http://www.truevalueon17th.com/', 'sun_close_time': '- 6:00 PM', 'tue_open_time': '8:00 AM', 'tvpaint': '1', '_distance': '1.32', 'wed_close_time': '- 7:30 PM', 'mon_open_time': '8:00 AM', 'google_email': None, 'ds': None, 'hgurl': None, 'localad': None, 'mon_close_time': '- 7:30 PM', 'thur_open_time': '8:00 AM', 'facebook': None, 'tvadvurl': None, 'tue_close_time': '- 7:30 PM', 'latitude': '38.911917', 'activeshiptostore': '1', 'bho': '[[\"1000\",\"1800\"],[\"0800\",\"1930\"],[\"0800\",\"1930\"],[\"0800\",\"1930\"],[\"0800\",\"1930\"],[\"0800\",\"1930\"],[\"0900\",\"1800\"]]', 'trurl': None, 'facebookurl': None, 'longitude': '-77.03849', 'ja': None, 'country': 'US', 'postalcode': '20009-2433', 'clientkey': 'L4ZK7Q8W-PA4X-4IS6-587Y-FRJLJ8Z5JF84', 'city': 'Washington', 'tvr': '1', 'dsurl': None, 'grurl': None, 'taylorrental': None, 'gr': None, 'url': 'http://www.truevalueon17th.com', 'sat_open_time': '9:00 AM', 'twitterurl': None, 'wed_open_time': '8:00 AM', 'google': None, 'main_id': 'TV', 'hg': None, 'creditcard': None, 'fax': None, 'foursquare': None, 'name': 'True Value On 17th', 'address1': '1623 17th St NW', 'sat_close_time': '- 6:00 PM', 'jaurl': None, 'state': 'DC'}, {'fri_open_time': '9:00 AM', 'giftcard': None, 'icon': 'default', 'tvadv': None, 'cs': None, 'csurl': None, 'phone': '(202) 659-8686', 'email': 'info@districthardware.com', 'google_notes': None, 'tv': '1', 'corronado': None, 'thur_close_time': '- 7:00 PM', 'google_shared': None, 'uid': 1051078615, 'address2': None, '_distanceuom': 'mile', 'province': None, 'fri_close_time': '- 6:00 PM', 'yelp': None, 'sun_open_time': '11:00 AM', 'tvurl': 'http://www.thebikeshopdc.com', 'sun_close_time': '- 5:00 PM', 'tue_open_time': '9:00 AM', 'tvpaint': None, '_distance': '1.50', 'wed_close_time': '- 7:00 PM', 'mon_open_time': '9:00 AM', 'google_email': 'social@districthardware.com', 'ds': None, 'hgurl': None, 'localad': None, 'mon_close_time': '- 7:00 PM', 'thur_open_time': '9:00 AM', 'facebook': None, 'tvadvurl': None, 'tue_close_time': '- 7:00 PM', 'latitude': '38.9038748979592', 'activeshiptostore': '1', 'bho': '[[\"1100\",\"1700\"],[\"0900\",\"1900\"],[\"0900\",\"1900\"],[\"0900\",\"1900\"],[\"0900\",\"1900\"],[\"0900\",\"1800\"],[\"0900\",\"1800\"]]', 'trurl': None, 'facebookurl': 'http://www.facebook.com/districthardwareandbike', 'longitude': '-77.0514083673469', 'ja': None, 'country': 'US', 'postalcode': '20037-1432', 'clientkey': '2VMQMTVK-8O6O-8EUD-BA78-S3N6DFM1QESS', 'city': 'Washington', 'tvr': None, 'dsurl': None, 'grurl': None, 'taylorrental': None, 'gr': None, 'url': 'http://www.districthardware.com', 'sat_open_time': '9:00 AM', 'twitterurl': 'http://www.twitter.com/dchardwarebike', 'wed_open_time': '9:00 AM', 'google': '1', 'main_id': 'TV', 'hg': None, 'creditcard': None, 'fax': None, 'foursquare': '1', 'name': 'District Hardware and Bike', 'address1': '1108 24th Street NW', 'sat_close_time': '- 6:00 PM', 'jaurl': None, 'state': 'DC'}, {'fri_open_time': '5:00 AM', 'giftcard': None, 'icon': 'default', 'tvadv': None, 'cs': None, 'csurl': None, 'phone': '(202) 636-1701', 'email': None, 'google_notes': None, 'tv': '1', 'corronado': None, 'thur_close_time': '- 4:00 PM', 'google_shared': 'Yes', 'uid': 1182202005, 'address2': None, '_distanceuom': 'mile', 'province': None, 'fri_close_time': '- 4:00 PM', 'yelp': None, 'sun_open_time': 'closed', 'tvurl': None, 'sun_close_time': None, 'tue_open_time': '5:00 AM', 'tvpaint': None, '_distance': '2.79', 'wed_close_time': '- 4:00 PM', 'mon_open_time': '5:00 AM', 'google_email': 'tom@websolutions.net', 'ds': None, 'hgurl': None, 'localad': None, 'mon_close_time': '- 4:00 PM', 'thur_open_time': '5:00 AM', 'facebook': None, 'tvadvurl': None, 'tue_close_time': '- 4:00 PM', 'latitude': '38.91536', 'activeshiptostore': None, 'bho': '[[\"9999\",\"9999\"],[\"0500\",\"1600\"],[\"0500\",\"1600\"],[\"0500\",\"1600\"],[\"0500\",\"1600\"],[\"0500\",\"1600\"],[\"0700\",\"1100\"]]', 'trurl': None, 'facebookurl': None, 'longitude': '-76.98028', 'ja': None, 'country': 'US', 'postalcode': '20002-1834', 'clientkey': 'WOM3OP7L-SBW9-JV36-8DN0-O7U209J7LRQL', 'city': 'WASHINGTON', 'tvr': None, 'dsurl': None, 'grurl': None, 'taylorrental': None, 'gr': None, 'url': 'http://www.truevalue.com/kamcobldgsply', 'sat_open_time': '7:00 AM', 'twitterurl': None, 'wed_open_time': '5:00 AM', 'google': '1', 'main_id': 'TV', 'hg': None, 'creditcard': None, 'fax': None, 'foursquare': None, 'name': 'KAMCO BUILDING SUPPLY', 'address1': '2100 W VIRGINIA AVENUE NE', 'sat_close_time': '- 11:00 AM', 'jaurl': None, 'state': 'DC'}, {'fri_open_time': '8:30 AM', 'giftcard': None, 'icon': 'default', 'tvadv': None, 'cs': None, 'csurl': None, 'phone': '(703) 524-2503', 'email': 'billstruevalue@truevalue.net', 'google_notes': None, 'tv': '1', 'corronado': None, 'thur_close_time': '- 7:00 PM', 'google_shared': 'YES', 'uid': 1051078648, 'address2': None, '_distanceuom': 'mile', 'province': None, 'fri_close_time': '- 7:00 PM', 'yelp': None, 'sun_open_time': '10:00 AM', 'tvurl': 'http://www.billstruevalue.com', 'sun_close_time': '- 5:00 PM', 'tue_open_time': '8:30 AM', 'tvpaint': '1', '_distance': '5.34', 'wed_close_time': '- 7:00 PM', 'mon_open_time': '8:30 AM', 'google_email': 'billstruevalue@truevalue.net', 'ds': None, 'hgurl': None, 'localad': None, 'mon_close_time': '- 7:00 PM', 'thur_open_time': '8:30 AM', 'facebook': None, 'tvadvurl': None, 'tue_close_time': '- 7:00 PM', 'latitude': '38.89758', 'activeshiptostore': '1', 'bho': '[[\"1000\",\"1700\"],[\"0830\",\"1900\"],[\"0830\",\"1900\"],[\"0830\",\"1900\"],[\"0830\",\"1900\"],[\"0830\",\"1900\"],[\"0830\",\"1800\"]]', 'trurl': None, 'facebookurl': 'http://www.facebook.com/billstruevalue', 'longitude': '-77.12483', 'ja': None, 'country': 'US', 'postalcode': '22207-2528', 'clientkey': '5JGZQATK-3G0D-KXC4-C6E3-TBAS768FPE51', 'city': 'Arlington', 'tvr': None, 'dsurl': None, 'grurl': None, 'taylorrental': None, 'gr': None, 'url': 'http://www.billstruevalue.com', 'sat_open_time': '8:30 AM', 'twitterurl': None, 'wed_open_time': '8:30 AM', 'google': None, 'main_id': 'TV', 'hg': None, 'creditcard': None, 'fax': None, 'foursquare': None, 'name': 'Bills True Value', 'address1': '2213 N. Buchanan Street', 'sat_close_time': '- 6:00 PM', 'jaurl': None, 'state': 'VA'}, {'fri_open_time': '7:00 AM', 'giftcard': None, 'icon': 'default', 'tvadv': '1', 'cs': None, 'csurl': None, 'phone': '(301) 229-3700', 'email': 'CGEH@TrueValue.net', 'google_notes': None, 'tv': '1', 'corronado': None, 'thur_close_time': '- 8:00 PM', 'google_shared': 'Yes', 'uid': 1051076988, 'address2': None, '_distanceuom': 'mile', 'province': None, 'fri_close_time': '- 8:00 PM', 'yelp': None, 'sun_open_time': '9:00 AM', 'tvurl': 'http://www.truevalue.com/christophershardware', 'sun_close_time': '- 5:00 PM', 'tue_open_time': '7:00 AM', 'tvpaint': None, '_distance': '7.98', 'wed_close_time': '- 8:00 PM', 'mon_open_time': '7:00 AM', 'google_email': 'mikec@truevalue.net', 'ds': None, 'hgurl': None, 'localad': '1', 'mon_close_time': '- 8:00 PM', 'thur_open_time': '7:00 AM', 'facebook': None, 'tvadvurl': 'http://truevalue.shoplocal.com/truevalue/new_user_entry.aspx?StoreRef=HZUK3INI-P5FB-M95A-U15P-BCWH5QWTK30V&forceview=y&Adref=xx', 'tue_close_time': '- 8:00 PM', 'latitude': '38.96912', 'activeshiptostore': None, 'bho': '[[\"0900\",\"1700\"],[\"0700\",\"2000\"],[\"0700\",\"2000\"],[\"0700\",\"2000\"],[\"0700\",\"2000\"],[\"0700\",\"2000\"],[\"0830\",\"1800\"]]', 'trurl': None, 'facebookurl': 'http://www.facebook.com/home.php#!/GlenEchoHardware', 'longitude': '-77.14004', 'ja': None, 'country': 'US', 'postalcode': '20816', 'clientkey': 'HZUK3INI-P5FB-M95A-U15P-BCWH5QWTK30V', 'city': 'Bethesda', 'tvr': '1', 'dsurl': None, 'grurl': None, 'taylorrental': None, 'gr': None, 'url': 'http://www.truevalue.com/christophershardware', 'sat_open_time': '8:30 AM', 'twitterurl': 'http://www.twitter.com/GlenEchoHdwe', 'wed_open_time': '7:00 AM', 'google': None, 'main_id': 'TV', 'hg': None, 'creditcard': '1', 'fax': None, 'foursquare': None, 'name': 'Christophers Glen Echo Hardware', 'address1': '7301 Mcarthur Blvd', 'sat_close_time': '- 6:00 PM', 'jaurl': None, 'state': 'MD'}, {'fri_open_time': '6:00 AM', 'giftcard': None, 'icon': 'default', 'tvadv': None, 'cs': None, 'csurl': None, 'phone': '(703) 823-8700', 'email': None, 'google_notes': None, 'tv': '1', 'corronado': None, 'thur_close_time': '- 5:00 PM', 'google_shared': None, 'uid': 1051078649, 'address2': None, '_distanceuom': 'mile', 'province': None, 'fri_close_time': '- 5:00 PM', 'yelp': None, 'sun_open_time': 'closed', 'tvurl': None, 'sun_close_time': None, 'tue_open_time': '6:00 AM', 'tvpaint': None, '_distance': '8.98', 'wed_close_time': '- 5:00 PM', 'mon_open_time': '6:00 AM', 'google_email': None, 'ds': None, 'hgurl': None, 'localad': None, 'mon_close_time': '- 5:00 PM', 'thur_open_time': '6:00 AM', 'facebook': None, 'tvadvurl': None, 'tue_close_time': '- 5:00 PM', 'latitude': '38.7984625564754', 'activeshiptostore': None, 'bho': '[[\"9999\",\"9999\"],[\"0600\",\"1700\"],[\"0600\",\"1700\"],[\"0600\",\"1700\"],[\"0600\",\"1700\"],[\"0600\",\"1700\"],[\"0730\",\"1100\"]]', 'trurl': None, 'facebookurl': None, 'longitude': '-77.1362345995378', 'ja': None, 'country': 'US', 'postalcode': '22304-4822', 'clientkey': 'ACULYL3K-A7QA-N6TD-3KO2-763059ZYGSEK', 'city': 'ALEXANDRIA', 'tvr': None, 'dsurl': None, 'grurl': None, 'taylorrental': None, 'gr': None, 'url': 'http://www.truevalue.com/kamcobldg', 'sat_open_time': '7:30 AM', 'twitterurl': None, 'wed_open_time': '6:00 AM', 'google': None, 'main_id': 'TV', 'hg': None, 'creditcard': None, 'fax': None, 'foursquare': None, 'name': 'KAMCO BLDG SPLY & TRUE VALUE', 'address1': '5860 FARINGTON AVE', 'sat_close_time': '- 11:00 AM', 'jaurl': None, 'state': 'VA'}, {'fri_open_time': '9:30 AM', 'giftcard': '1', 'icon': 'default', 'tvadv': None, 'cs': None, 'csurl': None, 'phone': '(703) 765-4110', 'email': 'hhvs@vacoxmail.com', 'google_notes': None, 'tv': '1', 'corronado': None, 'thur_close_time': '- 8:00 PM', 'google_shared': None, 'uid': 1051078650, 'address2': None, '_distanceuom': 'mile', 'province': None, 'fri_close_time': '- 8:00 PM', 'yelp': None, 'sun_open_time': '11:00 AM', 'tvurl': None, 'sun_close_time': '- 6:00 PM', 'tue_open_time': '9:30 AM', 'tvpaint': None, '_distance': '10.64', 'wed_close_time': '- 8:00 PM', 'mon_open_time': '9:30 AM', 'google_email': None, 'ds': None, 'hgurl': None, 'localad': None, 'mon_close_time': '- 8:00 PM', 'thur_open_time': '9:30 AM', 'facebook': None, 'tvadvurl': None, 'tue_close_time': '- 8:00 PM', 'latitude': '38.7436373198134', 'activeshiptostore': None, 'bho': '[[\"1100\",\"1800\"],[\"0930\",\"2000\"],[\"0930\",\"2000\"],[\"0930\",\"2000\"],[\"0930\",\"2000\"],[\"0930\",\"2000\"],[\"0930\",\"2000\"]]', 'trurl': None, 'facebookurl': None, 'longitude': '-77.0570290532158', 'ja': None, 'country': 'US', 'postalcode': '22308-1203', 'clientkey': 'GKNHJ7J6-3FNY-HHZS-DWCS-HYK8QW3I0JI7', 'city': 'Alexandria', 'tvr': None, 'dsurl': None, 'grurl': None, 'taylorrental': None, 'gr': None, 'url': 'http://www.truevalue.com/hollinhallvarietystore', 'sat_open_time': '9:30 AM', 'twitterurl': None, 'wed_open_time': '9:30 AM', 'google': None, 'main_id': 'TV', 'hg': None, 'creditcard': None, 'fax': None, 'foursquare': None, 'name': 'Hollin Hall Variety Store', 'address1': '7902 Fort Hunt Rd', 'sat_close_time': '- 8:00 PM', 'jaurl': None, 'state': 'VA'}, {'fri_open_time': '7:30 AM', 'giftcard': None, 'icon': 'default', 'tvadv': '1', 'cs': None, 'csurl': None, 'phone': '(301) 292-1900', 'email': None, 'google_notes': None, 'tv': '1', 'corronado': None, 'thur_close_time': '- 5:00 PM', 'google_shared': None, 'uid': 1051076986, 'address2': None, '_distanceuom': 'mile', 'province': None, 'fri_close_time': '- 5:00 PM', 'yelp': None, 'sun_open_time': 'closed', 'tvurl': 'http://www.truevalue.com/ford', 'sun_close_time': None, 'tue_open_time': '7:30 AM', 'tvpaint': '1', '_distance': '11.58', 'wed_close_time': '- 5:00 PM', 'mon_open_time': '7:30 AM', 'google_email': None, 'ds': None, 'hgurl': None, 'localad': '1', 'mon_close_time': '- 5:00 PM', 'thur_open_time': '7:30 AM', 'facebook': None, 'tvadvurl': 'http://truevalue.shoplocal.com/truevalue/new_user_entry.aspx?StoreRef=VB7GDJK3-IDS0-XKHT-PNGL-VK4UG394HK6E&forceview=y&Adref=xx', 'tue_close_time': '- 5:00 PM', 'latitude': '38.730159346246', 'activeshiptostore': None, 'bho': '[[\"9999\",\"9999\"],[\"0730\",\"1700\"],[\"0730\",\"1700\"],[\"0730\",\"1700\"],[\"0730\",\"1700\"],[\"0730\",\"1700\"],[\"0730\",\"1600\"]]', 'trurl': None, 'facebookurl': None, 'longitude': '-76.9922404673855', 'ja': None, 'country': 'US', 'postalcode': '20744-5148', 'clientkey': 'VB7GDJK3-IDS0-XKHT-PNGL-VK4UG394HK6E', 'city': 'FORT WASHINGTON', 'tvr': None, 'dsurl': None, 'grurl': None, 'taylorrental': None, 'gr': None, 'url': 'http://www.truevalue.com/ford', 'sat_open_time': '7:30 AM', 'twitterurl': None, 'wed_open_time': '7:30 AM', 'google': None, 'main_id': 'TV', 'hg': None, 'creditcard': '1', 'fax': None, 'foursquare': None, 'name': 'FORD LUMBER COMPANY', 'address1': '11616 LIVINGSTON RD', 'sat_close_time': '- 4:00 PM', 'jaurl': None, 'state': 'MD'}, {'fri_open_time': '7:30 AM', 'giftcard': '1', 'icon': 'default', 'tvadv': None, 'cs': None, 'csurl': None, 'phone': '(301) 570-1300', 'email': 'Christophers@ChristophersHW.com', 'google_notes': None, 'tv': '1', 'corronado': None, 'thur_close_time': '- 7:30 PM', 'google_shared': 'Yes', 'uid': 1051076989, 'address2': None, '_distanceuom': 'mile', 'province': None, 'fri_close_time': '- 7:30 PM', 'yelp': '1', 'sun_open_time': '9:00 AM', 'tvurl': None, 'sun_close_time': '- 5:00 PM', 'tue_open_time': '7:30 AM', 'tvpaint': None, '_distance': '17.50', 'wed_close_time': '- 7:30 PM', 'mon_open_time': '7:30 AM', 'google_email': 'christophers@christophershw.com', 'ds': None, 'hgurl': None, 'localad': None, 'mon_close_time': '- 7:30 PM', 'thur_open_time': '7:30 AM', 'facebook': None, 'tvadvurl': None, 'tue_close_time': '- 7:30 PM', 'latitude': '39.14891', 'activeshiptostore': None, 'bho': '[[\"0900\",\"1700\"],[\"0730\",\"1930\"],[\"0730\",\"1930\"],[\"0730\",\"1930\"],[\"0730\",\"1930\"],[\"0730\",\"1930\"],[\"0800\",\"1800\"]]', 'trurl': None, 'facebookurl': None, 'longitude': '-77.02204', 'ja': None, 'country': 'US', 'postalcode': '20860', 'clientkey': 'YOL39PAC-QNYN-CVFS-QHDO-1I4DOMOT4PCB', 'city': 'Sandy Spring', 'tvr': None, 'dsurl': None, 'grurl': None, 'taylorrental': None, 'gr': None, 'url': 'http://www.christophershardwarestore.com', 'sat_open_time': '8:00 AM', 'twitterurl': None, 'wed_open_time': '7:30 AM', 'google': '1', 'main_id': 'TV', 'hg': None, 'creditcard': None, 'fax': None, 'foursquare': None, 'name': 'Christophers Hardware', 'address1': '500 Olney Sandy Spring Rd', 'sat_close_time': '- 6:00 PM', 'jaurl': None, 'state': 'MD'}, {'fri_open_time': '8:00 AM', 'giftcard': None, 'icon': 'default', 'tvadv': '1', 'cs': None, 'csurl': None, 'phone': '(703) 361-3141', 'email': 'jerice@truevalue.net', 'google_notes': None, 'tv': '1', 'corronado': None, 'thur_close_time': '- 7:00 PM', 'google_shared': None, 'uid': 1051076977, 'address2': None, '_distanceuom': 'mile', 'province': None, 'fri_close_time': '- 7:00 PM', 'yelp': None, 'sun_open_time': 'closed', 'tvurl': 'http://www.jericeco.com/', 'sun_close_time': None, 'tue_open_time': '8:00 AM', 'tvpaint': None, '_distance': '25.53', 'wed_close_time': '- 7:00 PM', 'mon_open_time': '8:00 AM', 'google_email': None, 'ds': None, 'hgurl': None, 'localad': '1', 'mon_close_time': '- 7:00 PM', 'thur_open_time': '8:00 AM', 'facebook': None, 'tvadvurl': 'http://truevalue.shoplocal.com/truevalue/new_user_entry.aspx?StoreRef=7R0OKS9L-1M3V-03T6-T10M-F121Y326BYIU&forceview=y&Adref=xx', 'tue_close_time': '- 7:00 PM', 'latitude': '38.7579993877551', 'activeshiptostore': '1', 'bho': '[[\"9999\",\"9999\"],[\"0800\",\"1900\"],[\"0800\",\"1900\"],[\"0800\",\"1900\"],[\"0800\",\"1900\"],[\"0800\",\"1900\"],[\"0800\",\"1800\"]]', 'trurl': None, 'facebookurl': 'http://www.facebook.com/home.php?#!/pages/Manassas-VA/JE-Rice-Co/209185793694?ref=ts&__a=9&ajaxpipe=1', 'longitude': '-77.4656865306122', 'ja': None, 'country': 'US', 'postalcode': '20110', 'clientkey': '7R0OKS9L-1M3V-03T6-T10M-F121Y326BYIU', 'city': 'Manassas', 'tvr': None, 'dsurl': None, 'grurl': None, 'taylorrental': None, 'gr': None, 'url': 'http://www.jericeco.com', 'sat_open_time': '8:00 AM', 'twitterurl': None, 'wed_open_time': '8:00 AM', 'google': None, 'main_id': 'TV', 'hg': None, 'creditcard': '1', 'fax': None, 'foursquare': None, 'name': 'J E Rice Co.', 'address1': '9124 Mathis Ave', 'sat_close_time': '- 6:00 PM', 'jaurl': None, 'state': 'VA'}, {'fri_open_time': '8:00 AM', 'giftcard': '1', 'icon': 'default', 'tvadv': '1', 'cs': None, 'csurl': None, 'phone': '(443) 607-4162', 'email': 'Chesapeakehardware@truevalue.net', 'google_notes': None, 'tv': '1', 'corronado': None, 'thur_close_time': '- 7:00 PM', 'google_shared': None, 'uid': -2013484794, 'address2': None, '_distanceuom': 'mile', 'province': None, 'fri_close_time': '- 7:00 PM', 'yelp': None, 'sun_open_time': '8:00 AM', 'tvurl': None, 'sun_close_time': '- 5:00 PM', 'tue_open_time': '8:00 AM', 'tvpaint': '1', '_distance': '26.83', 'wed_close_time': '- 7:00 PM', 'mon_open_time': '8:00 AM', 'google_email': None, 'ds': None, 'hgurl': None, 'localad': '1', 'mon_close_time': '- 7:00 PM', 'thur_open_time': '8:00 AM', 'facebook': None, 'tvadvurl': 'http://truevalue.shoplocal.com/truevalue/new_user_entry.aspx?StoreRef=Q1ZGD4YP-NXGT-H3QM-2RUD-OFD11NRUZYNC&forceview=y&Adref=xx', 'tue_close_time': '- 7:00 PM', 'latitude': '38.8150991440624', 'activeshiptostore': '1', 'bho': '[[\"0800\",\"1700\"],[\"0800\",\"1900\"],[\"0800\",\"1900\"],[\"0800\",\"1900\"],[\"0800\",\"1900\"],[\"0800\",\"1900\"],[\"0800\",\"1900\"]]', 'trurl': None, 'facebookurl': None, 'longitude': '-76.5376268933135', 'ja': None, 'country': 'US', 'postalcode': '20733-9639', 'clientkey': 'Q1ZGD4YP-NXGT-H3QM-2RUD-OFD11NRUZYNC', 'city': 'CHURCHTON', 'tvr': '1', 'dsurl': None, 'grurl': None, 'taylorrental': None, 'gr': None, 'url': 'http://www.truevalue.com/chesapeakehardware', 'sat_open_time': '8:00 AM', 'twitterurl': None, 'wed_open_time': '8:00 AM', 'google': None, 'main_id': 'TV', 'hg': None, 'creditcard': '1', 'fax': None, 'foursquare': None, 'name': 'CHESAPEAKE HARDWARE', 'address1': '5570-C SHADY SIDE ROAD', 'sat_close_time': '- 7:00 PM', 'jaurl': None, 'state': 'MD'}, {'fri_open_time': '7:30 AM', 'giftcard': None, 'icon': 'default', 'tvadv': None, 'cs': None, 'csurl': None, 'phone': '(410) 647-4611', 'email': 'tony@clementhardware.com', 'google_notes': None, 'tv': '1', 'corronado': None, 'thur_close_time': '- 7:00 PM', 'google_shared': None, 'uid': -1478017856, 'address2': None, '_distanceuom': 'mile', 'province': None, 'fri_close_time': '- 7:00 PM', 'yelp': None, 'sun_open_time': '9:00 AM', 'tvurl': None, 'sun_close_time': '- 5:00 PM', 'tue_open_time': '7:30 AM', 'tvpaint': None, '_distance': '28.61', 'wed_close_time': '- 7:00 PM', 'mon_open_time': '7:30 AM', 'google_email': None, 'ds': None, 'hgurl': None, 'localad': None, 'mon_close_time': '- 7:00 PM', 'thur_open_time': '7:30 AM', 'facebook': None, 'tvadvurl': None, 'tue_close_time': '- 7:00 PM', 'latitude': '39.08077', 'activeshiptostore': '1', 'bho': '[[\"0900\",\"1700\"],[\"0730\",\"1900\"],[\"0730\",\"1900\"],[\"0730\",\"1900\"],[\"0730\",\"1900\"],[\"0730\",\"1900\"],[\"0700\",\"1800\"]]', 'trurl': None, 'facebookurl': 'http://www.facebook.com/pages/Clement-Hardware/218395654855700', 'longitude': '-76.54892', 'ja': None, 'country': 'US', 'postalcode': '21146-2954', 'clientkey': 'HCFP6UZL-75MR-TKAO-RP2Z-NMHJAEH1PCNM', 'city': 'SEVERNA PARK', 'tvr': '1', 'dsurl': None, 'grurl': None, 'taylorrental': None, 'gr': None, 'url': 'http://clementhardware.com/', 'sat_open_time': '7:00 AM', 'twitterurl': None, 'wed_open_time': '7:30 AM', 'google': None, 'main_id': 'TV', 'hg': None, 'creditcard': None, 'fax': None, 'foursquare': None, 'name': 'CLEMENT HARDWARE', 'address1': '500 RITCHIE HIGHWAY', 'sat_close_time': '- 6:00 PM', 'jaurl': None, 'state': 'MD'}, {'fri_open_time': '7:00 AM', 'giftcard': None, 'icon': 'default', 'tvadv': '1', 'cs': None, 'csurl': None, 'phone': '(301) 253-2131', 'email': None, 'google_notes': None, 'tv': '1', 'corronado': None, 'thur_close_time': '- 6:00 PM', 'google_shared': None, 'uid': 1051078622, 'address2': None, '_distanceuom': 'mile', 'province': None, 'fri_close_time': '- 6:00 PM', 'yelp': None, 'sun_open_time': '10:00 AM', 'tvurl': 'http://www.hyattbuildingsupply.com/', 'sun_close_time': '- 2:00 PM', 'tue_open_time': '7:00 AM', 'tvpaint': '1', '_distance': '28.77', 'wed_close_time': '- 6:00 PM', 'mon_open_time': '7:00 AM', 'google_email': None, 'ds': None, 'hgurl': None, 'localad': '1', 'mon_close_time': '- 6:00 PM', 'thur_open_time': '7:00 AM', 'facebook': None, 'tvadvurl': 'http://truevalue.shoplocal.com/truevalue/new_user_entry.aspx?StoreRef=N7IVBTZQ-HZ6O-KGV2-ZUEX-QAHBZXTIRS3E&forceview=y&Adref=xx', 'tue_close_time': '- 6:00 PM', 'latitude': '39.2873782648885', 'activeshiptostore': None, 'bho': '[[\"1000\",\"1400\"],[\"0700\",\"1800\"],[\"0700\",\"1800\"],[\"0700\",\"1800\"],[\"0700\",\"1800\"],[\"0700\",\"1800\"],[\"0800\",\"1600\"]]', 'trurl': None, 'facebookurl': None, 'longitude': '-77.2075308119523', 'ja': None, 'country': 'US', 'postalcode': '20872-1830', 'clientkey': 'N7IVBTZQ-HZ6O-KGV2-ZUEX-QAHBZXTIRS3E', 'city': 'DAMASCUS', 'tvr': None, 'dsurl': None, 'grurl': None, 'taylorrental': None, 'gr': None, 'url': 'http://www.hyattbuildingsupply.com', 'sat_open_time': '8:00 AM', 'twitterurl': None, 'wed_open_time': '7:00 AM', 'google': None, 'main_id': 'TV', 'hg': None, 'creditcard': None, 'fax': None, 'foursquare': None, 'name': 'HYATT TRUE VALUE', 'address1': '26200 RIDGE RD', 'sat_close_time': '- 4:00 PM', 'jaurl': None, 'state': 'MD'}, {'fri_open_time': '7:00 AM', 'giftcard': '1', 'icon': 'default', 'tvadv': '1', 'cs': None, 'csurl': None, 'phone': '(410) 268-3939', 'email': 'jared@kbtruevalue.com', 'google_notes': None, 'tv': '1', 'corronado': None, 'thur_close_time': '- 8:00 PM', 'google_shared': None, 'uid': 1051078638, 'address2': None, '_distanceuom': 'mile', 'province': None, 'fri_close_time': '- 8:00 PM', 'yelp': None, 'sun_open_time': '8:00 AM', 'tvurl': 'http://www.kbtruevalue.com/', 'sun_close_time': '- 6:00 PM', 'tue_open_time': '7:00 AM', 'tvpaint': '1', '_distance': '28.88', 'wed_close_time': '- 8:00 PM', 'mon_open_time': '7:00 AM', 'google_email': None, 'ds': None, 'hgurl': None, 'localad': '1', 'mon_close_time': '- 8:00 PM', 'thur_open_time': '7:00 AM', 'facebook': None, 'tvadvurl': 'http://truevalue.shoplocal.com/truevalue/new_user_entry.aspx?StoreRef=CH10ZM2D-T28I-979R-X1UQ-NYSIN4L4XT4P&forceview=y&Adref=xx', 'tue_close_time': '- 8:00 PM', 'latitude': '38.9502181524996', 'activeshiptostore': '1', 'bho': '[[\"0800\",\"1800\"],[\"0700\",\"2000\"],[\"0700\",\"2000\"],[\"0700\",\"2000\"],[\"0700\",\"2000\"],[\"0700\",\"2000\"],[\"0700\",\"1900\"]]', 'trurl': None, 'facebookurl': 'http://www.facebook.com/kbtruevalue', 'longitude': '-76.4927551456658', 'ja': None, 'country': 'US', 'postalcode': '21403-1756', 'clientkey': 'CH10ZM2D-T28I-979R-X1UQ-NYSIN4L4XT4P', 'city': 'Annapolis', 'tvr': '1', 'dsurl': None, 'grurl': None, 'taylorrental': None, 'gr': None, 'url': 'http://www.kbtruevalue.com', 'sat_open_time': '7:00 AM', 'twitterurl': 'http://twitter.com/kbtruevalue', 'wed_open_time': '7:00 AM', 'google': None, 'main_id': 'TV', 'hg': None, 'creditcard': '1', 'fax': None, 'foursquare': None, 'name': 'K & B True Value', 'address1': '912 Forest Dr', 'sat_close_time': '- 7:00 PM', 'jaurl': None, 'state': 'MD'}, {'fri_open_time': '7:30 AM', 'giftcard': None, 'icon': 'default', 'tvadv': None, 'cs': None, 'csurl': None, 'phone': '(410) 535-0442', 'email': 'lusbymot@verizon.net', 'google_notes': None, 'tv': '1', 'corronado': None, 'thur_close_time': '- 5:30 PM', 'google_shared': None, 'uid': 1051076983, 'address2': None, '_distanceuom': 'mile', 'province': None, 'fri_close_time': '- 7:30 PM', 'yelp': None, 'sun_open_time': 'closed', 'tvurl': 'http://www.lusbyhardware.com/', 'sun_close_time': None, 'tue_open_time': '7:30 AM', 'tvpaint': None, '_distance': '34.26', 'wed_close_time': '- 5:30 PM', 'mon_open_time': '7:30 AM', 'google_email': None, 'ds': None, 'hgurl': None, 'localad': None, 'mon_close_time': '- 5:30 PM', 'thur_open_time': '7:30 AM', 'facebook': None, 'tvadvurl': None, 'tue_close_time': '- 5:30 PM', 'latitude': '38.539349691187', 'activeshiptostore': '1', 'bho': '[[\"9999\",\"9999\"],[\"0730\",\"1730\"],[\"0730\",\"1730\"],[\"0730\",\"1730\"],[\"0730\",\"1730\"],[\"0730\",\"1930\"],[\"0730\",\"1730\"]]', 'trurl': None, 'facebookurl': None, 'longitude': '-76.5835104820298', 'ja': None, 'country': 'US', 'postalcode': '20678', 'clientkey': '9MZ7X579-CK0M-3IMU-TT63-OPZF55IIPXFZ', 'city': 'Prince Frederick', 'tvr': None, 'dsurl': None, 'grurl': None, 'taylorrental': None, 'gr': None, 'url': 'http://www.Lusbyhardware.com', 'sat_open_time': '7:30 AM', 'twitterurl': None, 'wed_open_time': '7:30 AM', 'google': None, 'main_id': 'TV', 'hg': None, 'creditcard': '1', 'fax': None, 'foursquare': None, 'name': 'Lusby Motor Company Inc.', 'address1': '155 Main St', 'sat_close_time': '- 5:30 PM', 'jaurl': None, 'state': 'MD'}, {'fri_open_time': '8:00 AM', 'giftcard': '1', 'icon': 'default', 'tvadv': '1', 'cs': None, 'csurl': None, 'phone': '(301) 259-2540', 'email': 'hungerford@olg.com', 'google_notes': None, 'tv': '1', 'corronado': None, 'thur_close_time': '- 5:00 PM', 'google_shared': 'YES', 'uid': 1051076982, 'address2': None, '_distanceuom': 'mile', 'province': None, 'fri_close_time': '- 5:00 PM', 'yelp': None, 'sun_open_time': 'closed', 'tvurl': 'http://www.aghungerford.com/', 'sun_close_time': None, 'tue_open_time': '8:00 AM', 'tvpaint': '1', '_distance': '35.93', 'wed_close_time': '- 5:00 PM', 'mon_open_time': '8:00 AM', 'google_email': 'hungerford@olg.com', 'ds': None, 'hgurl': None, 'localad': '1', 'mon_close_time': '- 5:00 PM', 'thur_open_time': '8:00 AM', 'facebook': None, 'tvadvurl': 'http://truevalue.shoplocal.com/truevalue/new_user_entry.aspx?StoreRef=N5VPHAFC-MVJV-T31M-RRXY-8PKZ81EMOPHM&forceview=y&Adref=xx', 'tue_close_time': '- 5:00 PM', 'latitude': '38.3786245621919', 'activeshiptostore': '1', 'bho': '[[\"9999\",\"9999\"],[\"0800\",\"1700\"],[\"0800\",\"1700\"],[\"0800\",\"1700\"],[\"0800\",\"1700\"],[\"0800\",\"1700\"],[\"0800\",\"1700\"]]', 'trurl': None, 'facebookurl': None, 'longitude': '-76.95422029337', 'ja': None, 'country': 'US', 'postalcode': '20664', 'clientkey': 'N5VPHAFC-MVJV-T31M-RRXY-8PKZ81EMOPHM', 'city': 'Newburg', 'tvr': '1', 'dsurl': None, 'grurl': None, 'taylorrental': None, 'gr': None, 'url': 'http://www.aghungerford.com', 'sat_open_time': '8:00 AM', 'twitterurl': None, 'wed_open_time': '8:00 AM', 'google': None, 'main_id': 'TV', 'hg': None, 'creditcard': '1', 'fax': None, 'foursquare': None, 'name': 'A G Hungerford & Son Inc.', 'address1': '12165 Rock Point Rd', 'sat_close_time': '- 5:00 PM', 'jaurl': None, 'state': 'MD'}], 'collectionname': 'poi'}, 'code': 1}\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "Case Study 2: getting API_KEY from html and then data from API\n",
    "\n",
    "The True Value Company is an American retailer-owned hardware cooperative with over 4,000 independent retail \n",
    "locations worldwide. Create scraper which gets all available True Value shops given post code. Scraper\n",
    "should not have any API key hardcoded, as it can change during site lifetime.\n",
    "\n",
    "Minimum data you should get:\n",
    "- address\n",
    "- city\n",
    "- country\n",
    "- latitude\n",
    "- longitude\n",
    "- name\n",
    "- postalcode\n",
    "- state\n",
    "\"\"\"\n",
    "main_page_url = 'http://hosted.where2getit.com/truevalue/index2015.html'\n",
    "main_page_text = requests.get(main_page_url).text\n",
    "api_key = re.findall(r\"appkey: '([0-9A-Z\\-]+)', \", main_page_text)[0]\n",
    "print('Got API KEY from main page: ', api_key)\n",
    "api_endpoint = 'http://hosted.where2getit.com/truevalue/rest/locatorsearch'\n",
    "POST_CODE = 20004\n",
    "body = {\n",
    "  \"request\": {\n",
    "    \"appkey\": api_key,\n",
    "    \"formdata\": {\n",
    "      \"geoip\": False,\n",
    "      \"dataview\": \"store_default\",\n",
    "      \"limit\": 40,\n",
    "      \"geolocs\": {\n",
    "        \"geoloc\": [\n",
    "          {\n",
    "            \"addressline\": str(POST_CODE)\n",
    "          }\n",
    "        ]\n",
    "      },\n",
    "      \"searchradius\": \"40|50|80\",\n",
    "      \"where\": {\n",
    "        \"and\": {\n",
    "          \"giftcard\": {\n",
    "            \"eq\": \"\"\n",
    "          },\n",
    "          \"tvpaint\": {\n",
    "            \"eq\": \"\"\n",
    "          },\n",
    "          \"creditcard\": {\n",
    "            \"eq\": \"\"\n",
    "          },\n",
    "          \"localad\": {\n",
    "            \"eq\": \"\"\n",
    "          },\n",
    "          \"ja\": {\n",
    "            \"eq\": \"\"\n",
    "          },\n",
    "          \"tvr\": {\n",
    "            \"eq\": \"\"\n",
    "          },\n",
    "          \"activeshiptostore\": {\n",
    "            \"eq\": \"\"\n",
    "          },\n",
    "          \"main_id\": {\n",
    "            \"eq\": \"\"\n",
    "          },\n",
    "          \"corronado\": {\n",
    "            \"eq\": \"\"\n",
    "          },\n",
    "          \"tv\": {\n",
    "            \"eq\": \"1\"\n",
    "          }\n",
    "        }\n",
    "      },\n",
    "      \"false\": \"0\"\n",
    "    }\n",
    "  }\n",
    "}\n",
    "r = requests.post(api_endpoint, data=json.dumps(body))\n",
    "data = json.loads(r.text)\n",
    "print('Raw response from API: ', data)\n",
    "shops = [{'name':entry['name'],\n",
    "          'address':entry['address1'],\n",
    "          'postalcode':entry['postalcode'],\n",
    "          'city':entry['city'],\n",
    "          'state':entry['state'],\n",
    "          'country':entry['country'],\n",
    "          'latitude':entry['latitude'],\n",
    "          'longitude':entry['longitude']\n",
    "         } for entry in data['response']['collection']]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>address</th>\n",
       "      <th>city</th>\n",
       "      <th>country</th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>name</th>\n",
       "      <th>postalcode</th>\n",
       "      <th>state</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1623 17th St NW</td>\n",
       "      <td>Washington</td>\n",
       "      <td>US</td>\n",
       "      <td>38.911917</td>\n",
       "      <td>-77.03849</td>\n",
       "      <td>True Value On 17th</td>\n",
       "      <td>20009-2433</td>\n",
       "      <td>DC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1108 24th Street NW</td>\n",
       "      <td>Washington</td>\n",
       "      <td>US</td>\n",
       "      <td>38.9038748979592</td>\n",
       "      <td>-77.0514083673469</td>\n",
       "      <td>District Hardware and Bike</td>\n",
       "      <td>20037-1432</td>\n",
       "      <td>DC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2100 W VIRGINIA AVENUE NE</td>\n",
       "      <td>WASHINGTON</td>\n",
       "      <td>US</td>\n",
       "      <td>38.91536</td>\n",
       "      <td>-76.98028</td>\n",
       "      <td>KAMCO BUILDING SUPPLY</td>\n",
       "      <td>20002-1834</td>\n",
       "      <td>DC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2213 N. Buchanan Street</td>\n",
       "      <td>Arlington</td>\n",
       "      <td>US</td>\n",
       "      <td>38.89758</td>\n",
       "      <td>-77.12483</td>\n",
       "      <td>Bills True Value</td>\n",
       "      <td>22207-2528</td>\n",
       "      <td>VA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7301 Mcarthur Blvd</td>\n",
       "      <td>Bethesda</td>\n",
       "      <td>US</td>\n",
       "      <td>38.96912</td>\n",
       "      <td>-77.14004</td>\n",
       "      <td>Christophers Glen Echo Hardware</td>\n",
       "      <td>20816</td>\n",
       "      <td>MD</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5860 FARINGTON AVE</td>\n",
       "      <td>ALEXANDRIA</td>\n",
       "      <td>US</td>\n",
       "      <td>38.7984625564754</td>\n",
       "      <td>-77.1362345995378</td>\n",
       "      <td>KAMCO BLDG SPLY &amp; TRUE VALUE</td>\n",
       "      <td>22304-4822</td>\n",
       "      <td>VA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7902 Fort Hunt Rd</td>\n",
       "      <td>Alexandria</td>\n",
       "      <td>US</td>\n",
       "      <td>38.7436373198134</td>\n",
       "      <td>-77.0570290532158</td>\n",
       "      <td>Hollin Hall Variety Store</td>\n",
       "      <td>22308-1203</td>\n",
       "      <td>VA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>11616 LIVINGSTON RD</td>\n",
       "      <td>FORT WASHINGTON</td>\n",
       "      <td>US</td>\n",
       "      <td>38.730159346246</td>\n",
       "      <td>-76.9922404673855</td>\n",
       "      <td>FORD LUMBER COMPANY</td>\n",
       "      <td>20744-5148</td>\n",
       "      <td>MD</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>500 Olney Sandy Spring Rd</td>\n",
       "      <td>Sandy Spring</td>\n",
       "      <td>US</td>\n",
       "      <td>39.14891</td>\n",
       "      <td>-77.02204</td>\n",
       "      <td>Christophers Hardware</td>\n",
       "      <td>20860</td>\n",
       "      <td>MD</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>9124 Mathis Ave</td>\n",
       "      <td>Manassas</td>\n",
       "      <td>US</td>\n",
       "      <td>38.7579993877551</td>\n",
       "      <td>-77.4656865306122</td>\n",
       "      <td>J E Rice Co.</td>\n",
       "      <td>20110</td>\n",
       "      <td>VA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>5570-C SHADY SIDE ROAD</td>\n",
       "      <td>CHURCHTON</td>\n",
       "      <td>US</td>\n",
       "      <td>38.8150991440624</td>\n",
       "      <td>-76.5376268933135</td>\n",
       "      <td>CHESAPEAKE HARDWARE</td>\n",
       "      <td>20733-9639</td>\n",
       "      <td>MD</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>500 RITCHIE HIGHWAY</td>\n",
       "      <td>SEVERNA PARK</td>\n",
       "      <td>US</td>\n",
       "      <td>39.08077</td>\n",
       "      <td>-76.54892</td>\n",
       "      <td>CLEMENT HARDWARE</td>\n",
       "      <td>21146-2954</td>\n",
       "      <td>MD</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>26200 RIDGE RD</td>\n",
       "      <td>DAMASCUS</td>\n",
       "      <td>US</td>\n",
       "      <td>39.2873782648885</td>\n",
       "      <td>-77.2075308119523</td>\n",
       "      <td>HYATT TRUE VALUE</td>\n",
       "      <td>20872-1830</td>\n",
       "      <td>MD</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>912 Forest Dr</td>\n",
       "      <td>Annapolis</td>\n",
       "      <td>US</td>\n",
       "      <td>38.9502181524996</td>\n",
       "      <td>-76.4927551456658</td>\n",
       "      <td>K &amp; B True Value</td>\n",
       "      <td>21403-1756</td>\n",
       "      <td>MD</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>155 Main St</td>\n",
       "      <td>Prince Frederick</td>\n",
       "      <td>US</td>\n",
       "      <td>38.539349691187</td>\n",
       "      <td>-76.5835104820298</td>\n",
       "      <td>Lusby Motor Company Inc.</td>\n",
       "      <td>20678</td>\n",
       "      <td>MD</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>12165 Rock Point Rd</td>\n",
       "      <td>Newburg</td>\n",
       "      <td>US</td>\n",
       "      <td>38.3786245621919</td>\n",
       "      <td>-76.95422029337</td>\n",
       "      <td>A G Hungerford &amp; Son Inc.</td>\n",
       "      <td>20664</td>\n",
       "      <td>MD</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML(pd.DataFrame(shops).to_html())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Got listing from:  https://www.airbnb.co.uk/api/v2/explore_tabs?metadata_only=false&items_per_grid=20&version=1.2.8&luxury_pre_launch=false&_intents=p1&screen_size=small&locale=en-GB&timezone_offset=60&key=d306zoyjsyarp7ifhu67rjxn52tv0t20&is_new_cards_experiment=false&is_standard_search=true&_format=for_explore_search_web&fetch_filters=true&currency=GBP&supports_for_you_v3=true&location=London&is_guided_search=true&s_tag=DOIPutuT&selected_tab_id=home_tab&section_offset=0&auto_ib=false&allow_override%5B%5D=&refinements%5B%5D=homes\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'\\nnote:\\nsome airbnb usefull endpoints\\n# get listings\\nhttps://www.airbnb.co.uk/api/v2/explore_tabs?version=1.2.8&_format=for_explore_search_web&items_per_grid=18&experiences_per_grid=20&guidebooks_per_grid=20&fetch_filters=true&is_guided_search=true&is_new_cards_experiment=false&supports_for_you_v3=true&screen_size=small&timezone_offset=60&auto_ib=false&luxury_pre_launch=false&metadata_only=false&is_standard_search=true&tab_id=home_tab&location=London&allow_override%5B%5D=&ne_lat=51.599363500119274&ne_lng=-0.06168207198925302&sw_lat=51.47626857868991&sw_lng=-0.289648380583003&zoom=12&search_by_map=true&federated_search_session_id=6d72b1e2-cb68-4877-b27c-8614e11fc5b0&_intents=p1&key=d306zoyjsyarp7ifhu67rjxn52tv0t20&currency=&locale=en-GB\\n# get booking detials\\nhttps://www.airbnb.co.uk/api/v2/pdp_listing_booking_details?guests=1&listing_id=13575756&_format=for_web_dateless&_interaction_type=pageload&_intents=p3_book_it&_parent_request_uuid=aed9f0ce-1534-4a89-9cf6-c6813adcb95b&_p3_impression_id=p3_1506465875_Q2VDMsV0pLs27%2BtX&show_smart_promotion=0&force_boost_unc_priority_message_type=&number_of_adults=1&number_of_children=0&number_of_infants=0&key=d306zoyjsyarp7ifhu67rjxn52tv0t20&currency=GBP&locale=en-GB\\n'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"\n",
    "Case Study 3: get available airbnb properties in London\n",
    "\n",
    "You want to visit London and airbnb looks like a nice option for you. As you are crazy data geek and you want to run\n",
    "some fancy algorithms to make a better choice of apartment to rent - you need data! Get all available airbnb \n",
    "properties in London. You are interested in pricing, location, rating, no. of reviews, images and more. \n",
    "You also don't like to repeate yourself, so you need to build scraper which will survive till your next trip.\n",
    "\"\"\"\n",
    "\n",
    "headers = {'accept-encoding': 'gzip, deflate, br',\n",
    "           'x-requested-with': 'XMLHttpRequest',\n",
    "           'accept-language': 'en-US,en;q=0.8,pl;q=0.6',\n",
    "           'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36',\n",
    "           'accept': 'application/json, text/javascript, */*; q=0.01',\n",
    "           'referer': 'https://www.airbnb.co.uk/s/London/homes',\n",
    "           'authority': 'www.airbnb.co.uk'}\n",
    "# get api key embeded in html (yes... same story again :))\n",
    "html = requests.get('https://www.airbnb.co.uk/s/London/homes', headers=headers).text\n",
    "api_key = re.findall(r'key\\&quot;:\\&quot;([a-zA-Z0-7]*)\\&quot;},\\&quot;deep_link', html)[0]\n",
    "# get first listing\n",
    "enpoint='https://www.airbnb.co.uk/api/v2/explore_tabs'\n",
    "params = {'version':'1.2.8',\n",
    "          '_format':'for_explore_search_web',\n",
    "          'items_per_grid':'20',\n",
    "          'fetch_filters':'true',\n",
    "          'is_guided_search':'true',\n",
    "          'is_new_cards_experiment':'false',\n",
    "          'supports_for_you_v3':'true',\n",
    "          'screen_size':'small',\n",
    "          'timezone_offset':'60',\n",
    "          'auto_ib':'false',\n",
    "          'luxury_pre_launch':'false',\n",
    "          'metadata_only':'false',\n",
    "          'is_standard_search':'true',\n",
    "          'refinements[]':'homes',\n",
    "          'selected_tab_id':'home_tab',\n",
    "          'location':'London',\n",
    "          'allow_override[]':'',\n",
    "          's_tag':'DOIPutuT',\n",
    "          'section_offset':'0',\n",
    "          '_intents':'p1',\n",
    "          'key':api_key,\n",
    "          'currency':'GBP',\n",
    "          'locale':'en-GB'}\n",
    "r = requests.get(enpoint, params=params)\n",
    "print('Got listing from: ', r.url)\n",
    "ds = json.loads(r.text)\n",
    "\n",
    "\"\"\"\n",
    "note:\n",
    "some airbnb usefull endpoints\n",
    "# get listings\n",
    "https://www.airbnb.co.uk/api/v2/explore_tabs?version=1.2.8&_format=for_explore_search_web&items_per_grid=18&experiences_per_grid=20&guidebooks_per_grid=20&fetch_filters=true&is_guided_search=true&is_new_cards_experiment=false&supports_for_you_v3=true&screen_size=small&timezone_offset=60&auto_ib=false&luxury_pre_launch=false&metadata_only=false&is_standard_search=true&tab_id=home_tab&location=London&allow_override%5B%5D=&ne_lat=51.599363500119274&ne_lng=-0.06168207198925302&sw_lat=51.47626857868991&sw_lng=-0.289648380583003&zoom=12&search_by_map=true&federated_search_session_id=6d72b1e2-cb68-4877-b27c-8614e11fc5b0&_intents=p1&key=d306zoyjsyarp7ifhu67rjxn52tv0t20&currency=&locale=en-GB\n",
    "# get booking detials\n",
    "https://www.airbnb.co.uk/api/v2/pdp_listing_booking_details?guests=1&listing_id=13575756&_format=for_web_dateless&_interaction_type=pageload&_intents=p3_book_it&_parent_request_uuid=aed9f0ce-1534-4a89-9cf6-c6813adcb95b&_p3_impression_id=p3_1506465875_Q2VDMsV0pLs27%2BtX&show_smart_promotion=0&force_boost_unc_priority_message_type=&number_of_adults=1&number_of_children=0&number_of_infants=0&key=d306zoyjsyarp7ifhu67rjxn52tv0t20&currency=GBP&locale=en-GB\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "will get page:  2\n",
      "No. of properties:  20\n",
      "will get page:  3\n",
      "No. of properties:  40\n",
      "will get page:  4\n",
      "No. of properties:  60\n",
      "will get page:  5\n",
      "No. of properties:  80\n",
      "will get page:  6\n",
      "No. of properties:  100\n"
     ]
    }
   ],
   "source": [
    "# paginate and get more properties\n",
    "props = []\n",
    "page_no = 1\n",
    "page_limit = 3\n",
    "while (ds['explore_tabs'][0]['pagination_metadata']['has_next_page'] == True) and (page_no<page_limit):\n",
    "    params['section_offset'] = str(page_no)\n",
    "    print('will get page: ', page_no+1)\n",
    "    r = requests.get(enpoint, params=params)\n",
    "    ds = json.loads(r.text)\n",
    "    for idx, section in enumerate(ds['explore_tabs'][0]['sections']):\n",
    "        for prop in ds['explore_tabs'][0]['sections'][idx]['listings']:\n",
    "            props.append({'name':prop['listing']['name'],\n",
    "                          'room_type':prop['listing']['room_type'],\n",
    "                          'person_capacity':prop['listing']['person_capacity'],\n",
    "                          'pic':prop['listing']['picture']['picture'],\n",
    "                          'rating':prop['listing']['star_rating'],\n",
    "                          'latitude':prop['listing']['lat'],\n",
    "                          'longitude':prop['listing']['lng'],\n",
    "                          'price':prop['pricing_quote']['rate']['amount'],\n",
    "                          'currency':prop['pricing_quote']['rate']['currency'],\n",
    "                          'price_type':prop['pricing_quote']['rate_type']})\n",
    "    page_no += 1\n",
    "    print('No. of properties: ', len(props))\n",
    "    time.sleep(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>currency</th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>name</th>\n",
       "      <th>person_capacity</th>\n",
       "      <th>pic</th>\n",
       "      <th>price</th>\n",
       "      <th>price_type</th>\n",
       "      <th>rating</th>\n",
       "      <th>room_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.516746</td>\n",
       "      <td>-0.050351</td>\n",
       "      <td>(HAR-A)PRIVATE ROOM FOR 5PPL CLOSE TO TOWER BR...</td>\n",
       "      <td>5</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/786aa625-2...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.524362</td>\n",
       "      <td>-0.116995</td>\n",
       "      <td>Double Room nr Soho | Russell Square |Kings Cross</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/affe8de1-a...</td>\n",
       "      <td>62</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.486756</td>\n",
       "      <td>-0.104479</td>\n",
       "      <td>1 double room in Central London</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/6bfb24c2-c...</td>\n",
       "      <td>18</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.511007</td>\n",
       "      <td>-0.226281</td>\n",
       "      <td>THE QUEENS HOSTEL, 6 BED MIXED DORM E</td>\n",
       "      <td>6</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/328d9beb-6...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.564283</td>\n",
       "      <td>-0.120809</td>\n",
       "      <td>Spacious Double room in Holloway, London</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/ae707340-f...</td>\n",
       "      <td>19</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.548527</td>\n",
       "      <td>-0.226324</td>\n",
       "      <td>Modern room 10 min from Central London</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/a0e3a2af-8...</td>\n",
       "      <td>40</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.491728</td>\n",
       "      <td>-0.014746</td>\n",
       "      <td>Double Room in Canary Wharf hs</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/ebd8e53d-0...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.452218</td>\n",
       "      <td>-0.025420</td>\n",
       "      <td>Comfortable, Clean London room</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/ae5469b5-9...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.501680</td>\n",
       "      <td>-0.052571</td>\n",
       "      <td>Double room, 2min from the station</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/4cfc5263-5...</td>\n",
       "      <td>57</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.624089</td>\n",
       "      <td>-0.054654</td>\n",
       "      <td>En-Suite Bedroom with Bathroom</td>\n",
       "      <td>3</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/97921358/8...</td>\n",
       "      <td>19</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.483282</td>\n",
       "      <td>-0.132566</td>\n",
       "      <td>GORGEOUS RIVERSIDE LUXURY, WITH SPA AND POOL</td>\n",
       "      <td>7</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/088b6118-2...</td>\n",
       "      <td>154</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Entire home/flat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.554006</td>\n",
       "      <td>-0.242773</td>\n",
       "      <td>Nice &amp; Equipped Double Bedroom in Dollis Hill!...</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/88df04b4-b...</td>\n",
       "      <td>31</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.497491</td>\n",
       "      <td>-0.060441</td>\n",
       "      <td>3. Lovely Room in Centre London+Wifi</td>\n",
       "      <td>1</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/d349ce62-e...</td>\n",
       "      <td>36</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.512330</td>\n",
       "      <td>-0.066251</td>\n",
       "      <td>Small Room 10 -near Tower of London and Shored...</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/4f4e9ad7-d...</td>\n",
       "      <td>40</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.476615</td>\n",
       "      <td>-0.132970</td>\n",
       "      <td>SMALL Stockwell station single room-£17</td>\n",
       "      <td>1</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/35f34bb4-1...</td>\n",
       "      <td>18</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.489384</td>\n",
       "      <td>-0.099203</td>\n",
       "      <td>Double Bedroom Apartment with Balcony in Zone 1</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/54855d11-1...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.525883</td>\n",
       "      <td>-0.100857</td>\n",
       "      <td>Bright and welcoming flat in central London</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/86509cc9-0...</td>\n",
       "      <td>91</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Entire home/flat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.517944</td>\n",
       "      <td>-0.069536</td>\n",
       "      <td>CS11 Nice Single Room Central London</td>\n",
       "      <td>1</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/0e601324-0...</td>\n",
       "      <td>33</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.523661</td>\n",
       "      <td>-0.172201</td>\n",
       "      <td>Cosy One Bed 3rd Floor Close to Marble Arch</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/2666a91d-0...</td>\n",
       "      <td>62</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Entire home/flat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.492525</td>\n",
       "      <td>-0.096608</td>\n",
       "      <td>Room Big Ben + Breakfast (R5a)</td>\n",
       "      <td>1</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/f589d5df-3...</td>\n",
       "      <td>19</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Shared room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.491423</td>\n",
       "      <td>-0.139555</td>\n",
       "      <td>Brand new studio in Victoria 11A</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/11490113/4...</td>\n",
       "      <td>72</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Entire home/flat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.510864</td>\n",
       "      <td>-0.181942</td>\n",
       "      <td>6 Bed Mixed Dormitory Ensuite</td>\n",
       "      <td>6</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/4d8c2dec-0...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Shared room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.451279</td>\n",
       "      <td>0.015362</td>\n",
       "      <td>Exceptional room zone 3 by station</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/4edf624d-0...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.510506</td>\n",
       "      <td>-0.129266</td>\n",
       "      <td>Trafalgar Square, Peaceful Room.\\nFemale frien...</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/e0265be9-4...</td>\n",
       "      <td>58</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.555259</td>\n",
       "      <td>-0.252571</td>\n",
       "      <td>Charming Room, 2 Double beds, ensuite (BA-II)</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/ddf9b001-b...</td>\n",
       "      <td>33</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.479652</td>\n",
       "      <td>-0.169762</td>\n",
       "      <td>Luxurious double @ heart of london</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/1528064c-5...</td>\n",
       "      <td>36</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.580548</td>\n",
       "      <td>-0.233645</td>\n",
       "      <td>SUPER VALUE, PRETTY &amp; SAFE, EASY ACCESS TO CENTRE</td>\n",
       "      <td>1</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/5a878555-6...</td>\n",
       "      <td>24</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.416261</td>\n",
       "      <td>-0.101107</td>\n",
       "      <td>The Snug - Completely independent studio</td>\n",
       "      <td>3</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/a7b7ac9c-6...</td>\n",
       "      <td>43</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Entire home/flat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.496717</td>\n",
       "      <td>-0.101200</td>\n",
       "      <td>Great central London flat</td>\n",
       "      <td>6</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/0c6c56c4-a...</td>\n",
       "      <td>119</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Entire home/flat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.470231</td>\n",
       "      <td>-0.090346</td>\n",
       "      <td>Cosy double in creative, social garden house</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/cb24dc8f-3...</td>\n",
       "      <td>33</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.530786</td>\n",
       "      <td>-0.056957</td>\n",
       "      <td>- Double Room Shoreditch \"Sparrow\", 2min to Train</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/5d305b15-f...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.521552</td>\n",
       "      <td>-0.045261</td>\n",
       "      <td>Central bedroom, close to underground, perfect!</td>\n",
       "      <td>3</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/c66b07ea-e...</td>\n",
       "      <td>36</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.499814</td>\n",
       "      <td>-0.113586</td>\n",
       "      <td>**AMAZING CITY CENTRE APARTMENT**</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/79e6ba4f-0...</td>\n",
       "      <td>71</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.492054</td>\n",
       "      <td>-0.096903</td>\n",
       "      <td>Safestay London Elephant &amp; Castle</td>\n",
       "      <td>6</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/d59e71e6-4...</td>\n",
       "      <td>15</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Shared room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.519988</td>\n",
       "      <td>-0.041878</td>\n",
       "      <td>(ION-B) Private room for 4 in London</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/58980ef0-1...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.516530</td>\n",
       "      <td>-0.062436</td>\n",
       "      <td>(4BFORD-3)PRIVATE ROOM FOR 2 CLOSE TO TOWER BR...</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/cf9a8ff6-5...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.489336</td>\n",
       "      <td>-0.230869</td>\n",
       "      <td>The Muse Haus II - Riverside Room</td>\n",
       "      <td>3</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/da06a628-0...</td>\n",
       "      <td>49</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.553402</td>\n",
       "      <td>-0.241072</td>\n",
       "      <td>Large Balcony Double Bedroom in Dollis Hill! BR5</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/64093166-7...</td>\n",
       "      <td>31</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.512739</td>\n",
       "      <td>-0.226962</td>\n",
       "      <td>THE QUEENS HOSTEL , 6 BED MIXED DORM C</td>\n",
       "      <td>6</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/d5752e25-1...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Shared room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.539388</td>\n",
       "      <td>-0.202903</td>\n",
       "      <td>Double Room in Contemporary Flat</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/26516a3a-1...</td>\n",
       "      <td>40</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.490277</td>\n",
       "      <td>-0.097092</td>\n",
       "      <td>Room Big Ben + Breakfast. Zone 1 (Q3)</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/82998940/6...</td>\n",
       "      <td>30</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.492041</td>\n",
       "      <td>-0.017314</td>\n",
       "      <td>Large Double with Private Bathroom-Canary Whar...</td>\n",
       "      <td>3</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/6df596d1-4...</td>\n",
       "      <td>31</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.533898</td>\n",
       "      <td>-0.130299</td>\n",
       "      <td>Double in Kings Cross Houseshare</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/57128460/a...</td>\n",
       "      <td>88</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.497440</td>\n",
       "      <td>-0.009651</td>\n",
       "      <td>Room next to Canary Wharf and Greenwich</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/e747b1f1-1...</td>\n",
       "      <td>31</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.551801</td>\n",
       "      <td>-0.239277</td>\n",
       "      <td>Garden Facing Double Bedroom in Dollis Hill! BR4</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/da6da5bc-7...</td>\n",
       "      <td>31</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.500920</td>\n",
       "      <td>-0.114502</td>\n",
       "      <td>LUXURY CITY CENTRE APARTMENT</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/050fc4a8-1...</td>\n",
       "      <td>51</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.548562</td>\n",
       "      <td>-0.225982</td>\n",
       "      <td>Clean Double Room 5min walk to Underground Sta...</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/bfe713da-b...</td>\n",
       "      <td>38</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.492564</td>\n",
       "      <td>-0.096076</td>\n",
       "      <td>Nice Room Big Ben + Breakfast (R7a)</td>\n",
       "      <td>1</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/60cf80f1-8...</td>\n",
       "      <td>22</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Shared room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.525872</td>\n",
       "      <td>-0.067217</td>\n",
       "      <td>(SP-C)PRIVATE ROOM FOR 4 NEAR SHOREDITCH</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/a4c8f54c-f...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.517147</td>\n",
       "      <td>-0.039077</td>\n",
       "      <td>(CROM-E)PRIVATE ROOM FOR 4 NEAR REGENT'S CANAL</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/e5d9092c-e...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.513970</td>\n",
       "      <td>-0.047868</td>\n",
       "      <td>(WE-1) ROOM FOR 4 NEAR STEPNEY GREEN PARK/GARDEN</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/fbf383e5-9...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.524006</td>\n",
       "      <td>-0.063918</td>\n",
       "      <td>(8RAM-A)Private Room for 4ppl near Victoria Park</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/06d594e2-a...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.517652</td>\n",
       "      <td>-0.038606</td>\n",
       "      <td>(92AST-2)Private rooms up to 4 near Mile End Park</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/6e5c7acc-d...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.513839</td>\n",
       "      <td>-0.055826</td>\n",
       "      <td>(ROB-B)PRIVATE ROOM FOR 4 PPL NEAR RIVERSIDE</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/29046148-5...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.527791</td>\n",
       "      <td>-0.068263</td>\n",
       "      <td>(MCD-D) PRIVATE ROOM UP TO 4 CLOSE TO BRICK LANE</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/f8d3886c-6...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.517796</td>\n",
       "      <td>-0.038177</td>\n",
       "      <td>(92AST-4)PRIVATE ROOM FOR 2 NEAR VICTORIA PARK</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/9f38c016-a...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.523549</td>\n",
       "      <td>-0.065682</td>\n",
       "      <td>(8RAM-C)Private Room for 3ppl near Victoria Park</td>\n",
       "      <td>3</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/f9c1ac2d-4...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.520828</td>\n",
       "      <td>-0.067859</td>\n",
       "      <td>(KING-D)PRIVATE ROOM FOR 3 PPL IN BRICK LANE</td>\n",
       "      <td>3</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/05413b50-5...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.515282</td>\n",
       "      <td>-0.047395</td>\n",
       "      <td>(WE-4)PRIVATE ROOM FOR 2 NEAR STEPNEY GREEN PARK</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/8686ff10-3...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.518104</td>\n",
       "      <td>-0.068196</td>\n",
       "      <td>(59CH-1) PRIVATE ROOM FOR 4 BRICK LANE</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/d8786c02-1...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.513845</td>\n",
       "      <td>-0.064276</td>\n",
       "      <td>(HAD-D)PRIVATE ROOM CLOSE TO TOWER BRIDGE</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/7f1928e5-6...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.509966</td>\n",
       "      <td>-0.061409</td>\n",
       "      <td>(BET-A)PRIVATE ROOM FOR 4 NEAR RIVERSIDE</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/10e3fcc8-c...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.527649</td>\n",
       "      <td>-0.068401</td>\n",
       "      <td>(MCD-C) PRIVATE ROOM UP TO 3 CLOSE TO BRICK LANE</td>\n",
       "      <td>3</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/59bbf736-6...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.519125</td>\n",
       "      <td>-0.069282</td>\n",
       "      <td>(43CHIC-B)PRIVATE ROOM FOR 2 IN BRICK LANE</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/9298d330-b...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.510940</td>\n",
       "      <td>-0.018134</td>\n",
       "      <td>(32GRU-D)PRIVATE ROOM FOR 2 PEOPLE NEAR RIVERSIDE</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/60424b3b-1...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.511244</td>\n",
       "      <td>-0.053994</td>\n",
       "      <td>(73SHAD-5)PRIVATE ROOM FOR 2 PEOPLE NEAR RIVER...</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/ff71eaa5-b...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.525535</td>\n",
       "      <td>-0.065950</td>\n",
       "      <td>(8RAM-B)Private Room for 2ppl near Victoria Park</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/4d783775-2...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.526277</td>\n",
       "      <td>-0.072128</td>\n",
       "      <td>(KR-D)PRIVATE ROOM FOR 2 IN SHOREDITCH</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/3e7bb23b-a...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.527191</td>\n",
       "      <td>-0.071177</td>\n",
       "      <td>(KR-A)PRIVATE ROOM FOR 4 IN SHOREDITCH/BALCONY</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/f7ffd8de-c...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.511228</td>\n",
       "      <td>-0.058300</td>\n",
       "      <td>(26MOR-B)PRIVATE ROOM FOR 4 PPL NEAR RIVERSIDE</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/7792ef9a-f...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.510086</td>\n",
       "      <td>-0.061221</td>\n",
       "      <td>(BET-D)PRIVATE ROOM FOR 2 NEAR RIVERSIDE</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/2a47f778-0...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.512000</td>\n",
       "      <td>-0.063476</td>\n",
       "      <td>(HAL-A)PRIVATE ROOM FOR 4 NEAR TOWER BRIDGE</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/8e7e39fd-8...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.512355</td>\n",
       "      <td>-0.065909</td>\n",
       "      <td>(HAD-C)PRIVATE ROOM CLOSE TO TOWER BRIDGE</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/2bfbd251-5...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.512548</td>\n",
       "      <td>-0.018926</td>\n",
       "      <td>(32GRU-B)PRIVATE ROOM FOR 2 PEOPLE NEAR RIVERSIDE</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/4030ee3b-8...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.515882</td>\n",
       "      <td>-0.037798</td>\n",
       "      <td>(50AST-1)PRIVATE ROOM FOR 4 NEAR MILE END PARK</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/d7437f04-4...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.523891</td>\n",
       "      <td>-0.052398</td>\n",
       "      <td>(BRAI-D)PRIVATE ROOM FOR 4 NEAR BRICK LANE</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/b11142b5-f...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.532256</td>\n",
       "      <td>-0.083022</td>\n",
       "      <td>(CHA-A) PRIVATE ROOM IN HOXTON WITH BALCONY FOR 4</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/6619ba13-b...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.535244</td>\n",
       "      <td>-0.089366</td>\n",
       "      <td>(CROP-A)PRIVATE ROOM FOR 4 PEOPLE IN SHOREDITCH</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/84969292-3...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.535351</td>\n",
       "      <td>-0.084282</td>\n",
       "      <td>(FUL-B)PRIVATE ROOM FOR 4 NEAR HOXTON</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/a7c18ecc-2...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.516583</td>\n",
       "      <td>-0.063056</td>\n",
       "      <td>(4AFORD-2)PRIVATE ROOM FOR 4PPL NEAR TOWER BRIDGE</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/291306f2-4...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.553648</td>\n",
       "      <td>-0.241361</td>\n",
       "      <td>Huge &amp; Bright Double Bedroom in Dollis Hill! BR1</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/6f44c440-8...</td>\n",
       "      <td>31</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.496208</td>\n",
       "      <td>-0.071039</td>\n",
       "      <td>New Central London double with private bathroom</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/7ddf6bf6-4...</td>\n",
       "      <td>58</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>82</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.498859</td>\n",
       "      <td>-0.086188</td>\n",
       "      <td>Single box room at london bridge</td>\n",
       "      <td>1</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/74d672bc-e...</td>\n",
       "      <td>22</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.518006</td>\n",
       "      <td>-0.167318</td>\n",
       "      <td>Cosy 2nd Floor 1 Bed Flat Close To Oxford Street</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/b9df22db-6...</td>\n",
       "      <td>67</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Entire home/flat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.510704</td>\n",
       "      <td>-0.050297</td>\n",
       "      <td>(GRD-A)PRIVATE ROOM UP TO 4 WITH CITY VIEWS</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/39cd4175-5...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.506991</td>\n",
       "      <td>-0.067635</td>\n",
       "      <td>* Zone 1 * FLAT * NO CLEANING FEES *</td>\n",
       "      <td>6</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/440765d6-8...</td>\n",
       "      <td>80</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Entire home/flat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.527842</td>\n",
       "      <td>-0.071902</td>\n",
       "      <td>(KR-C)PRIVATE ROOM FOR 4 IN SHOREDITCH</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/3a9cff13-d...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.521531</td>\n",
       "      <td>-0.139337</td>\n",
       "      <td>Room C Brilliant Location</td>\n",
       "      <td>3</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/bef5920b-7...</td>\n",
       "      <td>46</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.532407</td>\n",
       "      <td>-0.063612</td>\n",
       "      <td>(SBH-C)PRIVATE ROOM FOR 4 PEOPLE NEAR SHOREDITCH</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/e13000eb-6...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.466348</td>\n",
       "      <td>-0.192849</td>\n",
       "      <td>Beautiful 2 bed Fulham apartment</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/53d529ea-2...</td>\n",
       "      <td>98</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Entire home/flat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.490963</td>\n",
       "      <td>-0.096486</td>\n",
       "      <td>Room Big Ben + Breakfast (R5b)</td>\n",
       "      <td>1</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/1f4b6234-3...</td>\n",
       "      <td>18</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Shared room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.594978</td>\n",
       "      <td>-0.081565</td>\n",
       "      <td>Bright &amp; Spacious Double Room</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/72012623/a...</td>\n",
       "      <td>28</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.513420</td>\n",
       "      <td>-0.065957</td>\n",
       "      <td>(HAD-A)PRIVATE ROOM CLOSE TO TOWER BRIDGE</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/27e16557-8...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.534635</td>\n",
       "      <td>-0.072653</td>\n",
       "      <td>(GOD-A)PRIVATE ROOM FOR 3 NEAR REGENTS CANAL</td>\n",
       "      <td>3</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/d81ee2b9-6...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.492320</td>\n",
       "      <td>-0.015128</td>\n",
       "      <td>Double Room - Canary Wharf hp</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/4cebcd26-b...</td>\n",
       "      <td>30</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.516983</td>\n",
       "      <td>-0.027006</td>\n",
       "      <td>(TA-4) Private room for 2 close to Mile End Park</td>\n",
       "      <td>2</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/5f8370ba-c...</td>\n",
       "      <td>21</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.503962</td>\n",
       "      <td>-0.109114</td>\n",
       "      <td>5. Lovely Room Free WIFI Centre London</td>\n",
       "      <td>1</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/5de10973-b...</td>\n",
       "      <td>51</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.427872</td>\n",
       "      <td>-0.070807</td>\n",
       "      <td>Comfy, Clean Room For 1, Central London(FREE W...</td>\n",
       "      <td>1</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/ea2c17f9-9...</td>\n",
       "      <td>22</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.532777</td>\n",
       "      <td>-0.065245</td>\n",
       "      <td>(SBH-B)PRIVATE ROOM UP TO 4 PEOPLE NEAR SHORED...</td>\n",
       "      <td>4</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/c83c76d3-1...</td>\n",
       "      <td>25</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>GBP</td>\n",
       "      <td>51.474246</td>\n",
       "      <td>-0.194698</td>\n",
       "      <td>Little Corner of Relaxation &amp; Rest</td>\n",
       "      <td>3</td>\n",
       "      <td>https://a0.muscache.com/im/pictures/9a453323-2...</td>\n",
       "      <td>36</td>\n",
       "      <td>nightly</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Private room</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML(pd.DataFrame(props).to_html())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "will skip code... but you can find nice surprise in any wallmart product code :)\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "Case Study 4 - json embeded in html\n",
    "\n",
    "You want analyse and compare some retailing corporations. One of them is Wallmart. You want to get as much \n",
    "details, about products they are selling, as you can. You have investigated your target well, but infortunately\n",
    "you cannot find any hidden API... You need to get data from HTML. You already found nice sitemaps, got product\n",
    "pages and now you're ready to scrape.\n",
    "\n",
    "What will be most efficient and robust way to get all details about products?\n",
    "\"\"\"\n",
    "\n",
    "print(\"will skip code... but you can find nice surprise in any wallmart product code :)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Handling JavaScript\n",
    "\n",
    "### Why...?\n",
    "- Sometimes content of webpage can be dynamically presented/altered via JavaScript code\n",
    "- when you're dowlonading HTML, it can be completly different from what you see on browser\n",
    "- you need to perform some sort of interaction with page\n",
    "- your target have some fancy anti-scraping software detecting that you're a bot\n",
    "\n",
    "###  Selenium+PhantomJS\n",
    "- Selenium is browser automation tool most often used for testing web application.\n",
    "- It can be usefull while scraping\n",
    "- PhantomJS is just headless browser (there is no UI and it works in background)\n",
    "- BTW: you can use Selenium with any other browser (Firefox, Opera etc.)\n",
    "\n",
    "### It's often an overkill thoguh!\n",
    "- Scraping with Selenium+PhantomJS is much heavier than using simple Python libraries!\n",
    "    + you have to have all additional libraries and software installed\n",
    "    + it may be slower\n",
    "    + you have to navigate as you were a human (eg. find button element and click it programatically)\n",
    "    + you rely on page layout (not robust at all...)\n",
    "- Very often you can find work-around it. For example, if you try to deal with infinite scroll,\n",
    "  you can investigate what AJAX requests your browser is sending while scrolling (and emulate it)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No. of quotes is:  60\n",
      "“A day without sunshine is like, you know, night.”\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "Case Study 5 - how to (not)handle infinite scroll\n",
    "\n",
    "You are planning to spam you friends on Facebook with random quotes to show how smart and deep you are.\n",
    "You found an amazing page with quotes, but... it contains infinite scroll?! Don't worry though! \n",
    "As they say: \"You have to look through the rain to see the rainbow.\"\n",
    "\n",
    "Your task is to get all quotes from http://spidyquotes.herokuapp.com/\n",
    "\"\"\"\n",
    "spidyquotes_url = 'http://spidyquotes.herokuapp.com/scroll'\n",
    "# with Selenium+PhantomJS (in general - bad option. but yeah... may be fancy)\n",
    "driver = webdriver.PhantomJS('/Applications/phantomjs-2.1.1-macosx/bin/phantomjs')\n",
    "driver.get(spidyquotes_url)\n",
    "no_of_scrolls = 5\n",
    "scroll = 0\n",
    "while scroll < no_of_scrolls:\n",
    "    # do a fancy screenshoot here\n",
    "    driver.get_screenshot_as_file('/Users/stulski/Desktop/osobiste/pydata_meetup/shot_{}.jpg'.format(scroll))\n",
    "    # scroll down\n",
    "    driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n",
    "    time.sleep(1)\n",
    "    scroll += 1\n",
    "quote_elements = driver.find_elements_by_class_name('quote')\n",
    "all_quotes = [element.find_elements_by_class_name('text')[0].text for element in quote_elements]\n",
    "print('No. of quotes is: ', len(all_quotes))\n",
    "print(random.choice(all_quotes))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No. of quotes is:  90\n",
      "“Not all of us can do great things. But we can do small things with great love.”\n"
     ]
    }
   ],
   "source": [
    "# same as above, without unnecessery hassel\n",
    "p_idx = 1\n",
    "spidyquotes_better_url = 'http://spidyquotes.herokuapp.com/api/quotes?page='\n",
    "r = json.loads(requests.get(spidyquotes_better_url+str(p_idx)).text)\n",
    "time.sleep(1)\n",
    "all_quotes = []\n",
    "while r['has_next'] == True:\n",
    "    for quote in r['quotes']:\n",
    "        all_quotes.append(quote['text'])\n",
    "    p_idx += 1\n",
    "    r = json.loads(requests.get(spidyquotes_better_url+str(p_idx)).text)\n",
    "    time.sleep(1)\n",
    "print('No. of quotes is: ', len(all_quotes))\n",
    "print(random.choice(all_quotes))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Keynotes and advices\n",
    "* investigate you target well (sitemaps, hidden apis, how it works under-the-hood)\n",
    "* use incognito mode whie exploring\n",
    "* use developers tools\n",
    "* think about scraping as a \"hacking\" activity rather than parsing just getting html elements\n",
    "* change your user-agent\n",
    "* add time.sleep if you can afford it\n",
    "* same data are in different places at the website. find those easy to scrape!\n",
    "* if you need to parse HTML and get data from there - try to find something which will not break (avoid finding   general elements like DIVs and then finding Nth of those)\n",
    "* look for comonalities\n",
    "* websites are different are there is no one magical way to get your data\n",
    "* it looks easy when I'm showing it, but sometimes it takes time to reverse engineer websites\n",
    "\n",
    "** if you want production level scrapers - use proxy"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}