lesson-6-url-and-api/Lesson 6 Fetching Data from API.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lecture 6—Fetching data online"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Version 1.0, by [Makzan](https://makzan.net). Last updated 2021 March."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this series, we will use 3 lectures to learn fetching data online. This includes:\n",
    "\n",
    "- **Finding patterns in URL**\n",
    "- **Open web URL**\n",
    "- **Downloading files in Python**\n",
    "- **Fetch data with API**\n",
    "- Web scraping with Requests and BeautifulSoup\n",
    "- Web automation with Selenium\n",
    "- Converting Wikipedia tabular data into CSV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finding patterns in URL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "We need to know the URL In order to download files, or web scrap a web page. Usually it is finding the variable patterns in URL.\n",
    "Edit\n",
    "For example, from the following URL, we can find the pattern of the search query.\n",
    "\n",
    "\n",
    "- https://docs.python.org/3/search.html?q=namedtuple&check_keywords=yes&area=default\n",
    "- https://duckduckgo.com/?q=python+doc\n",
    "- https://www.google.com/maps/search/Libraries/@22.1612464,113.5303786,13z\n",
    "- http://macaodaily.com/html/2020-05/04/node_2.htm\n",
    "- http://www.dicj.gov.mo/web/cn/information/DadosEstat_mensal/2020/index.html\n",
    "- https://bis.dsat.gov.mo:37812/macauweb/routeLine.html?routeName=3&direction=0&language=zh-tw&ver=3.5.12"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s take a closer look at DSAT.gov.mo bus route page. If we can the bus routes, we can observe that the page URL doesn’t change. There may be 2 reasons:\n",
    "\n",
    "1. The page changes are generated via JavaScript rendering.\n",
    "2. The page is inside an iframe so that page changes do not change the top-level URL.\n",
    "\n",
    "\n",
    "If it is the first reason, we will need a more advanced browser driver technique. If it is the second reason, we can get the URL by opening the link in a new tab, or simply copying the link location via right-click.\n",
    "\n",
    "Now we can observe the URL for each route has the following pattern.\n",
    "\n",
    "https://bis.dsat.gov.mo:37812/macauweb/routeLine.html?routeName=3&direction=0&language=zh-tw&ver=3.5.12\n",
    "\n",
    "![](dsat-bus.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take DICJ.gov.mo example, the URL is:\n",
    "\n",
    "http://www.dicj.gov.mo/web/cn/information/DadosEstat_mensal/2020/index.html\n",
    "\n",
    "If we inspect the network requests, we can find the behind-the-scene XML URL:\n",
    "\n",
    "http://www.dicj.gov.mo/web/cn/information/DadosEstat_mensal/2020/report_cn.xml?id=10\n",
    "\n",
    "![](dicj.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Web Search Python Documentation Site"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, we can speed up our daily operation just by automatically opening the URL that we need. We can use `webbrowser` to do so.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import webbrowser\n",
    "\n",
    "query = \"webbrowser\"\n",
    "\n",
    "url = f\"https://docs.python.org/3/search.html?q={query}&check_keywords=yes&area=default\"\n",
    "\n",
    "webbrowser.open(url)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**✏️ Exercise Time**\n",
    "    \n",
    "Please try to turn the query into an input asking for the search query:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import webbrowser\n",
    "\n",
    "### Start writing your code here\n",
    "None\n",
    "### End writing your code\n",
    "\n",
    "webbrowser.open(url)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|Expected question to ask|\n",
    "|---|\n",
    "|Please input search query to search Python doc: |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Searching DuckDuckGo search engine\n",
    "\n",
    "DuckDuckGo search engine allows going to the first search result by adding an exclamation mark (!) in the query string. We will use this feature to create a Python script.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import webbrowser\n",
    "\n",
    "query = \"Python history\"\n",
    "\n",
    "url = f\"https://duckduckgo.com?q=!+{query}\"\n",
    "\n",
    "webbrowser.open(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**✏️ Exercise Time**\n",
    "\n",
    "Please try to turn the query into an input asking for the search query:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import webbrowser\n",
    "\n",
    "### Start writing your code here\n",
    "None\n",
    "### End writing your code\n",
    "\n",
    "webbrowser.open(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|Expected question to ask|\n",
    "|---|\n",
    "|Please input search query : |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Google map search near Macao"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import webbrowser\n",
    "\n",
    "query = \"Book store\"\n",
    "\n",
    "# A map search in Macao.\n",
    "url = f\"https://www.google.com/maps/search/{query}/@22.1612464,113.5303786,13z\"\n",
    "\n",
    "webbrowser.open(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](google-map-result.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**✏️ Exercise Time**\n",
    "\n",
    "Try to turn the map location into Shanghai."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import webbrowser\n",
    "\n",
    "query = \"Book store\"\n",
    "\n",
    "# Start writing your code here\n",
    "latitude = None\n",
    "longitude = None\n",
    "zoom_level = 13\n",
    "url = f\"https://www.google.com/maps/search/{query}/@{latitude},{longitude},{zoom_level}z\"\n",
    "\n",
    "webbrowser.open(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## URL for iOS apps\n",
    "\n",
    "\n",
    "In iOS, we can use x-callback-url to interact with apps in iOS by using Python and Pythonista.\n",
    "\n",
    "There are web site that collects x-callback-url for iOS apps:\n",
    "\n",
    "http://x-callback-url.com/apps/\n",
    "\n",
    "For example, Things—A tasks manager—provides x-callback-url API:\n",
    "\n",
    "https://culturedcode.com/things/support/articles/2803573/\n",
    "\n",
    "Another example that Bear—notes taking iOS app—provides x-callback-url API too.\n",
    "\n",
    "https://bear.app/faq/X-callback-url%20Scheme%20documentation/\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downloading files\n",
    "\n",
    "We can use `urlretrieve` from `urllib.request` module to download file.\n",
    "\n",
    "For example, we can download geckdriver.zip file from their Github repository with the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('chromedriver.zip', <http.client.HTTPMessage object at 0x1091cd350>)\n"
     ]
    }
   ],
   "source": [
    "'''Download chart from AAStock server with given stock numbers.'''\n",
    "\n",
    "from urllib.request import urlretrieve\n",
    "\n",
    "stock_numbers = ['0001','0005','0011','0700','3333','0002','0012']\n",
    "\n",
    "for stock_number in stock_numbers:\n",
    "    url = \"http://charts.aastocks.com/servlet/Charts?fontsize=12&15MinDelay=T&lang=1&titlestyle=1&vol=1&Indicator=1&indpara1=10&indpara2=20&indpara3=50&indpara4=100&indpara5=150&subChart1=2&ref1para1=14&ref1para2=0&ref1para3=0&subChart2=3&ref2para1=12&ref2para2=26&ref2para3=9&subChart3=12&ref3para1=0&ref3para2=0&ref3para3=0&scheme=3&com=100&chartwidth=660&chartheight=855&stockid=00{}.HK&period=6&type=1&logoStyle=1\".format(stock_number)\n",
    "    urlretrieve(url, '{}-chart.gif'.format(stock_number))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fetching XML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting untangle\n",
      "  Downloading untangle-1.1.1.tar.gz (3.1 kB)\n",
      "Building wheels for collected packages: untangle\n",
      "  Building wheel for untangle (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for untangle: filename=untangle-1.1.1-py3-none-any.whl size=3410 sha256=678ed047367a6d024ab37d3d424ef606a5d3de48f1d2aa254c5acdb9da946713\n",
      "  Stored in directory: /Users/makzan/Library/Caches/pip/wheels/b9/a9/9c/45580c8b7a00e3e79b889e8e78a4f3427fff5a4d48f1cfea0a\n",
      "Successfully built untangle\n",
      "Installing collected packages: untangle\n",
      "Successfully installed untangle-1.1.1\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install untangle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: SMG.gov.mo\n",
    "\n",
    "xml.smg.gov.mo\n",
    "\n",
    "![](xml.smg.gov.mo.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "現時澳門氣溫 30 度，濕度 81%。\n"
     ]
    }
   ],
   "source": [
    "import untangle\n",
    "import datetime\n",
    "\n",
    "obj = untangle.parse('https://xml.smg.gov.mo/c_actual_brief.xml')\n",
    "\n",
    "temperature = obj.ActualWeatherBrief.Custom.Temperature.Value.cdata\n",
    "humidity = obj.ActualWeatherBrief.Custom.Humidity.Value.cdata\n",
    "\n",
    "print(\"現時澳門氣溫 \" + temperature + \" 度，濕度 \" + humidity + \"%。\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There may be error when running the code above, depending on how many \"Temperature\" data are there from SMG.gov.mo.\n",
    "\n",
    "If there are only one `Temperature` data, it is a direct access. If there are more than one `Temperature` data, it becomes a list. We can determine if it is a list by checking `type(target) == list`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type([]) == list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "現時澳門氣溫 30 度，濕度 81%。\n"
     ]
    }
   ],
   "source": [
    "import untangle\n",
    "import datetime\n",
    "\n",
    "obj = untangle.parse('https://xml.smg.gov.mo/c_actual_brief.xml')\n",
    "\n",
    "humidity = obj.ActualWeatherBrief.Custom.Humidity.Value.cdata\n",
    "\n",
    "if type(obj.ActualWeatherBrief.Custom.Temperature) == list:\n",
    "    temperature = obj.ActualWeatherBrief.Custom.Temperature[0].Value.cdata\n",
    "else:\n",
    "    temperature = obj.ActualWeatherBrief.Custom.Temperature.Value.cdata\n",
    "\n",
    "\n",
    "print(\"現時澳門氣溫 \" + temperature + \" 度，濕度 \" + humidity + \"%。\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: 博彩月計毛收入\n",
    "\n",
    "http://www.dicj.gov.mo/web/cn/information/DadosEstat_mensal/index.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020 年 5 月份 毛收入 1,764 (2019:25,952), -93.2%\n",
      "2020 年 5 月份 累計毛收入 33,004 (2019:125,691), -73.7%\n"
     ]
    }
   ],
   "source": [
    "import untangle\n",
    "import datetime\n",
    "\n",
    "year = datetime.date.today().year\n",
    "\n",
    "# list begins at 0, and we look for previous month.\n",
    "month = datetime.date.today().month -1 -1\n",
    "\n",
    "if last_month < 0:\n",
    "    year = year - 1\n",
    "    last_month = 11 # list beings at 0.\n",
    "\n",
    "url = f\"http://www.dicj.gov.mo/web/cn/information/DadosEstat_mensal/{year}/report_cn.xml?id=8\"\n",
    "\n",
    "data = untangle.parse(url)\n",
    "\n",
    "month_data = data.STATISTICS.REPORT.DATA.RECORD[month]\n",
    "\n",
    "net_income = month_data.DATA[1].cdata\n",
    "last_net_income = month_data.DATA[2].cdata\n",
    "change_rate = month_data.DATA[3].cdata\n",
    "acc_net_income = month_data.DATA[4].cdata\n",
    "acc_last_net_income = month_data.DATA[5].cdata\n",
    "acc_change_rate = month_data.DATA[6].cdata\n",
    "\n",
    "print(f\"{year} 年 {month+1} 月份 毛收入 {net_income} ({year-1}:{last_net_income}), {change_rate}\")\n",
    "print(f\"{year} 年 {month+1} 月份 累計毛收入 {acc_net_income} ({year-1}:{acc_last_net_income}), {acc_change_rate}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 過去 12 個月博彩月計毛收入"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fetch_and_print_dicj_year_month(year, month):\n",
    "    url = f\"http://www.dicj.gov.mo/web/cn/information/DadosEstat_mensal/{year}/report_cn.xml?id=8\"\n",
    "\n",
    "    data = untangle.parse(url)\n",
    "\n",
    "    month_data = data.STATISTICS.REPORT.DATA.RECORD[month]\n",
    "\n",
    "    net_income = month_data.DATA[1].cdata\n",
    "    last_net_income = month_data.DATA[2].cdata\n",
    "    change_rate = month_data.DATA[3].cdata\n",
    "    acc_net_income = month_data.DATA[4].cdata\n",
    "    acc_last_net_income = month_data.DATA[5].cdata\n",
    "    acc_change_rate = month_data.DATA[6].cdata\n",
    "\n",
    "    print(f\"{year} 年 {month+1}  月份 毛收入\\t {net_income} \\t ({year-1}:{last_net_income}), {change_rate}\")\n",
    "#     print(f\"{year} 年 {month+1} 累計毛收入\\t {acc_net_income}\\t ({year-1}:{acc_last_net_income}), {acc_change_rate}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2019 年 6  月份 毛收入\t 23,812 \t (2018:22,490), 5.9%\n",
      "2019 年 7  月份 毛收入\t 24,453 \t (2018:25,327), -3.5%\n",
      "2019 年 8  月份 毛收入\t 24,262 \t (2018:26,559), -8.6%\n",
      "2019 年 9  月份 毛收入\t 22,079 \t (2018:21,952), 0.6%\n",
      "2019 年 10  月份 毛收入\t 26,443 \t (2018:27,328), -3.2%\n",
      "2019 年 11  月份 毛收入\t 22,877 \t (2018:24,995), -8.5%\n",
      "2019 年 12  月份 毛收入\t 22,838 \t (2018:26,468), -13.7%\n",
      "2020 年 1  月份 毛收入\t 22,126 \t (2019:24,942), -11.3%\n",
      "2020 年 2  月份 毛收入\t 3,104 \t (2019:25,370), -87.8%\n",
      "2020 年 3  月份 毛收入\t 5,257 \t (2019:25,840), -79.7%\n",
      "2020 年 4  月份 毛收入\t 754 \t (2019:23,588), -96.8%\n",
      "2020 年 5  月份 毛收入\t 1,764 \t (2019:25,952), -93.2%\n"
     ]
    }
   ],
   "source": [
    "import untangle\n",
    "import datetime\n",
    "\n",
    "for i in range(-12,0):    \n",
    "    date = datetime.date.today() + datetime.timedelta(days=i*30)    \n",
    "    fetch_and_print_dicj_year_month(date.year, date.month-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Exchange Rate API\n",
    "\n",
    "https://exchangeratesapi.io"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'rates': {'HKD': 1.0935529258}, 'base': 'CNY', 'date': '2020-06-17'}\n",
      "1.0935529258\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "import requests\n",
    "\n",
    "url = \"https://api.exchangeratesapi.io/latest?symbols=HKD&base=CNY\"\n",
    "\n",
    "response = requests.get(url)\n",
    "data = json.loads(response.text)\n",
    "print(data)\n",
    "\n",
    "print(data['rates']['HKD'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}