homework/Day_011_Regular_Expression_HW.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 正規表達式練習\n",
    "## 在網路爬蟲當中，正規表達式常常用來過濾以及搜尋特定的pattern字串。\n",
    "## 今天要來練習過濾IP address，以及URL。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re  #載入re模組"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 定義一個函數，用來測試是否能匹配正規表達式\n",
    "def RegexMatchingTest(regex, input_text):\n",
    "    #將正規表達式轉換成pattern\n",
    "    pattern = re.compile(regex)\n",
    "    \n",
    "    # 使轉換後的pattern，來測試是否匹配\n",
    "    result = re.search(pattern, input_text)\n",
    "\n",
    "    if result:\n",
    "        # 匹配完的結果會儲存在group()的屬性中，我們可以把匹配的結果列印出來\n",
    "        print(\"Matched: %s\" % (result.group()))\n",
    "        \n",
    "        if result.lastindex is not None:\n",
    "            # group(0)代表整個字串，group(1)、group(2)...代表分組中，匹配的內容\n",
    "            for i in range(0, result.lastindex+1):\n",
    "                print(\"  group(%d): %s\" % (i, result.group(i)))\n",
    "    else:\n",
    "        print(\"Not matched.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 用正規表達式過濾IP address。\n",
    "#### 一個合法的網路IP address，其格式為：X.X.X.X, 其中X是0~255的數字。我們可以用一個regex，來表達IP address的內容。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matched: 216.58.200.227\n",
      "  group(0): 216.58.200.227\n",
      "  group(1): 216\n",
      "  group(2): 58\n",
      "  group(3): 200\n",
      "  group(4): 227\n"
     ]
    }
   ],
   "source": [
    "test_string = \"Google IP address is 216.58.200.227\"\n",
    "\n",
    "# 過濾IP address的regex pattern\n",
    "regex = '(\\d{1,3}).(\\d{1,3}).(\\d{1,3}).(\\d{1,3})'\n",
    "RegexMatchingTest(regex, test_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 以上是最簡單的regex寫法。但深入思考，上面的regex也能夠匹配444.555.666.777這種無效的IP address。\n",
    "#### 我們必須再雕琢regex，只接受[0~255].[0~255].[0~255].[0~255]這種合法的IP address，而過濾不合法的IP。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matched: 216.58.200.22\n",
      "  group(0): 216.58.200.22\n",
      "  group(1): 200.\n",
      "  group(2): 200\n",
      "  group(3): 58\n",
      "  group(4): None\n",
      "  group(5): 200\n",
      "  group(6): None\n",
      "  group(7): 22\n",
      "Not matched.\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "    Your code here.\n",
    "    hint: 把IP可能出現的數字範圍，分開來思考\n",
    "          1. 000 ~ 199\n",
    "          2. 200 ~ 249\n",
    "          3. 250 ~ 255\n",
    "'''\n",
    "regex = '(((\\d{1,2})|(1\\d{2})|(2[0-4]\\d)|(25[0-5]))\\.){3}((\\d{1,2})|(1\\d{2})|(2[0-4]\\d)|(25[0-5]))'\n",
    "\n",
    "test_string1 = \"Test IP 216.58.200.227\"\n",
    "RegexMatchingTest(regex, test_string1)  #測試表達式是否會匹配此合法IP\n",
    "\n",
    "test_string2 = \"Test IP 999.888.777.666\"\n",
    "RegexMatchingTest(regex, test_string2)  #測試表達式是否會匹配此不合法IP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 用正規表達式過濾URL。\n",
    "#### 在網頁爬蟲中，常常會有外部連結的A tag，例如：\n",
    "< a href=\"https://movies.yahoo.com.tw/movietime_result.html/id=9467\"> 時刻表 < /a >\n",
    "#### 我們要把\"href=\"之後的URL擷取出來，用來做後續處理。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matched: https://movies.yahoo.com.tw/movietime_result.html/id=9467\n"
     ]
    }
   ],
   "source": [
    "html_a_tag = \"<a href=https://movies.yahoo.com.tw/movietime_result.html/id=9467> 時刻表 </a>\"\n",
    "\n",
    "#regex = '((http|https|ftp):\\/\\/)([a-z0-9]\\.|[a-z0-9][-a-z0-9]*[a-z0-9]\\.)+(tw|cn|com|edu|gov|net|org|biz|info|name)[-a-z0-9_:@&?=+,.!\\/~*\\'%$]*'\n",
    "regex = 'https\\://\\D*\\w*'\n",
    "RegexMatchingTest(regex, html_a_tag)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}