Craw_data use Selenium.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SELENIUM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: selenium in /home/tructt2/anaconda3/lib/python3.7/site-packages (3.141.0)\r\n",
      "Requirement already satisfied: urllib3 in /home/tructt2/anaconda3/lib/python3.7/site-packages (from selenium) (1.24.1)\r\n"
     ]
    }
   ],
   "source": [
    "!pip install selenium"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.webdriver.chrome.options import Options"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "chrome_options = Options()\n",
    "chrome_options.add_argument(\"--incognito\")\n",
    "chrome_options.add_argument(\"--window-size=1920x1080\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to install chrome driver in ubuntu, window\n",
    "I. Install in window\n",
    "https://sites.google.com/a/chromium.org/chromedriver/home\n",
    "dowload file chromedriver suitable with computer , and cursor direct to executable_path\n",
    "II. Install in Ubuntu\n",
    "Open terminal ( Ctr Alt T)\n",
    "1. wget https://chromedriver.storage.googleapis.com/2.41/chromedriver_linux64.zip\n",
    "2. unzip chromedriver_linux64.zip \\n\n",
    "3. sudo mv chromedriver /usr/bin/chromedriver \\n\n",
    "4. sudo chown root:root /usr/bin/chromedriver\n",
    "5. sudo chmod +x /usr/bin/chromedriver"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/tructt2/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:1: DeprecationWarning: use options instead of chrome_options\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    }
   ],
   "source": [
    "driver = webdriver.Chrome(chrome_options=chrome_options, executable_path=\"/usr/bin/chromedriver\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create request first"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://news.ycombinator.com/\"\n",
    "driver.get(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wait response from page\n",
    "Add latency to load javascript"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "url = \"https://news.ycombinator.com/\"\n",
    "driver.get(url)\n",
    "time.sleep(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interactive with data in web"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selenium cung cấp một số cách để truy cập các phần tử trên trang (https://selenium-python.readthedocs.io/locating-elements.html). Các phương pháp thường được sử dụng là:\n",
    "\n",
    "1. Elements by id: cách này bạn sẽ cần kiểm tra mã nguồn và tìm id của phần tử\n",
    "2. Elements by css_selector: css selector là một cách rất hiệu quả để chọn các phần tử trên một trang\n",
    "3. Elements by Xpath: là một ngôn ngữ truy vấn để chọn các nút từ một tài liệu XML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "##Find elements with Selenium\n",
    "#by_id\n",
    "els = driver.find_elements_by_id('.storylink')\n",
    "#by css\n",
    "# els = driver.find_elements_by_css_selector(element_css_selector)\n",
    "# #by xpath\n",
    "# els = driver.find_elements_by_xpath(element_x_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selenium trả về các đối tượng mà sau đó bạn có thể truy vấn. Ví dụ, nếu bạn muốn nhận:\n",
    "\n",
    "1. text hiển thị trong phần tử: els.text\n",
    "2. đường dẫn href url: el.get_attribute(“href”)\n",
    "3. thuộc tính src : el.get_attribute(“src”)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "storyTitles ['Post Mortem of Google Outage on 14 December 2020', 'Perceus: Garbage Free Reference Counting with Reuse [pdf]', 'Virtual Machine Detection in the Browser (2019)', \"Cyberpunk 2077: how 2020's biggest video game launch turned into a shambles\", 'Do It Yourself Bookbinding', 'Surgery, the Ultimate Placebo', 'Streaming Is Stalling: Can Music Keep Up in the Attention Economy?', \"FDA grants authorization to Moderna's Covid-19 vaccine\", 'Ganja.js: Geometric Algebra Generator for JavaScript, C++, C#, Rust, Python', 'Surrealist painting found in recycling bin at German airport', 'Show HN: Sqwok – A low-cruft, minimalist alternative to Reddit and Twitter', 'Zoom executive charged with disrupting meetings commemorating Tiananmen Square', 'The CPUs of Spacecraft Computers in Space', 'Postdocs under pressure: ‘Can I even do this any more?’', 'The Notation: Ken Iverson Centenary', 'That Dothraki Horde, Part III: Horse Fiddles', 'Adapting to Means Testing', 'DRY Is a Trade-Off', 'Stimulating Consciousness', 'Why can we smell copper?', 'Seuss-Star Trek mashup crashes and burns at Ninth Circuit', 'Cerulean: Experimental micro-blogging app for Matrix', 'A happy ending for King Lear? Trauma of plague caused Shakespeare to change it', 'Aphorisms on programming language design (2017)', 'Why Java Is Dying', 'A modern port of Turbo Vision 2.0', \"mRNA's next challenge: Will it work as a drug?\", 'Repl.it (YC W18) is hiring designers to build better programming environments', 'Improving Cross-Browser Testing, Part 1: Web Application Testing Today', 'It took two days to develop Moderna’s vaccine']\n",
      "storyUrls ['http://status.cloud.google.com/incident/zall/20013#20013004', 'https://www.microsoft.com/en-us/research/uploads/prod/2020/11/perceus-tr-v1.pdf', 'https://bannedit.github.io/Virtual-Machine-Detection-In-The-Browser.html', 'https://www.theguardian.com/games/2020/dec/18/cyberpunk-2077-how-2020s-biggest-video-game-launch-turned-into-a-shambles', 'http://www.diybookbinding.com/', 'https://www.skepdoc.info/ian-harris-on-surgery-the-ultimate-placebo/', 'https://www.billboard.com/articles/business/9500546/streaming-music-growth-stalled-factors-attention-economy', 'https://www.statnews.com/2020/12/18/fda-eua-moderna-vaccine-covid19/', 'https://github.com/enkimute/ganja.js', 'https://www.smithsonianmag.com/smart-news/340000-surrealist-painting-was-found-recycling-bin-180976542/', 'https://sqwok.im/', 'https://www.justice.gov/opa/pr/china-based-executive-us-telecommunications-company-charged-disrupting-video-meetings', 'http://www.cpushack.com/space-craft-cpu.html', 'https://www.nature.com/articles/d41586-020-03235-y', 'https://kx.com/blog/the-notation/', 'https://acoup.blog/2020/12/18/collections-that-dothraki-horde-part-iii-horse-fiddles/', 'https://www.jefftk.com/p/adapting-to-means-testing', 'https://orbifold.xyz/dry-trade-off.html', 'https://dana.org/article/stimulating-consciousness/', 'https://chemistry.stackexchange.com/a/7917', 'https://www.courthousenews.com/seuss-star-trek-mash-up-crash-lands-with-ninth-circuit/', 'https://matrix.org/blog/2020/12/18/introducing-cerulean', 'https://www.theguardian.com/culture/2020/dec/13/a-happy-ending-for-king-lear-trauma-of-plague-caused-shakespeare-to-change-plays-finale', 'http://www.rntz.net/post/2017-01-27-aphorisms-on-pl-design.html', 'https://medium.com/better-programming/why-java-is-dying-b02b5fd44db9', 'https://github.com/magiblot/tvision', 'https://science.sciencemag.org/content/370/6523/1388.full', 'https://jobs.lever.co/replit/4ef495be-6ced-42d9-bcc1-3c6f57a346bd', 'https://hacks.mozilla.org/2020/12/cross-browser-testing-part-1-web-app-testing-today/', 'https://theness.com/neurologicablog/index.php/it-took-two-days-to-develop-modernas-vaccine/']\n"
     ]
    }
   ],
   "source": [
    "# Get all title of document, url of document\n",
    "elements = driver.find_elements_by_css_selector(\".storylink\")\n",
    "storyTitles = [el.text for el in elements]\n",
    "storyUrls = [el.get_attribute(\"href\") for el in elements]\n",
    "print('storyTitles', storyTitles)\n",
    "print('storyUrls', storyUrls)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['505 points', '39 points', '86 points', '24 points', '132 points', '141 points', '23 points', '127 points', '155 points', '84 points', '150 points', '458 points', '128 points', '43 points', '51 points', '56 points', '29 points', '147 points', '11 points', '34 points', '91 points', '184 points', '74 points', '6 points', '7 points', '41 points', '190 points', '57 points', '307 points']\n",
      "['https://news.ycombinator.com/from?site=cloud.google.com', 'https://news.ycombinator.com/from?site=microsoft.com', 'https://news.ycombinator.com/from?site=bannedit.github.io', 'https://news.ycombinator.com/from?site=theguardian.com', 'https://news.ycombinator.com/from?site=diybookbinding.com', 'https://news.ycombinator.com/from?site=skepdoc.info', 'https://news.ycombinator.com/from?site=billboard.com', 'https://news.ycombinator.com/from?site=statnews.com', 'https://news.ycombinator.com/from?site=github.com/enkimute', 'https://news.ycombinator.com/from?site=smithsonianmag.com', 'https://news.ycombinator.com/from?site=sqwok.im', 'https://news.ycombinator.com/from?site=justice.gov', 'https://news.ycombinator.com/from?site=cpushack.com', 'https://news.ycombinator.com/from?site=nature.com', 'https://news.ycombinator.com/from?site=kx.com', 'https://news.ycombinator.com/from?site=acoup.blog', 'https://news.ycombinator.com/from?site=jefftk.com', 'https://news.ycombinator.com/from?site=orbifold.xyz', 'https://news.ycombinator.com/from?site=dana.org', 'https://news.ycombinator.com/from?site=stackexchange.com', 'https://news.ycombinator.com/from?site=courthousenews.com', 'https://news.ycombinator.com/from?site=matrix.org', 'https://news.ycombinator.com/from?site=theguardian.com', 'https://news.ycombinator.com/from?site=rntz.net', 'https://news.ycombinator.com/from?site=medium.com/better-programming', 'https://news.ycombinator.com/from?site=github.com/magiblot', 'https://news.ycombinator.com/from?site=science.sciencemag.org', 'https://news.ycombinator.com/from?site=lever.co', 'https://news.ycombinator.com/from?site=hacks.mozilla.org', 'https://news.ycombinator.com/from?site=theness.com']\n"
     ]
    }
   ],
   "source": [
    "# Get score and domain of document\n",
    "elements = driver.find_elements_by_css_selector(\".score\")\n",
    "scores = [el.text for el in elements]\n",
    "elements = driver.find_elements_by_css_selector(\".sitebit a\")\n",
    "sites = [el.get_attribute(\"href\") for el in elements]\n",
    "print(scores)\n",
    "print(sites)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Perform Javascript"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Click button"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "els = driver.find_element_by_css_selector(\".storylink\") \n",
    "#option 1: regular click\n",
    "els.click()\n",
    "#option 2: inject js\n",
    "# driver.execute_script(\"arguments[0].click();\", els)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scroll"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Connect all link in page"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "#find all the links\n",
    "links = driver.find_elements_by_partial_link_text('')\n",
    "#get the first one\n",
    "l = links[0]\n",
    "#click on it\n",
    "driver.execute_script(\"arguments[0].click();\", l)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tối ưu hóa lưu lượng tải"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vô hiệu hóa hiển thị hình ảnh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "chrome_options = Options()\n",
    "prefs={\"profile.managed_default_content_settings.images\": 2}\n",
    "chrome_options.add_experimental_option('prefs', prefs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tạo cache"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "chrome_options = Options()\n",
    "prefs={'disk-cache-size': 4096}\n",
    "chrome_options.add_experimental_option('prefs', prefs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chạy ngầm, không hiển thị UI trên trình duyệt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "chrome_options = Options()\n",
    "chrome_options.add_argument(\"--headless\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CHụp ảnh màn hình"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "driver = webdriver.Chrome()\n",
    "driver.get('https://twitter.com/')\n",
    "driver.save_screenshot(\"screenshot.png\")\n",
    "driver.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tải ảnh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import urllib\n",
    "from selenium import webdriver\n",
    "\n",
    "driver = webdriver.Chrome()\n",
    "driver.get('https://twitter.com/')\n",
    "\n",
    "# get the image source\n",
    "img = driver.find_element_by_css_selector('.Icon--bird')\n",
    "src = img.get_attribute('src')\n",
    "\n",
    "# download the image\n",
    "urllib.urlretrieve(src, \"twitter_icon.png\")\n",
    "driver.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Thực thi chuỗi hành động "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "menu = driver.find_element_by_css_selector(\".nav\") # get css class = nav\n",
    "hidden_submenu = driver.find_element_by_css_selector(\".nav #submenu1\") # get css class = nav, id = submenul\n",
    "\n",
    "ActionChains(driver).move_to_element(menu).click(hidden_submenu).perform()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cuộn chuột trong 1 thẻ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium.webdriver.support.ui import Select\n",
    "value = \"optionValue\"\n",
    "element = Select(driver.find_element_by_tag_name(\"select\")\n",
    "for option in element.options: \n",
    "if option.get_attribute(\"value\") == value: \n",
    "element.select_by_visible_text(option.text)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}