Web-Scraping-Lab.ipynb

{"cells":[{"cell_type":"markdown","metadata":{},"source":["<p style=\"text-align:center\">\n","    <a href=\"https://skills.network/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork928-2022-01-01\" target=\"_blank\">\n","    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\"  />\n","    </a>\n","</p>\n"]},{"cell_type":"markdown","metadata":{},"source":["# **Hands-on Lab : Web Scraping**\n"]},{"cell_type":"markdown","metadata":{},"source":["Estimated time needed: **30 to 45** minutes\n"]},{"cell_type":"markdown","metadata":{},"source":["## Objectives\n"]},{"cell_type":"markdown","metadata":{},"source":["In this lab you will perform the following:\n"]},{"cell_type":"markdown","metadata":{},"source":["* Extract information from a given web site \n","* Write the scraped data into a csv file.\n"]},{"cell_type":"markdown","metadata":{},"source":["## Extract information from the given web site\n","You will extract the data from the below web site: <br> \n"]},{"cell_type":"code","execution_count":17,"metadata":{},"outputs":[],"source":["#this url contains the data you need to scrape\n","url = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/datasets/Programming_Languages.html\""]},{"cell_type":"markdown","metadata":{},"source":["The data you need to scrape is the **name of the programming language** and **average annual salary**.<br> It is a good idea to open the url in your web broswer and study the contents of the web page before you start to scrape.\n"]},{"cell_type":"markdown","metadata":{},"source":["Import the required libraries\n"]},{"cell_type":"code","execution_count":18,"metadata":{},"outputs":[],"source":["# Your code here\n","\n","\n","import pandas as pd \n","\n","import requests\n","from bs4 import BeautifulSoup \n","import selenium "]},{"cell_type":"markdown","metadata":{},"source":["Download the webpage at the url\n"]},{"cell_type":"code","execution_count":19,"metadata":{},"outputs":[],"source":["#your code goes here\n","\n","table_dict = {}\n","\n","sal = [] \n","lang = []"]},{"cell_type":"code","execution_count":20,"metadata":{},"outputs":[],"source":["r = requests.get(url).text "]},{"cell_type":"markdown","metadata":{},"source":["Create a soup object\n"]},{"cell_type":"code","execution_count":21,"metadata":{},"outputs":[],"source":["#your code goes here\n","\n","soup = BeautifulSoup(r , 'lxml')\n"]},{"cell_type":"code","execution_count":22,"metadata":{},"outputs":[],"source":["table = soup.find_all('table')\n","my_table = table[0]"]},{"cell_type":"markdown","metadata":{},"source":["Scrape the `Language name` and `annual average salary`.\n"]},{"cell_type":"code","execution_count":23,"metadata":{},"outputs":[],"source":["for row in my_table.find_all('tr') : \n","    cols = row.find_all('td') \n","\n","    lang_col = cols[1].getText()\n","    lang.append(lang_col)\n","\n","    salary_col = cols[3].getText()\n","    sal.append(salary_col)"]},{"cell_type":"code","execution_count":24,"metadata":{},"outputs":[],"source":["table_dict = {\n","    'Languages' : lang , \n","    'Salary' : sal \n","}"]},{"cell_type":"markdown","metadata":{},"source":["Save the scrapped data into a file named *popular-languages.csv*\n"]},{"cell_type":"code","execution_count":25,"metadata":{},"outputs":[],"source":["table_df = pd.DataFrame(table_dict)\n","\n","df = table_df[1:]\n","\n","df.to_csv('popular-languages.csv')"]},{"cell_type":"markdown","metadata":{},"source":["## Authors\n"]},{"cell_type":"markdown","metadata":{},"source":["Ramesh Sannareddy\n"]},{"cell_type":"markdown","metadata":{},"source":["### Other Contributors\n"]},{"cell_type":"markdown","metadata":{},"source":["Rav Ahuja\n"]},{"cell_type":"markdown","metadata":{},"source":["## Change Log\n"]},{"cell_type":"markdown","metadata":{},"source":["|  Date (YYYY-MM-DD) |  Version | Changed By  |  Change Description |\n","|---|---|---|---|\n","| 2020-10-17  | 0.1  | Ramesh Sannareddy  |  Created initial version of the lab |\n"]},{"cell_type":"markdown","metadata":{},"source":[" Copyright &copy; 2020 IBM Corporation. This notebook and its source code are released under the terms of the [MIT License](https://cognitiveclass.ai/mit-license/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork928-2022-01-01).\n"]}],"metadata":{"kernelspec":{"display_name":"webscrapper","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.6"},"vscode":{"interpreter":{"hash":"a5de0eb3511747baffff99c3af738513a637dd1c1167328c5fd178c6b127ed74"}}},"nbformat":4,"nbformat_minor":4}