Data Collection with Web Scraping.ipynb

{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "# **Space X  Falcon 9 First Stage Landing Prediction**\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "## Web scraping Falcon 9 and Falcon Heavy Launches Records from Wikipedia\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "In this lab, you will be performing web scraping to collect Falcon 9 historical launch records from a Wikipedia page titled `List of Falcon 9 and Falcon Heavy launches`\n\n[https://en.wikipedia.org/wiki/List_of_Falcon\\_9\\_and_Falcon_Heavy_launches](https://en.wikipedia.org/wiki/List_of_Falcon\\_9\\_and_Falcon_Heavy_launches?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDS0321ENSkillsNetwork26802033-2021-01-01)\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "![](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DS0321EN-SkillsNetwork/labs/module\\_1\\_L2/images/Falcon9\\_rocket_family.svg)\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Falcon 9 first stage will land successfully\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "![](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DS0701EN-SkillsNetwork/api/Images/landing\\_1.gif)\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Several examples of an unsuccessful landing are shown here:\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "![](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DS0701EN-SkillsNetwork/api/Images/crash.gif)\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "More specifically, the launch records are stored in a HTML table shown below:\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "![](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DS0321EN-SkillsNetwork/labs/module\\_1\\_L2/images/falcon9-launches-wiki.png)\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "## Objectives\n\nWeb scrap Falcon 9 launch records with `BeautifulSoup`:\n\n*   Extract a Falcon 9 launch records HTML table from Wikipedia\n*   Parse the table and convert it into a Pandas data frame\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "First let's import required packages for this lab\n"
        },
        {
            "cell_type": "code",
            "execution_count": 1,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": "Requirement already satisfied: beautifulsoup4 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (4.9.3)\nRequirement already satisfied: soupsieve>1.2 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from beautifulsoup4) (2.2.1)\nRequirement already satisfied: requests in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (2.25.1)\nRequirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from requests) (2021.10.8)\nRequirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from requests) (1.26.6)\nRequirement already satisfied: idna<3,>=2.5 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from requests) (2.8)\nRequirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/envs/Python-3.8-main/lib/python3.8/site-packages (from requests) (3.0.4)\n"
                }
            ],
            "source": "!pip3 install beautifulsoup4\n!pip3 install requests"
        },
        {
            "cell_type": "code",
            "execution_count": 2,
            "metadata": {},
            "outputs": [],
            "source": "import sys\n\nimport requests\nfrom bs4 import BeautifulSoup\nimport re\nimport unicodedata\nimport pandas as pd\nimport numpy as np"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "and we will provide some helper functions for you to process web scraped HTML table\n"
        },
        {
            "cell_type": "code",
            "execution_count": 3,
            "metadata": {},
            "outputs": [],
            "source": "def date_time(table_cells):\n    \"\"\"\n    This function returns the data and time from the HTML  table cell\n    Input: the  element of a table data cell extracts extra row\n    \"\"\"\n    return [data_time.strip() for data_time in list(table_cells.strings)][0:2]\n\ndef booster_version(table_cells):\n    \"\"\"\n    This function returns the booster version from the HTML  table cell \n    Input: the  element of a table data cell extracts extra row\n    \"\"\"\n    out=''.join([booster_version for i,booster_version in enumerate( table_cells.strings) if i%2==0][0:-1])\n    return out\n\ndef landing_status(table_cells):\n    \"\"\"\n    This function returns the landing status from the HTML table cell \n    Input: the  element of a table data cell extracts extra row\n    \"\"\"\n    out=[i for i in table_cells.strings][0]\n    return out\n\n\ndef get_mass(table_cells):\n    mass=unicodedata.normalize(\"NFKD\", table_cells.text).strip()\n    if mass:\n        mass.find(\"kg\")\n        new_mass=mass[0:mass.find(\"kg\")+2]\n    else:\n        new_mass=0\n    return new_mass\n\n\ndef extract_column_from_header(row):\n    \"\"\"\n    This function returns the landing status from the HTML table cell \n    Input: the  element of a table data cell extracts extra row\n    \"\"\"\n    if (row.br):\n        row.br.extract()\n    if row.a:\n        row.a.extract()\n    if row.sup:\n        row.sup.extract()\n        \n    colunm_name = ' '.join(row.contents)\n    \n    # Filter the digit and empty names\n    if not(colunm_name.strip().isdigit()):\n        colunm_name = colunm_name.strip()\n        return colunm_name    \n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "To keep the lab tasks consistent, you will be asked to scrape the data from a snapshot of the  `List of Falcon 9 and Falcon Heavy launches` Wikipage updated on\n`9th June 2021`\n"
        },
        {
            "cell_type": "code",
            "execution_count": 4,
            "metadata": {},
            "outputs": [],
            "source": "static_url = \"https://en.wikipedia.org/w/index.php?title=List_of_Falcon_9_and_Falcon_Heavy_launches&oldid=1027686922\""
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Next, request the HTML page from the above URL and get a `response` object\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "### TASK 1: Request the Falcon9 Launch Wiki page from its URL\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "First, let's perform an HTTP GET method to request the Falcon9 Launch HTML page, as an HTTP response.\n"
        },
        {
            "cell_type": "code",
            "execution_count": 5,
            "metadata": {},
            "outputs": [],
            "source": "# use requests.get() method with the provided static_url\n# assign the response to a object\nresponse=requests.get(static_url).text"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Create a `BeautifulSoup` object from the HTML `response`\n"
        },
        {
            "cell_type": "code",
            "execution_count": 6,
            "metadata": {},
            "outputs": [],
            "source": "# Use BeautifulSoup() to create a BeautifulSoup object from a response text content\nsoup=BeautifulSoup(response,\"html.parser\")"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Print the page title to verify if the `BeautifulSoup` object was created properly\n"
        },
        {
            "cell_type": "code",
            "execution_count": 7,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/plain": "<title>List of Falcon 9 and Falcon Heavy launches - Wikipedia</title>"
                    },
                    "execution_count": 7,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": "# Use soup.title attribute\nsoup.title"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "### TASK 2: Extract all column/variable names from the HTML table header\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Next, we want to collect all relevant column names from the HTML table header\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Let's try to find all tables on the wiki page first. If you need to refresh your memory about `BeautifulSoup`, please check the external reference link towards the end of this lab\n"
        },
        {
            "cell_type": "code",
            "execution_count": 8,
            "metadata": {},
            "outputs": [],
            "source": "# Use the find_all function in the BeautifulSoup object, with element type `table`\n# Assign the result to a list called `html_tables`\nhtml_tables=[]\nhtml_tables=soup.find_all('table')"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Starting from the third table is our target table contains the actual launch records.\n"
        },
        {
            "cell_type": "code",
            "execution_count": 9,
            "metadata": {
                "scrolled": false
            },
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": "<table class=\"wikitable plainrowheaders collapsible\" style=\"width: 100%;\">\n<tbody><tr>\n<th scope=\"col\">Flight No.\n</th>\n<th scope=\"col\">Date and<br/>time (<a href=\"/wiki/Coordinated_Universal_Time\" title=\"Coordinated Universal Time\">UTC</a>)\n</th>\n<th scope=\"col\"><a href=\"/wiki/List_of_Falcon_9_first-stage_boosters\" title=\"List of Falcon 9 first-stage boosters\">Version,<br/>Booster</a> <sup class=\"reference\" id=\"cite_ref-booster_11-0\"><a href=\"#cite_note-booster-11\">[b]</a></sup>\n</th>\n<th scope=\"col\">Launch site\n</th>\n<th scope=\"col\">Payload<sup class=\"reference\" id=\"cite_ref-Dragon_12-0\"><a href=\"#cite_note-Dragon-12\">[c]</a></sup>\n</th>\n<th scope=\"col\">Payload mass\n</th>\n<th scope=\"col\">Orbit\n</th>\n<th scope=\"col\">Customer\n</th>\n<th scope=\"col\">Launch<br/>outcome\n</th>\n<th scope=\"col\"><a href=\"/wiki/Falcon_9_first-stage_landing_tests\" title=\"Falcon 9 first-stage landing tests\">Booster<br/>landing</a>\n</th></tr>\n<tr>\n<th rowspan=\"2\" scope=\"row\" style=\"text-align:center;\">1\n</th>\n<td>4 June 2010,<br/>18:45\n</td>\n<td><a href=\"/wiki/Falcon_9_v1.0\" title=\"Falcon 9 v1.0\">F9 v1.0</a><sup class=\"reference\" id=\"cite_ref-MuskMay2012_13-0\"><a href=\"#cite_note-MuskMay2012-13\">[7]</a></sup><br/>B0003.1<sup class=\"reference\" id=\"cite_ref-block_numbers_14-0\"><a href=\"#cite_note-block_numbers-14\">[8]</a></sup>\n</td>\n<td><a href=\"/wiki/Cape_Canaveral_Space_Force_Station\" title=\"Cape Canaveral Space Force Station\">CCAFS</a>,<br/><a href=\"/wiki/Cape_Canaveral_Space_Launch_Complex_40\" title=\"Cape Canaveral Space Launch Complex 40\">SLC-40</a>\n</td>\n<td><a href=\"/wiki/Dragon_Spacecraft_Qualification_Unit\" title=\"Dragon Spacecraft Qualification Unit\">Dragon Spacecraft Qualification Unit</a>\n</td>\n<td>\n</td>\n<td><a href=\"/wiki/Low_Earth_orbit\" title=\"Low Earth orbit\">LEO</a>\n</td>\n<td><a href=\"/wiki/SpaceX\" title=\"SpaceX\">SpaceX</a>\n</td>\n<td class=\"table-success\" style=\"background: LightGreen; color: black; vertical-align: middle; text-align: center;\">Success\n</td>\n<td class=\"table-failure\" style=\"background: #ffbbbb; color: black; vertical-align: middle; text-align: center;\">Failure<sup class=\"reference\" id=\"cite_ref-ns20110930_15-0\"><a href=\"#cite_note-ns20110930-15\">[9]</a></sup><sup class=\"reference\" id=\"cite_ref-16\"><a href=\"#cite_note-16\">[10]</a></sup><br/><small>(parachute)</small>\n</td></tr>\n<tr>\n<td colspan=\"9\">First flight of Falcon 9 v1.0.<sup class=\"reference\" id=\"cite_ref-sfn20100604_17-0\"><a href=\"#cite_note-sfn20100604-17\">[11]</a></sup> Used a boilerplate version of Dragon capsule which was not designed to separate from the second stage.<small>(<a href=\"#First_flight_of_Falcon_9\">more details below</a>)</small> Attempted to recover the first stage by parachuting it into the ocean, but it burned up on reentry, before the parachutes even deployed.<sup class=\"reference\" id=\"cite_ref-parachute_18-0\"><a href=\"#cite_note-parachute-18\">[12]</a></sup>\n</td></tr>\n<tr>\n<th rowspan=\"2\" scope=\"row\" style=\"text-align:center;\">2\n</th>\n<td>8 December 2010,<br/>15:43<sup class=\"reference\" id=\"cite_ref-spaceflightnow_Clark_Launch_Report_19-0\"><a href=\"#cite_note-spaceflightnow_Clark_Launch_Report-19\">[13]</a></sup>\n</td>\n<td><a href=\"/wiki/Falcon_9_v1.0\" title=\"Falcon 9 v1.0\">F9 v1.0</a><sup class=\"reference\" id=\"cite_ref-MuskMay2012_13-1\"><a href=\"#cite_note-MuskMay2012-13\">[7]</a></sup><br/>B0004.1<sup class=\"reference\" id=\"cite_ref-block_numbers_14-1\"><a href=\"#cite_note-block_numbers-14\">[8]</a></sup>\n</td>\n<td><a href=\"/wiki/Cape_Canaveral_Space_Force_Station\" title=\"Cape Canaveral Space Force Station\">CCAFS</a>,<br/><a href=\"/wiki/Cape_Canaveral_Space_Launch_Complex_40\" title=\"Cape Canaveral Space Launch Complex 40\">SLC-40</a>\n</td>\n<td><a href=\"/wiki/SpaceX_Dragon\" title=\"SpaceX Dragon\">Dragon</a> <a class=\"mw-redirect\" href=\"/wiki/COTS_Demo_Flight_1\" title=\"COTS Demo Flight 1\">demo flight C1</a><br/>(Dragon C101)\n</td>\n<td>\n</td>\n<td><a href=\"/wiki/Low_Earth_orbit\" title=\"Low Earth orbit\">LEO</a> (<a href=\"/wiki/International_Space_Station\" title=\"International Space Station\">ISS</a>)\n</td>\n<td><div class=\"plainlist\">\n<ul><li><a href=\"/wiki/NASA\" title=\"NASA\">NASA</a> (<a href=\"/wiki/Commercial_Orbital_Transportation_Services\" title=\"Commercial Orbital Transportation Services\">COTS</a>)</li>\n<li><a href=\"/wiki/National_Reconnaissance_Office\" title=\"National Reconnaissance Office\">NRO</a></li></ul>\n</div>\n</td>\n<td class=\"table-success\" style=\"background: LightGreen; color: black; vertical-align: middle; text-align: center;\">Success<sup class=\"reference\" id=\"cite_ref-ns20110930_15-1\"><a href=\"#cite_note-ns20110930-15\">[9]</a></sup>\n</td>\n<td class=\"table-failure\" style=\"background: #ffbbbb; color: black; vertical-align: middle; text-align: center;\">Failure<sup class=\"reference\" id=\"cite_ref-ns20110930_15-2\"><a href=\"#cite_note-ns20110930-15\">[9]</a></sup><sup class=\"reference\" id=\"cite_ref-20\"><a href=\"#cite_note-20\">[14]</a></sup><br/><small>(parachute)</small>\n</td></tr>\n<tr>\n<td colspan=\"9\">Maiden flight of <a class=\"mw-redirect\" href=\"/wiki/Dragon_capsule\" title=\"Dragon capsule\">Dragon capsule</a>, consisting of over 3 hours of testing thruster maneuvering and reentry.<sup class=\"reference\" id=\"cite_ref-spaceflightnow_Clark_unleashing_Dragon_21-0\"><a href=\"#cite_note-spaceflightnow_Clark_unleashing_Dragon-21\">[15]</a></sup> Attempted to recover the first stage by parachuting it into the ocean, but it disintegrated upon reentry, before the parachutes were deployed.<sup class=\"reference\" id=\"cite_ref-parachute_18-1\"><a href=\"#cite_note-parachute-18\">[12]</a></sup> <small>(<a href=\"#COTS_demo_missions\">more details below</a>)</small> It also included two <a href=\"/wiki/CubeSat\" title=\"CubeSat\">CubeSats</a>,<sup class=\"reference\" id=\"cite_ref-NRO_Taps_Boeing_for_Next_Batch_of_CubeSats_22-0\"><a href=\"#cite_note-NRO_Taps_Boeing_for_Next_Batch_of_CubeSats-22\">[16]</a></sup> and a wheel of <a href=\"/wiki/Brou%C3%A8re\" title=\"Brou\u00e8re\">Brou\u00e8re</a> cheese.\n</td></tr>\n<tr>\n<th rowspan=\"2\" scope=\"row\" style=\"text-align:center;\">3\n</th>\n<td>22 May 2012,<br/>07:44<sup class=\"reference\" id=\"cite_ref-BBC_new_era_23-0\"><a href=\"#cite_note-BBC_new_era-23\">[17]</a></sup>\n</td>\n<td><a href=\"/wiki/Falcon_9_v1.0\" title=\"Falcon 9 v1.0\">F9 v1.0</a><sup class=\"reference\" id=\"cite_ref-MuskMay2012_13-2\"><a href=\"#cite_note-MuskMay2012-13\">[7]</a></sup><br/>B0005.1<sup class=\"reference\" id=\"cite_ref-block_numbers_14-2\"><a href=\"#cite_note-block_numbers-14\">[8]</a></sup>\n</td>\n<td><a href=\"/wiki/Cape_Canaveral_Space_Force_Station\" title=\"Cape Canaveral Space Force Station\">CCAFS</a>,<br/><a href=\"/wiki/Cape_Canaveral_Space_Launch_Complex_40\" title=\"Cape Canaveral Space Launch Complex 40\">SLC-40</a>\n</td>\n<td><a href=\"/wiki/SpaceX_Dragon\" title=\"SpaceX Dragon\">Dragon</a> <a class=\"mw-redirect\" href=\"/wiki/Dragon_C2%2B\" title=\"Dragon C2+\">demo flight C2+</a><sup class=\"reference\" id=\"cite_ref-C2_24-0\"><a href=\"#cite_note-C2-24\">[18]</a></sup><br/>(Dragon C102)\n</td>\n<td>525\u00a0kg (1,157\u00a0lb)<sup class=\"reference\" id=\"cite_ref-25\"><a href=\"#cite_note-25\">[19]</a></sup>\n</td>\n<td><a href=\"/wiki/Low_Earth_orbit\" title=\"Low Earth orbit\">LEO</a> (<a href=\"/wiki/International_Space_Station\" title=\"International Space Station\">ISS</a>)\n</td>\n<td><a href=\"/wiki/NASA\" title=\"NASA\">NASA</a> (<a href=\"/wiki/Commercial_Orbital_Transportation_Services\" title=\"Commercial Orbital Transportation Services\">COTS</a>)\n</td>\n<td class=\"table-success\" style=\"background: LightGreen; color: black; vertical-align: middle; text-align: center;\">Success<sup class=\"reference\" id=\"cite_ref-26\"><a href=\"#cite_note-26\">[20]</a></sup>\n</td>\n<td class=\"table-noAttempt\" style=\"background: #ececec; color: black; vertical-align: middle; white-space: nowrap; text-align: center;\">No attempt\n</td></tr>\n<tr>\n<td colspan=\"9\">Dragon spacecraft demonstrated a series of tests before it was allowed to approach the <a href=\"/wiki/International_Space_Station\" title=\"International Space Station\">International Space Station</a>. Two days later, it became the first commercial spacecraft to board the ISS.<sup class=\"reference\" id=\"cite_ref-BBC_new_era_23-1\"><a href=\"#cite_note-BBC_new_era-23\">[17]</a></sup> <small>(<a href=\"#COTS_demo_missions\">more details below</a>)</small>\n</td></tr>\n<tr>\n<th rowspan=\"3\" scope=\"row\" style=\"text-align:center;\">4\n</th>\n<td rowspan=\"2\">8 October 2012,<br/>00:35<sup class=\"reference\" id=\"cite_ref-SFN_LLog_27-0\"><a href=\"#cite_note-SFN_LLog-27\">[21]</a></sup>\n</td>\n<td rowspan=\"2\"><a href=\"/wiki/Falcon_9_v1.0\" title=\"Falcon 9 v1.0\">F9 v1.0</a><sup class=\"reference\" id=\"cite_ref-MuskMay2012_13-3\"><a href=\"#cite_note-MuskMay2012-13\">[7]</a></sup><br/>B0006.1<sup class=\"reference\" id=\"cite_ref-block_numbers_14-3\"><a href=\"#cite_note-block_numbers-14\">[8]</a></sup>\n</td>\n<td rowspan=\"2\"><a href=\"/wiki/Cape_Canaveral_Space_Force_Station\" title=\"Cape Canaveral Space Force Station\">CCAFS</a>,<br/><a href=\"/wiki/Cape_Canaveral_Space_Launch_Complex_40\" title=\"Cape Canaveral Space Launch Complex 40\">SLC-40</a>\n</td>\n<td><a href=\"/wiki/SpaceX_CRS-1\" title=\"SpaceX CRS-1\">SpaceX CRS-1</a><sup class=\"reference\" id=\"cite_ref-sxManifest20120925_28-0\"><a href=\"#cite_note-sxManifest20120925-28\">[22]</a></sup><br/>(Dragon C103)\n</td>\n<td>4,700\u00a0kg (10,400\u00a0lb)\n</td>\n<td><a href=\"/wiki/Low_Earth_orbit\" title=\"Low Earth orbit\">LEO</a> (<a href=\"/wiki/International_Space_Station\" title=\"International Space Station\">ISS</a>)\n</td>\n<td><a href=\"/wiki/NASA\" title=\"NASA\">NASA</a> (<a href=\"/wiki/Commercial_Resupply_Services\" title=\"Commercial Resupply Services\">CRS</a>)\n</td>\n<td class=\"table-success\" style=\"background: LightGreen; color: black; vertical-align: middle; text-align: center;\">Success\n</td>\n<td rowspan=\"2\" style=\"background:#ececec; text-align:center;\"><span class=\"nowrap\">No attempt</span>\n</td></tr>\n<tr>\n<td><a href=\"/wiki/Orbcomm_(satellite)\" title=\"Orbcomm (satellite)\">Orbcomm-OG2</a><sup class=\"reference\" id=\"cite_ref-Orbcomm_29-0\"><a href=\"#cite_note-Orbcomm-29\">[23]</a></sup>\n</td>\n<td>172\u00a0kg (379\u00a0lb)<sup class=\"reference\" id=\"cite_ref-gunter-og2_30-0\"><a href=\"#cite_note-gunter-og2-30\">[24]</a></sup>\n</td>\n<td><a href=\"/wiki/Low_Earth_orbit\" title=\"Low Earth orbit\">LEO</a>\n</td>\n<td><a href=\"/wiki/Orbcomm\" title=\"Orbcomm\">Orbcomm</a>\n</td>\n<td class=\"table-partial\" style=\"background: wheat; color: black; vertical-align: middle; text-align: center;\">Partial failure<sup class=\"reference\" id=\"cite_ref-nyt-20121030_31-0\"><a href=\"#cite_note-nyt-20121030-31\">[25]</a></sup>\n</td></tr>\n<tr>\n<td colspan=\"9\">CRS-1 was successful, but the <a href=\"/wiki/Secondary_payload\" title=\"Secondary payload\">secondary payload</a> was inserted into an abnormally low orbit and subsequently lost. This was due to one of the nine <a href=\"/wiki/SpaceX_Merlin\" title=\"SpaceX Merlin\">Merlin engines</a> shutting down during the launch, and NASA declining a second reignition, as per <a href=\"/wiki/International_Space_Station\" title=\"International Space Station\">ISS</a> visiting vehicle safety rules, the primary payload owner is contractually allowed to decline a second reignition. NASA stated that this was because SpaceX could not guarantee a high enough likelihood of the second stage completing the second burn successfully which was required to avoid any risk of secondary payload's collision with the ISS.<sup class=\"reference\" id=\"cite_ref-OrbcommTotalLoss_32-0\"><a href=\"#cite_note-OrbcommTotalLoss-32\">[26]</a></sup><sup class=\"reference\" id=\"cite_ref-sn20121011_33-0\"><a href=\"#cite_note-sn20121011-33\">[27]</a></sup><sup class=\"reference\" id=\"cite_ref-34\"><a href=\"#cite_note-34\">[28]</a></sup>\n</td></tr>\n<tr>\n<th rowspan=\"2\" scope=\"row\" style=\"text-align:center;\">5\n</th>\n<td>1 March 2013,<br/>15:10\n</td>\n<td><a href=\"/wiki/Falcon_9_v1.0\" title=\"Falcon 9 v1.0\">F9 v1.0</a><sup class=\"reference\" id=\"cite_ref-MuskMay2012_13-4\"><a href=\"#cite_note-MuskMay2012-13\">[7]</a></sup><br/>B0007.1<sup class=\"reference\" id=\"cite_ref-block_numbers_14-4\"><a href=\"#cite_note-block_numbers-14\">[8]</a></sup>\n</td>\n<td><a href=\"/wiki/Cape_Canaveral_Space_Force_Station\" title=\"Cape Canaveral Space Force Station\">CCAFS</a>,<br/><a href=\"/wiki/Cape_Canaveral_Space_Launch_Complex_40\" title=\"Cape Canaveral Space Launch Complex 40\">SLC-40</a>\n</td>\n<td><a href=\"/wiki/SpaceX_CRS-2\" title=\"SpaceX CRS-2\">SpaceX CRS-2</a><sup class=\"reference\" id=\"cite_ref-sxManifest20120925_28-1\"><a href=\"#cite_note-sxManifest20120925-28\">[22]</a></sup><br/>(Dragon C104)\n</td>\n<td>4,877\u00a0kg (10,752\u00a0lb)\n</td>\n<td><a href=\"/wiki/Low_Earth_orbit\" title=\"Low Earth orbit\">LEO</a> (<a class=\"mw-redirect\" href=\"/wiki/ISS\" title=\"ISS\">ISS</a>)\n</td>\n<td><a href=\"/wiki/NASA\" title=\"NASA\">NASA</a> (<a href=\"/wiki/Commercial_Resupply_Services\" title=\"Commercial Resupply Services\">CRS</a>)\n</td>\n<td class=\"table-success\" style=\"background: LightGreen; color: black; vertical-align: middle; text-align: center;\">Success\n</td>\n<td class=\"table-noAttempt\" style=\"background: #ececec; color: black; vertical-align: middle; white-space: nowrap; text-align: center;\">No attempt\n</td></tr>\n<tr>\n<td colspan=\"9\">Last launch of the original Falcon 9 v1.0 <a href=\"/wiki/Launch_vehicle\" title=\"Launch vehicle\">launch vehicle</a>, first use of the unpressurized trunk section of Dragon.<sup class=\"reference\" id=\"cite_ref-sxf9_20110321_35-0\"><a href=\"#cite_note-sxf9_20110321-35\">[29]</a></sup>\n</td></tr>\n<tr>\n<th rowspan=\"2\" scope=\"row\" style=\"text-align:center;\">6\n</th>\n<td>29 September 2013,<br/>16:00<sup class=\"reference\" id=\"cite_ref-pa20130930_36-0\"><a href=\"#cite_note-pa20130930-36\">[30]</a></sup>\n</td>\n<td><a href=\"/wiki/Falcon_9_v1.1\" title=\"Falcon 9 v1.1\">F9 v1.1</a><sup class=\"reference\" id=\"cite_ref-MuskMay2012_13-5\"><a href=\"#cite_note-MuskMay2012-13\">[7]</a></sup><br/>B1003<sup class=\"reference\" id=\"cite_ref-block_numbers_14-5\"><a href=\"#cite_note-block_numbers-14\">[8]</a></sup>\n</td>\n<td><a class=\"mw-redirect\" href=\"/wiki/Vandenberg_Air_Force_Base\" title=\"Vandenberg Air Force Base\">VAFB</a>,<br/><a href=\"/wiki/Vandenberg_Space_Launch_Complex_4\" title=\"Vandenberg Space Launch Complex 4\">SLC-4E</a>\n</td>\n<td><a href=\"/wiki/CASSIOPE\" title=\"CASSIOPE\">CASSIOPE</a><sup class=\"reference\" id=\"cite_ref-sxManifest20120925_28-2\"><a href=\"#cite_note-sxManifest20120925-28\">[22]</a></sup><sup class=\"reference\" id=\"cite_ref-CASSIOPE_MDA_37-0\"><a href=\"#cite_note-CASSIOPE_MDA-37\">[31]</a></sup>\n</td>\n<td>500\u00a0kg (1,100\u00a0lb)\n</td>\n<td><a href=\"/wiki/Polar_orbit\" title=\"Polar orbit\">Polar orbit</a> <a href=\"/wiki/Low_Earth_orbit\" title=\"Low Earth orbit\">LEO</a>\n</td>\n<td><a href=\"/wiki/Maxar_Technologies\" title=\"Maxar Technologies\">MDA</a>\n</td>\n<td class=\"table-success\" style=\"background: LightGreen; color: black; vertical-align: middle; text-align: center;\">Success<sup class=\"reference\" id=\"cite_ref-pa20130930_36-1\"><a href=\"#cite_note-pa20130930-36\">[30]</a></sup>\n</td>\n<td class=\"table-no2\" style=\"background: #ffdddd; color: black; vertical-align: middle; text-align: center;\">Uncontrolled<br/><small>(ocean)</small><sup class=\"reference\" id=\"cite_ref-ocean_landing_38-0\"><a href=\"#cite_note-ocean_landing-38\">[d]</a></sup>\n</td></tr>\n<tr>\n<td colspan=\"9\">First commercial mission with a private customer, first launch from Vandenberg, and demonstration flight of Falcon 9 v1.1 with an improved 13-tonne to LEO capacity.<sup class=\"reference\" id=\"cite_ref-sxf9_20110321_35-1\"><a href=\"#cite_note-sxf9_20110321-35\">[29]</a></sup> After separation from the second stage carrying Canadian commercial and scientific satellites, the first stage booster performed a controlled reentry,<sup class=\"reference\" id=\"cite_ref-39\"><a href=\"#cite_note-39\">[32]</a></sup> and an <a href=\"/wiki/Falcon_9_first-stage_landing_tests\" title=\"Falcon 9 first-stage landing tests\">ocean touchdown test</a> for the first time. This provided good test data, even though the booster started rolling as it neared the ocean, leading to the shutdown of the central engine as the roll depleted it of fuel, resulting in a hard impact with the ocean.<sup class=\"reference\" id=\"cite_ref-pa20130930_36-2\"><a href=\"#cite_note-pa20130930-36\">[30]</a></sup> This was the first known attempt of a rocket engine being lit to perform a supersonic retro propulsion, and allowed SpaceX to enter a public-private partnership with <a href=\"/wiki/NASA\" title=\"NASA\">NASA</a> and its Mars entry, descent, and landing technologies research projects.<sup class=\"reference\" id=\"cite_ref-40\"><a href=\"#cite_note-40\">[33]</a></sup> <small>(<a href=\"#Maiden_flight_of_v1.1\">more details below</a>)</small>\n</td></tr>\n<tr>\n<th rowspan=\"2\" scope=\"row\" style=\"text-align:center;\">7\n</th>\n<td>3 December 2013,<br/>22:41<sup class=\"reference\" id=\"cite_ref-sfn_wwls20130624_41-0\"><a href=\"#cite_note-sfn_wwls20130624-41\">[34]</a></sup>\n</td>\n<td><a href=\"/wiki/Falcon_9_v1.1\" title=\"Falcon 9 v1.1\">F9 v1.1</a><br/>B1004\n</td>\n<td><a href=\"/wiki/Cape_Canaveral_Space_Force_Station\" title=\"Cape Canaveral Space Force Station\">CCAFS</a>,<br/><a href=\"/wiki/Cape_Canaveral_Space_Launch_Complex_40\" title=\"Cape Canaveral Space Launch Complex 40\">SLC-40</a>\n</td>\n<td><a href=\"/wiki/SES-8\" title=\"SES-8\">SES-8</a><sup class=\"reference\" id=\"cite_ref-sxManifest20120925_28-3\"><a href=\"#cite_note-sxManifest20120925-28\">[22]</a></sup><sup class=\"reference\" id=\"cite_ref-spx-pr_42-0\"><a href=\"#cite_note-spx-pr-42\">[35]</a></sup><sup class=\"reference\" id=\"cite_ref-aw20110323_43-0\"><a href=\"#cite_note-aw20110323-43\">[36]</a></sup>\n</td>\n<td>3,170\u00a0kg (6,990\u00a0lb)\n</td>\n<td><a href=\"/wiki/Geostationary_transfer_orbit\" title=\"Geostationary transfer orbit\">GTO</a>\n</td>\n<td><a href=\"/wiki/SES_S.A.\" title=\"SES S.A.\">SES</a>\n</td>\n<td class=\"table-success\" style=\"background: LightGreen; color: black; vertical-align: middle; text-align: center;\">Success<sup class=\"reference\" id=\"cite_ref-SNMissionStatus7_44-0\"><a href=\"#cite_note-SNMissionStatus7-44\">[37]</a></sup>\n</td>\n<td class=\"table-noAttempt\" style=\"background: #ececec; color: black; vertical-align: middle; white-space: nowrap; text-align: center;\">No attempt<br/><sup class=\"reference\" id=\"cite_ref-sf10120131203_45-0\"><a href=\"#cite_note-sf10120131203-45\">[38]</a></sup>\n</td></tr>\n<tr>\n<td colspan=\"9\">First <a href=\"/wiki/Geostationary_transfer_orbit\" title=\"Geostationary transfer orbit\">Geostationary transfer orbit</a> (GTO) launch for Falcon 9,<sup class=\"reference\" id=\"cite_ref-spx-pr_42-1\"><a href=\"#cite_note-spx-pr-42\">[35]</a></sup> and first successful reignition of the second stage.<sup class=\"reference\" id=\"cite_ref-46\"><a href=\"#cite_note-46\">[39]</a></sup> SES-8 was inserted into a <a href=\"/wiki/Geostationary_transfer_orbit\" title=\"Geostationary transfer orbit\">Super-Synchronous Transfer Orbit</a> of 79,341\u00a0km (49,300\u00a0mi) in apogee with an <a href=\"/wiki/Orbital_inclination\" title=\"Orbital inclination\">inclination</a> of 20.55\u00b0 to the <a href=\"/wiki/Equator\" title=\"Equator\">equator</a>.\n</td></tr></tbody></table>\n"
                }
            ],
            "source": "# Let's print the third table and check its content\nfirst_launch_table = html_tables[2]\nprint(first_launch_table)"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "You should able to see the columns names embedded in the table header elements `<th>` as follows:\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "```\n<tr>\n<th scope=\"col\">Flight No.\n</th>\n<th scope=\"col\">Date and<br/>time (<a href=\"/wiki/Coordinated_Universal_Time\" title=\"Coordinated Universal Time\">UTC</a>)\n</th>\n<th scope=\"col\"><a href=\"/wiki/List_of_Falcon_9_first-stage_boosters\" title=\"List of Falcon 9 first-stage boosters\">Version,<br/>Booster</a> <sup class=\"reference\" id=\"cite_ref-booster_11-0\"><a href=\"#cite_note-booster-11\">[b]</a></sup>\n</th>\n<th scope=\"col\">Launch site\n</th>\n<th scope=\"col\">Payload<sup class=\"reference\" id=\"cite_ref-Dragon_12-0\"><a href=\"#cite_note-Dragon-12\">[c]</a></sup>\n</th>\n<th scope=\"col\">Payload mass\n</th>\n<th scope=\"col\">Orbit\n</th>\n<th scope=\"col\">Customer\n</th>\n<th scope=\"col\">Launch<br/>outcome\n</th>\n<th scope=\"col\"><a href=\"/wiki/Falcon_9_first-stage_landing_tests\" title=\"Falcon 9 first-stage landing tests\">Booster<br/>landing</a>\n</th></tr>\n```\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Next, we just need to iterate through the `<th>` elements and apply the provided `extract_column_from_header()` to extract column name one by one\n"
        },
        {
            "cell_type": "code",
            "execution_count": 10,
            "metadata": {},
            "outputs": [],
            "source": "column_names = []\n\n# Apply find_all() function with `th` element on first_launch_table\nfirst_launch_table=first_launch_table.find_all('th')\n# Iterate each th element and apply the provided extract_column_from_header() to get a column name\nfor row in first_launch_table:\n    name=extract_column_from_header(row)\n    # Append the Non-empty column name (`if name is not None and len(name) > 0`) into a list called column_names\n    if name!=None and len(name)>0:\n        column_names.append(name)"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Check the extracted column names\n"
        },
        {
            "cell_type": "code",
            "execution_count": 11,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": "['Flight No.', 'Date and time ( )', 'Launch site', 'Payload', 'Payload mass', 'Orbit', 'Customer', 'Launch outcome']\n"
                }
            ],
            "source": "print(column_names)"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "## TASK 3: Create a data frame by parsing the launch HTML tables\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "We will create an empty dictionary with keys from the extracted column names in the previous task. Later, this dictionary will be converted into a Pandas dataframe\n"
        },
        {
            "cell_type": "code",
            "execution_count": 22,
            "metadata": {},
            "outputs": [],
            "source": "launch_dict= dict.fromkeys(column_names)\n\n# Remove an irrelvant column\ndel launch_dict['Date and time ( )']\n\n# Let's initial the launch_dict with each value to be an empty list\nlaunch_dict['Flight No.'] = []\nlaunch_dict['Launch site'] = []\nlaunch_dict['Payload'] = []\nlaunch_dict['Payload mass'] = []\nlaunch_dict['Orbit'] = []\nlaunch_dict['Customer'] = []\nlaunch_dict['Launch outcome'] = []\n# Added some new columns\nlaunch_dict['Version Booster']=[]\nlaunch_dict['Booster landing']=[]\nlaunch_dict['Date']=[]\nlaunch_dict['Time']=[]"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Next, we just need to fill up the `launch_dict` with launch records extracted from table rows.\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Usually, HTML tables in Wiki pages are likely to contain unexpected annotations and other types of noises, such as reference links `B0004.1[8]`, missing values `N/A [e]`, inconsistent formatting, etc.\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "To simplify the parsing process, we have provided an incomplete code snippet below to help you to fill up the `launch_dict`. Please complete the following code snippet with TODOs or you can choose to write your own logic to parse all launch tables:\n"
        },
        {
            "cell_type": "code",
            "execution_count": 23,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": "1\n4 June 2010\n18:45\nF9 v1.0B0003.1\nCCAFS\nDragon Spacecraft Qualification Unit\nDragon Spacecraft Qualification Unit\nLEO\nSpaceX\nSuccess\n\nFailure\n2\n8 December 2010\n15:43\nF9 v1.0B0004.1\nCCAFS\nDragon\nDragon\nLEO\nNASA\nSuccess\nFailure\n3\n22 May 2012\n07:44\nF9 v1.0B0005.1\nCCAFS\nDragon\nDragon\nLEO\nNASA\nSuccess\nNo attempt\n\n4\n8 October 2012\n00:35\nF9 v1.0B0006.1\nCCAFS\nSpaceX CRS-1\nSpaceX CRS-1\nLEO\nNASA\nSuccess\n\nNo attempt\n5\n1 March 2013\n15:10\nF9 v1.0B0007.1\nCCAFS\nSpaceX CRS-2\nSpaceX CRS-2\nLEO\nNASA\nSuccess\n\nNo attempt\n\n6\n29 September 2013\n16:00\nF9 v1.1B1003\nVAFB\nCASSIOPE\nCASSIOPE\nPolar orbit\nMDA\nSuccess\nUncontrolled\n7\n3 December 2013\n22:41\nF9 v1.1\nCCAFS\nSES-8\nSES-8\nGTO\nSES\nSuccess\nNo attempt\n8\n6 January 2014\n22:06\nF9 v1.1\nCCAFS\nThaicom 6\nThaicom 6\nGTO\nThaicom\nSuccess\nNo attempt\n9\n18 April 2014\n19:25\nF9 v1.1\nCape Canaveral\nSpaceX CRS-3\nSpaceX CRS-3\nLEO\nNASA\nSuccess\n\nControlled\n10\n14 July 2014\n15:15\nF9 v1.1\nCape Canaveral\nOrbcomm-OG2\nOrbcomm-OG2\nLEO\nOrbcomm\nSuccess\nControlled\n11\n5 August 2014\n08:00\nF9 v1.1\nCape Canaveral\nAsiaSat 8\nAsiaSat 8\nGTO\nAsiaSat\nSuccess\nNo attempt\n12\n7 September 2014\n05:00\nF9 v1.1\nCape Canaveral\nAsiaSat 6\nAsiaSat 6\nGTO\nAsiaSat\nSuccess\nNo attempt\n\n13\n21 September 2014\n05:52\nF9 v1.1\nCape Canaveral\nSpaceX CRS-4\nSpaceX CRS-4\nLEO\nNASA\nSuccess\nUncontrolled\n14\n10 January 2015\n09:47\nF9 v1.1\nCape Canaveral\nSpaceX CRS-5\nSpaceX CRS-5\nLEO\nNASA\nSuccess\nFailure \n15\n11 February 2015\n23:03\nF9 v1.1\nCape Canaveral\nDSCOVR\nDSCOVR\nHEO\nUSAF\nSuccess\n\nControlled\n16\n2 March 2015\n03:50\nF9 v1.1\nCape Canaveral\nABS-3A\nABS-3A\nGTO\nABS\nSuccess\n\nNo attempt\n17\n14 April 2015\n20:10\nF9 v1.1\nCape Canaveral\nSpaceX CRS-6\nSpaceX CRS-6\nLEO\nNASA\nSuccess\n\nFailure\n18\n27 April 2015\n23:03\nF9 v1.1\nCape Canaveral\nT\u00fcrkmen\u00c4lem 52\u00b0E / MonacoSAT\nT\u00fcrkmen\u00c4lem 52\u00b0E / MonacoSAT\nGTO\nNone\nSuccess\n\nNo attempt\n19\n28 June 2015\n14:21\nF9 v1.1\nCape Canaveral\nSpaceX CRS-7\nSpaceX CRS-7\nLEO\nNASA\nFailure\nPrecluded\n20\n22 December 2015\n01:29\nF9 FT\nCape Canaveral\nOrbcomm-OG2\nOrbcomm-OG2\nLEO\nOrbcomm\nSuccess\n\nSuccess\n21\n17 January 2016\n18:42\nF9 v1.1\nVAFB\nJason-3\nJason-3\nLEO\nNASA\nSuccess\n\nFailure\n22\n4 March 2016\n23:35\nF9 FT\nCape Canaveral\nSES-9\nSES-9\nGTO\nSES\nSuccess\n\nFailure\n23\n8 April 2016\n20:43\nF9 FT\nCape Canaveral\nSpaceX CRS-8\nSpaceX CRS-8\nLEO\nNASA\nSuccess\nSuccess\n24\n6 May 2016\n05:21\nF9 FT\nCape Canaveral\nJCSAT-14\nJCSAT-14\nGTO\nSKY Perfect JSAT Group\nSuccess\n\nSuccess\n25\n27 May 2016\n21:39\nF9 FT\nCape Canaveral\nThaicom 8\nThaicom 8\nGTO\nThaicom\nSuccess\n\nSuccess\n26\n15 June 2016\n14:29\nF9 FT\nCape Canaveral\nABS-2A\nABS-2A\nGTO\nABS\nSuccess\n\nFailure\n27\n18 July 2016\n04:45\nF9 FT\nCape Canaveral\nSpaceX CRS-9\nSpaceX CRS-9\nLEO\nNASA\nSuccess\n\nSuccess\n28\n14 August 2016\n05:26\nF9 FT\nCape Canaveral\nJCSAT-16\nJCSAT-16\nGTO\nSKY Perfect JSAT Group\nSuccess\n\nSuccess\n29\n14 January 2017\n17:54\nF9 FT\nVAFB\nIridium NEXT\nIridium NEXT\nPolar\nIridium Communications\nSuccess\n\nSuccess\n30\n19 February 2017\n14:39\nF9 FT\nKSC\nSpaceX CRS-10\nSpaceX CRS-10\nLEO\nNASA\nSuccess\n\nSuccess\n31\n16 March 2017\n06:00\nF9 FT\nKSC\nEchoStar 23\nEchoStar 23\nGTO\nEchoStar\nSuccess\n\nNo attempt\n32\n30 March 2017\n22:27\nF9 FT\u267a\nKSC\nSES-10\nSES-10\nGTO\nSES\nSuccess\nSuccess\n33\n1 May 2017\n11:15\nF9 FT\nKSC\nNROL-76\nNROL-76\nLEO\nNRO\nSuccess\n\nSuccess\n34\n15 May 2017\n23:21\nF9 FT\nKSC\nInmarsat-5 F4\nInmarsat-5 F4\nGTO\nInmarsat\nSuccess\n\nNo attempt\n35\n3 June 2017\n21:07\nF9 FT\nKSC\nSpaceX CRS-11\nSpaceX CRS-11\nLEO\nNASA\nSuccess\n\nSuccess\n36\n23 June 2017\n19:10\nF9 FTB1029.2\nKSC\nBulgariaSat-1\nBulgariaSat-1\nGTO\nBulsatcom\nSuccess\n\nSuccess\n37\n25 June 2017\n20:25\nF9 FT\nVAFB\nIridium NEXT\nIridium NEXT\nLEO\nIridium Communications\nSuccess\n\nSuccess\n38\n5 July 2017\n23:38\nF9 FT\nKSC\nIntelsat 35e\nIntelsat 35e\nGTO\nIntelsat\nSuccess\n\nNo attempt\n39\n14 August 2017\n16:31\nF9 B4\nKSC\nSpaceX CRS-12\nSpaceX CRS-12\nLEO\nNASA\nSuccess\n\nSuccess\n40\n24 August 2017\n18:51\nF9 FT\nVAFB\nFormosat-5\nFormosat-5\nSSO\nNSPO\nSuccess\n\nSuccess\n41\n7 September 2017\n14:00\nF9 B4\nKSC\nBoeing X-37B\nBoeing X-37B\nLEO\nUSAF\nSuccess\n\nSuccess\n42\n9 October 2017\n12:37\nF9 B4\nVAFB\nIridium NEXT\nIridium NEXT\nPolar\nIridium Communications\nSuccess\n\nSuccess\n43\n11 October 2017\n22:53:00\nF9 FTB1031.2\nKSC\nSES-11\nSES-11\nGTO\nSES S.A.\nSuccess\n\nSuccess\n44\n30 October 2017\n19:34\nF9 B4\nKSC\nKoreasat 5A\nKoreasat 5A\nGTO\nKT Corporation\nSuccess\n\nSuccess\n45\n15 December 2017\n15:36\nF9 FTB1035.2\nCape Canaveral\nSpaceX CRS-13\nSpaceX CRS-13\nLEO\nNASA\nSuccess\n\nSuccess\n46\n23 December 2017\n01:27\nF9 FTB1036.2\nVAFB\nIridium NEXT\nIridium NEXT\nPolar\nIridium Communications\nSuccess\nControlled\n47\n8 January 2018\n01:00\nF9 B4\nCCAFS\nZuma\nZuma\nLEO\nNorthrop Grumman\nSuccess\nSuccess\n48\n31 January 2018\n21:25\nF9 FTB1032.2\nCCAFS\nGovSat-1\nGovSat-1\nGTO\nSES\nSuccess\nControlled\n49\n22 February 2018\n14:17\nF9 FTB1038.2\nVAFB\nPaz\nPaz\nSSO\nHisdesat\nSuccess\nNo attempt\n50\n6 March 2018\n05:33\nF9 B4\nCCAFS\nHispasat 30W-6\nHispasat 30W-6\nGTO\nHispasat\nSuccess\nNo attempt\n51\n30 March 2018\n14:14\nF9 B4B1041.2\nVAFB\nIridium NEXT\nIridium NEXT\nPolar\nIridium Communications\nSuccess\nNo attempt\n52\n2 April 2018\n20:30\nF9 B4B1039.2\nCCAFS\nSpaceX CRS-14\nSpaceX CRS-14\nLEO\nNASA\nSuccess\nNo attempt\n53\n18 April 2018\n22:51\nF9 B4\nCCAFS\nTransiting Exoplanet Survey Satellite\nTransiting Exoplanet Survey Satellite\nHEO\nNASA\nSuccess\nSuccess\n54\n11 May 2018\n20:14\nF9 B5B1046.1\nKSC\nBangabandhu-1\nBangabandhu-1\nGTO\nThales-Alenia\nSuccess\nSuccess\n55\n22 May 2018\n19:47\nF9 B4B1043.2\nVAFB\nIridium NEXT\nIridium NEXT\nPolar\nIridium Communications\nSuccess\nNo attempt\n56\n4 June 2018\n04:45\nF9 B4B1040.2\nCCAFS\nSES-12\nSES-12\nGTO\nSES\nSuccess\nNo attempt\n57\n29 June 2018\n09:42\nF9 B4B1045.2\nCCAFS\nSpaceX CRS-15\nSpaceX CRS-15\nLEO\nNASA\nSuccess\nNo attempt\n58\n22 July 2018\n05:50\nF9 B5\nCCAFS\nTelstar 19V\nTelstar 19V\nGTO\nTelesat\nSuccess\nSuccess\n59\n25 July 2018\n11:39\nF9 B5B1048\nVAFB\nIridium NEXT\nIridium NEXT\nPolar\nIridium Communications\nSuccess\nSuccess\n60\n7 August 2018\n05:18\nF9 B5B1046.2\nCCAFS\nMerah Putih\nMerah Putih\nGTO\nTelkom Indonesia\nSuccess\nSuccess\n61\n10 September 2018\n04:45\nF9 B5\nCCAFS\nTelstar 18V\nTelstar 18V\nGTO\nTelesat\nSuccess\nSuccess\n62\n8 October 2018\n02:22\nF9 B5B1048.2\nVAFB\nSAOCOM 1A\nSAOCOM 1A\nSSO\nCONAE\nSuccess\nSuccess\n63\n15 November 2018\n20:46\nF9 B5B1047.2\nKSC\nEs'hail 2\nEs'hail 2\nGTO\nEs'hailSat\nSuccess\nSuccess\n64\n3 December 2018\n18:34:05\nF9 B5B1046.3\nVAFB\nSSO-A\nSSO-A\nSSO\nSpaceflight Industries\nSuccess\nSuccess\n65\n5 December 2018\n18:16\nF9 B5\nCCAFS\nSpaceX CRS-16\nSpaceX CRS-16\nLEO\nNASA\nSuccess\n\nFailure\n66\n23 December 2018\n13:51\nF9 B5\nCCAFS\nGPS III\nGPS III\nMEO\nUSAF\nSuccess\nNo attempt\n67\n11 January 2019\n15:31\nF9 B5B1049.2\nVAFB\nIridium NEXT\nIridium NEXT\nPolar\nIridium Communications\nSuccess\n\nSuccess\n68\n22 February 2019\n01:45\nF9 B5B1048.3\nCCAFS\nNusantara Satu\nNusantara Satu\nGTO\nPSN\nSuccess\n\nSuccess\n69\n2 March 2019\n07:49\nF9 B5[268]\nKSC\nCrew Dragon Demo-1\nCrew Dragon Demo-1\nLEO\nNASA\nSuccess\n\nSuccess\n70\n4 May 2019\n06:48\nF9 B5\nCCAFS\nSpaceX CRS-17\nSpaceX CRS-17\nLEO\nNASA\nSuccess\n\nSuccess\n71\n24 May 2019\n02:30\nF9 B5B1049.3\nCCAFS\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n72\n12 June 2019\n14:17\nF9 B5B1051.2\nVAFB\nRADARSAT Constellation\nRADARSAT Constellation\nSSO\nCanadian Space Agency\nSuccess\n\nSuccess\n73\n25 July 2019\n22:01\nF9 B5B1056.2\nCCAFS\nSpaceX CRS-18\nSpaceX CRS-18\nLEO\nNASA\nSuccess\n\nSuccess\n74\n6 August 2019\n23:23\nF9 B5B1047.3\nCCAFS\nAMOS-17\nAMOS-17\nGTO\nSpacecom\nSuccess\n\nNo attempt\n75\n11 November 2019\n14:56\nF9 B5\nCCAFS\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n76\n5 December 2019\n17:29\nF9 B5\nCCAFS\nSpaceX CRS-19\nSpaceX CRS-19\nLEO\nNASA\nSuccess\n\nSuccess\n77\n17 December 2019\n00:10\nF9 B5B1056.3\nCCAFS\nJCSat-18\nJCSat-18\nGTO\nSky Perfect JSAT\nSuccess\n\nSuccess\n78\n7 January 2020\n02:19:21\nF9 B5\nCCAFS\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n79\n19 January 2020\n15:30\nF9 B5\nKSC\nCrew Dragon in-flight abort test\nCrew Dragon in-flight abort test\nSub-orbital\nNASA\nSuccess\n\nNo attempt\n\n80\n29 January 2020\n14:07\nF9 B5\nCCAFS\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n81\n17 February 2020\n15:05\nF9 B5\nCCAFS\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nFailure\n82\n7 March 2020\n04:50\nF9 B5\nCCAFS\nSpaceX CRS-20\nSpaceX CRS-20\nLEO\nNASA\nSuccess\n\nSuccess\n83\n18 March 2020\n12:16\nF9 B5\nKSC\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nFailure\n84\n22 April 2020\n19:30\nF9 B5\nKSC\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n85\n30 May 2020\n19:22\nF9 B5\nKSC\nCrew Dragon Demo-2\nCrew Dragon Demo-2\nLEO\nNASA\nSuccess\n\nSuccess\n86\n4 June 2020\n01:25\nF9 B5\nCCAFS\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n87\n13 June 2020\n09:21\nF9 B5\nCCAFS\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n88\n30 June 2020\n20:10:46\nF9 B5\nCCAFS\nGPS III\nGPS III\nMEO\nU.S. Space Force\nSuccess\n\nSuccess\n89\n20 July 2020\n21:30\nF9 B5B1058.2\nCCAFS\nANASIS-II\nANASIS-II\nGTO\nRepublic of Korea Army\nSuccess\n\nSuccess\n90\n7 August 2020\n05:12\nF9 B5\nKSC\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n91\n18 August 2020\n14:31\nF9 B5B1049.6\nCCAFS\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n92\n30 August 2020\n23:18\nF9 B5\nCCAFS\nSAOCOM 1B\nSAOCOM 1B\nSSO\nCONAE\nSuccess\n\nSuccess\n93\n3 September 2020\n12:46:14\nF9 B5B1060.2\nKSC\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n94\n6 October 2020\n11:29:34\nF9 B5B1058.3\nKSC\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n95\n18 October 2020\n12:25:57\nF9 B5B1051.6\nKSC\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n96\n24 October 2020\n15:31:34\nF9 B5\nCCAFS\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n97\n5 November 2020\n23:24:23\nF9 B5\nCCAFS\nGPS III\nGPS III\nMEO\nUSSF\nSuccess\n\nSuccess\n98\n16 November 2020\n00:27\nF9 B5\nKSC\nCrew-1\nCrew-1\nLEO\nNASA\nSuccess\n\nSuccess\n99\n21 November 2020\n17:17:08\nF9 B5\nVAFB\nSentinel-6 Michael Freilich (Jason-CS A)\nSentinel-6 Michael Freilich (Jason-CS A)\nLEO\nNASA\nSuccess\n\nSuccess\n100\n25 November 2020\n02:13\nF9 B5 \u267a\nCCAFS\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n101\n6 December 2020\n16:17:08\nF9 B5 \u267a\nKSC\nSpaceX CRS-21\nSpaceX CRS-21\nLEO\nNASA\nSuccess\n\nSuccess\n102\n13 December 2020\n17:30:00\nF9 B5 \u267a\nCCSFS\nSXM-7\nSXM-7\nGTO\nSirius XM\nSuccess\n\nSuccess\n103\n19 December 2020\n14:00:00\nF9 B5 \u267a\nKSC\nNROL-108\nNROL-108\nLEO\nNRO\nSuccess\n\nSuccess\n104\n8 January 2021\n02:15\nF9 B5\nCCSFS\nT\u00fcrksat 5A\nT\u00fcrksat 5A\nGTO\nT\u00fcrksat\nSuccess\n\nSuccess\n105\n20 January 2021\n13:02\nF9 B5B1051.8\nKSC\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n106\n24 January 2021\n15:00\nF9 B5B1058.5\nCCSFS\nTransporter-1\nTransporter-1\nSSO\nVarious\n\nSuccess\n\nSuccess\n107\n4 February 2021\n06:19\nF9 B5 \u267a\nCCSFS\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n108\n16 February 2021\n03:59:37\nF9 B5 \u267a\nCCSFS\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nFailure\n109\n4 March 2021\n08:24\nF9 B5 \u267a\nKSC\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n110\n11 March 2021\n08:13:29\nF9 B5 \u267a\nCCSFS\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n111\n14 March 2021\n10:01\nF9 B5 \u267a\nKSC\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n112\n24 March 2021\n08:28\nF9 B5B1060.6\nCCSFS\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n113\n7 April 2021\n16:34\nF9 B5 \u267a\nCCSFS\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n114\n23 April 2021\n9:49\nF9 B5B1061.2\nKSC\nCrew-2\nCrew-2\nLEO\nNASA\nSuccess\n\nSuccess\n115\n29 April 2021\n03:44\nF9 B5B1060.7\nCCSFS\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n116\n4 May 2021\n19:01\nF9 B5B1049.9\nKSC\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n117\n9 May 2021\n06:42\nF9 B5B1051.10\nCCSFS\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n118\n15 May 2021\n22:56\nF9 B5B1058.8\nKSC\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n119\n26 May 2021\n18:59\nF9 B5B1063.2\nCCSFS\nStarlink\nStarlink\nLEO\nSpaceX\nSuccess\n\nSuccess\n120\n3 June 2021\n17:29\nF9 B5B1067.1\nKSC\nSpaceX CRS-22\nSpaceX CRS-22\nLEO\nNASA\nSuccess\n\nSuccess\n121\n6 June 2021\n04:26\nF9 B5\nCCSFS\nSXM-8\nSXM-8\nGTO\nSirius XM\nSuccess\n\nSuccess\n"
                }
            ],
            "source": "extracted_row = 0\n#Extract each table \nfor table_number,table in enumerate(soup.find_all('table',\"wikitable plainrowheaders collapsible\")):\n   # get table row \n    for rows in table.find_all(\"tr\"):\n        #check to see if first table heading is as number corresponding to launch a number \n        if rows.th:\n            if rows.th.string:\n                flight_number=rows.th.string.strip()\n                flag=flight_number.isdigit()\n        else:\n            flag=False\n        #get table element \n        row=rows.find_all('td')\n        #if it is number save cells in a dictonary \n        if flag:\n            extracted_row += 1\n            # Flight Number value\n            # TODO: Append the flight_number into launch_dict with key `Flight No.`\n            launch_dict['Flight No.'].append(flight_number)\n            print(flight_number)\n            datatimelist=date_time(row[0])\n            \n            # Date value\n            # TODO: Append the date into launch_dict with key `Date`\n            date = datatimelist[0].strip(',')\n            launch_dict['Date'].append(date)\n            print(date)\n            \n            # Time value\n            # TODO: Append the time into launch_dict with key `Time`\n            time = datatimelist[1]\n            print(time)\n            launch_dict['Time'].append(time)\n              \n            # Booster version\n            # TODO: Append the bv into launch_dict with key `Version Booster`\n            bv=booster_version(row[1])\n            \n            if not(bv):\n                bv=row[1].a.string\n            print(bv)\n            launch_dict['Version Booster'].append(bv)\n            # Launch Site\n            # TODO: Append the bv into launch_dict with key `Launch Site`\n            launch_site = row[2].a.string\n            launch_dict['Launch site'].append(launch_site)\n            print(launch_site)\n            \n            # Payload\n            # TODO: Append the payload into launch_dict with key `Payload`\n            payload = row[3].a.string\n            launch_dict['Payload'].append(payload)\n            print(payload)\n            \n            # Payload Mass\n            # TODO: Append the payload_mass into launch_dict with key `Payload mass`\n            payload_mass = get_mass(row[4])\n            launch_dict['Payload mass'].append(payload_mass)\n            print(payload)\n            \n            # Orbit\n            # TODO: Append the orbit into launch_dict with key `Orbit`\n            orbit = row[5].a.string\n            launch_dict['Orbit'].append(orbit)\n            print(orbit)\n            \n            # Customer\n            # TODO: Append the customer into launch_dict with key `Customer`\n            if (row[6].a is not None):\n                customer=row[6].a.string\n            else:\n                customer=row[6].string\n            print(customer)\n            launch_dict['Customer'].append(customer)\n            \n            # Launch outcome\n            # TODO: Append the launch_outcome into launch_dict with key `Launch outcome`\n            launch_outcome = list(row[7].strings)[0]\n            launch_dict['Launch outcome'].append('launch_outcome')\n            print(launch_outcome)\n            \n            # Booster landing\n            # TODO: Append the launch_outcome into launch_dict with key `Booster landing`\n            booster_landing = landing_status(row[8])\n            launch_dict['Booster landing'].append(launch_outcome)\n            print(booster_landing)"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "After you have fill in the parsed launch record values into `launch_dict`, you can create a dataframe from it.\n"
        },
        {
            "cell_type": "code",
            "execution_count": 24,
            "metadata": {},
            "outputs": [],
            "source": "df=pd.DataFrame(launch_dict)"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "We can now export it to a <b>CSV</b> for the next section, but to make the answers consistent and in case you have difficulties finishing this lab.\n\nFollowing labs will be using a provided dataset to make each lab independent.\n"
        },
        {
            "cell_type": "code",
            "execution_count": 25,
            "metadata": {},
            "outputs": [],
            "source": "df.to_csv('spacex_web_scraped.csv', index=False)\n"
        },
        {
            "cell_type": "code",
            "execution_count": 26,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Flight No.</th>\n      <th>Launch site</th>\n      <th>Payload</th>\n      <th>Payload mass</th>\n      <th>Orbit</th>\n      <th>Customer</th>\n      <th>Launch outcome</th>\n      <th>Version Booster</th>\n      <th>Booster landing</th>\n      <th>Date</th>\n      <th>Time</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>CCAFS</td>\n      <td>Dragon Spacecraft Qualification Unit</td>\n      <td>0</td>\n      <td>LEO</td>\n      <td>SpaceX</td>\n      <td>launch_outcome</td>\n      <td>F9 v1.0B0003.1</td>\n      <td>Success\\n</td>\n      <td>4 June 2010</td>\n      <td>18:45</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>CCAFS</td>\n      <td>Dragon</td>\n      <td>0</td>\n      <td>LEO</td>\n      <td>NASA</td>\n      <td>launch_outcome</td>\n      <td>F9 v1.0B0004.1</td>\n      <td>Success</td>\n      <td>8 December 2010</td>\n      <td>15:43</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>3</td>\n      <td>CCAFS</td>\n      <td>Dragon</td>\n      <td>525 kg</td>\n      <td>LEO</td>\n      <td>NASA</td>\n      <td>launch_outcome</td>\n      <td>F9 v1.0B0005.1</td>\n      <td>Success</td>\n      <td>22 May 2012</td>\n      <td>07:44</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>4</td>\n      <td>CCAFS</td>\n      <td>SpaceX CRS-1</td>\n      <td>4,700 kg</td>\n      <td>LEO</td>\n      <td>NASA</td>\n      <td>launch_outcome</td>\n      <td>F9 v1.0B0006.1</td>\n      <td>Success\\n</td>\n      <td>8 October 2012</td>\n      <td>00:35</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>5</td>\n      <td>CCAFS</td>\n      <td>SpaceX CRS-2</td>\n      <td>4,877 kg</td>\n      <td>LEO</td>\n      <td>NASA</td>\n      <td>launch_outcome</td>\n      <td>F9 v1.0B0007.1</td>\n      <td>Success\\n</td>\n      <td>1 March 2013</td>\n      <td>15:10</td>\n    </tr>\n  </tbody>\n</table>\n</div>",
                        "text/plain": "  Flight No. Launch site                               Payload Payload mass  \\\n0          1       CCAFS  Dragon Spacecraft Qualification Unit            0   \n1          2       CCAFS                                Dragon            0   \n2          3       CCAFS                                Dragon       525 kg   \n3          4       CCAFS                          SpaceX CRS-1     4,700 kg   \n4          5       CCAFS                          SpaceX CRS-2     4,877 kg   \n\n  Orbit Customer  Launch outcome Version Booster Booster landing  \\\n0   LEO   SpaceX  launch_outcome  F9 v1.0B0003.1       Success\\n   \n1   LEO     NASA  launch_outcome  F9 v1.0B0004.1         Success   \n2   LEO     NASA  launch_outcome  F9 v1.0B0005.1         Success   \n3   LEO     NASA  launch_outcome  F9 v1.0B0006.1       Success\\n   \n4   LEO     NASA  launch_outcome  F9 v1.0B0007.1       Success\\n   \n\n              Date   Time  \n0      4 June 2010  18:45  \n1  8 December 2010  15:43  \n2      22 May 2012  07:44  \n3   8 October 2012  00:35  \n4     1 March 2013  15:10  "
                    },
                    "execution_count": 26,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": "df.head()"
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3.8",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.8.11"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 4
}