price_scraping.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyPJ7rMS/ph0vnlWvxoMoUeb",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/KelvinLam05/price_tracker/blob/main/price_scraping.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Goal of the project**"
      ],
      "metadata": {
        "id": "fmgpLCnefjfl"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In ecommerce, it’s very common for retailers to need to monitor the prices of their competitors. Prices make a big difference to sales and if they’re set too high then customers will go elsewhere, so monitoring them is crucial for ensuring the sales keeping coming in.\n",
        "In this project, we’ll create a price tracker with Webscraper.io. We’ll scrape an ecommerce store, extract the product prices, and store them in a CSV file."
      ],
      "metadata": {
        "id": "OYGezW0zfZUr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Scrape the site**"
      ],
      "metadata": {
        "id": "dCIXtKRDfoLh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Webscraper.io is a chrome browser extension built for data extraction from webpages. Using this extension we can create a plan (sitemap) how a website should be traversed and what should be extracted."
      ],
      "metadata": {
        "id": "gS1SQUoGfs7O"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "`{\"_id\":\"price_scraping\",\"startUrl\":[\"https://www.primark.com/en-us/all-products/womenswear/c/womens\",\"https://www.primark.com/en-us/all-products/menswear/c/mens\",\"https://www.primark.com/en-us/all-products/kids/c/kids\",\"https://www.primark.com/en-us/all-products/cosmetics/c/beauty\",\"https://www.primark.com/en-us/all-products/homeware/c/homeware\",\"https://www.primark.com/en-us/all-products/new-arrivals/c/newarrival\"],\"selectors\":[{\"id\":\"product_wrappers\",\"parentSelectors\":[\"_root\"],\"type\":\"SelectorElementClick\",\"clickElementSelector\":\"button.button--invisible-border\",\"clickElementUniquenessType\":\"uniqueCSSSelector\",\"clickType\":\"clickMore\",\"delay\":4000,\"discardInitialElements\":\"do-not-discard\",\"multiple\":true,\"selector\":\"li.product-listing__item\"},{\"id\":\"name\",\"parentSelectors\":[\"product_wrappers\"],\"type\":\"SelectorText\",\"selector\":\"p.product-item__name\",\"multiple\":false,\"regex\":\"\"},{\"id\":\"price\",\"parentSelectors\":[\"product_wrappers\"],\"type\":\"SelectorText\",\"selector\":\"div\",\"multiple\":false,\"regex\":\"\"}]}`"
      ],
      "metadata": {
        "id": "J3qhMwgzfPkb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Load the package**"
      ],
      "metadata": {
        "id": "UL7xyraQe5q6"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GvomePyYi9YC"
      },
      "source": [
        "# Importing library\n",
        "import pandas as pd"
      ],
      "execution_count": 70,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Load the data**"
      ],
      "metadata": {
        "id": "vMkdJn4Na0nE"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ySPd5Z2vjB3M"
      },
      "source": [
        "# Load dataset\n",
        "df = pd.read_csv('/content/price_scraping.csv')"
      ],
      "execution_count": 71,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Only include the relevant columns\n",
        "df = df[['name', 'price']]"
      ],
      "metadata": {
        "id": "OIZsoIk0Y5Nw"
      },
      "execution_count": 72,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Examine the data\n",
        "df.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "z1SiIxa-0iXy",
        "outputId": "ea9ee15c-381c-4fc3-a786-1c9115be5647"
      },
      "execution_count": 73,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                              name   price\n",
              "0  3-Pack Lace Trim Hipster Briefs   $8.00\n",
              "1       3-Pack Elegant Lace Thongs   $6.00\n",
              "2     Faux Fur Checkerboard Slides  $10.00\n",
              "3     Faux Fur Checkerboard Slides  $10.00\n",
              "4              3-Piece Shacket Set  $24.00"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-99c622d8-6ed2-4460-9fbb-9559a7de5333\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>name</th>\n",
              "      <th>price</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>3-Pack Lace Trim Hipster Briefs</td>\n",
              "      <td>$8.00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>3-Pack Elegant Lace Thongs</td>\n",
              "      <td>$6.00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Faux Fur Checkerboard Slides</td>\n",
              "      <td>$10.00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Faux Fur Checkerboard Slides</td>\n",
              "      <td>$10.00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>3-Piece Shacket Set</td>\n",
              "      <td>$24.00</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-99c622d8-6ed2-4460-9fbb-9559a7de5333')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-99c622d8-6ed2-4460-9fbb-9559a7de5333 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-99c622d8-6ed2-4460-9fbb-9559a7de5333');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 73
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Overview of all variables, their datatypes\n",
        "df.info()"
      ],
      "metadata": {
        "id": "13p_WXbKIlj5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b92bbed4-9ed5-4703-a08e-b3b3ca78ce72"
      },
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 2316 entries, 0 to 2315\n",
            "Data columns (total 2 columns):\n",
            " #   Column  Non-Null Count  Dtype \n",
            "---  ------  --------------  ----- \n",
            " 0   name    2316 non-null   object\n",
            " 1   price   2316 non-null   object\n",
            "dtypes: object(2)\n",
            "memory usage: 36.3+ KB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Data preprocessing**"
      ],
      "metadata": {
        "id": "RhIPuk5NZTQ4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Remove dollar sign\n",
        "df['price'] = df['price'].str.replace('$', '')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ivrn3X5fytda",
        "outputId": "282bb91a-4aaa-42b1-98fc-0e379f0d1d0f"
      },
      "execution_count": 75,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:2: FutureWarning: The default value of regex will change from True to False in a future version. In addition, single character regular expressions will *not* be treated as literal strings when regex=True.\n",
            "  \n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Convert price to float\n",
        "df['price'] = pd.to_numeric(df['price'])"
      ],
      "metadata": {
        "id": "21px0QFfZ7Vg"
      },
      "execution_count": 76,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df.head()"
      ],
      "metadata": {
        "id": "G4iUH_YczAGF",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "3df1b132-885f-4b4f-f2f9-33fde04ad6ad"
      },
      "execution_count": 77,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                              name  price\n",
              "0  3-Pack Lace Trim Hipster Briefs    8.0\n",
              "1       3-Pack Elegant Lace Thongs    6.0\n",
              "2     Faux Fur Checkerboard Slides   10.0\n",
              "3     Faux Fur Checkerboard Slides   10.0\n",
              "4              3-Piece Shacket Set   24.0"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-347bbb9e-21d9-4104-9f8a-afc0716e4dd9\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>name</th>\n",
              "      <th>price</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>3-Pack Lace Trim Hipster Briefs</td>\n",
              "      <td>8.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>3-Pack Elegant Lace Thongs</td>\n",
              "      <td>6.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Faux Fur Checkerboard Slides</td>\n",
              "      <td>10.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Faux Fur Checkerboard Slides</td>\n",
              "      <td>10.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>3-Piece Shacket Set</td>\n",
              "      <td>24.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-347bbb9e-21d9-4104-9f8a-afc0716e4dd9')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-347bbb9e-21d9-4104-9f8a-afc0716e4dd9 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-347bbb9e-21d9-4104-9f8a-afc0716e4dd9');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 77
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Export Pandas DataFrame to CSV\n",
        "df.to_csv('/content/price_tracker.csv', index = None, header = True)"
      ],
      "metadata": {
        "id": "HgMn6artbTVn"
      },
      "execution_count": 78,
      "outputs": []
    }
  ]
}