project_1/01_from_web_to_csv_togit.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "825547f2",
   "metadata": {},
   "source": [
    "# Hispasonic (from web to csv)\n",
    "\n",
    "\n",
    "<br>\n",
    "\n",
    "As a fan of electronic musical instruments, browsing through these types of pages has always been a enjoyable pastime. However, delving into data patterns is another aspect that strongly captivates me. This project perfectly merges both of my passions.\n",
    "\n",
    "Hispasonic holds significant importance in Spain as a hub for musical instruments, recording equipment, and everything within the realm of music. The platform includes a second-hand market where users can sell, purchase, exchange, or even give away their musical instruments.\n",
    "\n",
    "This initial phase of the project primarily involves gathering relevant advertisements, with a specific focus on the category of electronic musical instruments.\n",
    "\n",
    "<br>\n",
    "\n",
    "Before start obtaining information, the first thing we must know is to understand how the announcement page is organized.\n",
    "\n",
    "***\n",
    "\n",
    "- *Image of one of the pages of hispasonic*\n",
    "\n",
    "\n",
    "![hispa_1e.png](images/hispa_1e.png)\n",
    "\n",
    "<br>\n",
    "\n",
    "<br>\n",
    "\n",
    "We can see several important things:\n",
    "\n",
    "- Selected category is on \"teclados y sintetizadores\".\n",
    "\n",
    "- Know the number of pages that we are going to analyze to get **all the ads**.\n",
    "\n",
    "\n",
    "## 1. Function library loading. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7d10127f-ef47-4cc8-b71c-02a3739c1f51",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/ion/Documentos/albertjimrod/personal_proj_hispasonic/project_1\n"
     ]
    }
   ],
   "source": [
    "!pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9ed4b345",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests               # Is an elegant and simple HTTP library for Python\n",
    "from bs4 import BeautifulSoup # library for pulling data out of HTML and XML files\n",
    "import re                     # regular expressions operations\n",
    "import pandas as pd           # A fast, powerful, flexible and easy to use open source data analysis tool\n",
    "import os                     # A versatile way to use operating system-dependent functionality.\n",
    "import datetime as dt         # module for manipulating dates and times.\n",
    "import time                   # This module provides various time-related functions.\n",
    "import random                 # This module implements pseudo-random number generators for various distributions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa21dd00",
   "metadata": {},
   "source": [
    "### First contact\n",
    "\n",
    "First of all we must to know if we have a proper response from the server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "72632b9f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "table {float:left}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html \n",
    "<style>\n",
    "table {float:left}\n",
    "</style>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8289422a",
   "metadata": {},
   "source": [
    "These are the main possible answers we can get from the server:\n",
    "\n",
    "|||\n",
    "|:--|:--|\n",
    "|**1xx informational response –** |the request was received, continuing process|\n",
    "|**2xx successful –** |the request was successfully received, understood, and accepted|\n",
    "|**3xx redirection –** |further action needs to be taken in order to complete the request|\n",
    "|**4xx client error –** |the request contains bad syntax or cannot be fulfilled|\n",
    "|**5xx server error –** |the server failed to fulfil an apparently valid request|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "307a8a47",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [200]>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Enter the address and see the response from the server.\n",
    "\n",
    "url = \"https://www.hispasonic.com/anuncios/teclados-sintetizadores\"\n",
    "page = requests.get(url)\n",
    "page"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b636246b",
   "metadata": {},
   "source": [
    "#### *<Response [200]> means correct connection.*\n",
    "\n",
    "## 2. Main strategy, get the number of pages to analyze.\n",
    "\n",
    "Once we have communication, we have to know how to determine how to obtain the **total number of pages** to scrap.\n",
    "\n",
    "In each of the pages are the ads that we want to analyze, so it is very important to know how to obtain that value, since it can vary depending on the number of ads that are offered.\n",
    "\n",
    "![cantidad_iteraciones.png](images/cantidad_iteraciones.png)\n",
    "\n",
    "The item is identified as follows.\n",
    "\n",
    "       'ul', class_='pagination'\n",
    "       \n",
    "<br>\n",
    "\n",
    "**ul** means *'unordered list'* with a **class** name called `pagination`.\n",
    "\n",
    "<br>\n",
    "\n",
    "To determine the number of iterations, that is, the number of pages on which to extract the information, I must:\n",
    "\n",
    "- Find this element inside the html content.\n",
    "\n",
    "- Know the final value.\n",
    "\n",
    "<br>\n",
    "\n",
    "We will do this with [**Beautifulsoup**](https://beautiful-soup-4.readthedocs.io/en/latest/#) use to extract the contents of an element."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1ce0da73",
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(page.content, 'html.parser')\n",
    "# soup <- all site is stored in this variable"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db2af5f5",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "So inside `soup` variable we have all the site code, we are looking for `'ul', class_='pagination'`\n",
    "\n",
    "The following code refers to:\n",
    "\n",
    "- **first 5 links of the pages**\n",
    "\n",
    "- the **next 10 pages** and the **last one**, which is the one that interests us.\n",
    "\n",
    "Save it in a variable, called `unordered_list`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d8fa198d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<ul class=\"pagination\">\n",
       "<li>\n",
       "<span class=\"selected\">1</span>\n",
       "</li>\n",
       "<li>\n",
       "<a href=\"/anuncios/teclados-sintetizadores/pagina2\" rel=\"next\">2</a>\n",
       "</li>\n",
       "<li>\n",
       "<a href=\"/anuncios/teclados-sintetizadores/pagina3\">3</a>\n",
       "</li>\n",
       "<li>\n",
       "<a href=\"/anuncios/teclados-sintetizadores/pagina4\">4</a>\n",
       "</li>\n",
       "<li>\n",
       "<a href=\"/anuncios/teclados-sintetizadores/pagina5\">5</a>\n",
       "</li>\n",
       "<li>\n",
       "<a href=\"/anuncios/teclados-sintetizadores/pagina11\" title=\"Siguientes 10 páginas\">›</a>\n",
       "</li>\n",
       "</ul>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unordered_list = soup.find('ul', class_='pagination') # into variable unordered_list\n",
    "unordered_list"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c423556-b070-4eb0-ba19-2ada80b963c0",
   "metadata": {},
   "source": [
    "[contents and childrens (Beautiful Soup)](https://beautiful-soup-4.readthedocs.io/en/latest/index.html?highlight=contents#contents-and-children)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "af9b9665-00ac-4166-b891-5e83811aeb86",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['\\n',\n",
       " <li>\n",
       " <span class=\"selected\">1</span>\n",
       " </li>,\n",
       " '\\n',\n",
       " <li>\n",
       " <a href=\"/anuncios/teclados-sintetizadores/pagina2\" rel=\"next\">2</a>\n",
       " </li>,\n",
       " '\\n',\n",
       " <li>\n",
       " <a href=\"/anuncios/teclados-sintetizadores/pagina3\">3</a>\n",
       " </li>,\n",
       " '\\n',\n",
       " <li>\n",
       " <a href=\"/anuncios/teclados-sintetizadores/pagina4\">4</a>\n",
       " </li>,\n",
       " '\\n',\n",
       " <li>\n",
       " <a href=\"/anuncios/teclados-sintetizadores/pagina5\">5</a>\n",
       " </li>,\n",
       " '\\n',\n",
       " <li>\n",
       " <a href=\"/anuncios/teclados-sintetizadores/pagina11\" title=\"Siguientes 10 páginas\">›</a>\n",
       " </li>,\n",
       " '\\n']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unordered_list = unordered_list.contents # tag's children available in a list called .content. from variable to list\n",
    "unordered_list"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a66716e9",
   "metadata": {},
   "source": [
    "### 2.1 Exploring `unordered_list` variable.\n",
    "\n",
    "`unordered_list` is a list, therefore we know what its length is and know in what position the elements that compose it are."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ab13a82a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(unordered_list) # number of elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "1cd2fa7b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unordered_list[0] # first element"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e9e2b616",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unordered_list[-1] # last element"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "dcbd42c3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<li>\n",
       "<a href=\"/anuncios/teclados-sintetizadores/pagina11\" title=\"Siguientes 10 páginas\">›</a>\n",
       "</li>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unordered_list[-2] # this is the one I'm interested in"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "682ce4cc",
   "metadata": {},
   "source": [
    "### 2.2 How to get the value number from `unordered_list`?\n",
    "\n",
    "<br>\n",
    "\n",
    "I need to access the value within the list, so the strategy  will be the following:\n",
    "\n",
    "- 1. Convert the list to a text string.\n",
    "\n",
    "- 2. Filter the characters that correspond to numeric values, just the max ones.\n",
    "\n",
    "- 3. Convert those numeric characters to numbers (int).\n",
    "\n",
    "<br>\n",
    "\n",
    "I will convert the contents of the list into a text string and have the numeric characters extracted together with highest values by using regular expressions.\n",
    "\n",
    "**1. Converting the content of `paginas` into a text string**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "57f5ef56",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<li>\\n<a href=\"/anuncios/teclados-sintetizadores/pagina11\" title=\"Siguientes 10 páginas\">›</a>\\n</li>'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test = str(unordered_list[-2])\n",
    "test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1da0c8e8",
   "metadata": {},
   "source": [
    "**2/3. Filter the characters that correspond to numeric values, just the max ones** and **Convert those numeric characters to numbers (int)**.\n",
    "\n",
    "`extractMax` A function that gets the numbers contained in the lowercase text and converts them to integer numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "56138a8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extractMax(input):\n",
    "     # get a list of all numbers separated by \n",
    "     # lower case characters \n",
    "     # \\d+ is a regular expression which means\n",
    "     # one or more digit\n",
    "     # output will be like ['100','564','365']\n",
    "    numbers = re.findall(r'\\d+',input)\n",
    "     # now we need to convert each number into integer\n",
    "     # int(string) converts string into integer\n",
    "     # we will map int() function onto all elements \n",
    "     # of numbers list\n",
    "    numbers = map(int,numbers)\n",
    "    return max(numbers) # returns a int number"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b4bec164",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "page_numbers = extractMax(test)\n",
    "page_numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a93e1f3e",
   "metadata": {},
   "source": [
    "We already have the number of pages that we will have to analyze. \n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b8367e4",
   "metadata": {},
   "source": [
    "## 3. Getting and save all links (ads and not ads)\n",
    "\n",
    "Once we have the number of pages in which we must extract the ads, the next step is to extract those ads from each of the pages looking inside the code of each of them.\n",
    "\n",
    "So what we have to do is:\n",
    "\n",
    "- Extracting **everything that is a link**.\n",
    "\n",
    "\n",
    "- From the links extracted, the most important thing is get the final number which is the way to **identify those who are ads and what are not**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "d3ee1408",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://www.hispasonic.com/anuncios/teclados-sintetizadores/pagina11\n",
      "https://www.hispasonic.com/anuncios/teclados-sintetizadores/pagina10\n",
      "https://www.hispasonic.com/anuncios/teclados-sintetizadores/pagina9\n",
      "https://www.hispasonic.com/anuncios/teclados-sintetizadores/pagina8\n",
      "https://www.hispasonic.com/anuncios/teclados-sintetizadores/pagina7\n",
      "https://www.hispasonic.com/anuncios/teclados-sintetizadores/pagina6\n",
      "https://www.hispasonic.com/anuncios/teclados-sintetizadores/pagina5\n",
      "https://www.hispasonic.com/anuncios/teclados-sintetizadores/pagina4\n",
      "https://www.hispasonic.com/anuncios/teclados-sintetizadores/pagina3\n",
      "https://www.hispasonic.com/anuncios/teclados-sintetizadores/pagina2\n",
      "https://www.hispasonic.com/anuncios/teclados-sintetizadores/pagina1\n"
     ]
    }
   ],
   "source": [
    "links_ads = []        # all the ads on the page\n",
    "listado_enlaces = []  # all the links on the page\n",
    "\n",
    "pattern=\"([0-9]{4,9})\" # filtering all links with number, that mean choosing the page number related to and ad.\n",
    "\n",
    "for pagina in range(page_numbers, 0, -1): \n",
    "    url = \"https://www.hispasonic.com/anuncios/teclados-sintetizadores/pagina{pagina}\".format(pagina=pagina)\n",
    "    print(url)\n",
    "    page = requests.get(url)\n",
    "    soup = BeautifulSoup(page.content, 'html.parser')\n",
    "\n",
    "    \n",
    "    for link in soup.find_all('a'):       # filter everything that is a link on soup variable\n",
    "        links_ads.append(link.get('href'))\n",
    "        fecha = soup.find_all('span', class_='miniicon miniicon-date')\n",
    "        \n",
    "    \n",
    "    for link_ad in links_ads:                   # of those links what I do is stay with what ends in number\n",
    "        if re.search(pattern, link_ad):\n",
    "            listado_enlaces.append(link_ad)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e8ad853",
   "metadata": {},
   "source": [
    "#### This is a small sample of the contents of the lists\n",
    "\n",
    "It can be seen as being links in both cases, in the first we only have **links that do not interest us**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "7e69fd15",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['/musica',\n",
       " '/productos',\n",
       " '/anuncios',\n",
       " '/anuncios/todo',\n",
       " '/anuncios/todo/f/compra-protegida',\n",
       " '/anuncios',\n",
       " '/anuncios/compraventa',\n",
       " '/anuncios/teclados-sintetizadores',\n",
       " '/anuncios/todo/f/compra-protegida',\n",
       " '/compra-protegida',\n",
       " '/anuncios/todo/f/compra-protegida',\n",
       " '/index.php?controller=ad&action=new_ad_form',\n",
       " '/anuncios/teclados-sintetizadores',\n",
       " '/anuncios/teclados-sintetizadores',\n",
       " '/anuncios/teclados-sintetizadores/pagina9']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "links_ads[5:20] # example: row 5 to 20 of everything is a link on soup variable"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c7d2c1e",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "However in the second list `listado_enlaces` what we have are the **links we want to get in each of the pages**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "abe14e16",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['/anuncios/slider-cap-arp-odyssey-mkiii/1124687',\n",
       " '/anuncios/compro-linndrum/1131733',\n",
       " '/anuncios/compro-linndrum/1131733',\n",
       " '/anuncios/roland-jd-xa/1123891',\n",
       " '/anuncios/roland-jd-xa/1123891',\n",
       " '/anuncios/roland-xv-3080-128-voice-rackmount-synthesizer-module/1132282',\n",
       " '/anuncios/roland-xv-3080-128-voice-rackmount-synthesizer-module/1132282',\n",
       " '/anuncios/roland-fantom-s/1118566',\n",
       " '/anuncios/roland-fantom-s/1118566',\n",
       " '/anuncios/korg-x3-modulo/1118568',\n",
       " '/anuncios/korg-x3-modulo/1118568',\n",
       " '/anuncios/taclado-casio-celviano-ap80r/1122261',\n",
       " '/anuncios/taclado-casio-celviano-ap80r/1122261',\n",
       " '/anuncios/behringer-cat/1118903',\n",
       " '/anuncios/behringer-cat/1118903']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "listado_enlaces[5:20] # example: of those links what I do is stay with what ends in number"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bebc69f5",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 3.1 Cleaning links.\n",
    "\n",
    "Taking a look into `listado_enlaces` it is striking that there are links that are repeated.\n",
    "\n",
    "            '...\n",
    "            '/anuncios/korg-vocoder-vc10/866556',\n",
    "             '/anuncios/korg-vocoder-vc10/866556',\n",
    "             '/anuncios/polyend-tracker/1057403',\n",
    "             '/anuncios/polyend-tracker/1057403',\n",
    "             '/anuncios/trajetas-teclados/949462',\n",
    "             '/anuncios/trajetas-teclados/949462',\n",
    "                                             ...',\n",
    "\n",
    "<br>\n",
    "\n",
    "### We need to do a couple of things.\n",
    "\n",
    "![regex_expression.png](images/regex_expression.png)\n",
    "\n",
    "- 1. **Extract** the brand name from the url using regular expressions.\n",
    "\n",
    "- 2. **Filter** the amount of url repeated.\n",
    "\n",
    "\n",
    "\n",
    "<br>\n",
    "\n",
    "<br>\n",
    "\n",
    "To get **not repeated url**, we will make a **filter with a dictionary**.\n",
    "\n",
    "The main idea is filter the url repeated as `key` and asign it a synth brand for this unique url as `value`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "534a5ee0-d385-4dd6-89ef-34bccea96c0b",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.chdir('/home/ion/Documentos/albertjimrod/hispaok/htmls') # folder where htmls folder is ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "f0ae73d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "diccionario_enlaces = {} # dict\n",
    "listado_marcas = []      # synth_brand\n",
    "\n",
    "brand_pattern = r\"((?<=anuncios\\/)[1-9][a-z]{1,})|((?<=anuncios\\/)[a-z]{1,})\" # filter brand regex\n",
    "\n",
    "for enlace in listado_enlaces:\n",
    "    if enlace not in diccionario_enlaces:  \n",
    "        try:\n",
    "            marca = re.search(brand_pattern, enlace).group()\n",
    "            diccionario_enlaces[enlace] = marca\n",
    "        except AttributeError:\n",
    "            #marca = re.search(brand_pattern, enlace)\n",
    "            pass # voy a ver si funciona, lo que aprendi del try except"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8024069",
   "metadata": {},
   "source": [
    "With the dictionary that we have just created we are going to download all the ads locally.\n",
    "\n",
    "The reason is not to overload the server and run the risk of being banned."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e95e9f22",
   "metadata": {},
   "source": [
    "## 3.2 Download all the ads.\n",
    "\n",
    "\n",
    "To avoid the inconvenience that would suppose the overload of the server, we will download all the ads in local mode adding a delay in the download time. In this way we will work with more comfort."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "53a85560",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.64 s, sys: 212 ms, total: 3.85 s\n",
      "Wall time: 17min 28s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "main_path='https://www.hispasonic.com'\n",
    "local_path = '/home/ion/Documentos/albertjimrod/hispaok/htmls/'\n",
    "\n",
    "for enlace in diccionario_enlaces:\n",
    "    # sampling interval\n",
    "    time.sleep(random.uniform(1, 4))       \n",
    "    page = requests.get(main_path + enlace) # https://www.hispasonic.com/anuncios/polyend-tracker/1057403.html\n",
    "    # filter for extracting\n",
    "    enlace = enlace.split(\"/\")  \n",
    "    # name ad\n",
    "    enlace= enlace[2]         \n",
    "    # Open a file in write mode (\"w+\"). If the file doesn't exist, it's created. If it already exists, it is overwritten\n",
    "    with open(local_path + enlace + '.html',\"w+\") as f: \n",
    "        #Write to the file that was opened earlier (f). page.text\n",
    "        f.write(page.text)\n",
    "    \n",
    "    print(local_path + enlace)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16bb6c3f",
   "metadata": {},
   "source": [
    "## 3.3 It's not all about sales.\n",
    "\n",
    "<br>\n",
    "\n",
    "When ads have been downloaded, the next step is doing a quick scan inside the downloaded ads, so there's no only sales.\n",
    "\n",
    "\n",
    "A starting point is to look in the description of the titles and see if some of these words exist.\n",
    "\n",
    "\n",
    "By using `find` and `grep` together we can see if these words we are looking for are inside the files.\n",
    "\n",
    "<br>\n",
    "\n",
    "\n",
    "\n",
    "![vendo.png](images/vendo.png)\n",
    "\n",
    "- **vendo : *sell***\n",
    "\n",
    "<br>\n",
    "\n",
    "\n",
    "![busco_piezas.png](images/busco_piezas.png)\n",
    "\n",
    "- **busco, se busca: *looking for*** and - **piezas: *parts***\n",
    "\n",
    "<br>\n",
    "\n",
    "\n",
    "![cambio.png](images/cambio.png)\n",
    "\n",
    "- **cambio: *change***\n",
    "\n",
    "<br>\n",
    "    \n",
    "![compro.png](images/compro.png)\n",
    "\n",
    "- **compro: *buy***\n",
    "\n",
    "<br>\n",
    "\n",
    "![regalo.png](images/regalo.png)\n",
    "\n",
    "- **regalo: *for free***\n",
    "\n",
    "<br>\n",
    "\n",
    "This information will be very useful because these are **the actions**, and **it will allow us to classify** if the ad is for sale, purchase or any other concept that we have discovered.\n",
    "\n",
    "## 3.4 Elements of the ad that we are going to extract.\n",
    "\n",
    "<br>\n",
    "\n",
    "Another step to take into account is to obtain:\n",
    "\n",
    "- **description**\n",
    "\n",
    "- **user**\n",
    "\n",
    "- **price**\n",
    "\n",
    "- **brand**\n",
    "\n",
    "- **city** \n",
    "\n",
    "- **date published** \n",
    "\n",
    "- **date expire** \n",
    "\n",
    "- **times seen**\n",
    "\n",
    "<br>\n",
    "\n",
    "\n",
    "![hispa_4.png](images/hispa_4.png)\n",
    "\n",
    "\n",
    "\n",
    "<br>\n",
    "\n",
    "This is an ad as example and the fields we want to get:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0123e433",
   "metadata": {},
   "source": [
    "## 3.5 Extraction of the `action` and the `brand` name from the description.\n",
    "\n",
    "\n",
    "<br>\n",
    "\n",
    "### 3.5-1 Extraction of `action`\n",
    "\n",
    "<br>\n",
    "\n",
    "\n",
    "The extraction contained in the fields is not very complicated, however in the main description we find a problem to solve. It is about **how to differentiate a `sale`, a `purchase` or a `change`.\n",
    "\n",
    "\n",
    "To do this the strategy carried out has been to use a series of keywords in the meaning of the ad as triggers of **accion: *action*** in the event that those words exist in the description of the advertisement. \n",
    "\n",
    "\n",
    "In the same way as we (humans) would do to see if the ad is a sale or on the contrary a gift.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "98e52429",
   "metadata": {},
   "outputs": [],
   "source": [
    "accion = [\"compro\",\"cambio\",\"vendo\",\"regalo\",\"busco\",\"busca\",'reparar','piezas']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de5bb2d3",
   "metadata": {},
   "source": [
    "Once we have the `accion` keywords list, the next step is to make them as a trigger, that is, manage to make a certain action.\n",
    "\n",
    "<br>\n",
    "\n",
    "Using the words contained in `accion` list as the **key**, and the **value of the dictionary** a **call to a function depending the action on acción**.\n",
    "\n",
    "<br>\n",
    "\n",
    "    func_dict = {                      # the key give us the action (function)\n",
    "        \"compro\":func_compro,\n",
    "        \"cambio\":func_cambio,\n",
    "        ...\n",
    "\n",
    "\n",
    "    def func_compro(clave_func_dict):  # if `compro` means I am not selling, and so on...\n",
    "\n",
    "    if list_compro[-1] == \"0\":  \n",
    "        list_vendo.pop(-1)\n",
    "        list_vendo.append(\"0\")\n",
    "        list_compro.pop(-1)\n",
    "        list_compro.append(\"1\")\n",
    "    else:\n",
    "        pass\n",
    "        \n",
    "\n",
    "<br>\n",
    "\n",
    "### 3.5-1 Extraction of synthesizer name.\n",
    "\n",
    "<br>\n",
    "\n",
    "\n",
    "The next step to implement is to get all the possible brands of synthesizer manufacturers that we can find in the ads. \n",
    "\n",
    "\n",
    "To do this by doing an internet search I could find a list of a large number of them, at least to the date.\n",
    "\n",
    "\n",
    "However due to the time I have been working with the project I already have a **list** of names `sintes` with which I have been working but that when I reached this point I realized that I had to modify and merge with the new list.\n",
    "\n",
    "link where I obtain the brand synth: https://www.perfectcircuit.com/modular-synths\n",
    "\n",
    "#### Synthesizer manufacturers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "03145eb8",
   "metadata": {},
   "outputs": [],
   "source": [
    "sintes = ['coast 0_coast', '000', '4ms', 'avp a-v-p_synth', 'acces', 'access', 'digitakt elektron','voix la-voix-du-luthier', 'luthier la-voix-du-luthier', \n",
    "'oktatrak elektron','analog elektron','heat elektron','rythm elektron','digitone elektron','keys elektron','cycles elektron','samples elektron','acidlab', 'akai',\n",
    "'mpc akai','alembic', 'alesis', 'allen allen_&_heath', 'analogaudio1', 'analogue solutions', 'analogue systems', 'arp', 'arturia', 'asm ashun_soundmachines', \n",
    "'atomo atomo_synth', 'damage audio_damage', 'audiophile audiophile_circuits', 'axoloty', 'balaguer', 'baloran', 'bastl bastl_instruments', 'befaco', 'behringer', \n",
    "'beringer', 'bheringer', 'bitbox', 'black corporation', 'boss', 'bubblesound instruments', 'buchla', 'casio', 'charlie lab', 'charvel', 'chronograf', 'circuit abbey', 'clavia', \n",
    "'club knobs', 'corsynth', 'cre8audio', 'crumar', 'custom made', 'cyclone', 'cyclone', 'dave jones', 'dave smith', \n",
    "'dave smith instruments', 'deepmind', 'delptronics', 'delta music', 'denon dj', 'dexibell', 'dexibell', 'digitack', 'doepfer', \n",
    "'dreadbox', 'dubreq', 'dynacord', 'e mu', 'e-mu', 'e-mu', 'e:m:c', 'elby designs', 'electribe', 'electronic music laboratories (eml)', \n",
    "'electrovoice', 'elektron', 'elka', 'emc', 'emu', 'endorphin.es', 'endorphines', 'ensoniq', 'eowave', 'epiphone', 'erica synth', 'erica synths', \n",
    "'ernie ball music man', 'esp ltd', 'eurorack', 'eventide', 'evh', 'evolver', 'exodus digital', 'farfisa', 'fender', 'fishman', 'fodera', 'formanta', \n",
    "'frap tool', 'frequency central', 'fretlight', 'friedman', 'future retro', 'futuresonus', 'gator', 'gemini', 'generalmusic', 'gibson', 'godin', 'gotharman', \n",
    "'graph tech', 'gretsch', 'guild', 'hammond', 'hartmann', 'hexinverter', 'hinton instruments', 'hofner', 'hypersynth', 'ibanez', 'ik', 'instruo', 'iomega', \n",
    "'isla', 'jackson', 'jaspers', 'john bowen synth design', 'jomox', 'kawai', 'kenton', 'ketron', 'kilpatrick audio', 'knobula', \n",
    "'koma elektronik', 'komplete', 'korg', 'kramer', 'kurzweil', 'kurzweil', 'lakland', 'line 6', 'linn electronics', 'livid', 'logan electronics', 'm-audio', \n",
    "'macbeth studio systems', 'make', 'malekko', 'manikin electronic', 'maschine', 'mellotron', 'mfb', 'micro modular', 'miditech', \n",
    "'models', 'modor', 'modular', 'modulus', 'monome', 'moog', 'mpc', 'mpc', 'mutable instruments', 'mutant', 'native instruments', 'neutron', 'noise engineering', \n",
    "'nord', 'nord electro', 'nord lead 2 rack', 'nord lead 3', 'nord lead 3', 'nord lead 4', 'nord micro modular', 'nord modular', 'nord rack', 'nord stage', \n",
    "'nord wave', 'novation', 'numark', 'oberheim', 'octatrack', 'orthogonal devices', 'paratek', 'pearl', 'peavey', 'pioneer dj', 'pittsburgh', 'pittsburgh modular', \n",
    "'polyend', 'polygraf', 'ppg (palm products gmbh)', 'prs', 'qu bit', 'qu-bit', 'qu-bit electronix', 'quasimidi', 'qubit', 'quiklok', 'radikal technologies', \n",
    "'rhodes', 'rickenbacker', 'roland', 'roli', 'sanson', 'schecter', 'sensel', 'sequencial', 'sequential circuits', 'sequentix', 'shakmat', 'simmons', 'soma', 'sonicware', 'special waves', 'spector', 'spectral audio', 'sputnik', 'squarp instruments', \n",
    "'squier', 'ssff', 'stanton', 'steinberger', 'sterling', 'strymon', 'studio electronics', 'synamodec', 'synthesis technology', \n",
    "'synthrotek', 'synthstrom', 'synthstrom', 'synthtech','swissonic', 'tascam', 'taylor', 'technos', 'teenage', 'teenage engineering', 'tiptop', 'tiptop audio', \n",
    "'traveler guitar', 'udo audio', 'uno synth ', 'vermona', 'vermona', 'virus', 'viscount', 'volca', 'vox', 'waldorf', 'warwick', 'washburn', 'waves grendel', \n",
    "'wersi', 'wersi music', 'winter modular', 'wmd', 'wmd / ssf', 'wurlitzer', 'yamaha', 'yocto', 'zeppelin design labs', 'zoom','1010 music', '2hp', '4ms', 'acid rain technology', \n",
    "'acl', 'addac system', 'after later audio', 'aion modular', 'ajh synth', 'cosmos soma', 'divina soma', 'enner soma', 'ether soma', 'flux soma', 'illuminator soma', \n",
    "'lyra-8 soma', 'lyra8-fx soma', 'metaconformer soma', 'ornament-8 soma', 'pulsar-23 soma', 'qo soma', 'reflex soma', 'roat soma', 'terra soma', 'the pipe soma',\n",
    "'alm busy circuits', 'alright devices', 'analogue solutions', 'bastl instruments', 'befaco', 'blackhole cases', 'blue lantern', 'boredbrain music', \n",
    "'bubblesound', 'buchla', 'cosmotronic', 'cre8audio', 'divkid', 'dnipro modular', 'doepfer', 'dreadbox', 'e-rm','LinnDrum','Linn Electronics', 'electrosmith', 'emblematic systems', \n",
    "'empress effects', 'endorphin.es', 'eowave', 'erica synths', 'erogenous tones', 'eskatonic modular', 'eventide', 'five12', 'frap tools', 'future sound systems', \n",
    "'gieskes', 'grayscale', 'hexinverter', 'industrial music electronics','voix du luthier', 'instruo', 'io instruments', 'jomox', 'joranalogue', 'klavis', \n",
    "'koma elektronik', 'l-1', 'lmntl', 'low-gain electronics', 'lzx industries', 'make noise','eloquencer', 'malekko heavy industry', 'manhattan analog', 'meng qi', \n",
    "'michigan synth works', 'modbap modular', 'moog', 'mordax', 'mosaic', 'mrseri', 'mutable instruments', 'nano modules', 'noise engineering', \n",
    "'patching panda', 'percussa', 'pittsburgh modular', 'plankton electronics', 'poly effects', 'qu-bit electronix', 'random source', 'ritual electronics', \n",
    "'rossum', 'schlappi engineering', 'shakmat modular', 'soundforce', 'soundmachines', 'squarp', 'steady state fate', 'strymon', 'studio electronics', \n",
    "'supercritical', 'synthesis technology', 'system 80', 'tall dog electronics', 'tasty chips', 'tenderfoot electronics', 'tesseract modular', 'tiptop audio', \n",
    "'trogotronic', 'tubbutec', 'u-he', 'verbos electronics', 'vermona', 'voicas', 'vpme.de', 'winter modular', 'wmd', 'worng electronics', 'xaoc devices', \n",
    "'xor electronics', 'zlob modular',\"ASM\",\"Elektron\",\"Moog\",\"Teenage Engineering\",\"Korg\",\"Novation\",\"Modal Electronics\",\n",
    "\"Black Corporation\",\"Roland\",\"Arturia\",\"Critter & Guitari\",\"Polyend\",\"UDO\",\"Waldorf\",\"Nord\",\"Yamaha\",\"Vermona\",\"Crumar\",\"JMT Synth\",\"Modor\",\n",
    "\"Studio Electronics\",\"Trogotronic\",\"Gieskes\",\"Akai\",\"Dreadbox\",\"Herbs and Stones\",\"IK Multimedia\",\"Tasty Chips\",\"Buchla\",\"Soundmachines\",\n",
    "\"Access\",\"Grp\",\"Analogue Solutions\",\"The Division Department\",\"Norand\",\"Jomox\",\"Sonicware\",\"Radikal Technologies\",\"Playtime Engineering\",\n",
    "\"1010 Music\",\"Fred's Lab\",\"Kilpatrick Audio\",\"Eowave\",\"Electrosmith\",\"Meng Qi\",\"Studiologic\",\"Suzuki\",\"Nonlinear Labs\",\"Dato\",\"Artiphon\",\n",
    "\"Malekko Heavy Industry\",\"Kodamo\",\"Hikari Instruments\",\"Manikin Electronic\",\"Second Sound\",\"Arturia\",\"Squarp\",\"Polyend\",\"Novation\",\"Akai\",\n",
    "\"Roger Linn Design\",\"Conductive Labs\",\"Native Instruments\",\"Faderfox\",\"Sensel\",\"Roland\",\"Keith McMillen\",\"Pioneer\",\"E-RM\",\"Expressive E\",\"Korg\",\n",
    "\"M-Audio\",\"Alesis\",\"JouÃ©\",\"Soundforce\",\"Yamaha\",\"Genki\",\"Erica Synths\",\"Make Noise\",\"Doepfer\",\"Elektron\",\"Moog\",\n",
    "\"Teenage Engineering\",\"1010 Music\",\"Expert Sleepers\",\"BASTL Instruments\",\"Kenton\",\"Circuit Happy\",\"MOTU\",\"MIDI Solutions\",\"Solid State Logic\",\n",
    "\"Nord\",\"Malekko Heavy Industry\",\"Koma Elektronik\",\"Random Source\",\"Eowave\",\"Zoom\",\"Crumar\",\"Electro-Harmonix\",\"Grp\",\"Michigan Synth Works\",\n",
    "\"Analogue Solutions\",\"Knas\",\"iConnectivity\",\"Soundmachines\",\"Eurodesk-Z\",\"Presonus\",\"Torso Electronics\",\"IK Multimedia\",\"ESI Audiotechnik\",\n",
    "\"Low-Gain Electronics\",\"Artiphon\",\"Instruments of Things\",\"Apogee\",\"SND\",\"Moffenzeef\",\"CME\",\"Embodme\",\n",
    "\"Tech 21\",\"Snyderphonics\",\"Tricks Magic Shop\",\"Strymon\",\"Vermona\",\"OTO Machines\",\"Dreadbox\",\"Chase Bliss Audio\",\"Boss\",\"GFI\",\"Meris\",\n",
    "\"Eventide\",\"SOMA Laboratory\",\"Echo Fix\",\"Fairfield Circuitry\",\"Universal Audio\",\"Gamechanger Audio\",\"EarthQuaker Devices\",\"Death By Audio\",\n",
    "\"Sherman\",\"Electro-Harmonix\",\"Old Blood Noise Endeavors\",\"Knas\",\"Red Panda\",\"Malekko Heavy Industry\",\"Kemper\",\"DigiTech\",\"JAM Pedals\",\n",
    "\"Erica Synths\",\"Elektron\",\"WMD\",\"1010 Music\",\"Roland\",\"Korg\",\"Poly Effects\",\"Jomox\",\"Thermionic Culture\",\"Warm Audio\",\n",
    "\"Zoom\",\"Boredbrain Music\",\"Meng Qi\",\"Electrosmith\",\"Benidub\",\"BAE\",\"Trogotronic\",\"MIDI Solutions\",\"Plankton Electronics\",\"Vongon\",\n",
    "\"ART\",\"Hungry Robot\",\"Walrus Audio\",\"Enjoy Electronics\",\"CIOKS\",\"TK Audio\",\"Source Audio\",\"API\",\"Voodoo Lab\",\n",
    "\"FMR Audio\",\"JHS Pedals\",\"MOD Devices\",\"Cooper FX\",\"Finegear\",\"Ezhi & Aka\",\"Truetone\",\"LastGasp Art Laboratories\",\"Origin Effects\",\n",
    "\"Rainger FX\",\"Line 6\",\"PedalTrain\",\"Dr. Scientist\",\"Elta Music\",\"Keeley\",\"Recovery\",\"Glou-Glou\",\"Retro Mechanical Labs\",\"Electro-Faustus\",\n",
    "\"Animal Factory\",\"Hologram\",\"Caroline Guitar Company\",\"MXR\",\"Second Sound\",\"Xotic\",\"Dunlop\",\"Adventure Audio\",\"ISP Technologies\",\n",
    "\"Industrialectric\",\"Tech 21\",\"Collision Devices\",\"Orgeldream\",\"Universal Audio\",\"API\",\"Solid State Logic\",\"Rupert Neve Designs\",\"Shure\",\"MOTU\",\n",
    "\"Warm Audio\",\"Focusrite\",\"Vermona\",\"Focal\",\"Neumann\",\"Roland\",\"Thermionic Culture\",\"Arturia\",\"Zoom\",\"Presonus\",\"Adam\",\"ART\",\"Yamaha\",\n",
    "\"TASCAM\",\"Furman\",\"Antelope Audio\",\"Dangerous Music\",\"Pioneer\",\"Echo Fix\",\"Native Instruments\",\"Eventide\",\"Allen & Heath\",\n",
    "\"Meris\",\"Sherman\",\"dbx\",\"BAE\",\"Maag Audio\",\"Empirical Labs\",\"Avantone Pro\",\"iConnectivity\",\"Mackie\",\"Audient\",\"beyerdynamic\",\"TK Audio\",\n",
    "\"IK Multimedia\",\"Black Lion Audio\",\"RME\",\"Keith McMillen\",\"Golden Age Project\",\"Audio-Technica\",\"Fredenstein\",\"A-Designs\",\"Rosson Audio\",\n",
    "\"Daking\",\"Looptrotter\",\"Rode\",\"Prism Sound\",\"Samson\",\"Cranborne Audio\",\"ESI Audiotechnik\",\"Elysia\",\"HEDD\",\"FMR Audio\",\"Heritage Audio\",\n",
    "\"Avedis Audio\",\"Sennheiser\",\"Lindell Audio\",\"Blue Microphones\",\"Apogee\",\"Recovery\",\"M-Audio\",\"Zeppelin Design Labs\",\"KRK\",\"AKG\",\n",
    "\"Cloud Microphones\",\"Steinberg\",\"Alesis\",\"Dynaudio\",\"Austrian Audio\",\"Auralex\",\"IsoAcoustics\",\"Aston Microphones\",\"Auratone\",\"sE Electronics\",\"SE Electronics\",\n",
    "\"Tech 21\",\"Lauten Audio\",\"Cascade Microphones\",\"Soundrise\",\"Pioneer\",\"Allen & Heath\",\"Pro-Ject\",\"PLAYdifferently\",\"U-Turn Audio\",\"Audio-Technica\",\n",
    "\"Thorens\",\"Audioengine\",\"Technics\",\"Rane\",\"AKG\",\"Music Hall\",\"Native Instruments\",\"Numark\",\"Sennheiser\",\"Jesse Dean Designs\",\"Ortofon\",\n",
    "\"Rosson Audio\",\"MWM\",\"Gator\",\"IK Multimedia\",\"ART\",\"Yamaha\",\"Ultimate Support\",\"RME\",\"Roland\",\"KRK\",\"Austrian Audio\",\"Shure\",\"Odyssey\",\n",
    "\"Teenage Engineering\",\"Denon\",\"Record Props\",\"Presonus\",\"Hosa\",\"Hosa\",\"Mogami \",\"Roland \",\"Voodoo Lab \",\"CIOKS \",\"LMNTL \",\"Warm Audio \"\n",
    "\"Teenage Engineering \",\"myVolts \",\"Gator \",\"Truetone \",\"Strymon \",\"Eurodesk-Z\",\"Furman\",\"Elektron\",\"Tiptop Audio\",\"Retrokits\",\"4MS\",\"EBS\",\n",
    "\"Pomona Electronics\",\"Modbang\",\"Intellijel Designs\",\"Plankton Electronics\",\"Radial Engineering\",\"1010 Music\",\"Native Instruments\",\n",
    "\"Expert Sleepers\",\"Buchla\",\"iConnectivity\",\"Modbap Modular\",\"Boredbrain Music\",\"Make Noise\",\"Korg\",\"Moog \",\"Rode \",\"Shure \",\n",
    "\"LabLab Audio \",\"Zoom \",\"Doepfer \",\"Koma Elektronik \",\"ADDAC System \",\"Frap Tools \",\"Endorphin.es \",\"ART s\",\"Yamaha \",\"Walrus Audio\",\n",
    "\"ALM Busy Circuits \",\"Analogue Solutions \",\"Trogotronic \",\"Befaco \",\"Boss \",\"Soundmachines \",\"LZX Industries \",\"Cyclone Analogic \",\n",
    "\"M-Audio \",\"E-RM \",\"Pulp Logic \",\"Electro-Harmonix \",\"ESI Audiotechnik \",\"Eskatonic Modular \",\"Eventide \",\"Instruo \",\"Keith McMillen\",\n",
    "\"Malekko Heavy Industry \",\"Dunlop\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "203f4150",
   "metadata": {},
   "outputs": [],
   "source": [
    "# cleaning names in sintes list.\n",
    "\n",
    "lista_criba = []\n",
    "\n",
    "for marca in sintes:\n",
    "    # Switch to lowercase\n",
    "    marca = marca.lower()\n",
    "    if marca not in lista_criba:\n",
    "        # Filter out the repeated names and put them in another list\n",
    "        lista_criba.append(marca)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "99ae7396",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['4ms',\n",
       " 'avp a-v-p_synth',\n",
       " 'acces',\n",
       " 'access',\n",
       " 'digitakt elektron',\n",
       " 'voix la-voix-du-luthier',\n",
       " 'luthier la-voix-du-luthier',\n",
       " 'oktatrak elektron']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lista_criba[2:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1b4422a",
   "metadata": {},
   "source": [
    "- Once I clean the list of possible repeated names, what I do next is **the names composed of two terms are a list of two elements**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "cddbef42",
   "metadata": {},
   "outputs": [],
   "source": [
    "# split double names in sintes list\n",
    "\n",
    "lista_sintes= []\n",
    "\n",
    "for marca in lista_criba:\n",
    "    marca = marca.lower()\n",
    "    if marca not in lista_sintes:\n",
    "        marca=marca.split()\n",
    "        lista_sintes.append(marca)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "92bdeb4e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['4ms'],\n",
       " ['avp', 'a-v-p_synth'],\n",
       " ['acces'],\n",
       " ['access'],\n",
       " ['digitakt', 'elektron'],\n",
       " ['voix', 'la-voix-du-luthier'],\n",
       " ['luthier', 'la-voix-du-luthier'],\n",
       " ['oktatrak', 'elektron']]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lista_sintes[2:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "125dbd04",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 3.6 How to identify manufacturer brands?.\n",
    "\n",
    "<br>\n",
    "\n",
    "In this whole process, one of the things I wanted to implement was the possibility of being able to correctly read and identify the brand names of the synthesizer manufacturers from the description of the ad.\n",
    "\n",
    "\n",
    "\n",
    "If we solely rely on a dictionary capabilyties to obtain the manufacturer's name, a clear issue may arise if two manufacturers share the first part of their name, like: \n",
    "<br>\n",
    "\n",
    "    ...analogue systems, analogue solutions...\n",
    "    \n",
    "<br>\n",
    "\n",
    "The selection criterion would not be based on the correct name but rather on the position of that name within the dictionary so, to solve this problem we have to do:\n",
    "\n",
    "\n",
    "- Building a manufacturers' dictionary.\n",
    "\n",
    "- Implement an algorithm that differentiates between multiple manufacturer brands\n",
    "\n",
    "<br>\n",
    "\n",
    "As an example we will use this small dictionary as if it were our dictionary of synthesizer manufacturers:\n",
    "\n",
    "  - Manufacturers' dictionary:\n",
    "\n",
    "\n",
    "    sint3 = {\"analogue\":[\"solutions\",\"systems\"]} \n",
    "    \n",
    "<br>\n",
    "\n",
    "\n",
    "The implementation of the algorithm will be based on detecting:\n",
    "\n",
    "- ***single** names* like :`Roland`\n",
    "\n",
    "- **double** and unique names* like :  `Dave Smith`\n",
    "\n",
    "- ***double** names with a **first name common** to different manufacturers* like : `Analogue Systems or Analogue Solutions`\n",
    "\n",
    "\n",
    "\n",
    "\n",
    " \n",
    "<br>\n",
    "\n",
    "The particularity of **this** example is that it allows us to see one of the most controversial cases when it comes to the extraction of a name. \n",
    "\n",
    "---\n",
    "\n",
    "### Example of an implementation of the algorithm that detects brand names in the description:\n",
    "\n",
    "With this examples what is intended is just to understand how to read differents types of `ad description`:\n",
    "\n",
    "- `Analogue Systems`\n",
    "\n",
    "- ![analogue-systems](gif/analogue-systems.gif)\n",
    "\n",
    "<br>\n",
    "\n",
    "- `Analogue Solutions`\n",
    "\n",
    "![analogue-solutions](gif/analogue-solutions.gif)\n",
    "\n",
    "\n",
    "<br>\n",
    "\n",
    "- `Doepfer`\n",
    "\n",
    "- ![doepfer](gif/doepfer.gif)\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "daba6816",
   "metadata": {},
   "source": [
    "in the previous example we made use of a small dictionary that we could make by hand, but we need to implement a dictionary with all manufacturers.\n",
    "\n",
    "\n",
    "## Building the manufacturer's dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "22abe3d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "marcas_nombres = []\n",
    "\n",
    "def sint_word(sintex):\n",
    "    marcas_nombres.append(sintex)\n",
    "    return [marcas_nombres[-1]]\n",
    "\n",
    "def sint_more_word_rep(sintex):\n",
    "    marcas_nombres.append(sintex)\n",
    "    return marcas_nombres[-1]\n",
    "\n",
    "\n",
    "dict_funct = {\"sint_word\":sint_word,\n",
    "            \"sint_more_word_rep\":sint_more_word_rep\n",
    "}\n",
    "\n",
    "dict_marca = {}\n",
    "tag_mark = ''\n",
    "\n",
    "for marcas in lista_sintes:\n",
    "    if len(marcas) == 1:\n",
    "        if marcas[0] not in dict_marca:\n",
    "            tag_mark = 'sint_word'\n",
    "            brand = marcas[0]\n",
    "            ret = dict_funct[tag_mark](brand)\n",
    "            \n",
    "            dict_marca[brand] = ret\n",
    "            #print(\"x\")\n",
    "            \n",
    "    elif len(marcas) > 1:                           # aqui la marca tiene este formato: ['0', 'coast']\n",
    "        if marcas[0] not in dict_marca:\n",
    "            tag_mark = 'sint_word'\n",
    "            #print(marcas[0])\n",
    "            #print(marcas[1])\n",
    "\n",
    "           \n",
    "            ret = dict_funct[tag_mark](marcas[1])\n",
    "            dict_marca[marcas[0]] = ret\n",
    "\n",
    "        elif marcas[0] in dict_marca:\n",
    "            tag_mark = 'sint_more_word_rep'\n",
    "            ret = dict_funct[tag_mark](marcas[1])\n",
    "            dict_marca[marcas[0]].append(ret)\n",
    "            #print(\"x\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "e256db68",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'coast': ['0_coast'], '000': ['000'], '4ms': ['4ms'], 'avp': ['a-v-p_synth'], 'acces': ['acces'], 'access': ['access'], 'digitakt': ['elektron'], 'voix': ['la-voix-du-luthier', 'du'], 'luthier': ['la-voix-du-luthier'], 'oktatrak': ['elektron'], 'analog': ['elektron'], 'heat': ['elektron'], 'rythm': ['elektron'], 'digitone': ['elektron'], 'keys': ['elektron'], 'cycles': ['elektron'], 'samples': ['elektron'], 'acidlab': ['acidlab'], 'akai': ['akai'], 'mpc': ['akai'], 'alembic': ['alembic'], 'alesis': ['alesis'], 'allen': ['allen_&_heath', '&'], 'analogaudio1': ['analogaudio1'], 'analogue': ['solutions', 'systems', 'solutions'], 'arp': ['arp'], 'arturia': ['arturia'], 'asm': ['ashun_soundmachines'], 'atomo': ['atomo_synth'], 'damage': ['audio_damage'], 'audiophile': ['audiophile_circuits'], 'axoloty': ['axoloty'], 'balaguer': ['balaguer'], 'baloran': ['baloran'], 'bastl': ['bastl_instruments', 'instruments'], 'befaco': ['befaco'], 'behringer': ['behringer'], 'beringer': ['beringer'], 'bheringer': ['bheringer'], 'bitbox': ['bitbox'], 'black': ['corporation', 'lion'], 'boss': ['boss'], 'bubblesound': ['instruments'], 'buchla': ['buchla'], 'casio': ['casio'], 'charlie': ['lab'], 'charvel': ['charvel'], 'chronograf': ['chronograf'], 'circuit': ['abbey', 'happy'], 'clavia': ['clavia'], 'club': ['knobs'], 'corsynth': ['corsynth'], 'cre8audio': ['cre8audio'], 'crumar': ['crumar'], 'custom': ['made'], 'cyclone': ['cyclone', 'analogic'], 'dave': ['jones', 'smith', 'smith'], 'deepmind': ['deepmind'], 'delptronics': ['delptronics'], 'delta': ['music'], 'denon': ['dj'], 'dexibell': ['dexibell'], 'digitack': ['digitack'], 'doepfer': ['doepfer'], 'dreadbox': ['dreadbox'], 'dubreq': ['dubreq'], 'dynacord': ['dynacord'], 'e': ['mu'], 'e-mu': ['e-mu'], 'e:m:c': ['e:m:c'], 'elby': ['designs'], 'electribe': ['electribe'], 'electronic': ['music'], 'electrovoice': ['electrovoice'], 'elektron': ['elektron'], 'elka': ['elka'], 'emc': ['emc'], 'emu': ['emu'], 'endorphin.es': ['endorphin.es'], 'endorphines': ['endorphines'], 'ensoniq': ['ensoniq'], 'eowave': ['eowave'], 'epiphone': ['epiphone'], 'erica': ['synth', 'synths'], 'ernie': ['ball'], 'esp': ['ltd'], 'eurorack': ['eurorack'], 'eventide': ['eventide'], 'evh': ['evh'], 'evolver': ['evolver'], 'exodus': ['digital'], 'farfisa': ['farfisa'], 'fender': ['fender'], 'fishman': ['fishman'], 'fodera': ['fodera'], 'formanta': ['formanta'], 'frap': ['tool', 'tools', 'tools'], 'frequency': ['central'], 'fretlight': ['fretlight'], 'friedman': ['friedman'], 'future': ['retro', 'sound'], 'futuresonus': ['futuresonus'], 'gator': ['gator'], 'gemini': ['gemini'], 'generalmusic': ['generalmusic'], 'gibson': ['gibson'], 'godin': ['godin'], 'gotharman': ['gotharman'], 'graph': ['tech'], 'gretsch': ['gretsch'], 'guild': ['guild'], 'hammond': ['hammond'], 'hartmann': ['hartmann'], 'hexinverter': ['hexinverter'], 'hinton': ['instruments'], 'hofner': ['hofner'], 'hypersynth': ['hypersynth'], 'ibanez': ['ibanez'], 'ik': ['ik', 'multimedia'], 'instruo': ['instruo'], 'iomega': ['iomega'], 'isla': ['isla'], 'jackson': ['jackson'], 'jaspers': ['jaspers'], 'john': ['bowen'], 'jomox': ['jomox'], 'kawai': ['kawai'], 'kenton': ['kenton'], 'ketron': ['ketron'], 'kilpatrick': ['audio'], 'knobula': ['knobula'], 'koma': ['elektronik', 'elektronik'], 'komplete': ['komplete'], 'korg': ['korg'], 'kramer': ['kramer'], 'kurzweil': ['kurzweil'], 'lakland': ['lakland'], 'line': ['6'], 'linn': ['electronics'], 'livid': ['livid'], 'logan': ['electronics'], 'm-audio': ['m-audio'], 'macbeth': ['studio'], 'make': ['make', 'noise'], 'malekko': ['malekko', 'heavy', 'heavy'], 'manikin': ['electronic'], 'maschine': ['maschine'], 'mellotron': ['mellotron'], 'mfb': ['mfb'], 'micro': ['modular'], 'miditech': ['miditech'], 'models': ['models'], 'modor': ['modor'], 'modular': ['modular'], 'modulus': ['modulus'], 'monome': ['monome'], 'moog': ['moog'], 'mutable': ['instruments'], 'mutant': ['mutant'], 'native': ['instruments'], 'neutron': ['neutron'], 'noise': ['engineering'], 'nord': ['nord', 'electro', 'lead', 'lead', 'lead', 'micro', 'modular', 'rack', 'stage', 'wave'], 'novation': ['novation'], 'numark': ['numark'], 'oberheim': ['oberheim'], 'octatrack': ['octatrack'], 'orthogonal': ['devices'], 'paratek': ['paratek'], 'pearl': ['pearl'], 'peavey': ['peavey'], 'pioneer': ['dj'], 'pittsburgh': ['pittsburgh', 'modular'], 'polyend': ['polyend'], 'polygraf': ['polygraf'], 'ppg': ['(palm'], 'prs': ['prs'], 'qu': ['bit'], 'qu-bit': ['qu-bit', 'electronix'], 'quasimidi': ['quasimidi'], 'qubit': ['qubit'], 'quiklok': ['quiklok'], 'radikal': ['technologies'], 'rhodes': ['rhodes'], 'rickenbacker': ['rickenbacker'], 'roland': ['roland'], 'roli': ['roli'], 'sanson': ['sanson'], 'schecter': ['schecter'], 'sensel': ['sensel'], 'sequencial': ['sequencial'], 'sequential': ['circuits'], 'sequentix': ['sequentix'], 'shakmat': ['shakmat', 'modular'], 'simmons': ['simmons'], 'soma': ['soma', 'laboratory'], 'sonicware': ['sonicware'], 'special': ['waves'], 'spector': ['spector'], 'spectral': ['audio'], 'sputnik': ['sputnik'], 'squarp': ['instruments'], 'squier': ['squier'], 'ssff': ['ssff'], 'stanton': ['stanton'], 'steinberger': ['steinberger'], 'sterling': ['sterling'], 'strymon': ['strymon'], 'studio': ['electronics'], 'synamodec': ['synamodec'], 'synthesis': ['technology'], 'synthrotek': ['synthrotek'], 'synthstrom': ['synthstrom'], 'synthtech': ['synthtech'], 'swissonic': ['swissonic'], 'tascam': ['tascam'], 'taylor': ['taylor'], 'technos': ['technos'], 'teenage': ['teenage', 'engineering'], 'tiptop': ['tiptop', 'audio'], 'traveler': ['guitar'], 'udo': ['audio'], 'uno': ['synth'], 'vermona': ['vermona'], 'virus': ['virus'], 'viscount': ['viscount'], 'volca': ['volca'], 'vox': ['vox'], 'waldorf': ['waldorf'], 'warwick': ['warwick'], 'washburn': ['washburn'], 'waves': ['grendel'], 'wersi': ['wersi', 'music'], 'winter': ['modular'], 'wmd': ['wmd', '/'], 'wurlitzer': ['wurlitzer'], 'yamaha': ['yamaha'], 'yocto': ['yocto'], 'zeppelin': ['design'], 'zoom': ['zoom'], '1010': ['music'], '2hp': ['2hp'], 'acid': ['rain'], 'acl': ['acl'], 'addac': ['system', 'system'], 'after': ['later'], 'aion': ['modular'], 'ajh': ['synth'], 'cosmos': ['soma'], 'divina': ['soma'], 'enner': ['soma'], 'ether': ['soma'], 'flux': ['soma'], 'illuminator': ['soma'], 'lyra-8': ['soma'], 'lyra8-fx': ['soma'], 'metaconformer': ['soma'], 'ornament-8': ['soma'], 'pulsar-23': ['soma'], 'qo': ['soma'], 'reflex': ['soma'], 'roat': ['soma'], 'terra': ['soma'], 'the': ['pipe', 'division'], 'alm': ['busy', 'busy'], 'alright': ['devices'], 'blackhole': ['cases'], 'blue': ['lantern', 'microphones'], 'boredbrain': ['music'], 'cosmotronic': ['cosmotronic'], 'divkid': ['divkid'], 'dnipro': ['modular'], 'e-rm': ['e-rm'], 'linndrum': ['linndrum'], 'electrosmith': ['electrosmith'], 'emblematic': ['systems'], 'empress': ['effects'], 'erogenous': ['tones'], 'eskatonic': ['modular', 'modular'], 'five12': ['five12'], 'gieskes': ['gieskes'], 'grayscale': ['grayscale'], 'industrial': ['music'], 'io': ['instruments'], 'joranalogue': ['joranalogue'], 'klavis': ['klavis'], 'l-1': ['l-1'], 'lmntl': ['lmntl'], 'low-gain': ['electronics'], 'lzx': ['industries', 'industries'], 'eloquencer': ['eloquencer'], 'manhattan': ['analog'], 'meng': ['qi'], 'michigan': ['synth'], 'modbap': ['modular'], 'mordax': ['mordax'], 'mosaic': ['mosaic'], 'mrseri': ['mrseri'], 'nano': ['modules'], 'patching': ['panda'], 'percussa': ['percussa'], 'plankton': ['electronics'], 'poly': ['effects'], 'random': ['source'], 'ritual': ['electronics'], 'rossum': ['rossum'], 'schlappi': ['engineering'], 'soundforce': ['soundforce'], 'soundmachines': ['soundmachines'], 'steady': ['state'], 'supercritical': ['supercritical'], 'system': ['80'], 'tall': ['dog'], 'tasty': ['chips'], 'tenderfoot': ['electronics'], 'tesseract': ['modular'], 'trogotronic': ['trogotronic'], 'tubbutec': ['tubbutec'], 'u-he': ['u-he'], 'verbos': ['electronics'], 'voicas': ['voicas'], 'vpme.de': ['vpme.de'], 'worng': ['electronics'], 'xaoc': ['devices'], 'xor': ['electronics'], 'zlob': ['modular'], 'modal': ['electronics'], 'critter': ['&'], 'jmt': ['synth'], 'herbs': ['and'], 'grp': ['grp'], 'norand': ['norand'], 'playtime': ['engineering'], \"fred's\": ['lab'], 'studiologic': ['studiologic'], 'suzuki': ['suzuki'], 'nonlinear': ['labs'], 'dato': ['dato'], 'artiphon': ['artiphon'], 'kodamo': ['kodamo'], 'hikari': ['instruments'], 'second': ['sound'], 'roger': ['linn'], 'conductive': ['labs'], 'faderfox': ['faderfox'], 'keith': ['mcmillen'], 'expressive': ['e'], 'jouã©': ['jouã©'], 'genki': ['genki'], 'expert': ['sleepers'], 'motu': ['motu'], 'midi': ['solutions'], 'solid': ['state'], 'electro-harmonix': ['electro-harmonix'], 'knas': ['knas'], 'iconnectivity': ['iconnectivity'], 'eurodesk-z': ['eurodesk-z'], 'presonus': ['presonus'], 'torso': ['electronics'], 'esi': ['audiotechnik', 'audiotechnik'], 'instruments': ['of'], 'apogee': ['apogee'], 'snd': ['snd'], 'moffenzeef': ['moffenzeef'], 'cme': ['cme'], 'embodme': ['embodme'], 'tech': ['21'], 'snyderphonics': ['snyderphonics'], 'tricks': ['magic'], 'oto': ['machines'], 'chase': ['bliss'], 'gfi': ['gfi'], 'meris': ['meris'], 'echo': ['fix'], 'fairfield': ['circuitry'], 'universal': ['audio'], 'gamechanger': ['audio'], 'earthquaker': ['devices'], 'death': ['by'], 'sherman': ['sherman'], 'old': ['blood'], 'red': ['panda'], 'kemper': ['kemper'], 'digitech': ['digitech'], 'jam': ['pedals'], 'thermionic': ['culture'], 'warm': ['audio', 'audio'], 'benidub': ['benidub'], 'bae': ['bae'], 'vongon': ['vongon'], 'art': ['art', 's'], 'hungry': ['robot'], 'walrus': ['audio'], 'enjoy': ['electronics'], 'cioks': ['cioks'], 'tk': ['audio'], 'source': ['audio'], 'api': ['api'], 'voodoo': ['lab', 'lab'], 'fmr': ['audio'], 'jhs': ['pedals'], 'mod': ['devices'], 'cooper': ['fx'], 'finegear': ['finegear'], 'ezhi': ['&'], 'truetone': ['truetone'], 'lastgasp': ['art'], 'origin': ['effects'], 'rainger': ['fx'], 'pedaltrain': ['pedaltrain'], 'dr.': ['scientist'], 'elta': ['music'], 'keeley': ['keeley'], 'recovery': ['recovery'], 'glou-glou': ['glou-glou'], 'retro': ['mechanical'], 'electro-faustus': ['electro-faustus'], 'animal': ['factory'], 'hologram': ['hologram'], 'caroline': ['guitar'], 'mxr': ['mxr'], 'xotic': ['xotic'], 'dunlop': ['dunlop'], 'adventure': ['audio'], 'isp': ['technologies'], 'industrialectric': ['industrialectric'], 'collision': ['devices'], 'orgeldream': ['orgeldream'], 'rupert': ['neve'], 'shure': ['shure'], 'focusrite': ['focusrite'], 'focal': ['focal'], 'neumann': ['neumann'], 'adam': ['adam'], 'furman': ['furman'], 'antelope': ['audio'], 'dangerous': ['music'], 'dbx': ['dbx'], 'maag': ['audio'], 'empirical': ['labs'], 'avantone': ['pro'], 'mackie': ['mackie'], 'audient': ['audient'], 'beyerdynamic': ['beyerdynamic'], 'rme': ['rme'], 'golden': ['age'], 'audio-technica': ['audio-technica'], 'fredenstein': ['fredenstein'], 'a-designs': ['a-designs'], 'rosson': ['audio'], 'daking': ['daking'], 'looptrotter': ['looptrotter'], 'rode': ['rode'], 'prism': ['sound'], 'samson': ['samson'], 'cranborne': ['audio'], 'elysia': ['elysia'], 'hedd': ['hedd'], 'heritage': ['audio'], 'avedis': ['audio'], 'sennheiser': ['sennheiser'], 'lindell': ['audio'], 'krk': ['krk'], 'akg': ['akg'], 'cloud': ['microphones'], 'steinberg': ['steinberg'], 'dynaudio': ['dynaudio'], 'austrian': ['audio'], 'auralex': ['auralex'], 'isoacoustics': ['isoacoustics'], 'aston': ['microphones'], 'auratone': ['auratone'], 'se': ['electronics'], 'lauten': ['audio'], 'cascade': ['microphones'], 'soundrise': ['soundrise'], 'pro-ject': ['pro-ject'], 'playdifferently': ['playdifferently'], 'u-turn': ['audio'], 'thorens': ['thorens'], 'audioengine': ['audioengine'], 'technics': ['technics'], 'rane': ['rane'], 'music': ['hall'], 'jesse': ['dean'], 'ortofon': ['ortofon'], 'mwm': ['mwm'], 'ultimate': ['support'], 'odyssey': ['odyssey'], 'record': ['props'], 'hosa': ['hosa'], 'mogami': ['mogami'], 'myvolts': ['myvolts'], 'retrokits': ['retrokits'], 'ebs': ['ebs'], 'pomona': ['electronics'], 'modbang': ['modbang'], 'intellijel': ['designs'], 'radial': ['engineering'], 'lablab': ['audio'], 'pulp': ['logic']}\n"
     ]
    }
   ],
   "source": [
    "print(dict_marca)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45ca6b6e",
   "metadata": {
    "tags": []
   },
   "source": [
    "It will give us:\n",
    "\n",
    "<br>\n",
    "\n",
    "- What was inside of `sint3[idx]` ['solutions', 'systems'] thanks to a print.\n",
    "\n",
    "\n",
    "- Finally we have the we have the right name of the description, as we expected.\n",
    "\n",
    "<br>\n",
    "\n",
    "Once we understand the operation we are going to implement the necessary code.\n",
    "\n",
    "\n",
    "## 3.7 Detecting manufacturer brands.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "b38bd4c2",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "compare = ''                    #variable where the middle name is saved\n",
    "marca_del_sinte = ''            # empty variable for store synth brand \n",
    "texto_descriptivo = ''          #ad descriptive text\n",
    "list_temp = []                  #temporary list to detect the middle name \n",
    "\n",
    "                                # buy, sell, change... lists.\n",
    "list_compro = []\n",
    "list_cambio = []\n",
    "list_vendo = []\n",
    "list_regalo = []\n",
    "list_busco = []\n",
    "list_rebaja = []\n",
    "list_reparar = []\n",
    "list_piezas = []\n",
    "list_urgente = []\n",
    "list_oferta = []\n",
    "\n",
    "list_brand = []                 # manufacturers synth brand\n",
    "list_descripcion = []           # final ad description on dataframe output \n",
    "texto_descriptivo_salida = []   # esto es el contenido del anuncio\n",
    "\n",
    "list_price = []                 # price\n",
    "list_user = []                  # user\n",
    "list_city = []                  # city\n",
    "list_published = []             # date published\n",
    "list_expire = []                # data expire ad\n",
    "list_times_seen= []             # times seen ad\n",
    "\n",
    "list_original=[]\n",
    "\n",
    "lista_palabras_para_eliminar = [] # In this list I'm going to add the words that I should remove from the ad. Stocks, synth brand.\n",
    "\n",
    "def func_compro(clave_func_dict): \n",
    "    if list_compro[-1] == \"0\":\n",
    "        list_vendo.pop(-1)\n",
    "        list_vendo.append(\"0\")\n",
    "        list_compro.pop(-1)\n",
    "        list_compro.append(\"1\")\n",
    "    else:\n",
    "        pass\n",
    "    \n",
    "def func_cambio(clave_func_dict):\n",
    "    if list_cambio[-1] == \"0\":\n",
    "        list_vendo.pop(-1)\n",
    "        list_vendo.append(\"0\")\n",
    "        list_cambio.pop(-1)\n",
    "        list_cambio.append(\"1\")\n",
    "        #list_price.append(\"0\")\n",
    "    else:\n",
    "        pass\n",
    "\n",
    "def func_vendo(clave_func_dict):\n",
    "    if list_vendo[-1] == \"0\":\n",
    "        list_vendo.pop(-1)\n",
    "        list_vendo.append(\"1\")\n",
    "    else:\n",
    "        pass\n",
    "\n",
    "def func_regalo(clave_func_dict): \n",
    "    if list_regalo[-1] == \"0\":\n",
    "        list_vendo.pop(-1)\n",
    "        list_vendo.append(\"0\")\n",
    "        list_regalo.pop(-1)\n",
    "        list_regalo.append(\"1\")\n",
    "    else:\n",
    "        pass\n",
    "\n",
    "def func_busco(clave_func_dict):  # if looking for, then is not a sell...\n",
    "    if list_busco[-1] == \"0\":\n",
    "        list_vendo.pop(-1)\n",
    "        list_vendo.append(\"0\")\n",
    "        list_busco.pop(-1)\n",
    "        list_busco.append(\"1\")\n",
    "    else:\n",
    "        pass\n",
    "\n",
    "def func_reparar(clave_func_dict):\n",
    "    if list_busco[-1] == \"0\":\n",
    "        list_reparar.pop(-1)\n",
    "        list_reparar.append(\"1\")\n",
    "    else:\n",
    "        pass\n",
    "\n",
    "def func_piezas(clave_func_dict):\n",
    "    if list_busco[-1] == \"0\":\n",
    "        list_piezas.pop(-1)\n",
    "        list_piezas.append(\"1\")\n",
    "    else:\n",
    "        pass\n",
    "\n",
    "def func_rebaja(clave_func_dict):\n",
    "    if list_rebaja[-1] == \"0\":\n",
    "        list_vendo.pop(-1)\n",
    "        list_vendo.append(\"0\")\n",
    "        list_rebaja.pop(-1)\n",
    "        list_rebaja.append(\"1\")\n",
    "    else:\n",
    "        pass\n",
    "\n",
    "def func_oferta(clave_func_dict):\n",
    "    if list_oferta[-1] == \"0\":\n",
    "        list_oferta.pop(-1)\n",
    "        list_oferta.append(\"1\")\n",
    "    else:\n",
    "        pass\n",
    "\n",
    "func_dict = {                                                        # function dictionary\n",
    "    \"compro\":func_compro,\n",
    "    \"cambio\":func_cambio,\n",
    "    \"vendo\":func_vendo,\n",
    "    \"vende\":func_vendo,\n",
    "    \"regalo\":func_regalo,\n",
    "    \"busco\":func_busco,\n",
    "    \"busca\":func_busco,\n",
    "    \"reparar\":func_reparar,\n",
    "    \"piezas\":func_piezas,\n",
    "    \"rebajado\":func_rebaja,\n",
    "    \"rebaja\":func_rebaja,\n",
    "    \"oferta\":func_oferta\n",
    "}\n",
    "\n",
    "def remove_compro(clave_func_dict):\n",
    "    #list_compro.append(clave_func_dict\n",
    "    list_compro.remove(clave_func_dict)\n",
    "\n",
    "#rmv_func = {\"compro\":remove_compro}\n",
    "\n",
    "\n",
    "def urgente():                                                       # if some \"accion\" word is repeated on description, means urgency\n",
    "    list_urgente.remove('0')\n",
    "    list_urgente.append(\"1\")\n",
    "\n",
    "def eliminar_signos(txt): \n",
    "    # cleaning text\n",
    "    txt = txt.lower()\n",
    "    description = txt.replace(\":\",\" \")\n",
    "    descripcion = description.replace(\";\",\" \")\n",
    "    descripcion_1 = descripcion.replace(\"(\",\" \")\n",
    "    descripcion_2 = descripcion_1.replace(\")\",\" \")\n",
    "    descripcion_3 = descripcion_2.replace(\"/\",\" \")\n",
    "    descripcion_4 = descripcion_3.replace(\".\",\" \")\n",
    "    descripcion_5 = descripcion_4.split()\n",
    "    return descripcion_5\n",
    "\n",
    "def default_atributes():                                            # default actions, means all is selling, if not then function will be called.\n",
    "    \"\"\"\n",
    "    Añade contenido a las diferentes listas con las que se trabaja en cada fila.\n",
    "    \"\"\"\n",
    "    list_cambio.append(\"0\")\n",
    "    list_compro.append(\"0\")\n",
    "    list_urgente.append(\"0\")\n",
    "    list_vendo.append(\"1\")\n",
    "    list_regalo.append(\"0\")\n",
    "    list_reparar.append(\"0\")\n",
    "    list_piezas.append(\"0\")\n",
    "    list_busco.append(\"0\")\n",
    "    list_brand.append(\"-\")\n",
    "\n",
    "### Inicio\n",
    "\n",
    "\n",
    "for pagina_anuncio in os.listdir('.'): # Read the contents of the directory with a for in the current \".\" folder\n",
    "    with open(pagina_anuncio, 'r') as pagina_bruto:\n",
    "        pagina_analizar = pagina_bruto.read()                        # Converts to the 'pagina_analizar' object with python's read() method\n",
    "        soup = BeautifulSoup(pagina_analizar, 'html.parser')         # With Beautifulsoup, the pagina_analizar is parsed with the 'html.parser' and a variable named soup is passed\n",
    "        node = soup.find('h1')                                       # All the contents of the H1 tag are searched within SOUP, and that content is fed into the NODE variable.\n",
    "\n",
    "    if  node is not None:                                            # avoiding skipping an error related to None. An if is used to check that \"node\" is not empty using the condition \"if node is not None\"\n",
    "        descripcion = node.text                                      # Using the.text method, I extract the text from Node and pass it to the Description variable\n",
    "        descripcion = eliminar_signos(descripcion)                   # function that removes punctuation ;,:,(,),/... and lowercase the text\n",
    "                             \n",
    "\n",
    "        default_atributes()                                          # Calling the function default_atributes().\n",
    "        \n",
    "        # --- synt_brand\n",
    "\n",
    "        for word_1 in descripcion:                                  \n",
    "            if word_1 in accion:                                     \n",
    "                func_dict[word_1](word_1)                           \n",
    "                lista_palabras_para_eliminar.append(word_1)\n",
    "\n",
    "            elif word_1 in compare:   \n",
    "                list_temp.append(word_1)                            \n",
    "\n",
    "                for marca_sinte in list_temp:                       \n",
    "                    marca_del_sinte += marca_sinte + ' '             \n",
    "                    lista_palabras_para_eliminar.append(marca_sinte) \n",
    "\n",
    "                list_brand.pop(-1)\n",
    "                list_brand.append(marca_del_sinte)\n",
    "\n",
    "                compare = '' \n",
    "\n",
    "            elif word_1 in dict_marca:                        \n",
    "                size_brand = len(dict_marca[word_1])\n",
    "\n",
    "                if ((size_brand == 1) and (list_brand != \"-\")) :\n",
    "                    list_brand.pop(-1)\n",
    "                    list_brand.append(word_1)\n",
    "                    break\n",
    "\n",
    "                elif ((size_brand == 1) and (list_brand == \"-\")) :\n",
    "                    list_descripcion.append(word_1)\n",
    "\n",
    "                if ((size_brand >= 1) and (list_brand == \"-\") and (size_brand != 0)) :  \n",
    "                    list_descripcion.append(word_1) \n",
    "                    list_brand.pop(-1) \n",
    "                    x = dict_marca[word_1]\n",
    "                    list_brand.append(x)\n",
    "                    break\n",
    "\n",
    "                elif size_brand >= 1:                               \n",
    "                    compare = dict_marca[word_1]                    \n",
    "                    list_temp.append(word_1)                         \n",
    "\n",
    "                elif list_brand != \"-\":                              \n",
    "                    list_descripcion.append(word_1)\n",
    "\n",
    "            marca_del_sinte = ''\n",
    "        list_temp.clear()\n",
    "        \n",
    "        # --- urgente\n",
    "        \n",
    "        duplicates = [element for element in lista_palabras_para_eliminar if lista_palabras_para_eliminar.count(element) > 1] # Detecta caracteres repetidos dentro de 'lista_de_palabras_para_eliminar' siempre que el tamaño de la lista sea superior a 1\n",
    "        unique_duplicates = list(set(duplicates))                                                                             # Muestra el elemento duplicado\n",
    "        size_unique_duplicates = len(duplicates)                                                                              # Muestra la longitud de esos dos elementos sumados 'size_unique_duplicates'\n",
    "        if size_unique_duplicates > 3:                                                                                        # Si la longitud 'size_unique_duplicates' es superior a 3 entonces llama a la función urgente.\n",
    "            urgente()                                                                                                         # Pinta un 1 en la columna urgente\n",
    "\n",
    "        for eliminar in lista_palabras_para_eliminar:\n",
    "            try:\n",
    "                descripcion.remove(eliminar)                # As actions are identified, and the synth's name is removed from the ad description\n",
    "            except:\n",
    "                pass\n",
    "\n",
    "            \n",
    "        for palabras in descripcion:                       # The description is traversed after it has been deleted and what remains is entered into a variable 'texto_descriptivo'\n",
    "            texto_descriptivo += palabras + ' '\n",
    "\n",
    "        texto_descriptivo_salida.append(texto_descriptivo) # The variable with the content of 'texto_descriptivo' will be the text that will finally remain as a description in the final csv\n",
    "\n",
    "        texto_descriptivo =''                              # I write the content of the variable 'texto_descriptivo' on top of it by way of a reset.\n",
    "\n",
    "\n",
    "        # --- price\n",
    "        \n",
    "        try:\n",
    "            # Try to find the element with the 'ad-price' class and extract the text\n",
    "            price = soup.find('div', class_='ad-price').text\n",
    "            # Quita el símbolo € del texto del precio\n",
    "            price = price.replace(\"€\", \"\")\n",
    "            \n",
    "        except AttributeError:\n",
    "            # If the item is missing, assign \"N/A\" to the price variable\n",
    "            price = 0\n",
    "            # Delete the last item in list_price if it exists (there may be an error if the list is empty)\n",
    "        \n",
    "        finally:\n",
    "            # Add the price value (either the found price or \"N/A\") to list_price\n",
    "            list_price.append(price)\n",
    "        \n",
    "\n",
    "        # --- user name\n",
    "\n",
    "        user = soup.find('div',class_='col-lg-7').a.text\n",
    "        list_user.append(user)\n",
    "        \n",
    "\n",
    "        # --- city\n",
    "\n",
    "        city = soup.find('div',class_='col-lg-7').div.strong.text\n",
    "        list_city.append(city)\n",
    "\n",
    "    \n",
    "        # --- published\n",
    "\n",
    "        publish = ' '\n",
    "\n",
    "        try:\n",
    "            # Find the div element with the class 'col-lg-7' and extract the text from the inner div\n",
    "            published = soup.find('div', class_='col-lg-7').div.text.split()[-5:-2]\n",
    "\n",
    "            for indx in published:\n",
    "                list_original.append(indx)  # Add indx to the list list_original\n",
    "\n",
    "                # Check to see if there's a forward slash on the item\n",
    "                if '/' in indx:\n",
    "                    # indx = indx.replace(\"/\", \"-\")  # Reemplaza \"/\" por \"-\"\n",
    "                    DD = indx[0:2]  # Extract the first two characters (day)\n",
    "                    MM = indx[3:5]  # Extract the next two characters (month)\n",
    "                    YYYY = indx[6:]  # Extract the remaining characters (year)\n",
    "                    publish = f'{YYYY}/{MM}/{DD}'  # Create the date string in YYYY-MM-DD format\n",
    "                    #print(\"YYYY\", publish)\n",
    "                    \n",
    "\n",
    "                # If \"hace\" is in the element, it means that it is no longer a date that is extracted, but the reference to how long ago.\n",
    "                elif 'hace' in indx:\n",
    "                    #indx = indx.replace(\"/\", \"-\")  # Replace \"/\" por \"-\"\n",
    "                    a = published.index(indx)  # Gets the index of the current item\n",
    "\n",
    "                    # Combine the numerical value and the unit of time (2 hours ago, 5 days ago, 2 weeks ago...)\n",
    "                    publish = published[a + 1] + ' ' + published[a + 2] # <- With this I get the format of: 1 week ago or 19 hours ago...\n",
    "                    #  I put the Publish content into the dataframe, later I'll modify that annoying format\n",
    "                    #print(\"publish\", publish)\n",
    "                    \n",
    "\n",
    "        except (AttributeError, IndexError):\n",
    "            # If exceptions occur due to attribute or index issues, assign \"N/A\" to the publish variable\n",
    "            publish = \" \"\n",
    "\n",
    "        finally:\n",
    "            # Add the final value of \"publish\" to the list list_published\n",
    "            list_published.append(publish)\n",
    "\n",
    "    \n",
    "        \n",
    "        # --- expire \n",
    "\n",
    "        expire = soup.find('div',class_=\"expira\").text.split()[1]\n",
    "        #expire = expire.replace(\"/\",\"-\")\n",
    "        DD = expire[0:2]\n",
    "        MM = expire[3:5]\n",
    "        YYYY = expire[6:]\n",
    "        date_corrected = f'{YYYY}-{MM}-{DD}'\n",
    "        list_expire.append(date_corrected)\n",
    "        \n",
    "        \n",
    "\n",
    "        # --- times seen\n",
    "        \n",
    "        seen = soup.find('div',class_=\"expira\").text.split()[4]\n",
    "        list_times_seen.append(seen)\n",
    "\n",
    "        lista_palabras_para_eliminar.clear()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44c06356-a30b-4b40-86ce-90d4e5e8c772",
   "metadata": {},
   "source": [
    "### About BeautifulSoup Warning:\n",
    "\n",
    "This bug report is a duplicate of:  Bug #1873787: Suppress UserWarning * looks like a URL. Edit Remove\n",
    "\n",
    "https://bugs.launchpad.net/beautifulsoup/+bug/1955450\n",
    "\n",
    "    ...Beautiful Soup generally takes the approach of trying to give \"helpful\" error/warning codes so that a user understands why things are not working the way they expect. While every developer may have a different opinion on how helpful error/warnings should be done, Beautiful Soup has taken a more ambitious approach..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c41c71c6",
   "metadata": {},
   "source": [
    "## 3.8 Date extraction\n",
    "\n",
    "The next step is to know what is the extraction date. This is an important fact since because it will serve as a reference to know how long means 3 days, 1 week, 5 hours since the records were made."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "a9310eb0",
   "metadata": {},
   "outputs": [],
   "source": [
    "hoy = dt.datetime.now()\n",
    "year=str(hoy.year)\n",
    "month=str(hoy.month)\n",
    "day=str(hoy.day)\n",
    "date_scrapped = year + '/' + month + '/' + day"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a35460d7",
   "metadata": {},
   "source": [
    "- Dataframe created in `df` variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "cf12bc24",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame({'urgent':list_urgente,\n",
    "                   'buy':list_compro,\n",
    "                   'change':list_cambio,\n",
    "                   'sell':list_vendo,\n",
    "                   'price':list_price,\n",
    "                   'gift':list_regalo,\n",
    "                   'search':list_busco,\n",
    "                   'repair':list_reparar,\n",
    "                   'parts':list_piezas,\n",
    "                   'synt_brand':list_brand,\n",
    "                   'description':texto_descriptivo_salida,\n",
    "                   'city':list_city,\n",
    "                   'published':list_published,\n",
    "                   'expire':list_expire,\n",
    "                   'date_scrapped':date_scrapped,\n",
    "                   'seen':list_times_seen\n",
    "                  },index = list(range(1,len(texto_descriptivo_salida)+1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30b4cd01",
   "metadata": {},
   "source": [
    "## 3.9 Clean the column of the publication dates.\n",
    "\n",
    "\n",
    "As we can see sometimes the format is correct and sometimes indicates moments related to the date we are on. so it has to be corrected.\n",
    "\n",
    "The solution is to create a function that reads that format and converts it to the correct date and format.\n",
    "\n",
    "\n",
    "For this we have to implement all the cases that can be given."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "8b629d3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "semanas = ['1 semana', '2 semanas', '3 semanas', '4 semanas']\n",
    "dias = ['1 día', '2 días', '3 días', '4 días', '5 días', '6 días', '7 días']\n",
    "horas = ['1 hora','2 horas', '3 horas', '4 horas', '5 horas', '6 horas',\n",
    "        '7 horas','8 horas', '9 horas', '10 horas', '11 horas', '12 horas',\n",
    "        '13 horas', '14 horas','15 horas', '16 horas', '17 horas', '18 horas',\n",
    "        '19 horas', '20 horas', '21 horas', '22 horas','23 horas', '24 horas']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "ad2198c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "minutes=[]\n",
    "for mint in range(1,61):\n",
    "    if mint < 2:\n",
    "        texto = str(mint) + ' minuto'\n",
    "        minutes.append(texto)\n",
    "    else:\n",
    "        texto = str(mint) + ' minutos'\n",
    "        minutes.append(texto)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1226207",
   "metadata": {},
   "source": [
    "- `nice_format` is the function that is responsible for identifying the time intervals that the web gives us and making a time conversion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "d673457a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def nice_format(parameter):\n",
    "\n",
    "    days_inweek = 7\n",
    "    hoy = dt.datetime.now()\n",
    "    year=str(hoy.year)\n",
    "    month=str(hoy.month)\n",
    "    day=str(hoy.day)\n",
    "\n",
    "    date_scrapped = year + '/' + month + '/' + day\n",
    "    \n",
    "    current_datetime = dt.datetime.strptime(date_scrapped,\"%Y/%m/%d\")  \n",
    "    \n",
    "    \n",
    "    if parameter in semanas:\n",
    "    \n",
    "        num_semana = parameter.split()\n",
    "        num_semana = int(num_semana[0])\n",
    "        cambio_semana = semanas[num_semana-1]\n",
    "        \n",
    "        dias_semana = (num_semana * days_inweek)\n",
    "        \n",
    "        fecha_real_semana = current_datetime - dt.timedelta(dias_semana)\n",
    "        \n",
    "        fecha_real_semana = fecha_real_semana.strftime(\"%Y/%m/%d\")\n",
    "                \n",
    "        df['published'] = df['published'].replace( to_replace = cambio_semana, value = fecha_real_semana) #+ ' semana'\n",
    "        \n",
    "        \n",
    "    if parameter in dias:\n",
    "        num_dia = parameter.split()\n",
    "        num_dias = int(num_dia[0])\n",
    "        cambio_dia = dias[num_dias-1]\n",
    "\n",
    "        fecha_real_dia = current_datetime - dt.timedelta(num_dias)\n",
    "        fecha_real_dia = fecha_real_dia.strftime(\"%Y/%m/%d\")\n",
    "        \n",
    "        df['published'] = df['published'].replace( to_replace = cambio_dia, value = fecha_real_dia) #+ ' semana'\n",
    "        \n",
    "        \n",
    "    if parameter in horas:\n",
    "        num_hora = parameter.split()\n",
    "        num_hora = int(num_hora[0])\n",
    "        \n",
    "        if (parameter != '24 horas'):\n",
    "            hora_real = current_datetime\n",
    "            hora_real = hora_real.strftime(\"%Y/%m/%d\")\n",
    "            \n",
    "            df['published'] = df['published'].replace(to_replace = parameter,\n",
    "                                              value = hora_real)\n",
    "        \n",
    "        elif parameter == '24 horas':\n",
    "            horas_24 = 1\n",
    "            hora_real = current_datetime - dt.timedelta(horas_24)\n",
    "            hora_real = hora_real.strftime(\"%Y/%m/%d\")\n",
    "            \n",
    "            df['published'] = df['published'].replace( to_replace = parameter,value = hora_real ) #+ ' semana'\n",
    "    \n",
    "    \n",
    "    if parameter in minutes:\n",
    "        horas_24 = 1\n",
    "        hora_real = current_datetime - dt.timedelta(horas_24)\n",
    "        hora_real = hora_real.strftime(\"%Y/%m/%d\")\n",
    "            \n",
    "        df['published'] = df['published'].replace( to_replace = parameter,value = hora_real ) #+ ' semana'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "91ea4b72-4330-4954-b916-7b8c15301c7f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "df['published'].apply(nice_format) # Make changes on Series\n",
    "print('')                          # Avoiding verbosing print\n",
    "\n",
    "df['expire'] = df['expire'].str.replace('-', '/')\n",
    "df['date_scrapped'] = df['date_scrapped'].str.replace('-', '/')\n",
    "\n",
    "# Function to convert lists to text strings\n",
    "def convert_to_string(lista):\n",
    "    return str(lista)\n",
    "\n",
    "# Function to modify the punctuation marks of a series\n",
    "def remove_punctuation_marks(serie):\n",
    "    serie = serie.str.replace(r',', '', regex=True)\n",
    "    serie = serie.str.replace(r'\\[', '', regex=True)\n",
    "    serie = serie.str.replace(r'\\]', '', regex=True)\n",
    "    serie = serie.str.replace(r'\\'', '', regex=True)\n",
    "    return serie\n",
    "\n",
    "df['synt_brand'] = df['synt_brand'].apply(convert_to_string)\n",
    "df['synt_brand'] = remove_punctuation_marks(df[\"synt_brand\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd121700",
   "metadata": {},
   "source": [
    "### Last step\n",
    "\n",
    "We already have all the data inside the dataframe now the only thing left to do is to save the content in a csv file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "7e8ea201-bfc1-401c-b678-7e64eea14eb4",
   "metadata": {},
   "outputs": [],
   "source": [
    "mark = \"hpw\"+ year + month  + day + \".csv\"\n",
    "ruta = '/home/ion/Documentos/albertjimrod/hispaok/csv/'\n",
    "df.to_csv(ruta + mark, index = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "d06ee019-e619-4e7f-b40d-520f8fa122fa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>urgent</th>\n",
       "      <th>buy</th>\n",
       "      <th>change</th>\n",
       "      <th>sell</th>\n",
       "      <th>price</th>\n",
       "      <th>gift</th>\n",
       "      <th>search</th>\n",
       "      <th>repair</th>\n",
       "      <th>parts</th>\n",
       "      <th>synt_brand</th>\n",
       "      <th>description</th>\n",
       "      <th>city</th>\n",
       "      <th>published</th>\n",
       "      <th>expire</th>\n",
       "      <th>date_scrapped</th>\n",
       "      <th>seen</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>korg</td>\n",
       "      <td>korg 05r w módulo</td>\n",
       "      <td>Madrid</td>\n",
       "      <td>2023/07/27</td>\n",
       "      <td>2024/06/14</td>\n",
       "      <td>2023/12/20</td>\n",
       "      <td>456</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>360</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>waldorf</td>\n",
       "      <td>waldorf pulse rack</td>\n",
       "      <td>Barcelona</td>\n",
       "      <td>2023/12/18</td>\n",
       "      <td>2024/06/15</td>\n",
       "      <td>2023/12/20</td>\n",
       "      <td>88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>495</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>roland</td>\n",
       "      <td>sampler s750 roland</td>\n",
       "      <td>Madrid</td>\n",
       "      <td>2023/09/30</td>\n",
       "      <td>2024/06/17</td>\n",
       "      <td>2023/12/20</td>\n",
       "      <td>440</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>roland</td>\n",
       "      <td>roland rd 2000</td>\n",
       "      <td>Madrid</td>\n",
       "      <td>2023/11/19</td>\n",
       "      <td>2024/06/12</td>\n",
       "      <td>2023/12/20</td>\n",
       "      <td>286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>340</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>studiologic</td>\n",
       "      <td>studiologic numa compact 2</td>\n",
       "      <td>Castellón</td>\n",
       "      <td>2023/12/16</td>\n",
       "      <td>2024/06/13</td>\n",
       "      <td>2023/12/20</td>\n",
       "      <td>124</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>650</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>kurzweil</td>\n",
       "      <td>kurzweil pc3 le6</td>\n",
       "      <td>Madrid</td>\n",
       "      <td>2023/11/05</td>\n",
       "      <td>2024/06/17</td>\n",
       "      <td>2023/12/20</td>\n",
       "      <td>252</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>425</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>korg</td>\n",
       "      <td>korg drumlogue</td>\n",
       "      <td>Bizkaia</td>\n",
       "      <td>2023/01/24</td>\n",
       "      <td>2024/06/16</td>\n",
       "      <td>2023/12/20</td>\n",
       "      <td>1530</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>roland</td>\n",
       "      <td>sonidos roland jupiter x xm series y zenology ...</td>\n",
       "      <td>Pontevedra</td>\n",
       "      <td>2022/07/15</td>\n",
       "      <td>2024/06/16</td>\n",
       "      <td>2023/12/20</td>\n",
       "      <td>1744</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>90</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>doepfer</td>\n",
       "      <td>doepfer dual quantizer vintage edition</td>\n",
       "      <td>Madrid</td>\n",
       "      <td>2023/11/29</td>\n",
       "      <td>2024/06/13</td>\n",
       "      <td>2023/12/20</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>75</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>m-audio</td>\n",
       "      <td>controlador m-audio 5 octavas evolution mk461c</td>\n",
       "      <td>Madrid</td>\n",
       "      <td>2023/02/15</td>\n",
       "      <td>2024/06/16</td>\n",
       "      <td>2023/12/20</td>\n",
       "      <td>629</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>ensoniq</td>\n",
       "      <td>necesito programacion eprom para ensoniq esq1</td>\n",
       "      <td>Albacete</td>\n",
       "      <td>2023/12/13</td>\n",
       "      <td>2024/06/10</td>\n",
       "      <td>2023/12/20</td>\n",
       "      <td>140</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>129</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>roland</td>\n",
       "      <td>roland eg 101</td>\n",
       "      <td>Castellón</td>\n",
       "      <td>2023/12/20</td>\n",
       "      <td>2024/06/17</td>\n",
       "      <td>2023/12/20</td>\n",
       "      <td>88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>260</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>fairfield</td>\n",
       "      <td>fairfield circuitry shallow water</td>\n",
       "      <td>Barcelona</td>\n",
       "      <td>2023/09/21</td>\n",
       "      <td>2024/06/15</td>\n",
       "      <td>2023/12/20</td>\n",
       "      <td>322</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>500</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>modal</td>\n",
       "      <td>modal cobalt + extras, en garantía</td>\n",
       "      <td>Sevilla</td>\n",
       "      <td>2023/12/13</td>\n",
       "      <td>2024/06/15</td>\n",
       "      <td>2023/12/20</td>\n",
       "      <td>133</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>280</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>boss</td>\n",
       "      <td>boss re 202 space echo</td>\n",
       "      <td>Sevilla</td>\n",
       "      <td>2023/11/07</td>\n",
       "      <td>2024/06/15</td>\n",
       "      <td>2023/12/20</td>\n",
       "      <td>398</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1300</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>yamaha</td>\n",
       "      <td>yamaha psr-sx-900</td>\n",
       "      <td>Madrid</td>\n",
       "      <td>2023/12/16</td>\n",
       "      <td>2024/06/13</td>\n",
       "      <td>2023/12/20</td>\n",
       "      <td>68</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>360</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>doepfer</td>\n",
       "      <td>doepfer dark energy mki chip curtis cem3394 co...</td>\n",
       "      <td>Barcelona</td>\n",
       "      <td>2023/12/06</td>\n",
       "      <td>2024/06/16</td>\n",
       "      <td>2023/12/20</td>\n",
       "      <td>129</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>470</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>yamaha</td>\n",
       "      <td>sampler yamaha a5000 con extras</td>\n",
       "      <td>Madrid</td>\n",
       "      <td>2023/12/15</td>\n",
       "      <td>2024/06/17</td>\n",
       "      <td>2023/12/20</td>\n",
       "      <td>156</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>295</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>roland</td>\n",
       "      <td>jx-08 roland boutique</td>\n",
       "      <td>Barcelona</td>\n",
       "      <td>2023/07/10</td>\n",
       "      <td>2024/06/14</td>\n",
       "      <td>2023/12/20</td>\n",
       "      <td>814</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>275</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>behringer</td>\n",
       "      <td>behringer pro 800</td>\n",
       "      <td>Barcelona</td>\n",
       "      <td>2023/12/20</td>\n",
       "      <td>2024/06/17</td>\n",
       "      <td>2023/12/20</td>\n",
       "      <td>122</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   urgent buy change sell price gift search repair parts   synt_brand  \\\n",
       "1       0   0      0    1   200    0      0      0     0         korg   \n",
       "2       0   0      0    1   360    0      0      0     0      waldorf   \n",
       "3       0   0      0    1   495    0      0      0     0       roland   \n",
       "4       0   0      1    0     0    0      0      0     0       roland   \n",
       "5       0   0      0    1   340    0      0      0     0  studiologic   \n",
       "6       0   0      0    1   650    0      0      0     0     kurzweil   \n",
       "7       0   0      0    1   425    0      0      0     0         korg   \n",
       "8       0   0      0    1    22    0      0      0     0       roland   \n",
       "9       0   0      0    1    90    0      0      0     0      doepfer   \n",
       "10      0   0      0    1    75    0      0      0     0      m-audio   \n",
       "11      0   1      0    0     0    0      0      0     0      ensoniq   \n",
       "12      0   0      0    1   129    0      0      0     0       roland   \n",
       "13      0   0      0    1   260    0      0      0     0    fairfield   \n",
       "14      0   0      0    1   500    0      0      0     0        modal   \n",
       "15      0   0      0    1   280    0      0      0     0         boss   \n",
       "16      0   0      0    1  1300    0      0      0     0       yamaha   \n",
       "17      0   0      0    1   360    0      0      0     0      doepfer   \n",
       "18      0   0      0    1   470    0      0      0     0       yamaha   \n",
       "19      0   0      0    1   295    0      0      0     0       roland   \n",
       "20      0   0      0    1   275    0      0      0     0    behringer   \n",
       "\n",
       "                                          description        city   published  \\\n",
       "1                                  korg 05r w módulo       Madrid  2023/07/27   \n",
       "2                                 waldorf pulse rack    Barcelona  2023/12/18   \n",
       "3                                sampler s750 roland       Madrid  2023/09/30   \n",
       "4                                     roland rd 2000       Madrid  2023/11/19   \n",
       "5                         studiologic numa compact 2    Castellón  2023/12/16   \n",
       "6                                   kurzweil pc3 le6       Madrid  2023/11/05   \n",
       "7                                     korg drumlogue      Bizkaia  2023/01/24   \n",
       "8   sonidos roland jupiter x xm series y zenology ...  Pontevedra  2022/07/15   \n",
       "9             doepfer dual quantizer vintage edition       Madrid  2023/11/29   \n",
       "10    controlador m-audio 5 octavas evolution mk461c       Madrid  2023/02/15   \n",
       "11     necesito programacion eprom para ensoniq esq1     Albacete  2023/12/13   \n",
       "12                                     roland eg 101    Castellón  2023/12/20   \n",
       "13                 fairfield circuitry shallow water    Barcelona  2023/09/21   \n",
       "14                modal cobalt + extras, en garantía      Sevilla  2023/12/13   \n",
       "15                            boss re 202 space echo      Sevilla  2023/11/07   \n",
       "16                                 yamaha psr-sx-900       Madrid  2023/12/16   \n",
       "17  doepfer dark energy mki chip curtis cem3394 co...   Barcelona  2023/12/06   \n",
       "18                   sampler yamaha a5000 con extras       Madrid  2023/12/15   \n",
       "19                             jx-08 roland boutique    Barcelona  2023/07/10   \n",
       "20                                 behringer pro 800    Barcelona  2023/12/20   \n",
       "\n",
       "        expire date_scrapped  seen  \n",
       "1   2024/06/14    2023/12/20   456  \n",
       "2   2024/06/15    2023/12/20    88  \n",
       "3   2024/06/17    2023/12/20   440  \n",
       "4   2024/06/12    2023/12/20   286  \n",
       "5   2024/06/13    2023/12/20   124  \n",
       "6   2024/06/17    2023/12/20   252  \n",
       "7   2024/06/16    2023/12/20  1530  \n",
       "8   2024/06/16    2023/12/20  1744  \n",
       "9   2024/06/13    2023/12/20    82  \n",
       "10  2024/06/16    2023/12/20   629  \n",
       "11  2024/06/10    2023/12/20   140  \n",
       "12  2024/06/17    2023/12/20    88  \n",
       "13  2024/06/15    2023/12/20   322  \n",
       "14  2024/06/15    2023/12/20   133  \n",
       "15  2024/06/15    2023/12/20   398  \n",
       "16  2024/06/13    2023/12/20    68  \n",
       "17  2024/06/16    2023/12/20   129  \n",
       "18  2024/06/17    2023/12/20   156  \n",
       "19  2024/06/14    2023/12/20   814  \n",
       "20  2024/06/17    2023/12/20   122  "
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(20)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "1b08ca7490639a610c09048f473d1b1514aaa85648d7e388c5f947d92672b22b"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}