scraping_experiments.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just running some experiments in a notebook for convenience"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re, requests\n",
    "\n",
    "import bs4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MA_BASE_URL = 'https://www.metal-archives.com'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to get a list of all bands?\n",
    "\n",
    "Based on https://github.com/jonchar/ma-scraper,\n",
    "it looks like MA uses AJAX internally.\n",
    "\n",
    "So browse by band alphabetically and open dev tools.  The Network tab shows the following request:\n",
    "https://www.metal-archives.com/browse/ajax-letter/l/A/json/1?sEcho=1&iColumns=4&sColumns=&iDisplayStart=0&iDisplayLength=500&mDataProp_0=0&mDataProp_1=1&mDataProp_2=2&mDataProp_3=3&iSortCol_0=0&sSortDir_0=asc&iSortingCols=1&bSortable_0=true&bSortable_1=true&bSortable_2=true&bSortable_3=false&_=1551813490451\n",
    "\n",
    "This returns a JSON object with all the info used to populate the table.\n",
    "It looks like the we can use `iDisplayStart`, `iDisplayLength` to get all the data.\n",
    "\n",
    "Looks like `iDisplayLength` must be 500 and `iDisplayStart` must be a multiple of 500.\n",
    "Otherwise metal-archives will just return the first page of data.\n",
    "As Jon Charest noticed, we must also specify `sEcho` to get a valid JSON response.\n",
    "Looks like the value can be just `''`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ma_band_list_url(letter):\n",
    "    \"\"\"\n",
    "    Construct the GET request URL for the alphabetical band list.\n",
    "    Looks like letter can be A-Z, NBR, or ~.\n",
    "    \"\"\"\n",
    "    url = MA_BASE_URL + '/browse/ajax-letter/l/' + str(letter) + '/json/1'\n",
    "    return url\n",
    "\n",
    "headers = {'user-agent': 'bot'} # give a user agent so we don't 403\n",
    "params = {'sEcho': '',\n",
    "          'iDisplayStart': 500,\n",
    "          'iDisplayLength': 500,\n",
    "         }\n",
    "print(ma_band_list_url('A'))\n",
    "response = requests.get(ma_band_list_url('A'), params=params, headers=headers)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response.json()['aaData'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response.headers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Okay, now let's munge the response data a little bit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "band_info = [\"<a href='https://www.metal-archives.com/bands/Abducted/3540381624'>Abducted</a>\",\n",
    " 'Spain',\n",
    " 'Thrash Metal',\n",
    " '<span class=\"split_up\">Split-up</span>']\n",
    "band_info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = bs4.BeautifulSoup(band_info[0], 'html5lib')\n",
    "soup2 = bs4.BeautifulSoup(band_info[3], 'html5lib')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "name = soup.a.text\n",
    "url = soup.a.get('href')\n",
    "country = band_info[1]\n",
    "genre = band_info[2]\n",
    "status = soup2.span.text\n",
    "print(name, url, country, genre, status)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Similar munging for the review JSON"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "review_info = ['January 31',\n",
    "               '<a href=\"https://www.metal-archives.com/reviews/Toxik_Attack/Assassinos_em_S%C3%A9rie/746017/Cosmic_Mystery/407515\" title=\"ole skool thrash metal!\" class=\"iconContainer ui-state-default ui-corner-all\"><span class=\"ui-icon ui-icon-search\">Read</span></a>',\n",
    "               '<a href=\"https://www.metal-archives.com/bands/Toxik_Attack/3540389184\">Toxik Attack</a>',\n",
    "               '<a href=\"https://www.metal-archives.com/albums/Toxik_Attack/Assassinos_em_S%C3%A9rie/746017\">Assassinos em Série</a>',\n",
    "               '67%',\n",
    "               '<a href=\"https://www.metal-archives.com/users/Cosmic%20Mystery\" class=\"profileMenu\">Cosmic Mystery</a>',\n",
    "               '23:18']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "year = 2019\n",
    "month = 1\n",
    "day = int(review_info[0].split()[1])\n",
    "hour, minute = map(int, review_info[6].split(':'))\n",
    "\n",
    "score = int(review_info[4][:-1])\n",
    "\n",
    "soup = bs4.BeautifulSoup(review_info[1], 'html5lib')\n",
    "soup2 = bs4.BeautifulSoup(review_info[2], 'html5lib')\n",
    "soup3 = bs4.BeautifulSoup(review_info[3], 'html5lib')\n",
    "soup5 = bs4.BeautifulSoup(review_info[5], 'html5lib')\n",
    "\n",
    "review_url = soup.a.get('href')\n",
    "review = soup.a.get('title')\n",
    "\n",
    "band_url = soup2.a.get('href')\n",
    "band = soup2.a.text\n",
    "\n",
    "album_url = soup3.a.get('href')\n",
    "album = soup3.a.text\n",
    "\n",
    "reviewer_url = soup5.a.get('href')\n",
    "reviewer = soup5.a.text\n",
    "\n",
    "print(year, month, day, hour, minute, band, band_url, album, album_url, review, review_url, score, reviewer, reviewer_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_ID_from_band_URL(band_url):\n",
    "    return int(band_url.split('/')[-1])\n",
    "get_ID_from_band_URL(band_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now let's scrape a band page & relevent AJAX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "band_url = 'https://www.metal-archives.com/bands/Panopticon/126117'\n",
    "band_url = 'https://www.metal-archives.com/bands/Cattle_Decapitation/2840'\n",
    "headers = {'user-agent': 'bot'} # give a user agent so we don't 403\n",
    "response = requests.get(band_url, headers=headers)\n",
    "print(response.status_code)\n",
    "#print(response.text)\n",
    "soup = bs4.BeautifulSoup(response.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the band name (to eventually check that it matches what we requested)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "band_name_list = soup.find_all('h1', 'band_name')\n",
    "assert len(band_name_list) == 1, 'should only have one band name header element'\n",
    "band_name_tag = band_name_list[0]\n",
    "print(band_name_tag)\n",
    "print(band_name_tag.a.get('href'))\n",
    "print(band_name_tag.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get lyrical themes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#print(soup.body)\n",
    "stats_div = soup.find('div', {'id': 'band_stats'})\n",
    "right_stuff = stats_div.find('dl', 'float_right')\n",
    "dds = right_stuff.find_all('dd')\n",
    "themes = dds[1].text\n",
    "print(themes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get current label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_tag = dds[2].a\n",
    "label = label_tag.text\n",
    "label_url = label_tag.get('href')\n",
    "print(label, label_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the added/modified dates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "audit_div = soup.find('div', {'id': 'auditTrail'})\n",
    "td_list = audit_div.find_all('td')\n",
    "added_on_td = td_list[2]\n",
    "modified_on_td = td_list[3]\n",
    "date_matcher = re.compile(r'\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}')\n",
    "added_on_matcher = re.compile(r'Added on: (\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2})')\n",
    "\n",
    "added_on_match = re.match(added_on_matcher, added_on_td.text)\n",
    "print(added_on_match)\n",
    "added_on = added_on_match.group(1)\n",
    "print(added_on)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get band members"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "band_members_tag = soup.find('div', {'id': 'band_members'})\n",
    "tab_links = band_members_tag.ul.find_all('li')\n",
    "tab_links[0].a."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "complete_members_tag = soup.find('div', {'id': 'band_tab_members_all'}) # complete lineup\n",
    "current_members_tag = soup.find('div', {'id': 'band_tab_members_current'}) # current lineup\n",
    "#past_members_tag = soup.find('div', {'id': 'band_tab_members_past'}) # past lineup\n",
    "#live_members_tag = soup.find('div', {'id': 'band_tab_members_live'}) # live lineup\n",
    "#print(complete_members_tag)\n",
    "\n",
    "# Complete members table has everybody (duh) and is delimited by Current, Past, Current (Live), Past (Live)\n",
    "# We should be able to get everybody by iterating over the complete members table.\n",
    "#print(complete_members_tag.table.tbody)\n",
    "for row in complete_members_tag.table.tbody.find_all('tr'):\n",
    "    c = row.get('class')[0]\n",
    "    \n",
    "    if c == 'lineupHeaders':\n",
    "        text = str(row.td.text).strip().rstrip()\n",
    "        text = re.sub(r'\\s+', ' ', text)\n",
    "        print(text)\n",
    "    elif c == 'lineupRow':\n",
    "        artist_tag = row.td\n",
    "        print(artist_tag.find_all('a'))\n",
    "        \n",
    "        pass\n",
    "    elif c == 'lineupBandsRow':\n",
    "        pass\n",
    "    else:\n",
    "        raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the band comment/\"read-more\", which requires a separate request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "band_id = get_ID_from_band_URL(band_url)\n",
    "read_more_url = 'https://www.metal-archives.com/band/read-more/id/' + str(band_id)\n",
    "response = requests.get(read_more_url, headers=headers)\n",
    "print(response.status_code)\n",
    "soup = bs4.BeautifulSoup(response.text, 'html5lib')\n",
    "print(str(soup.body)) # with body tag\n",
    "print(''.join(map(str, soup.body.children))) # without body tag\n",
    "#Probably want to keep the whole body HTML for future stuff.  For example, \n",
    "#this has A.Lunn's artist page linked, so we could search for artist, band, etc. pages.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get all similar artists by users' votes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "band_id = get_ID_from_band_URL(band_url)\n",
    "#similar_bands_url = 'https://www.metal-archives.com/band/ajax-recommendations/id/' + str(band_id) + '?showMoreSimilar=1'\n",
    "#response = requests.get(similar_bands_url, headers=headers)\n",
    "\n",
    "similar_bands_url = 'https://www.metal-archives.com/band/ajax-recommendations/id/' + str(band_id)\n",
    "params = {'showMoreSimilar': '1'}\n",
    "response = requests.get(similar_bands_url, headers=headers, params=params)\n",
    "\n",
    "print(response.status_code)\n",
    "print(response.url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = bs4.BeautifulSoup(response.text)\n",
    "table = soup.find('table', {'id': 'artist_list'})\n",
    "#print(table)\n",
    "for row in table.tbody.find_all('tr'):\n",
    "    cells = row.find_all('td')\n",
    "    if len(cells) < 4: break\n",
    "    artist_tag = cells[0]\n",
    "    score = cells[3].text\n",
    "    print(artist_tag.text, score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get discography"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "discog_url = f'https://www.metal-archives.com/band/discography/id/{band_id}/tab/all'\n",
    "response = requests.get(discog_url, headers=headers)\n",
    "if response.status_code != 200:\n",
    "    raise RuntimeError('Got response status {}, bailing.'.format(response.status_code))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = bs4.BeautifulSoup(response.text, 'html5lib')\n",
    "table = soup.find('table', {'class': 'display discog'})\n",
    "for row in table.tbody.find_all('tr'):\n",
    "    cells = row.find_all('td')\n",
    "    album_tag = cells[0].a\n",
    "    album = album_tag.text\n",
    "    album_url = album_tag.get('href')\n",
    "    #album_id = ...\n",
    "    \n",
    "    type_tag = cells[1]\n",
    "    type_str = type_tag.text\n",
    "    \n",
    "    year_tag = cells[2]\n",
    "    year = int(year_tag.text)\n",
    "    \n",
    "    print(album, album_url, type_str, year)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}