Get_Pages_Doc.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6bfa1473",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3e533ae",
   "metadata": {},
   "source": [
    "This notebook is part of the BAINSA Wikipedia Knowledge Graph Project. Using the Wiki Dumps, we aim to construct an unweighted directed graph containg the Wikipedia articles as nodes and page links as edges. \n",
    "In this particular notebook, we use the file \"enwiki-20221020-page.sql\". To parse it, we exploit the structure of the SQL language as shown in the example below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f0ad76f",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"enwiki-20221020-page.sql\") as f:\n",
    "    line = \"a\"\n",
    "    while \"INSERT INTO\" not in line:\n",
    "        line = f.readline()\n",
    "    print(line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "358f1578",
   "metadata": {},
   "outputs": [],
   "source": [
    "line = line[26:]\n",
    "print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3919f8b",
   "metadata": {},
   "source": [
    "## Line processing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d9bf569",
   "metadata": {},
   "source": [
    "We define a function to process individual lines of the SQL file. The schema of the table created by the SQL statement contained in the file can be found at https://www.mediawiki.org/wiki/Manual:Page_table. The fields that are of interest to us are page_id (0), page_namespace (1), page_title (2), page_is_redirect (3). To obtain a file containing only proper articles, we eliminate the pages for which the flag page_is redirect is 1, as well as all pages not contained in the article namespace (i.e., namespace != 0). We store the page_id and the page_title in a dictionary defined outside the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "e1e2717d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_line(line):\n",
    "    n = len(line)\n",
    "    i = 0\n",
    "    while i < n:\n",
    "        if line[i] == \"(\":\n",
    "            start = i + 1\n",
    "        else:\n",
    "            raise Exception(f\"Something went wrong {first_line[i-10:i+10]}\")\n",
    "        while not(line[i] == \")\" and line[i + 1] == \",\" and line[i + 2] == \"(\"):\n",
    "            i += 1\n",
    "            if i == n - 2:\n",
    "                i += 2\n",
    "                break\n",
    "        end = i\n",
    "        i += 2\n",
    "        block = line[start:end]\n",
    "        block = block.split(\",\")\n",
    "        if block[1] != '0': \n",
    "            count[0] += 1\n",
    "            continue\n",
    "        if block[3] != '0':\n",
    "            count[1] += 1\n",
    "            continue\n",
    "        page_id = int(block[0])\n",
    "        title = block[2].replace(\" \", \"_\")\n",
    "        if page_id in g:\n",
    "            g[page_id].append(title)\n",
    "        else:\n",
    "            g[page_id] = [title]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b5d1ade",
   "metadata": {},
   "source": [
    "## Parsing the file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b935763f",
   "metadata": {},
   "source": [
    "Using the previously defined function, we want to parse the entire file line by line, store the relevant information in a dictionary and write it to the output text file. To avoid memory overflow, we will write evey 100 titles in the file and than reinitialize the data structure with an empty dictionary. Furthermore, we eliminate pages that are disambiguations, as these are not proper articles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "deea5b7c",
   "metadata": {},
   "outputs": [],
   "source": [
    "g = dict()\n",
    "i = 0\n",
    "output = open(\"articles.txt\", \"w\")\n",
    "with open(\"/Users/mateicosa/Downloads/enwiki-20221020-page.sql\", \"r\") as f:\n",
    "    line = \"\"\n",
    "    while \"INSERT INTO\" not in line:\n",
    "        line = f.readline()\n",
    "    line = line[26:]\n",
    "    process_line(line)\n",
    "    line = \"INSERT INTO\"\n",
    "    while line:\n",
    "        i += 1\n",
    "        j += 1\n",
    "        line = f.readline()\n",
    "        if \"INSERT INTO\" not in line:\n",
    "            break\n",
    "        line = line[26:]\n",
    "        process_line_complete(line)\n",
    "        if i == 100:\n",
    "            i = 0\n",
    "            for page_id in g:\n",
    "                if \"(disambiguation)\" not in g[page_id][0]:\n",
    "                    output.write(str(page_id) + \" \" + ' '.join(g[page_id]) + \"\\n\")\n",
    "            g = dict()\n",
    "output.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6ac541d",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "340be2f6",
   "metadata": {},
   "source": [
    "We are left with a file that contains rows having the following structure: page_id page_title. We will later use this file together with another one containg the links in order to produce the graph."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}