Home

Table of Contents Introduction Usage Wikipeda dumps Related Work

Introduction

The Wikipedia extractor tool extracts plain text from a Wikipedia database dump, discarding any other information or annotation present in Wikipedia pages, such as images, tables, references and lists.

Each document in the dump of the encyclopedia is representend as a single XML element, encoded as illustrated in the following example from the document titled Armonium:

  L''''armonium'''' (in francese, ''harmonium'') è uno [[strumenti musicali|
  strumento musicale]] azionato con una [[tastiera (musica)|tastiera]], detta
  manuale. Sono stati costruiti anche alcuni armonium con due manuali.
  
  ==Armonium occidentale==
  Come l[[organo (musica)|organo]], l'armonium è utilizzato tipicamente in
  [[chiesa (architettura)|chiesa]], per l'esecuzione di [[musica sacra]], ed è
  fornito di pochi registri, quando addirittura in certi casi non ne possiede
  nemmeno uno: il suo [[timbro (musica)|timbro]] è molto meno ricco di quello
  organistico e così pure la sua estensione.
  
  ...
  
  ==Armonium indiano==
  {{S sezione}}
  
  == Voci correlate ==
  *[[Musica]]
  *[[Generi musicali]]</text>

For this document the Wikipedia extractor produces the following plain text:

 Armonium occidentale.
 Come l'organo, l'armonium è utilizzato tipicamente in chiesa, per l'esecuzione
 di musica sacra, ed è fornito di pochi registri, quando addirittura in certi
 casi non ne possiede nemmeno uno: il suo timbro è molto meno ricco di quello
 organistico e così pure la sua estensione.
 ...
 </doc>

The extraction tool is written in Python and requires no additional library.

Wikipedia articles are written in the MediaWiki Markup Language which provides a simple notation for formatting text (bolds, italics, underlines, images, tables, etc.). It is also posible to insert HTML markup in the documents. Wiki and HTML tags are often misused (unclosed tags, wrong attributes, etc.), therefore the extractor deploys several heuristics in order to circumvent such problems. A currently missing feature for the extractor is template expansion.

Usage

WikiExtractor.py is a Python script that extracts and cleans text from a Wikipedia database dump. The output is stored in a number of files of similar size in a given directory. Each file will contains several documents in this document format.

This is a beta version that performs template expansion by preprocesssng the whole dump and extracting template definitions.

Usage:

 WikiExtractor.py [options] xml-dump-file

optional arguments:

  -h, --help            show this help message and exit
  -o OUTPUT, --output OUTPUT
                        output directory
  -b n[KM], --bytes n[KM]
                        put specified bytes per output file (default is 1M)
  -B BASE, --base BASE  base URL for the Wikipedia pages
  -c, --compress        compress output files using bzip
  -l, --links           preserve links
  -ns ns1,ns2, --namespaces ns1,ns2
                        accepted namespaces
  -q, --quiet           suppress reporting progress info
  -s, --sections        preserve sections
  -a, --article         analyze a file containing a single article
  --templates TEMPLATES
                        use or create file containing templates
  -v, --version         print program version

Wikipeda dumps

• All Wikipedia database dumps
• torrents for use with a BitTorrent client such as uTorrent

Related Work

• WikiPrep A Perl tool for preprocessing Wikipedia XML dumps.
• Extracting Text from Wikipedia Another Python tool for text extracting from Wikipedia XML dumps.
• Alternative Parsers A list of links, descriptions, and status reports of the various alternative MediaWiki parsers.