Wikiassoc is a tool for generating term associations by analyzing the link structure of Wikipedia (or any other wiki based on the MediaWiki software). You put in a Wikipedia database dump, and get out a table of terms (article titles) and the most strongly related terms.
To build and install Wikiassoc, you need a fairly modern C++ compiler (tested with GCC 4.3 and Open64 4.2.2.2), the Boost libraries (specifically Boost.IOStreams and Boost.Regex), zlib, bzlib and the GNU autools (autoconf and automake).
It is highly advisable to use
- a C++ compiler that supports OpenMP
- google-sparsehash
as Wikiassoc will be very slow or consume huge amounts of memory without these.
Wikiassoc uses the GNU build tools. Enter
./prepare
./configure && make && make install
or see the file INSTALL for more detailed instructions.
To compile an associative thesaurus with Wikiassoc, first download some files from the Wikimedia dump repository. For example, say you want word associations in Latin. Fetch the files
lawiki-YYYYMMDD-page.sql.gz
lawiki-YYYYMMDD-pagelinks.sql.gz
and run the Wikiassoc program as
wikiassoc lawiki-YYYYMMDD-page.sql.gz lawiki-YYYYMMDD-pagelinks.sql.gz \
| gzip -c > lawiki-associations.gz
(Using gzip is highly recommended, as Wikiassoc produces a lot of output.)
You will get a log of what's happening on stderr. Note that Wikiassoc takes a lot of memory; on the larger Wikipedias, it may be as much as 12GB or more.
In lawiki-associations.gz
, you will find a text file with terms and indented
associations for the term:
Astronomia
Scientia
Physica
Universum
Galaxias
Planeta
Geologia
Mathematica
Stella
Luna
Terra
See the manpage for details (man wikiassoc
).
For each article in the Wikipedia database dump, Wikiassoc looks at all the articles that can be reached by following at most two links. It then weighs all these articles by a scheme called pf-ibf, or path frequency-inverse backlink frequency. For further explanation, refer to:
Nakayama, K., Hara, T. and Nishio, S. (2007) Wikipedia Mining for an Association Web Thesaurus Construction. In Proc. International Conference on Web Information Systems Engineering (WISE), pp. 322-334.