Wikipedia graph mining: dynamic structure of collective memory
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
ipython
project
src/main/scala
.gitignore
LICENSE
README.md
build.sbt

README.md

WikiBrain

Implementation of the graph learning algorithm presented in Wikipedia graph mining: dynamic structure of collective memory. The learning algorithm is inspired by the Hebbian learning theory.

We also reported the results with interactive graph visualizations in an accompanying blog post.

Dataset

To reproduce the experiments, download the dataset from DOI.

Clone this project and extract the downloaded .zip files to /src/main/resources/ folder.

Change PATH_RESOURCES in Globals.scala to the path to this project on your computer.

Runing the experiments

Open WikiBrainHebbStatic.scala and run the code (Shift+F10 in Intellij Idea).

You may have to change your Spark configuration according to RAM availability on your computer.

val spark = SparkSession.builder
  .master("local[*]") // use all available cores
  .appName("Wiki Brain")
  .config("spark.driver.maxResultSize", "20g") // change this if needed
  .config("spark.executor.memory", "50g") // change this if needed
  .getOrCreate()

Resulting graphs

You will find the resulting graph graph.gexf in PATH_RESOURCES. This file can be opened in Gephi.