Code for A Kernel Independence Test for Geographical Language Variation, Nguyen and Eisenstein, 2017.
Jupyter Notebook Python R
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.

A Kernel Independence Test for Geographical Language Variation.

Code for the following paper:

D. Nguyen and J. Eisenstein. A Kernel Independence Test for Geographical Language Variation. 
To appear in Computational Linguistics.

Getting Started

The code is in Python 2.7 and makes use of several Python packages:

  • descartes
  • fiona
  • numpy
  • pyproj
  • scipy
  • shapely


The following file runs some unit tests



The data can be downloaded from (274 MB)

  • synthetic_experiments: The synthetic datasets. Each directory corresponds to one experiment. Each directory contains a results.txt file with the raw results.
  • shapefiles: Shapefiles of the Netherlands for plotting the synthetic datasets, aggregating data into bins (for Moran's I), etc. You'll still need them if you would like to experiment with the synthetic data.


  • synthetic_data.ipynb: This notebook plots several selected synthetic datasets and shows how to apply the methods to the different types of data (binary, categorical and frequency data).
  • HSIC.ipynb: This notebook illustrates HSIC with several synthetic (non-geographical) datasets.
  • plots.r: Shows how to generate the plots in the paper based on the result files in the synthetic_experiments directory.

Command line tool


Frequency data (should return 0.00653):

python -d freq -l sample_data/locs1.txt -f sample_data/data1.txt

Binary data (should return 0.00241):

python -d bin -l sample_data/locs2.txt -f sample_data/data2.txt

Categorical data (should return 0.00111):

python -d cat -l sample_data/locs3.txt -f sample_data/data3.txt