Package for Statistically significant linguistic change
Python Shell Makefile
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.

README.rst

langchangetrack

https://badge.fury.io/py/langchangetrack.png https://travis-ci.org/viveksck/langchangetrack.png?branch=master https://pypip.in/d/langchangetrack/badge.png

https://github.com/viveksck/langchangetrack/blob/master/langchangetrack/images/gay_invisible.png

Package for Statistically Significant Language Change.

Features

  • This package provides tools to detect linguistic change in temporal corpora.

  • The meta algorithm works in 2 main steps

    1. Time series construction:Given a word, we construct a time series that tracks the displacement of a word through time. We track the displacement of a word using either Frequency, Part of Speech Distribution or Co-occurrences.
    2. Change point detection: We then use change point detection methods to detect if the time series contains a change point and if so what the change point is.

The details of the above steps are outlined in : http://arxiv.org/abs/1411.3315

Visualization Demo

Please see this for a cool visualization of words moving through time: http://tinyurl.com/wordvis

Usage

Input

We assume a temporal corpus of text files (appropriately tokenized) to be present in a directory. In addition we assume list of words in a single text file that one is interested in tracking. This could just be the set of words in the common vocabulary of the temporal corpus.

Output

The output consists of the pvalues for each word indicating the significance of the changepoint detected.

Sample Usage

$ngrams_pipeline.py --corpus-dir data/temporal_corpus/ --file-extension "ngrams" --working-dir ./working --output-dir ./output --context-size 5 --epochs 3 --start-time-point 1900 --end-time-point 2000 --step-size 5 --vocabulary-file data/temporal_corpus/common_vocab.txt --workers 16

$pos_pipeline.py --corpus-dir data/temporal_corpus/ --file-extension "ngrams" --working-dir ./working --output-dir ./output --start-time-point 1900 --end-time-point 1930 --step-size 5 --vocabulary-file data/temporal_corpus/common_vocab.txt --workers 16

$freq_pipeline.py --corpus-dir data/temporal_corpus/ --file-extension "ngrams" --working-dir ./working --output-dir ./output --start-time-point 1900 --end-time-point 2000 --step-size 5 --vocabulary-file data/temporal_corpus/common_vocab.txt --workers 16

You might need to tune the hyper parameters as per your specific need.

Detailed Usage

Usage: ngrams_pipeline.py

optional arguments:
-h, --help show this help message and exit
--corpus-dir CORPUS_DIR
 Corpus directory
--file-extension EXT
 Corpus file extension
--working-dir WORKING_DIR
 Working directory
--output-dir OUTPUT_DIR
 Output directory
--context-size WINDOW
 Context size to use for training embeddings
--epochs EPOCHS
 Number of epochs to training embeddings
--start-time-point START
 Start time point
--end-time-point END
 End time point
--step-size STEP
 Step size for timepoints
--model-family MODEL_FAMILY
 Model family default (locallinear)
--number-nearest-neighbors KNN
 

Number of nearest neighbors to use for mapping to joint space (default:1000)

--vocabulary-file VOCAB_FILE

Common vocabulary file

--threshold THRESHOLD
 Threshold for mean shift model for change point detection (default: 1.75)
--bootstrap-samples BOOTSTRAP
 Number of bootstrap samples to draw (default: 1000)
--workers WORKERS
 Maximum number of workers (default: 1)
-l LOG, --log LOG
 log verbosity level

Usage: pos_pipeline.py

optional arguments:
-h, --help show this help message and exit
--corpus-dir CORPUS_DIR
 Corpus directory
--file-extension EXT
 Corpus file extension
--working-dir WORKING_DIR
 Working directory
--output-dir OUTPUT_DIR
 Output directory
--start-time-point START
 Start time point
--end-time-point END
 End time point
--step-size STEP
 Step size for timepoints
--vocabulary-file VOCAB_FILE
 Common vocabulary file
--threshold THRESHOLD
 Threshold for mean shift model for change point detection
--bootstrap-samples BOOTSTRAP
 Number of bootstrap samples to draw
--workers WORKERS
 Maximum number of workers
-l LOG, --log LOG
 log verbosity level

usage: freq_pipeline.py

optional arguments:
-h, --help show this help message and exit
--corpus-dir CORPUS_DIR
 Corpus directory
--file-extension EXT
 Corpus file extension
--working-dir WORKING_DIR
 Working directory
--output-dir OUTPUT_DIR
 Output directory
--start-time-point START
 Start time point
--end-time-point END
 End time point
--step-size STEP
 Step size for timepoints
--vocabulary-file VOCAB_FILE
 Common vocabulary file
--threshold THRESHOLD
 Threshold for mean shift model for change point detection
--bootstrap-samples BOOTSTRAP
 Number of bootstrap samples to draw
--workers WORKERS
 Maximum number of workers
-l LOG, --log LOG
 log verbosity level

Requirements

  • wheel==0.23.0
  • argparse>=1.2.1
  • numpy>=0.9.1
  • scipy>=0.15.1
  • more_itertools>=2.2
  • joblib>=0.8.3-r1
  • gensim==0.10.3
  • statsmodels>=0.5.0
  • changepoint>=0.1.0
  • nltk>=3.0.0
  • textblob>=0.9.0
  • textblob-aptagger>=0.2.0
  • psutil>=2.2.0
  • GNU Parallel
  • R (good to have)
  • rpy2 (good to have)

Installation

  1. Install GNU Parallel from here: www.gnu.org/software/software.html
  2. cd langchangetrack
  3. pip install -r requirements.txt
  4. python setup.py install