Skip to content
Code for Blodgett et al. 2016, Demographic dialectal variation in social media
Python
Branch: master
Clone or download
Latest commit db4d6d1 Nov 9, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
code readme Oct 27, 2019
model change dir name Oct 27, 2019
LICENSE Update LICENSE Nov 6, 2019
README.md usage Nov 9, 2019

README.md

This repository contains the learned demographic model and script to predict demographic proportions using the model from the paper "Demographic Dialectal Variation in Social Media: A Case Study of African-American English" by Su Lin Blodgett, Lisa Green, and Brendan O'Connor, EMNLP 2016.

The demographic model's full vocabulary and count tables (averaged over the last 50 Gibbs sampling iterations) are in model/vocab.txt and model/model_count_table.txt, respectively. Demographic predictions for a tweet can be calculated by loading the model with

predict.load_model(modelfile)

and calling

predict.predict(tweet)

for a given tweet. Output proportions are for the African-American, Hispanic, Asian, and White topics, respectively. The model only needs to be loaded once per session. Note that these are not confidence values about the probability of the author's demographics. Rather, these represent an inference of the proportion of words in the text that come from a demographically-associated language/dialect. As we state in the paper, we only consider the AA and White categories to be reliable and useful, and conducted linguistic validation only for those two categories.

Tokenizing for the paper was done using code/twokenize.py and emoji detection done with code/emoji.py.

Python 2.7 is assumed.

Example usage:

$ cd code
$ python2.7

>>> import predict

>>> predict.load_model()

>>> predict.predict(u"hello there".split())
array([0.16603718, 0.27450171, 0.22886004, 0.33060107])

>>> predict.predict(u"af af af".split())
array([8.42944382e-01, 1.32149345e-01, 1.11905398e-04, 2.47943681e-02])

If you use this demographic model or code, please cite the paper:

@inproceedings{blodgett2016demographic,
author = {Blodgett, Su Lin and Green, Lisa and O'Connor, Brendan}, 
title = {{Demographic Dialectal Variation in Social Media: A Case Study of African-American English}},
booktitle = {Proceedings of EMNLP},
year = 2016}

More information is available at: http://slanglab.cs.umass.edu/TwitterAAE/

Please send questions to: blodgett@cs.umass.edu and brenocon@cs.umass.edu

You can’t perform that action at this time.