Scrap real time posts from twitter through the streaming api
Switch branches/tags
Nothing to show
Clone or download
Pull request Compare This branch is even with Marsan-Ma-zz:master.
Latest commit a66ce03 Sep 30, 2016
Permalink
Failed to load latest commit information.
.gitignore [New] Initial commit. Sep 16, 2016
README.md [Fix] Readme. Sep 16, 2016
config.yml.default [New] Initial commit. Sep 16, 2016
dump.py [Fix] args bugs. Sep 30, 2016
twitter.py [Fix] args bugs. Sep 30, 2016

README.md

Twitter Scraper

Scraping twitter content from twitter streaming API, in python3.

How it works

Setup

  1. Copy the config.yml.default to config.yml, and fill your twitter application tokens you got from twitter developers.

  2. just type python3 twitter.py, the listener will start to dump corpus in: corpus/<YYYYMMDD_HHMMSS.txt>

Output

I build this corpus for training the neural network model ChatBot, thus the corpus is arranged as consecutive dialog where even sentences are the response of odd sentences, like:

Line number Sentences
1 happy birthday!
2 thank you :)
3 game of throne is the best drama i've seen.
4 I'll say the walking dead is even better.

Where (1,2), (3,4) are two independent dialog pairs.

Content Filters

Twitter streaming API supports some filter, and the most useful ones are:

Location filter

simply place the logitude and altitude in [South,West,North,East] format, for example:

Place filter
San Francisco [-122.75,36.8,-121.75,37.8])
New York City [-74,40,-73,41]
San Francisco or New York City [-122.75,36.8,-121.75,37.8, -74,40,-73,41]

Language filter

there are threads questioning about language filter is not work, actually is does, it just need to accompanied with track filter, like:

This won't work.
stream.filter(languages=["en"])

This works. stream.filter(languages=["en"], track=['machine', 'learning'])

If you want language other than English

Twitter can't tokenize some languages like Japanese or Chinese correctly, your track parameter won't work for these languages. For example, you might setup:

stream.filter(languages=["ja"], track=['バイト'])

While you expect you will get a lot of tweets about バイト, but you simply can't get it, because twitter can't tokenize Japenses and Chinese correctly.

If you just want some corpus, regardless of the topics, here are the work around: use some generic English keywords instead, like:

stream.filter(languages=["zh"], track=['I', 'you', 'http', 'www', '@', '。', ',', '!', '』', ')', '...', '-', '/'])

According to document, you could add up to 400 keywords on this list, even some emoji also works.