Dataset of threads and comments from reddit
Switch branches/tags
Nothing to show
Clone or download
Permalink
Failed to load latest commit information.
.gitignore originals Apr 11, 2016
README.md fix link Feb 22, 2018
entertainment_anime.csv cleaned up Jun 17, 2017
entertainment_comicbooks.csv cleaned up Jun 17, 2017
entertainment_harrypotter.csv cleaned up Jun 17, 2017
entertainment_movies.csv cleaned up Jun 17, 2017
entertainment_music.csv cleaned up Jun 17, 2017
entertainment_starwars.csv cleaned up Jun 17, 2017
gaming_dota2.csv cleaned up Jun 17, 2017
gaming_gaming.csv cleaned up Jun 17, 2017
gaming_leagueoflegends.csv cleaned up Jun 17, 2017
gaming_minecraft.csv cleaned up Jun 17, 2017
gaming_pokemon.csv cleaned up Jun 17, 2017
gaming_skyrim.csv cleaned up Jun 17, 2017
gaming_starcraft.csv cleaned up Jun 17, 2017
gaming_tf2.csv cleaned up Jun 17, 2017
headers.txt originals Apr 11, 2016
humor_adviceanimals.csv cleaned up Jun 17, 2017
humor_circlejerk.csv cleaned up Jun 17, 2017
humor_facepalm.csv cleaned up Jun 17, 2017
humor_funny.csv cleaned up Jun 17, 2017
humor_imgoingtohellforthis.csv cleaned up Jun 17, 2017
humor_jokes.csv cleaned up Jun 17, 2017
learning_askhistorians.csv cleaned up Jun 17, 2017
learning_askscience.csv cleaned up Jun 17, 2017
learning_explainlikeimfive.csv cleaned up Jun 17, 2017
learning_science.csv cleaned up Jun 17, 2017
learning_space.csv cleaned up Jun 17, 2017
learning_todayilearned.csv cleaned up Jun 17, 2017
learning_youshouldknow.csv cleaned up Jun 17, 2017
lifestyle_drunk.csv cleaned up Jun 17, 2017
lifestyle_food.csv cleaned up Jun 17, 2017
lifestyle_frugal.csv cleaned up Jun 17, 2017
lifestyle_guns.csv cleaned up Jun 17, 2017
lifestyle_lifehacks.csv cleaned up Jun 17, 2017
lifestyle_motorcycles.csv cleaned up Jun 17, 2017
lifestyle_progresspics.csv cleaned up Jun 17, 2017
lifestyle_sex.csv cleaned up Jun 17, 2017
news_conservative.csv cleaned up Jun 17, 2017
news_conspiracy.csv cleaned up Jun 17, 2017
news_libertarian.csv cleaned up Jun 17, 2017
news_news.csv cleaned up Jun 17, 2017
news_offbeat.csv cleaned up Jun 17, 2017
news_politics.csv cleaned up Jun 17, 2017
news_truereddit.csv cleaned up Jun 17, 2017
news_worldnews.csv cleaned up Jun 17, 2017
television_breakingbad.csv cleaned up Jun 17, 2017
television_community.csv cleaned up Jun 17, 2017
television_doctorwho.csv cleaned up Jun 17, 2017
television_gameofthrones.csv cleaned up Jun 17, 2017
television_himym.csv cleaned up Jun 17, 2017
television_mylittlepony.csv cleaned up Jun 17, 2017
television_startrek.csv cleaned up Jun 17, 2017
television_thewalkingdead.csv cleaned up Jun 17, 2017

README.md

Reddit Comment and Thread Datas

Around 260,000 threads / comments scraped from Reddit. Useful dataset for NLP projects.

Quick Start

Scraped using omega-red

The .csvs are named <metareddit>_<subreddit>.csv. The headers are described here and in headers.txt.

Headers are:

text,id,subreddit,meta,time,author,ups,downs,authorlinkkarma,authorkarma,authorisgold
  • text: Text of the comment / thread
  • id: Unique reddit id for the comment / thread
  • subreddit: Subreddit that the comment / thread belongs to
  • meta: Metareddit that the comment / thread belongs to. Subreddits belong to metareddits. A subreddit can be leagueoflegends. The metareddit for that subreddit would be gaming, which can also include the subreddit dota2
  • time: UNIX timestamp of the comment / thread
  • author: Username of the author of the comment / thread
  • ups: Number of upvotes the comment / thread received
  • downs: Number of downvotes the comment / thread received
  • authorlinkkarma: The author's link karma. What is Link Karma?
  • authorkarma: The author's karma. Reddit FAQ explaining karma.
  • authorisgold: Boolean indicator for the gold status of the user. 1 for gold users, 0 for non-gold (normal) users. Reddit FAQ explaining gold status.

Original Texts

If you prefer the texts with punctuation, they are included (as the original output files from omega-red) at https://mega.nz/#F!NtsCGTgD!urXdXLJ6yITYdWEdWN-H1w

Threads

are in /threads.csv at https://mega.nz/#F!NtsCGTgD!urXdXLJ6yITYdWEdWN-H1w

Headers are:

'text', 'title', 'url', 'id', 'subreddit', 'meta', 'time', 'author', 'ups', 'downs', 'authorlinkkarma', 'authorcommentkarma', 'authorisgold'
  • text: text of the thread
  • title: title of the thread
  • url: url of the thread
  • id: unique ID of the thread
  • subreddit: subreddit that the thread belongs to
  • meta: meta tag assigned to the subreddit of the thread in config.json
  • time: timestamp of the thread
  • author: username of the author of the thread
  • ups: number of ups the thread has received
  • downs: number of downs the thread has received
  • authorlinkkarma: the author's link karma
  • authorcommentkarma: the author's comment karma
  • authorisgold: 1 if the author has gold status, 0 otherwise

Comments

are in comments.csv at https://mega.nz/#F!NtsCGTgD!urXdXLJ6yITYdWEdWN-H1w

Heades are (different from threads.csv)

'text', 'id', 'subreddit', 'meta', 'time', 'author', 'ups', 'downs', 'authorlinkkarma', 'authorcommentkarma', 'authorisgold'
  • text: text of the comment
  • id: unique ID of the comment
  • subreddit: subreddit that the thread belongs to
  • meta: meta tag assigned to the subreddit of the thread in config.json
  • time: timestamp of the thread
  • author: username of the author of the thread
  • ups: number of ups the thread has received
  • downs: number of downs the thread has received
  • authorlinkkarma: the author's link karma
  • authorcommentkarma: the author's comment karma
  • authorisgold: 1 if the author has gold status, 0 otherwise

All text is normalized to lower case, tokenized using a TreebankTokenizer from natural, then joined with spaces. This results in punctuation being separated from words, a desired effect.


The MIT License (MIT)

Copyright (c) 2016, Linan Qiu

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.