-
Notifications
You must be signed in to change notification settings - Fork 76
Available data sources
Sal Hagen edited this page Aug 6, 2021
·
18 revisions
On this page we list the scripts for data sources. Some of these are fully functional, others are deprecated. Let us know if you have a new data source to add.
For datasource-specific information, check the README files in the folder of the respective data source.
| Name | Source | Objects | Local | Notes |
|---|---|---|---|---|
| 4chan | 4chan API | Comments + OPs | Yes | We wrote several scripts to import data from 4chan archives in the helper-scripts folder, e.g. this script to import csv dumps from 4plebs. |
| 8chan | 4chan API | Comments + OPs | Yes | 8chan is now defunct. We scraped live data when it was still online. Let us know in case you are interested in a database copy. |
| 8kun | 8chan API | Comments + OPs | Yes | |
| Bitchute | Scraping | Commments + videos | No | Uses BitChutes search endpoint. |
| Douban | Scraping | Comments + OPs | No | |
| Import from tool | Files from other tools | - | No | This to import files from tools like CrowdTangle. |
| Instaloader | Comments | No | Deprecated. | |
| Parler | Parler API | Posts | No | |
| Parliament | PENELOPE API | Speeches | No | Parliamentary speeches from the UK and Germany. |
| Pushshift API | Comments + OPs | No | Data retrieved via Pushshift. | |
| Telegram | Telegram API | Posts in open groups | No | Required personal API code. |
| The Guardian climate change | - | Articles + comments | Yes | A static data source of The Guardian articles and comments concerning climate change (collected by VUB). |
| TikTok | Scraping | Comments | No | Deprecated. |
| Tumblr | Tumblr API | Posts + text reblogs | No | |
| Twitter v2 API | Tweets | No | Requires personal academic track API key. | |
| Usenet | - | Comments + OPs | Yes | Static Usenet database. |