Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Java Python Shell Other
Fetching latest commit…
Cannot retrieve the latest commit at this time.
|Failed to load latest commit information.|
Feedsucker is an application for collecting texts from a set of feeds over a period of time and storing them in a relational database. Main use case is collection of texts from web media outlets. Feedsucker is a free software, licensed under the Apache License, Version 2.0 The app was developed while its author was an employee of Ruđer Bošković Institute (http://www.irb.hr/eng). Documentation is contained in the following folders: doc/structure/ : Structure of the main application code, and supporting functionality. doc/deploy/ : Deployment instructons doc/todo/ : List of functionality to add, todo lists and bug lists. Feedsucker is written in Java (Java 7 or above is required), with interface to newspaper written in Python and with control and support functionality written in Bash. For this reason, it is deployable on Linux only, without some modifications (these are on todo list, see doc/todo/new_functionality.txt). Supported feeds are rss and atom feeds and a class of html pages (containing URLs with specific structure common for news sites). For text scraping, feedsucker relies on newspaper, a tool written in Python. In general, a feed can be viewed as any source of URLs or other addresses pointing to resources containing text that can be scraped/extracted. Feedsucker is extensible and new scrapers (IArticleScraper classes) and feed readers (IFeedReader classes) can be written and used within app workflow. Feedsucker was developed as a research tool for creating media text corpora. It enables a researcher to collect texts from a set of outlets/feeds of interest. Following research articles are based on data collected with Feedsucker: "Getting the Agenda Right: Measuring Media Agenda using Topic Models" "Issues and their Salience in the 2015 Parliamentary Election in Croatia: A Topic Model based Analysis of the Media Agenda" (to appear) Largest deployment up to date collected 1.1 million texts from 73 feeds (25 outlets) over a period of 14 months, but the app should easily handle at least hundreds of feeds.