Full details of this software can be found at AITopics.
The main script
AINews.py can be called with one of three
python AINews.py crawl
This crawls the sources list and stores its results in the database
python AINews.py prepare
This filters and processes the news, and creates an XML file export.
python AINews.py email
This generates the weekly email as an HTML submission form.
Our configuration has a script that uses the
that order) commands once each day, and the
NewsFinder is primarily coded in Python and requires the following libraries:
- MySQL Python bindings (MySQLdb)
- Beautiful Soup
Packages for Ubuntu:
sudo apt-get install python-mysqldb libsvm-tools python-libsvm \ python-cheetah python-nltk python-beautifulsoup \ python-pyrss2gen python-feedparser python-unidecode
"Installation" of NewsFinder should only involve downloading the code in
this repository. Assuming Python includes the NewsFinder code in its path,
execution of NewsFinder should be as simple as running
There are various supplementary files that NewsFinder expects. These files can be obtained from another repository: AINewsSupplementary
In our working configuration, we have a directory in our home called
NewsFinder. In this directory are the AINewsSupplementary
resource, etc.). The NewsFinder code
found in this repository is stored in a
code directory in the main
NewsFinder folder. In any event, the
paths.ini file (in the
config directory of this project) can be modified to point to
whereever you keep your files.
tables.sql to create tables needed by NewsFinder.
All configuration is provided in the
config.inihas miscellaneous configuration options, described in the file
db.inihas database connectivity information (host, username, password, etc.)
paths.inistores the paths of various data and output directories used by NewsFinder. This file allows you to store data and output in whatever way makes sense for a particular installation (for example, data files should be outside of a webserver root)
Make a copy of the
.ini.sample files (rename to
.ini) to build
your own configuration.
Sources are specified in a CSV file obtained from a URL provided in
paths.ini config file. Here is an example short sources CSV file:
"SourceID","Title","Link","Parser" "62671","Forbes Technology","http://www.forbes.com/technology/index.xml","RSS" "62673","BBC Technology","http://feeds.bbci.co.uk/news/technology/rss.xml","RSS"
Only RSS sources are supported at this time.
The most recent version (as of this writing) of NewsFinder was written by Joshua Eckroth. The prior iteration was written by Liang Dong. Before that, NewsFinder was coded by Tom Charytoniuk. The project has been supervised by Bruce Buchanan and Reid Smith.
Copyright (c) 2011 by the Association for the Advancement of Artificial Intelligence. This program and parts of it may be used and distributed without charge for non-commercial purposes as long as this notice is included.
arff.py is pulled from the
laic-arff package, which is
distributed under the MIT License.