Fetching latest commit…
Cannot retrieve the latest commit at this time.
|Failed to load latest commit information.|
Copyright (c) 2007-2010 The Kaphan Foundation This file contains instructions for running and configuring the Winnow content recommendation engine. For information on building and installing the engine see the INSTALL file. Please see the pages of the wiki our github repository http://github.com/WinnowTag/winnow/wiki for detailed explanations of Winnow's architecture and operation. Creating an item cache ============================================ Before running Winnow, you'll need to create the persistent item cache. This is a directory "item_cache" containing the three sqlite3 databases atom.db, tokens.db and catalog.db. See the "Item Cache" document page in the github repository wiki for more information. To initialize the database schemas, use sqlite3 to execute the SQL shown below in each corresponding newly created sqlite3 database file. After you've used the classifier on a sufficient number of items, you'll need to set up the random background. The classifier accuracy will be poor until you do this. See the section "Setting up random background" at the end of this file. TODO: Create a script that does this. TODO: Provide a reasonably small item_cache already populated and with random background initialized that can be used for testing and initial development. atom.db ------- CREATE TABLE entry_atom (id integer not null primary key, atom BLOB); tokens.db --------- CREATE TABLE entry_tokens (id integer not null primary key, tokens BLOB); catalog.db ---------- CREATE TABLE entries (id integer NOT NULL PRIMARY KEY, full_id text, updated real, created_at real, last_used_at real); CREATE TABLE "random_backgrounds" ( "entry_id" integer NOT NULL PRIMARY KEY, constraint "random_backgrounds_entry_id" foreign key ("entry_id") references "entries" ("id") ); CREATE TABLE "tokens" ( "id" integer NOT NULL PRIMARY KEY, "token" text NOT NULL UNIQUE ); CREATE UNIQUE INDEX entries_full_id on entries(full_id); CREATE INDEX entries_last_used_at on entries(last_used_at); CREATE INDEX entries_updated on entries(updated); CREATE TRIGGER random_backgrounds_entry_id BEFORE DELETE ON entries FOR EACH ROW BEGIN SELECT RAISE(ROLLBACK, 'delete on table "entries" violates foreign key constraint "random_backgrounds_entry_id"') WHERE (SELECT entry_id FROM random_backgrounds WHERE entry_id = OLD.id) IS NOT NULL; END; Running the Classifier ============================================ The classifier can be run using the "classifier" executable. If you performed the "make install" of the build instructions this is likely in your PATH. Run "classifier -h" to see a list of supported arguments. A typical usage will be to start the classifier on a specified port and item cache directory and a positive cutoff threshold of 0.9. You probably also want a log file to be generated. This will be enough for most development cases. This can be done using the following command: classifier --db <item_cache_dir> --port 8008 -t 0.9 --log-file classifier.log You can then tail -f classifier.log to see the log messages the classifier produces. Once a message appears in the log saying the classifier has completed initialization you can then access it's http server on port 8008. Initialization can sometimes take a few minutes depending on the size of the item cache and whether the file system has cached item cache file access. Once things are up and running Winnow will be able to talk to the classifier using the standard classifier UI elements. You can also do a simple test by entering http://localhost:8009/classifier.xml into a web browser to make sure everything is working. Shutting it down ============================================= The classifier is shutdown by sending it a SIGTERM signal. This can be done using CTRL-C or kill <pid>. Setting up the random background ============================================= After you have classified more than 1000-5000 items, set up the random background. This is done by executing the following SQL in the item.db sqlite3 database: INSERT INTO RANDOM_BACKGROUNDS SELECT ID FROM ENTRIES ORDER BY random() LIMIT N ...where N is the size that you want. We recommend between 1000 and 5000 items depending upon the size of the corpus that you classify. For winnowtag.org, which works with over 3/4 million items, we use a random background of 5000 items. Until you set up the random background, Winnow's classification accuracy will be poor.