Home
This is a work in progress, I’m documenting my installation process as I go.
sudo apt-get install postgresql postgresql-client postgresql-contrib libpq-dev
If you just try to do
createdb opencongress_dev
you’ll get something like this:
FATAL: ident authentication failed
This is explained and solved here: http://www.depesz.com/index.php/2007/10/04/ident/
Long story short: edit /etc/postgresql/8.3/main/pg_hba.conf
and change your local
lines to have trust
instead of ident sameuser
as the authentication method. Read the article for security implications.
After that you need to restart postgres:
sudo /etc/init.d/postgresql-8.3 restart
By default, postgres comes only with the admin user, called postgres
, so we’ll use it from here on. To create the database:
createdb -U postgres opencongress_dev
Now add the tsearch2 stuff. The file that’s in the opencongress repo (db/tsearch2.sql
) blows up with all sorts of errors for me. This is what worked:
psql -d opencongress_dev -U postgres < /usr/share/postgresql/8.3/contrib/tsearch2.sql
Repeat the same steps to create the opencongress_test
database.
Install necessary gems:
sudo gem install ferret bluecloth simple-rss postgres-pr
Set up your database.yml
file to use adapter: postgresql
and host: /var/run/postgresql
.
The environment config tries to read config/api_keys.yml
which isn’t present by default. Just copy the dummy over from config/api_keys.example.yml
for now. For migrating that’s all you need. You can fill out the api keys later.
Before you run your migrations, you’ll also need to set up Solr, the search server. Thankfully, it already comes bundled with the acts_as_solr plugin, which is in our vendor/plugins directory.
Create the logs and tmp directories:
mkdir vendor/plugins/acts_as_solr/solr/logs
mkdir vendor/plugins/acts_as_solr/solr/tmp
And fire it up:
rake solr:start
It should now be running on port 8982.
And then:
rake db:migrate
Ok, not so fast:
== PolymorphicPageviews: migrating ===========================================
-- add_index(:page_views, [:viewable_id, :viewable_type, :created_at])
rake aborted!
An error has occurred, this and all later migrations canceled:
RuntimeError: ERROR C42703 Mcolumn "viewable_id" does not exist Findexcmds.c L779 RComputeIndexAttrs: CREATE INDEX "index_page_views_on_viewable_id_and_viewable_type_and_created_at" ON "page_views" ("viewable_id", "viewable_type", "created_at")
Open db/migrate/2009010145551_polymorphic_pageviews.rb
, uncomment all the code in self.up
and remove the line add_index :page_views, :ip_address
(this index was already created at migration #063). (or just use this patch: http://github.com/obvio171/opencongress/commit/8305fabb1e4160c0df7ea94b0bf6f1679794f6f9.diff )
You’ll also need RedCloth installed for the EditableBlogroll migration.
sudo gem install RedCloth
And now, what we’ve all been waiting for:
rake db:migrate
That one should be a winner! All migrations run! w00t!
Now let’s import some data.
(From the README)
Next, you will have to get the data from govtrack and fill in your database with the parsed govtrack data. Right now, you do have to use the /data/… absolute path (relative to the root of the filesystem). You’ll probably need sudo for this.
mkdir -p /data/govtrack/111
rsync -az govtrack.us::govtrackdata/us/111/bills /data/govtrack/111
rsync -az govtrack.us::govtrackdata/us/111/repstats /data/govtrack/111
rsync -az govtrack.us::govtrackdata/us/111/bills.index.xml /data/govtrack/111
rsync -az govtrack.us::govtrackdata/us/111/committeeschedule.xml /data/govtrack/111
rsync -az govtrack.us::govtrackdata/us/111/rolls /data/govtrack/111
Now, you will have to actually parse the data. You’ll need the htree library for that:
cd /tmp
cvs -d :pserver:anonymous@cvs.m17n.org:/cvs/ruby co htree
cd htree
sudo ruby install.rb
Bizarre, yes. Anyone wanna package that as a gem?
You also need to fix the data paths in the data import scripts. This patch will take care of that for you:
http://github.com/obvio171/opencongress/commit/de85309520f9663a88022a5712062838edd5650b
So now go back to your application root directory and run:
Note: this will take a REALLY long time. Go get some coffee.
ruby bin/govtrack_parse_people.rb
The following doesn’t work, because govtrack_parse_bills.rb
expects a different format from the one in the file it tries to read (bills.index.xml). It seems to expect the format of a bill file (like this one). The other parsers also break.
ruby bin/govtrack_parse_bills.rb ruby bin/govtrack_parse_committees.rb ruby bin/govtrack_parse_committee_schedules.rb ruby parse_individual_bills.rb