HTTP API for datanommer and the fedmsg bus
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.



A webapp to retrieve historical information about messages on the fedmsg bus. It is a JSON api for the datanommer message store.

Production Instance

Hacking on datagrepper


  • virtualenvwrapper
  • Postgresql

Install postgresql and virtualenvwrapper:

$ sudo yum install -y postgresql-server python-virtualenvwrapper

Setting up the stack

Use a virtualenv:

$ mkvirtualenv datagrepper
$ workon datagrepper

Install dependencies:

$ pip install -r requirements.txt
$ pip install psycopg2

Configuring Postgresql (and getting some data)

In normal operations, the datanommer consumer daemon will be running somewhere and continuously stuff each new fedmsg message that it sees into a postgres DB. If you're just sitting down to hack on datagrepper, that won't be your situation so you'll need a dump of the database.


If you've tried installing postgres before and think you've messed it up, you'll need to blow away the old databases with $ rm -rf /var/lib/pgsql

Install postgres (and fedmsg, while we're at it):

$ sudo yum install -y postgresql-server fedmsg
$ sudo postgresql-setup initdb

Make sure postgres is set to allow connections over tcp/ip using password authentication. Edit the /var/lib/pgsql/data/pg_hba.conf. You might find a line like this:

host    all             all               ident

Instead of that line, you need one that looks like this:

host    all             all               md5

Become yourself again (not the postgres user) and start up postgres:

$ sudo systemctl restart postgresql.service

Become the postgres user (again) and run the psql command. Use that psql shell to setup the DB, the user, and privileges:

$ sudo su - postgres
$ psql
# create database datanommer;
# create user datanommer with password 'bunbunbun';
# grant all privileges on database datanommer to datanommer;
# \q

Back in the bash shell (but still as the postgres user), grab a DB dump and restore it:

$ wget
$ xzcat datanommer.dump.xz | psql datanommer

Last step, run datagrepper

You have to configure your development datagrepper instance to talk to postgres (by default, it looks for a sqlite database). Edit fedmsg.d/ and give it these contents:

config = {
    'datanommer.enabled': False,
    'datanommer.sqlalchemy.url': 'postgresql+psycopg2://datanommer:bunbunbun@localhost:5432/datanommer',
    'fedmsg.consumers.datagrepper-runner.enabled': True,

As your normal old user self, run the development server:

$ workon datagrepper
$ python

In a browser, visit http://localhost:5000 to see the docs.

You can quick test that you can get data by running:

$ sudo yum install -y httpie
$ http get localhost:5000/raw delta==1000000 rows_per_page==1