heroku-log-store is an evented Ruby service for receiving and storing syslog messages over HTTP. Its purpose is to function as a Heroku https log drain which stores log data in a Postgres database for long-term log storage and very simple querying.
heroku-log-store makes no assumptions about the format of your data, merely dumping payload data into a Postgres table which you can then query yourself. Think of heroku-log-store as a long-term but queryable place to store your log data. Future versions may built simple introspection functionality on top of this data, but I reserve the right to keep it real dumb.
Currently, heroku-log-store is seeing lite action in production. Please report any issues in the project's Github issue tracker.
To have heroku-log-store store your logs, deploy an instance of heroku-log-store to Heroku. Yes, a log-drain app on Heroku, draining logs from another app on Heroku. Just don't create a circular reference or else the universe might rip.
Note: This assumes you have the Heroku Toolbelt installed locally.
Also, these steps install the
dev heroku-postgresql add-on which
is limited to 10K rows. This is not a lot of capacity as logs can be quite chatty. It is recommended that you use at least
basic postgres database. If you want to do this just specify
heroku-postgresql:basic below instead.
$ git clone git://github.com/rwdaigle/heroku-log-store.git $ cd heroku-log-store $ heroku create log-store $ heroku config:set RACK_ENV=production $ heroku addons:add heroku-postgresql:dev $ git push heroku master
Your drain should now be deployed and running on Heroku, although it will need to be configured before use.
To configure your heroku-log-store instance you need to create the basic table structure in the database.
There is a process type called
migrate you can invoke with
heroku run that will do this for you.
$ heroku run migrate Running `migrate` attached to terminal... up, run.4179 WARNING: => is deprecated as an operator name DETAIL: This name may be disallowed altogether in future versions of PostgreSQL.
Now open the app to see instance-specific instructions such as setting up http basic authentication and setting the instance to be a log drain for a source app.
$ heroku open Opening log-store... done
You will be prompted for a username/password. Until the credentials are explicitly set, you can login with anything.
Leave both fields blank and click "Log in". You should now see instructions similar to:
Follow the instructions in the app to properly setup security and to set the app as a log drain for another app. If you weren't able to reach this point, please submit an issue to the project's Github issue tracker.
heroku-log-store is not a replacement for Papertrail or any other true log-aggregation and eventing service. As such, its interface is limited to SQL.
Once your heroku-log-store is receiving log data you can query/filter it by opening a Postgres shell from the log-drain app.
$ heroku pg:psql u7ejsaid=#
events table is where the goods are. Here is its schema:
=# \d events Table "public.events" Column | Type | Modifiers -----------------+--------------------------+----------------------------------------------------- id | bigint | not null default nextval('events_id_seq'::regclass) emitted_at | timestamp with time zone | received_at | timestamp with time zone | default now() priority | integer | syslog_version | integer | hostname | character(255) | appname | character(255) | proc_id | character(255) | msg_id | character(255) | structured_data | text | message | text | Indexes: "events_pkey" PRIMARY KEY, btree (id) "events_emitted_at" btree (emitted_at)
The rest is up to you.