Pull messages from NSQ and write to a Postgres JSONB column.
$ nsq_to_postgres --config config.yml
select log->'program' as program, log->'level' as level, count(*) from logs group by program, level order by count desc;
program | level | count ---------+---------+-------- "site" | "info" | 142276 "api" | "info" | 8176 "api" | "error" | 1638
A configuration file must be specified via
--config, I prefer making the path explicit so no one is left guessing of its whereabouts (unlike most programs, grr!).
Two sections are available for tweaking, first the
postgres section which defines the connection information, the target table name, target column name, and verbosity. For example:
postgres: connection: user=tj password=sloth host=localhost port=5432 sslmode=disable table: logs column: log max_open_connections: 10 verbose: no
When nsq_to_postgres first establishes a connect the table will be automatically created for you, if you have not already done so.
The next section available is
nsq which defines the topic to consume from, the nsqd or nsqlookupd addresses, max number of retry attempts and so on.
For more nsq configuration options visit segmentio/go-queue.
nsq: topic: logs nsqd: localhost:4150 max_attempts: 5 msg_timeout: 15s max_in_flight: 300 concurrency: 50
Boot postgres, nsqd, and nsqlookupd (requires docker and docker-compose):
$ make client