pqstream turns your postgres database into an event stream
Switch branches/tags
Nothing to show
Clone or download
tmc Merge pull request #82 from tmc/tmc.contrib
contrib: add contrib/cmd, move pqsamq into it
Latest commit f638150 Mar 3, 2018
Permalink
Failed to load latest commit information.
.circleci clients/ruby: add basic rspec and coverage Jan 1, 2018
clients clients/ruby: remove accidental doc commit Jan 2, 2018
cmd contrib: add contrib/cmd, move pqsamq into it Mar 2, 2018
contrib/cmd/pqsamq contrib: add contrib/cmd, move pqsamq into it Mar 2, 2018
ctxutil initial commit Sep 4, 2017
pqs patch: rename to changes, include more useful diff Sep 30, 2017
vendor vendor: vendor protobuf to make client code generation easier (#77) Dec 29, 2017
CONTRIBUTING.md dos2unix (#23) Sep 28, 2017
Dockerfile docker: add Dockerfile (#24) Oct 17, 2017
LICENSE remove execute bit Sep 6, 2017
README.md README: docs improvement (#76) Dec 29, 2017
build ci: include test dependencies in build script (#20) Sep 22, 2017
check_generated_code ci: add ci Sep 8, 2017
codecov.yml codecov: fixups Jan 1, 2018
gen.go initial commit Sep 4, 2017
install-go clients/python: improve tools and ci for python client (#79) Jan 1, 2018
lint Remove gometalinter duplicated errcheck option and increase deadline (#… Oct 3, 2017
patch.go patch: include a json merge patch field for UPDATE operations (#22) Sep 29, 2017
patch_test.go lint: add linting Oct 1, 2017
pqstream.proto patch: rename to changes, include more useful diff Sep 30, 2017
queries.go cleanup: unreachable code, pull out handleEvent Sep 30, 2017
redactions.go move redaction decoding (#74) Oct 24, 2017
redactions_test.go move redaction decoding (#74) Oct 24, 2017
server.go lint: add linting Oct 1, 2017
server_test.go test: enhance trigger tests Jan 1, 2018
test clients/python: improve tools and ci for python client (#79) Jan 1, 2018

README.md

pqstream

pqstream is a program that streams changes out of a postgres database with the intent of populating other systems and enabling stream processing of data sets.

ci status go report card coverage

installation

$ go get -u github.com/tmc/pqstream/cmd/{pqs,pqsd}

basic usage

create an example database:

$ createdb dbname
# echo "create table notes (id serial, created_at timestamp, note text)" | psql dbname

connect the agent:

$ pqsd -connect postgresql://user:pass@host/dbname

connect the cli:

$ pqs

at this point you will see streams of database operations rendered to stdout:

(in a psql shell):

dbname=# insert into notes values (default, default, 'here is a sample note');
INSERT 0 1
dbname=# insert into notes values (default, default, 'here is a sample note');
INSERT 0 1
dbname=# update notes set note = 'here is an updated note' where id=1;
UPDATE 1
dbname=# delete from notes where id = 1;
DELETE 1
dbname=#

our client should now show our operations:

$ pqs
{"schema":"public","table":"notes","op":"INSERT","id":"1","payload":{"created_at":null,"id":1,"note":"here is a sample note"}}
{"schema":"public","table":"notes","op":"INSERT","id":"2","payload":{"created_at":null,"id":2,"note":"here is a sample note"}}
{"schema":"public","table":"notes","op":"UPDATE","id":"1","payload":{"created_at":null,"id":1,"note":"here is an updated note"},"changes":{"note":"here is a sample note"}}
{"schema":"public","table":"notes","op":"DELETE","id":"1","payload":{"created_at":null,"id":1,"note":"here is an updated note"}}

field redaction

If there's a need to prevent sensitive fields (i.e. PII) from being exported the redactions flag can be used with pqsd:

$ pqsd -connect postgresql://user:pass@host/dbname -redactions='{"public":{"users":["first_name","last_name","email"]}}'

The redactions is encoded in JSON and conforms to the following layout:

'{"schema":{"table":["field1","field2"]}}'`