Skip to content
pqstream turns your postgres database into an event stream
Go Ruby Python Shell Makefile JavaScript Dockerfile
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci clients/ruby: add basic rspec and coverage Jan 1, 2018
clients clients/ruby: remove accidental doc commit Jan 2, 2018
cmd contrib: add contrib/cmd, move pqsamq into it Mar 2, 2018
contrib/cmd/pqsamq contrib: add contrib/cmd, move pqsamq into it Mar 2, 2018
ctxutil initial commit Sep 4, 2017
pqs Revert "change pqstream.pb.go as the protobuf asks" Oct 14, 2019
vendor vendor: vendor protobuf to make client code generation easier (#77) Dec 29, 2017
CONTRIBUTING.md dos2unix (#23) Sep 28, 2017
Dockerfile Fix typo Oct 14, 2019
LICENSE remove execute bit Sep 6, 2017
README.md README: docs improvement (#76) Dec 29, 2017
build ci: include test dependencies in build script (#20) Sep 22, 2017
check_generated_code ci: add ci Sep 8, 2017
codecov.yml codecov: fixups Jan 1, 2018
gen.go initial commit Sep 4, 2017
install-go clients/python: improve tools and ci for python client (#79) Jan 1, 2018
lint
patch.go patch: include a json merge patch field for UPDATE operations (#22) Sep 29, 2017
patch_test.go lint: add linting Oct 1, 2017
pqstream.proto patch: rename to changes, include more useful diff Sep 30, 2017
queries.go cleanup: unreachable code, pull out handleEvent Sep 30, 2017
redactions.go move redaction decoding (#74) Oct 24, 2017
redactions_test.go move redaction decoding (#74) Oct 24, 2017
server.go lint: add linting Oct 1, 2017
server_test.go test: enhance trigger tests Jan 1, 2018
test clients/python: improve tools and ci for python client (#79) Jan 1, 2018

README.md

pqstream

pqstream is a program that streams changes out of a postgres database with the intent of populating other systems and enabling stream processing of data sets.

ci status go report card coverage

installation

$ go get -u github.com/tmc/pqstream/cmd/{pqs,pqsd}

basic usage

create an example database:

$ createdb dbname
# echo "create table notes (id serial, created_at timestamp, note text)" | psql dbname

connect the agent:

$ pqsd -connect postgresql://user:pass@host/dbname

connect the cli:

$ pqs

at this point you will see streams of database operations rendered to stdout:

(in a psql shell):

dbname=# insert into notes values (default, default, 'here is a sample note');
INSERT 0 1
dbname=# insert into notes values (default, default, 'here is a sample note');
INSERT 0 1
dbname=# update notes set note = 'here is an updated note' where id=1;
UPDATE 1
dbname=# delete from notes where id = 1;
DELETE 1
dbname=#

our client should now show our operations:

$ pqs
{"schema":"public","table":"notes","op":"INSERT","id":"1","payload":{"created_at":null,"id":1,"note":"here is a sample note"}}
{"schema":"public","table":"notes","op":"INSERT","id":"2","payload":{"created_at":null,"id":2,"note":"here is a sample note"}}
{"schema":"public","table":"notes","op":"UPDATE","id":"1","payload":{"created_at":null,"id":1,"note":"here is an updated note"},"changes":{"note":"here is a sample note"}}
{"schema":"public","table":"notes","op":"DELETE","id":"1","payload":{"created_at":null,"id":1,"note":"here is an updated note"}}

field redaction

If there's a need to prevent sensitive fields (i.e. PII) from being exported the redactions flag can be used with pqsd:

$ pqsd -connect postgresql://user:pass@host/dbname -redactions='{"public":{"users":["first_name","last_name","email"]}}'

The redactions is encoded in JSON and conforms to the following layout:

'{"schema":{"table":["field1","field2"]}}'`
You can’t perform that action at this time.