Skip to content
A template for PostgreSQL High Availability with ZooKeeper, etcd, or Consul
Python Gherkin Dockerfile Shell Groovy
Branch: master
Clone or download
CyberDem0n Build recovery params in a separate method (#1219)
In addition to that try to protect from the case when some recovery parameters are set in one of included files by explicitly setting their value to an empty string on postgres 12.

Simplifies #1208
Latest commit f4623c4 Oct 11, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
docker Fix a command in the docker readme. (#1138) Aug 6, 2019
docs Refactor Patroni REST API communication (#1197) Oct 11, 2019
extras Add confd template for pgbouncer (#844) Nov 21, 2018
features A few small fixes (#1221) Oct 11, 2019
kubernetes Grant deletecollection to patroni serviceaccount (#1033) Apr 11, 2019
patroni Build recovery params in a separate method (#1219) Oct 11, 2019
tests Build recovery params in a separate method (#1219) Oct 11, 2019
.gitignore Ingore VSCode files (#1007) Mar 22, 2019
.travis.yml Remove psycopg2 from requirements (#1023) Apr 15, 2019
Dockerfile Fix some small issues in the Dockerfile (#1074) May 28, 2019
LICENSE Rename LICENCE to LICENSE Feb 17, 2016
MAINTAINERS Improve quality of code by resolving issues found by quantifiedcode a… Feb 12, 2016
MANIFEST.in Refactoring to enable package building and succesfull installation of… Sep 3, 2015
README.rst Remove psycopg2 from requirements (#1023) Apr 15, 2019
TODO.md
docker-compose.yml Rename cluster name to demo (#1000) Mar 11, 2019
haproxy.cfg Improve haproxy.cfg (#473) Jul 13, 2017
mkbinary.sh Try to get rid from hardcoded names when building binary Sep 5, 2016
patroni.py Add patroni.py executable file Sep 7, 2015
patroni.spec Try to get rid from hardcoded names when building binary Sep 5, 2016
patronictl.py Improve quality of code by resolving issues found by quantifiedcode a… Feb 12, 2016
postgres0.yml Make it possible to run pg_rewind without superuser on pg11+ (#1035) May 2, 2019
postgres1.yml Make it possible to run pg_rewind without superuser on pg11+ (#1035) May 2, 2019
postgres2.yml Make it possible to run pg_rewind without superuser on pg11+ (#1035) May 2, 2019
release.sh Tag on github is prefixed with v. Sep 8, 2015
requirements.txt A few small fixes (#1221) Oct 11, 2019
setup.py Remove psycopg2 from requirements (#1023) Apr 15, 2019
tox.ini Make sure that we are running and testing latest versions of everythi… Sep 19, 2016

README.rst

Build Status Coverage Status

Patroni: A Template for PostgreSQL HA with ZooKeeper, etcd or Consul

You can find a version of this documentation that is searchable and also easier to navigate at patroni.readthedocs.io.

There are many ways to run high availability with PostgreSQL; for a list, see the PostgreSQL Documentation.

Patroni is a template for you to create your own customized, high-availability solution using Python and - for maximum accessibility - a distributed configuration store like ZooKeeper, etcd, Consul or Kubernetes. Database engineers, DBAs, DevOps engineers, and SREs who are looking to quickly deploy HA PostgreSQL in the datacenter-or anywhere else-will hopefully find it useful.

We call Patroni a "template" because it is far from being a one-size-fits-all or plug-and-play replication system. It will have its own caveats. Use wisely.

Note to Kubernetes users: Patroni can run natively on top of Kubernetes. Take a look at the Kubernetes chapter of the Patroni documentation.

How Patroni Works

Patroni originated as a fork of Governor, the project from Compose. It includes plenty of new features.

For an example of a Docker-based deployment with Patroni, see Spilo, currently in use at Zalando.

For additional background info, see:

Development Status

Patroni is in active development and accepts contributions. See our Contributing section below for more details.

We report new releases information here.

Community

There are two places to connect with the Patroni community: on github, via Issues and PRs, and on channel #patroni in the PostgreSQL Slack. If you're using Patroni, or just interested, please join us.

Technical Requirements/Installation

Pre-requirements for Mac OS

To install requirements on a Mac, run the following:

brew install postgresql etcd haproxy libyaml python

Psycopg2

Starting from psycopg2-2.8 the binary version of psycopg2 will no longer be installed by default. Installing it from the source code requires C compiler and postgres+python dev packages. Since in the python world it is not possible to specify dependency as psycopg2 OR psycopg2-binary you will have to decide how to install it.

There are a few options available:

  1. Use the package manager from your distro
sudo apt-get install python-psycopg2   # install python2 psycopg2 module on Debian/Ubuntu
sudo apt-get install python3-psycopg2  # install python3 psycopg2 module on Debian/Ubuntu
sudo yum install python-psycopg2       # install python2 psycopg2 on RedHat/Fedora/CentOS
  1. Install psycopg2 from the binary package
pip install psycopg2-binary
  1. Install psycopg2 from source
pip install psycopg2>=2.5.4

General installation for pip

Patroni can be installed with pip:

pip install patroni[dependencies]

where dependencies can be either empty, or consist of one or more of the following:

etcd
python-etcd module in order to use Etcd as DCS
consul
python-consul module in order to use Consul as DCS
zookeeper
kazoo module in order to use Zookeeper as DCS
exhibitor
kazoo module in order to use Exhibitor as DCS (same dependencies as for Zookeeper)
kubernetes
kubernetes module in order to use Kubernetes as DCS in Patroni
aws
boto in order to use AWS callbacks

For example, the command in order to install Patroni together with dependencies for Etcd as a DCS and AWS callbacks is:

pip install patroni[etcd,aws]

Note that external tools to call in the replica creation or custom bootstap scripts (i.e. WAL-E) should be installed independently of Patroni.

Running and Configuring

To get started, do the following from different terminals:

> etcd --data-dir=data/etcd
> ./patroni.py postgres0.yml
> ./patroni.py postgres1.yml

You will then see a high-availability cluster start up. Test different settings in the YAML files to see how the cluster's behavior changes. Kill some of the components to see how the system behaves.

Add more postgres*.yml files to create an even larger cluster.

Patroni provides an HAProxy configuration, which will give your application a single endpoint for connecting to the cluster's leader. To configure, run:

> haproxy -f haproxy.cfg
> psql --host 127.0.0.1 --port 5000 postgres

YAML Configuration

Go here for comprehensive information about settings for etcd, consul, and ZooKeeper. And for an example, see postgres0.yml.

Environment Configuration

Go here for comprehensive information about configuring(overriding) settings via environment variables.

Replication Choices

Patroni uses Postgres' streaming replication, which is asynchronous by default. Patroni's asynchronous replication configuration allows for maximum_lag_on_failover settings. This setting ensures failover will not occur if a follower is more than a certain number of bytes behind the leader. This setting should be increased or decreased based on business requirements. It's also possible to use synchronous replication for better durability guarantees. See replication modes documentation for details.

Applications Should Not Use Superusers

When connecting from an application, always use a non-superuser. Patroni requires access to the database to function properly. By using a superuser from an application, you can potentially use the entire connection pool, including the connections reserved for superusers, with the superuser_reserved_connections setting. If Patroni cannot access the Primary because the connection pool is full, behavior will be undesirable.

You can’t perform that action at this time.