pg_auto_failover includes the command line tool pg_autoctl
that implements many commands to manage your Postgres nodes. To implement the Postgres architectures described in this documentation, and more, it is generally possible to use only some of the many pg_autoctl
commands.
This section of the documentation is a short introduction to the main commands that are useful when getting started with pg_auto_failover. More commands are available and help deal with a variety of situations, see the manual
for the whole list.
To understand which replication settings to use in your case, see architecture_basics
section and then the multi_node_architecture
section.
To follow a step by step guide that you can reproduce on your own Azure subscription and create a production Postgres setup from VMs, see the tutorial
section.
To understand how to setup pg_auto_failover in a way that is compliant with your internal security guide lines, read the security
section.
As a command line tool pg_autoctl
depends on some environment variables. Mostly, the tool re-uses the Postgres environment variables that you might already know.
To manage a Postgres node pg_auto_failover needs to know its data directory location on-disk. For that, some users will find it easier to export the PGDATA
variable in their environment. The alternative consists of always using the --pgdata
option that is available to all the pg_autoctl
commands.
To get started with the simplest Postgres failover setup, 3 nodes are needed: the pg_auto_failover monitor, and 2 Postgres nodes that will get assigned roles by the monitor. One Postgres node will be assigned the primary role, the other one will get assigned the secondary role.
To create the monitor use the command:
$ pg_autoctl create monitor
The create the Postgres nodes use the following command on each node you want to create:
$ pg_autoctl create postgres
While those create commands initialize your nodes, now you have to actually run the Postgres service that are expected to be running. For that you can manually run the following command on every node:
$ pg_autoctl run
It is also possible (and recommended) to integrate the pg_auto_failover service in your usual service management facility. When using systemd the following commands can be used to produce the unit file configuration required:
$ pg_autoctl show systemd
INFO HINT: to complete a systemd integration, run the following commands:
INFO pg_autoctl -q show systemd --pgdata "/tmp/pgaf/m" | sudo tee /etc/systemd/system/pgautofailover.service
INFO sudo systemctl daemon-reload
INFO sudo systemctl enable pgautofailover
INFO sudo systemctl start pgautofailover
[Unit]
...
While it is expected that for a production deployment each node actually is a separate machine (virtual or physical, or even a container), it is also possible to run several Postgres nodes all on the same machine for testing or development purposes.
Tip
When running several pg_autoctl
nodes on the same machine for testing or contributing to pg_auto_failover, each Postgres instance needs to run on its own port, and with its own data directory. It can make things easier to then set the environment variables PGDATA
and PGPORT
in each terminal, shell, or tab where each instance is started.
Once your Postgres nodes have been created, and once each pg_autoctl
service is running, it is possible to inspect the current state of the formation with the following command:
$ pg_autoctl show state
The pg_autoctl show state
commands outputs the current state of the system only once. Sometimes it would be nice to have an auto-updated display such as provided by common tools such as watch(1) or top(1) and the like. For that, the following commands are available (see also pg_autoctl_watch
):
$ pg_autoctl watch
$ pg_autoctl show state --watch
To analyze what's been happening to get to the current state, it is possible to review the past events generated by the pg_auto_failover monitor with the following command:
$ pg_autoctl show events
Hint
The pg_autoctl show
commands can be run from any node in your system. Those command need to connect to the monitor and print the current state or the current known list of events as per the monitor view of the system.
Use pg_autoctl show state --local
to have a view of the local state of a given node without connecting to the monitor Postgres instance.
The option --json
is available in most pg_autoctl
commands and switches the output format from a human readable table form to a program friendly JSON pretty-printed output.
When creating a node it is possible to use the --candidate-priority
and the --replication-quorum
options to set the replication properties as required by your choice of Postgres architecture.
To review the current replication settings of a formation, use one of the two following commands, which are convenient aliases (the same command with two ways to invoke it):
$ pg_autoctl show settings
$ pg_autoctl get formation settings
It is also possible to edit those replication settings at any time while your nodes are in production: you can change your mind or adjust to new elements without having to re-deploy everything. Just use the following commands to adjust the replication settings on the fly:
$ pg_autoctl set formation number-sync-standbys
$ pg_autoctl set node replication-quorum
$ pg_autoctl set node candidate-priority
Important
The pg_autoctl get
and pg_autoctl set
commands always connect to the monitor Postgres instance.
The pg_autoctl set
command then changes the replication settings on the node registration on the monitor. Then the monitor assigns the APPLY_SETTINGS state to the current primary node in the system for it to apply the new replication settings to its Postgres streaming replication setup.
As a result, the pg_autoctl set
commands requires a stable state in the system to be allowed to proceed. Namely, the current primary node in the system must have both its Current State and its Assigned State set to primary, as per the pg_autoctl show state
output.
When a Postgres node must be taken offline for a maintenance operation, such as e.g. a kernel security upgrade or a minor Postgres update, it is best to make it so that the pg_auto_failover monitor knows about it.
- For one thing, a node that is known to be in maintenance does not participate in failovers. If you are running with two Postgres nodes, then failover operations are entirely prevented while the standby node is in maintenance.
- Moreover, depending on your replication settings, enabling maintenance on your standby ensures that the primary node switches to async replication before Postgres is shut down on the secondary, avoiding write queries to be blocked.
To implement maintenance operations, use the following commands:
$ pg_autoctl enable maintenance
$ pg_autoctl disable maintenance
The main pg_autoctl run
service that is expected to be running in the background should continue to run during the whole maintenance operation. When a node is in the maintenance state, the pg_autoctl
service is not controlling the Postgres service anymore.
Note that it is possible to enable maintenance on a primary Postgres node, and that operation then requires a failover to happen first. It is possible to have pg_auto_failover orchestrate that for you when using the command:
$ pg_autoctl enable maintenance --allow-failover
Important
The pg_autoctl enable
and pg_autoctl disable
commands requires a stable state in the system to be allowed to proceed. Namely, the current primary node in the system must have both its Current State and its Assigned State set to primary, as per the pg_autoctl show state
output.
In the cases when a failover is needed without having an actual node failure, the pg_auto_failover monitor can be used to orchestrate the operation. Use one of the following commands, which are synonyms in the pg_auto_failover design:
$ pg_autoctl perform failover
$ pg_autoctl perform switchover
Finally, it is also possible to “elect” a new primary node in your formation with the command:
$ pg_autoctl perform promotion
Important
The pg_autoctl perform
commands requires a stable state in the system to be allowed to proceed. Namely, the current primary node in the system must have both its Current State and its Assigned State set to primary, as per the pg_autoctl show state
output.
This section of the documentation is meant to help users get started by focusing on the main commands of the pg_autoctl
tool. Each command has many options that can have very small impact, or pretty big impact in terms of security or architecture. Read the rest of the manual to understand how to best use the many pg_autoctl
options to implement your specific Postgres production architecture.