From version 9.0 onwards, PostgreSQL ships with streaming replication as part of the core product, reversing a long held policy of leaving replication to third-party add-ons such as Slony or pgpool.
This blueprint proposes to support streaming replication since it is shipped with the core product, although a discussion of some of the other PostgreSQL options can be found in the alternatives section.
https://blueprints.launchpad.net/trove/+spec/postgresql-replication
In order to achieve feature parity with MySQL, Trove should provide support for at least one of the replication solutions for PostgreSQL.
The standard parameters for enabling replication strategies,
replication_strategy
and replication_namespace
, will be added to point
to the strategy code for PostgreSQL guests.
None.
As with other datastores that support replication, the create instance
operation will support the replica_of
and replica_count
fields for
PostgreSQL guests:
POST http://127.0.0.1:8779/v1.0/<tenant id>/instances { "instance": { "volume": {"size": <size>}, "flavorRef": "<flavor-id>", "name": "s", "replica_of": "<master id>", *"replica_count": "<n>"* } }
For guests with pg_rewind
[9] support, the following instance actions are
supported:
POST http://127.0.0.1:8779/v1.0/<tenant id>/instance/<instance id>/action { *"detach_replica": null* } POST http://127.0.0.1:8779/v1.0/<tenant id>/instance/<instance id>/action { *"eject_replica_source": null* } POST http://127.0.0.1:8779/v1.0/<tenant id>/instance/<instance id>/action { *"promote_to_replica_source": null* }
None.
None.
As with other datastores that support replication, this will enable the commands:
trove create <inst> .. --replica_of <master_inst>
on PostgreSQL guests. With pg_rewind
[9] support on the guest, failover
and reconfiguration commands are supported:
trove detach-replica <inst> trove eject-replica-source <inst> trove promote-to-replica-source <inst>
None.
PostgreSQL provides both high performance and crash recovery capability through the use of a write-ahead log. The WAL files are appended to for every change to the database, with periodic checkpoints to purge the logs and integrate the changes into the database files [4].
PostgreSQL streaming replication is built on top of the recovery system: a master database operates in continuous archiving mode, sending the contents of its WAL files to one or more slave servers. These slave servers, in turn, operate in continuous recovery mode, "recovering" from the logs as they are received. A slave can be promoted in place of a failed master, and can be configured to support read-only transactions while in recovery mode.
The PostgreSQL guestagent currently provides a backup and restore strategy
which makes use of the pg_dump
and pg_restore
commands. These produce
logical dumps which cannot be used as part of a replication system that
depends on continuous archiving. As such, a backup/restore strategy using the
pg_basebackup
tool is a requirement for replication.
In order to support hot standby slaves, a PostgreSQL master must have
wal_level
set to hot_standby, which is the most verbose mode. Replication
is handled through the use of a user that has been given REPLICATION privilege
[6] and has been explicitly allowed to access the special replication
database in the pg_hba.conf file. After enabling these changes, a
configuration reload is done.
Enabling a slave requires a recent backup to be restored. Since streaming
replication bootstraps the recovery system, a recovery.conf file is written to
the PGDATA
directory containing the connection details for the master that
should be replicated from. A restart is required to enable continuous recovery
mode.
To detach a slave in PostgreSQL means to stop recovery mode. This is done by
writing a special trigger file, configured with the trigger_file
option in
the recovery configuration.
To demote a master requires no special action other than to revert configuration changes to their defaults.
The failover process in Trove is controlled by the task-manager, but the guest agent must implement functions that allow the task manager to determine the best slave to promote and when it can proceed.
Standard PostgreSQL does not support an equivalent of the GTID in MySQL, so the combination of host + WAL location [8] will be used as a transaction identifier where necessary.
A simple polling mechanism will be implemented to determine when a slave has caught up to the point of a particular change.
Failback in postgresql is complicated by recovery timelines [2] . When a slave is triggered out of recovery mode, it jumps to a new timeline, generating new WAL data into a "fork" of the previous database state. This can be seen in the example of these 24-character WAL filenames:
000000010000000000000006 000000020000000000000007
These represent WAL files 6 and 7, but the 7th file is on a second timeline forked from the first.
When a master is demoted, however, it does not change timelines, and so in order to safely reattach this demoted master to a newly-promoted slave, a timeline resync is required.
This can only be done safely through the use of the tool pg_rewind [9]. This tool is supported for PostgreSQL 9.4, but must be compiled separately. In PostgreSQL 9.5 it will be shipped with the core product.
For guests that have pg_rewind available, failback can be done, otherwise a manual recreate of another slave from the master is required.
A number of third-party replication options exist for PostgreSQL, including Slony, pgpool-II and a number of commercially-available solutions [1].
pgpool-II depends on middleware that inserts itself between the client and the underlying database instances. It provides the benefit of multi-master replication, however conflict-resolution may be required in some cases.
Slony provides master-slave replication using table-level triggers. It has greater overhead on the master database than standard streaming replication, but has the benefit of table-level granularity.
TBD (section added after approval)
Primary assignee: atomic77
Target Milestone for completion:
mitaka-1
- implement basic streaming replication
- implement failover-related APIs
- add postgresql-specific hooks as necessary to enable generic int-tests for replication to run against PostgreSQL guests
None.
pg_basebackup incremental backup and restore strategy for PostgreSQL. [3] [10]
Postgresql-specific hooks to the generic int-test framework will be added as necessary.
The documentation will need to be updated to indicate that the PostgreSQL guest supports replication.
None.
[1] | http://www.postgresql.org/docs/9.4/static/different-replication-solutions.html |
[2] | http://www.postgresql.org/docs/9.4/static/continuous-archiving.html |
[3] | http://www.postgresql.org/docs/current/static/app-pgbasebackup.html |
[4] | http://www.postgresql.org/docs/9.0/static/wal-configuration.html |
[5] http://www.postgresql.org/docs/9.4/static/warm-standby-failover.html
[6] | http://www.postgresql.org/docs/current/static/sql-createrole.html |
- [7] http://www.postgresql.org/message-id/flat/CA+TgmobWQJ-GCa_tWUc4=80A
- 1RJ2_+Rq3w_MqaVguk_q018dqw@mail.gmail.com#CA+TgmobWQJ-GCa_tWUc4=80A1RJ 2_+Rq3w_MqaVguk_q018dqw@mail.gmail.com
[8] | By "WAL location" we mean the position in the WAL file, as would be
returned by the pg_current_xlog_location() system
administration function |
[9] | (1, 2, 3) https://github.com/vmware/pg_rewind/tree/REL9_4_STABLE |
[10] | https://blueprints.launchpad.net/trove/+spec/postgresql-incremental-backup |