New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for streaming replication protocol #322

Merged
merged 68 commits into from Aug 15, 2016

Conversation

Projects
None yet
7 participants
@a1exsh
Contributor

a1exsh commented Jun 1, 2015

Introduce ReplicationConnection and ReplicationCursor classes, that
incapsulate initiation of special type of PostgreSQL connection and
handling of special replication commands only available in this special
connection mode.

The handling of stream of replication data from the server is modelled
largely after the existing support for "COPY table TO file" command and
pg_recvlogical tool supplied with PostgreSQL (though, it can also be
used for physical replication.)

Add support for streaming replication protocol
Introduce ReplicationConnection and ReplicationCursor classes, that
incapsulate initiation of special type of PostgreSQL connection and
handling of special replication commands only available in this special
connection mode.

The handling of stream of replication data from the server is modelled
largely after the existing support for "COPY table TO file" command and
pg_recvlogical tool supplied with PostgreSQL (though, it can also be
used for physical replication.)
Show outdated Hide outdated lib/extras.py Outdated
Show outdated Hide outdated psycopg/pqpath.c Outdated
Show outdated Hide outdated psycopg/pqpath.c Outdated
@dvarrazzo

This comment has been minimized.

Show comment
Hide comment
@dvarrazzo

dvarrazzo Jun 1, 2015

Member

I'd like to look better into this patch: I don't have much experience about replication.

I guess a test in the test suite would be hard to fit, but it'd be nice to have at least an example script.

Member

dvarrazzo commented Jun 1, 2015

I'd like to look better into this patch: I don't have much experience about replication.

I guess a test in the test suite would be hard to fit, but it'd be nice to have at least an example script.

Show outdated Hide outdated lib/extras.py Outdated
@dvarrazzo

This comment has been minimized.

Show comment
Hide comment
@dvarrazzo

dvarrazzo Jun 1, 2015

Member

It would be cool if you wanted to have a chat on the ML about what this new feature would provide. Is it something you are already using in production?

Member

dvarrazzo commented Jun 1, 2015

It would be cool if you wanted to have a chat on the ML about what this new feature would provide. Is it something you are already using in production?

@a1exsh

This comment has been minimized.

Show comment
Hide comment
@a1exsh

a1exsh Jun 1, 2015

Contributor

This is experimental code, it is not yet used in production.

The main goal is to provide a python client for logical decoding stream of WAL log entries, a feature that has been added in 9.4.

Contributor

a1exsh commented Jun 1, 2015

This is experimental code, it is not yet used in production.

The main goal is to provide a python client for logical decoding stream of WAL log entries, a feature that has been added in 9.4.

@a1exsh

This comment has been minimized.

Show comment
Hide comment
@a1exsh

a1exsh Jun 2, 2015

Contributor

Speaking of example code, this is what I'm currently using only for testing:

from io import TextIOBase
import sys
import psycopg2
from psycopg2.extras import ReplicationConnection, REPLICATION_LOGICAL

class LogicalWriter(TextIOBase): #(object)
    def write(self, buf):
        sys.stdout.write(buf)
        sys.stdout.write("\n")
        return True

writer = LogicalWriter()

conn = psycopg2.connect(dsn, connection_factory=ReplicationConnection)
cur = conn.cursor()

#cur.create_replication_slot(REPLICATION_LOGICAL, slot, "bottledwater")
cur.start_replication(writer, REPLICATION_LOGICAL, slot_name=slot,
                      options={'format': 'json'})

On the server side we use a fork of bottledwater, compiled with JSON output support: https://github.com/a1exsh/bottledwater-pg/tree/feature/json-smallest-workable

What raises the most questions for us is the protocol for the Writer object to be passed to start_replication method. I think it would be nice to pass some metadata to the write() callback that is available to the pqpath.c only at the moment, such as: values of dataStart, walEnd and sendTime fields from the message header, probably also written_lsn and fsync_lsn.

One can either pass these as positional parameters or keyword arguments, but that isn't very extensible, and it breaks the contract for sys.stdout (which is useful only for testing, but still useful).

I believe the most extensible way is to pass an object that can have attributes for the metadata, as well as methods to control reporting the fsync_lsn to the server, stopping replication (right now the only way for that is to throw an exception and catch it around start_replication call), and so on. To make it compatible with sys.stdout usecase, one can even inherit the class from str, like this:

class ReplicationMessage(str):
    #wal_end
    #data_start
    #send_time
    #def commit(self):
    #def stop(self):
    ...

So instead of cryptic "return True" the client would signal that it has stored the incoming message with msg.commit(). And if for some reason it needs to stop processing the stream, there is msg.stop().

Contributor

a1exsh commented Jun 2, 2015

Speaking of example code, this is what I'm currently using only for testing:

from io import TextIOBase
import sys
import psycopg2
from psycopg2.extras import ReplicationConnection, REPLICATION_LOGICAL

class LogicalWriter(TextIOBase): #(object)
    def write(self, buf):
        sys.stdout.write(buf)
        sys.stdout.write("\n")
        return True

writer = LogicalWriter()

conn = psycopg2.connect(dsn, connection_factory=ReplicationConnection)
cur = conn.cursor()

#cur.create_replication_slot(REPLICATION_LOGICAL, slot, "bottledwater")
cur.start_replication(writer, REPLICATION_LOGICAL, slot_name=slot,
                      options={'format': 'json'})

On the server side we use a fork of bottledwater, compiled with JSON output support: https://github.com/a1exsh/bottledwater-pg/tree/feature/json-smallest-workable

What raises the most questions for us is the protocol for the Writer object to be passed to start_replication method. I think it would be nice to pass some metadata to the write() callback that is available to the pqpath.c only at the moment, such as: values of dataStart, walEnd and sendTime fields from the message header, probably also written_lsn and fsync_lsn.

One can either pass these as positional parameters or keyword arguments, but that isn't very extensible, and it breaks the contract for sys.stdout (which is useful only for testing, but still useful).

I believe the most extensible way is to pass an object that can have attributes for the metadata, as well as methods to control reporting the fsync_lsn to the server, stopping replication (right now the only way for that is to throw an exception and catch it around start_replication call), and so on. To make it compatible with sys.stdout usecase, one can even inherit the class from str, like this:

class ReplicationMessage(str):
    #wal_end
    #data_start
    #send_time
    #def commit(self):
    #def stop(self):
    ...

So instead of cryptic "return True" the client would signal that it has stored the incoming message with msg.commit(). And if for some reason it needs to stop processing the stream, there is msg.stop().

@dvarrazzo

This comment has been minimized.

Show comment
Hide comment
@dvarrazzo

dvarrazzo Jun 2, 2015

Member

Hello,

thank you very much for the example. The problem is that this bug tracker is only monitored by few people: I'm sure the mailing list is a better place to discuss about this feature. Some times ago I spoke with some core developer, can't really remember who, and he indeed asked for COPY_BOTH support to play with replication. I couldn't work on the feature yet.

If you post on the ML and also reference the message to -hackers I'm sure you'll get some better look at your patch.

I'll review what you have made anyway but can't do it just now.

Member

dvarrazzo commented Jun 2, 2015

Hello,

thank you very much for the example. The problem is that this bug tracker is only monitored by few people: I'm sure the mailing list is a better place to discuss about this feature. Some times ago I spoke with some core developer, can't really remember who, and he indeed asked for COPY_BOTH support to play with replication. I couldn't work on the feature yet.

If you post on the ML and also reference the message to -hackers I'm sure you'll get some better look at your patch.

I'll review what you have made anyway but can't do it just now.

@a1exsh

This comment has been minimized.

Show comment
Hide comment
@a1exsh

a1exsh Jun 2, 2015

Contributor
Contributor

a1exsh commented Jun 2, 2015

@dvarrazzo

This comment has been minimized.

Show comment
Hide comment
@dvarrazzo

dvarrazzo Jun 2, 2015

A question here: cannot we just use copy_expert here? We may relax the hasattr checks performed there (and just let the pqpath code fail if the file object doesn't have the right attrib). For the rest the two methods look very similar.

dvarrazzo commented on psycopg/cursor_type.c in e32e1b8 Jun 2, 2015

A question here: cannot we just use copy_expert here? We may relax the hasattr checks performed there (and just let the pqpath code fail if the file object doesn't have the right attrib). For the rest the two methods look very similar.

@dvarrazzo

This comment has been minimized.

Show comment
Hide comment
@dvarrazzo

dvarrazzo Jun 2, 2015

maybe all these backend functions and constants are better defined in a pair of file of their own? They also have a different naming convention from the rest of the library and pqpath is already fat enough :)

dvarrazzo commented on psycopg/pqpath.c in e32e1b8 Jun 2, 2015

maybe all these backend functions and constants are better defined in a pair of file of their own? They also have a different naming convention from the rest of the library and pqpath is already fat enough :)

@dvarrazzo dvarrazzo added this to the psycopg 2.7 milestone Jun 2, 2015

@a1exsh

This comment has been minimized.

Show comment
Hide comment
@a1exsh

a1exsh Jun 2, 2015

Contributor

Unfortunately, we cannot use copy_expert because the two-way communication via keepalive messages from client to the server is critical for proper implementation of streaming replication.

One can consume data in the same way using copy_expert, but the server will never know how far the client is in terms of data stored reliably (fsync'ed). As such, the server will retain the WAL files indefinitely, ultimately exhausting the disk space. (Less annoying is the fact that if you stop and restart consuming the repl. stream, you will get the same stream of data again from the beginning, since the server was not told you don't need it anymore.) There is a note about this in the doc part of this patch and it is explained in the Postgres streaming replication docs.

Contributor

a1exsh commented Jun 2, 2015

Unfortunately, we cannot use copy_expert because the two-way communication via keepalive messages from client to the server is critical for proper implementation of streaming replication.

One can consume data in the same way using copy_expert, but the server will never know how far the client is in terms of data stored reliably (fsync'ed). As such, the server will retain the WAL files indefinitely, ultimately exhausting the disk space. (Less annoying is the fact that if you stop and restart consuming the repl. stream, you will get the same stream of data again from the beginning, since the server was not told you don't need it anymore.) There is a note about this in the doc part of this patch and it is explained in the Postgres streaming replication docs.

@a1exsh

This comment has been minimized.

Show comment
Hide comment
@Ormod

This comment has been minimized.

Show comment
Hide comment
@Ormod

Ormod Jun 17, 2015

It would also be nice if there were a convenience function for streaming a basebackup tar a la pg_basebackup. (to a given file like object)

Ormod commented Jun 17, 2015

It would also be nice if there were a convenience function for streaming a basebackup tar a la pg_basebackup. (to a given file like object)

a1exsh added some commits Jun 30, 2015

Rework replication protocol
This change exposes lower level functions for operating the
(logical) replication protocol, while keeping the high-level
start_replication function that does all the job for you in
case of a synchronous connection.

A number of other changes and fixes are put into this commit.

Ormod added a commit to Ormod/pghoard that referenced this pull request Jun 2, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jun 2, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jun 2, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jun 3, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jun 3, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jun 3, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jun 6, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:
psycopg/psycopg2#322

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jun 6, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:
psycopg/psycopg2#322

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jun 6, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:
psycopg/psycopg2#322

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jul 5, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:
psycopg/psycopg2#322

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

    psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jul 5, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:
psycopg/psycopg2#322

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

    psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jul 5, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:
psycopg/psycopg2#322

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

    psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jul 5, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:
psycopg/psycopg2#322

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

    psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jul 5, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:
psycopg/psycopg2#322

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

    psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jul 5, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:
psycopg/psycopg2#322

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

    psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jul 5, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:
psycopg/psycopg2#322

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

    psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jul 5, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:
psycopg/psycopg2#322

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

    psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)
@PJMODOS

This comment has been minimized.

Show comment
Hide comment
@PJMODOS

PJMODOS Jul 6, 2016

I've been discussing this patch with @dvarrazzo at pgconf.uk and I just want to repeat my opinion here. I have been using this for a while in testing scripts for pglogical and am generally happy with it in the current form. I think it's ready for merge.

PJMODOS commented Jul 6, 2016

I've been discussing this patch with @dvarrazzo at pgconf.uk and I just want to repeat my opinion here. I have been using this for a while in testing scripts for pglogical and am generally happy with it in the current form. I think it's ready for merge.

Ormod added a commit to Ormod/pghoard that referenced this pull request Jul 6, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:
psycopg/psycopg2#322

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

    psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jul 6, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:
psycopg/psycopg2#322

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

    psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jul 14, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:
psycopg/psycopg2#322

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

    psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jul 14, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:
psycopg/psycopg2#322

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

    psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jul 14, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:
psycopg/psycopg2#322

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

    psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jul 14, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:
psycopg/psycopg2#322

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

    psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jul 14, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:
psycopg/psycopg2#322

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

    psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jul 14, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:
psycopg/psycopg2#322

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

    psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

Ormod added a commit to Ormod/pghoard that referenced this pull request Jul 14, 2016

walreceiver: Add experimental support for replicating WALs via PG rep…
…lication protocol

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:
psycopg/psycopg2#322

You can now use "active_backup_mode": "walreceiver" in case you have a psycopg2
that supports the PostgreSQL replication protocol. At this time the support hasn't
been merged into psycopg2 master but can be found from:

    psycopg/psycopg2#322

The backup method "walreceiver" writes no extra WAL data to disk on the machine
taking the backup.

Before this when using pg_receivexlog mode pghoard first wrote the files uncompressed
on disk once and then compressed them and wrote them on disk again causing roughly
1.5x the size of WAL writes just so pghoard could back the files up.

We now also write the last flush_lsn position into pghoard state file and read
it back from there when creating a walreceiver so we can continue from the last
known position. Note that this will fail in case pghoard was shutdown uncleanly.
(kill -9 or such)

@dvarrazzo dvarrazzo merged commit d5443c6 into psycopg:master Aug 15, 2016

@dvarrazzo

This comment has been minimized.

Show comment
Hide comment
@dvarrazzo

dvarrazzo Aug 15, 2016

Member

Merged to master. Testing would be appreciated. Thank you very much to all the contributors!

Member

dvarrazzo commented Aug 15, 2016

Merged to master. Testing would be appreciated. Thank you very much to all the contributors!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment