Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
There appears to be a regression introduced in 2.8.3 (#913) where a logical replication cursor will be advanced whether the user wants it to be or not. This may be specific to
Our use case involves aggregating several logical replication messages before sending them on to our consumer , and we only want to advance the WAL
Here's a program that reproduces the behavior:
Create a table in
However, once the status interval elapses (set here to 1 second so it's quick), the WAL position is advanced beyond the event. Restarting the script doesn't print the event. This happens with both
If the slot is created with
Our current workaround is to
As I understand it, #913 aims to cover that latter behavior (advancing the WAL position when we've read everything), but there may be a bug in the implementation.
 AWS Kinesis. We have many small database transactions, and we want to group them into one big
This is a very interesting bug.
Meanwhile in wal2json there is only one message per change and
Please pay attention on
The fix is very simple and straitforward, I will create a separate variable where will record the flush LSN explicitly set by calling the
167: Update psycopg2 requirement from ~=2.8.3 to ~=2.8.4 in /config r=MiklerGM a=dependabot-preview[bot] Updates the requirements on [psycopg2](https://github.com/psycopg/psycopg2) to permit the latest version. <details> <summary>Changelog</summary> *Sourced from [psycopg2's changelog](https://github.com/psycopg/psycopg2/blob/master/NEWS).* > Current release > --------------- > > What's new in psycopg 2.8.4 > ^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > - Fixed building with Python 3.8 (
🎫`[#854](psycopg/psycopg2#854). > - Don't swallow keyboard interrupts on connect when a password is specified > in the connection string ( 🎫`[#898](psycopg/psycopg2#898). > - Don't advance replication cursor when the message wasn't confirmed > ( 🎫`[#940](psycopg/psycopg2#940). > - Fixed inclusion of ``time.h`` on linux ( 🎫`[#951](psycopg/psycopg2#951). > - Fixed int overflow for large values in `~psycopg2.extensions.Column.table_oid` > and `~psycopg2.extensions.Column.type_code` ( 🎫`[#961](psycopg/psycopg2#961). > - `~psycopg2.errorcodes` map and `~psycopg2.errors` classes updated to > PostgreSQL 12. > - Wheel package compiled against OpenSSL 1.1.1d and PostgreSQL at least 11.4. > > > What's new in psycopg 2.8.3 > ^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > - Added *interval_status* parameter to > `~psycopg2.extras.ReplicationCursor.start_replication()` method and other > facilities to send automatic replication keepalives at periodic intervals > ( 🎫`[#913](psycopg/psycopg2#913). > - Fixed namedtuples caching introduced in 2.8 ( 🎫`[#928](psycopg/psycopg2#928). > > > What's new in psycopg 2.8.2 > ^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > - Fixed `~psycopg2.extras.RealDictCursor` when there are repeated columns > ( 🎫`[#884](psycopg/psycopg2#884). > - Binary packages built with openssl 1.1.1b. Should fix concurrency problems > ( 🎟`[#543](psycopg/psycopg2#543), [#836](psycopg/psycopg2#836). > > > What's new in psycopg 2.8.1 > ^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > - Fixed `~psycopg2.extras.RealDictRow` modifiability ( 🎫`[#886](psycopg/psycopg2#886). > - Fixed "there's no async cursor" error polling a connection with no cursor > ( 🎫`[#887](psycopg/psycopg2#887). > > > What's new in psycopg 2.8 > ------------------------- > > New features: ></tr></table> ... (truncated) </details> <details> <summary>Commits</summary> - See full diff in [compare view](https://github.com/psycopg/psycopg2/commits) </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com): - Update frequency (including time of day and day of week) - Pull request limits (per update run and/or open at any time) - Automerge options (never/patch/minor, and dev/runtime dependencies) - Out-of-range updates (receive only lockfile updates, if desired) - Security updates (receive only security updates, if desired) </details> Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
I am still getting this behavior with version 2.8.4
Even though I am not sending any feedback to the server, the
I performed another test and kept better record of the positions
It seems it only automatically advances the
No, it doesn't. There is no direct relation between
Hi @CyberDem0n ,
I am not really following what you mean.
RE 1. I don't think postgres sends a keep-alive message. It is up to the client to send the keep-alive message. Postgres will simply disconnect the client if no keep-alive was received before
RE 2. Regardless of the reason the WAL position (
I have performed another test and set the
This had the same result of the auto advancing the
This is just your assumption, not based on facts. When you are unsure about something - read the documentation. If it is still not clear - read the source code. I did both. If you still don't trust me - compile and run psycopg2 with debug, it will produce a lot of messages like here.
The psycopg2 is sending Standby status update messages, not keep-alive!
Does it makes sense?
This behavior shouldn't break your application logic. Your app doesn't get the
The documentation link you sent is for the Streaming Replication Protocol , in other words the way the client interacts with the server. The client is required to send the keep-alive message.
I appreciate you wanting to help, but please, no need to be rude.
As far as I know (I dug into the postgres/psycopg2/wal2json source when I was debugging this issue a few months ago), everything @CyberDem0n said above is correct and relevant. The server is sending keepalive messages . His response was (helpfully) verbose and precise and, in my opinion, not really rude. I believe it explains what you're seeing. I suggest rereading it.
 Note "The payload of each CopyData message from server to the client contains a message of one of the following formats:" in the linked docs.
Exactly on this page there is a section about logical replication protocol:
Logical replication protocol is using messages in the same format as physical replication. Also the documentation is 100% clear about terminology, the postgres server sends
I am very sorry about that, I don't really want to be rude.
@jbergknoff-rival I hope the issue you reported was solved in 2.8.4.
Apology completely accepted ... lets meet for a beer sometime, I am buying.
Also ... please let's continue the discussion
I believe, when I said the client sends the keep-alive message, what I really meant is the client sends a status-update message. And this status-update message is required to avoid the server disconnecting the session when
I have spent many hours reading documentation and source code, but many more testing behavior, as that is what I need to understand when making design decisions.
I see this behavior only when position X is the same or higher than
I thank you for this useful information, and especially thank you for your time.