Add Support for Streaming Replication Protocol #50
Conversation
This change introduces new messages types to support the streaming protocol. The change also modifies the framer to handle the new types of messages Introduce `executeSimple` for simple queries Unlike execute, this method returns data if there's any; otherwise it returns affected rows. Fix affected rows parsing in CommandCompleteMessage When using non-standard command such as 'IDENTIFY_SYSTEM', the parsing had an issue where it'll parse `SYSTEM` as the integer. A new reliable regex was added to parse the affected rows correctly if there's any
|
@osaxma: thank you for working on this, I think this feature is a good thing to have here. However, the PR is huge, and I'd be more comfortable reviewing it smaller chunks. Do you have any suggestion how to split it up or restructure? |
|
Sure -- I'll split them into multiple smaller PRs... Tho adding all the new message types will be a huge PR nevertheless but I guess you'll notice that as the majority will be in new files. |
|
Added the following PRs so far:
I'll keep this one open as I'll push the final changes to it later |
|
Thank you so much for the smaller-sized PRs, they are easy to follow and review so far, much less frightening than this one :) |
|
I guess that was about it? Let me know if there is anything else, otherwise I shall release the 2.5.0 with this great feature! :) |
I guess it's time to close this one for sure. Thanks a lot for your time in reviewing all my PRs in such a short period and maintaing this package. It's funny because this whole thing was not planned at all -- I was just tinkering with Logical Replication until I found myself writing the entire functionality 😅 Cheers! |
This PR adds support for Streaming Replication Protocol.
The following repository contains an example usage: postgresql-dart-replication-example. While the protocol is intended for creating database replicas, the Logical Replication can be used to listen to changes in the database (e.g. inserts, updates, and deletes).
A summary of additions and changes:
.gitignorequerySimplemethodexecute, but it'll return whatever data the server gives.executedoes not return any data.querySimplewas introduced to avoid breaking the existing API and to allow querying the database in Replication Mode which is necessary (i.e. executingIDENTIFY_SYSTEM;and retrieving the system info data)addMessagemethod to PostgreSQLConnectionStream<ServerMessage> messagesto PostgreSQLConnectionsocketis a single listener stream, there's no way to listen to server messages after opening the connection.notificationsexcept that it'll pass all messages to any listeners.addMessage, the user can take over the communication with the server.START_REPLICATION)shared_messages.dartandlogical_replication_messages.dartclient_messages.dartandserver_messages.dartmessagesfolder but I left that for review.MessageFramer.addBytesto handle new message types and special casesCommandCompleteIDENTIFY_SYSTEM;, theSYSTEMwas parsed as the affected row integerI believe these were the major additions with few changes.
About testing:
I added a test for
logicl_replication_test.dart-- the test might need to be disabled for CI since the test restarts the docker container (necessary afterALTER SYSTEM) which could make any concurrent tests fail. Though I wanted to see how the CI reacts first.I also had difficulty running the tests locally (before adding the test mentioned above). Many tests throw ConnectionError when ran concurrently but when I ran them individually, they pass (had to start docker container manually for those that don't call
usePostgresDocker). Also, all the SSL tests failed.I'm not sure if there's something that I'm missing on how I should run the tests.