Segmentation fault when using last_update_wins

[jouwerkerk]

Hello,

I'm experimenting with a 2-node bidirectional replication using pglogical 1.1.2. I'm able to consistently cause a seg fault when I set pglogical.conflict_resolution = 'last_update_wins', and when either:

1. Executing simultaneous identical UPDATE statements on both nodes (using psql and terminator) or
2. Disabling both subscriptions, executing UPDATE statements on both nodes (that will cause a conflict) and then re-enabling one or both subscriptions.

I'm expecting that the last_update_wins strategy will do the right thing and either 'keep local' or 'apply remote' based on the commit timestamp. Instead, I'm seeing an attempt to resolve the conflict (e.g. keep local), a crash of PostgreSQL, an automatic restart, and then a second attempt to resolve the conflict (e.g. apply remote).

For example, in pg_log/postgresql-Wed.log:

worker process: pglogical apply 16384:1871150101 (PID 19795) was terminated by signal 11: Segmentation fault

Question: Is using pglogical in this way (bidirectional) not recommended at this point in time?

Some findings: Using this 2-node bidirectional-subscription setup, I have had success with simple INSERT, UPDATE and DELETE statements that don't require conflict resolution. And I've had success with INSERT and DELETE statements that do require conflict resolution. It's the UPDATE statements that are blocking me. The first_update_wins strategy also triggers the seg fault. The apply_remote does not.

Thank you,

...Jelle

p.s. RPMS:
postgresql95-server-9.5.3-2PGDG.rhel6.x86_64
postgresql95-contrib-9.5.3-2PGDG.rhel6.x86_64
postgresql95-pglogical-1.1.2-1_2ndQuadrant.el6.x86_64
postgresql95-pglogical-output-1.1.0-1_2ndQuadrant.el6.x86_64

p.p.s. My config

wal_level = 'logical'
max_worker_processes = 10
max_replication_slots = 10
max_wal_senders = 10
shared_preload_libraries = 'pglogical'
track_commit_timestamp = on
pglogical.conflict_resolution = 'last_update_wins'

p.p.p.s.
I tried to use gdb to get a backtrace but the resulting gdb.txt was full of "No symbol table info available."
This may help:

Core was generated by `postgres: bgworker: pglogical apply'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000000007c683b in ?? ()

p.p.p.p.s.
Using bdr would be ideal: I'm looking for async multi-master replication. But using a patched PostgreSQL 9.4 is a show-stopper. Should I just wait for a future version of PostgreSQL to have in-core BDR support?

[PJMODOS]

This is interesting. I would be grateful if you could provide some script for reproducing this.

About your questions, bidirectional should work with some limitations compared to BDR:

• the initial data sync is not going to be fully consistent in case of having multiple nodes that are being written to
• there are no global sequences
• there is no transparent DDL rep

If you can live with those you can use it in bidirectional mode (provided we fix this issue).

[PJMODOS]

Actually I believe I have fixed the issue in commit fd8e505

[jouwerkerk]

Thank you for the prompt fix. I've tested again using a patched pglogical and 'last_update_wins' appears to be behaving as expected. 👍

[pratmeht]

Hi, we too are testing pglogical in bidirectional. We are using pglogical 2.3.2 on Postgres 12.3. Need one help.

While creating Subscription, I am setting forward_origins := '{}', and my Subscriptions are enabled, still if I do UPDATE on any 1 of the 2 bi-directional sites, I see a messages of "CONFLICT: remote UPDATE on relation <schema_name>.<table_name>" in Postgres log file.

Is this normal or am I missing something ? Do we know why does this happen ? Is there a way to stop these messages ?

BTW, I too had experienced Sgementation Fault once (not due to 'last_update_wins' but when I was trying to resync a table. It was kicked-out of psql prompt. It happened only once, Not sure what would be the reason!