Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"conflict state 7 after post commit" with async replication and CC #181

Closed
philip-galera opened this issue Sep 2, 2015 · 4 comments
Closed
Assignees
Labels
bug
Milestone

Comments

@philip-galera
Copy link
Contributor

@philip-galera philip-galera commented Sep 2, 2015

A CC while async replication is running can cause the following message to be printed repeatedly:

2015-09-02 11:45:23 45002 [Warning] WSREP: thd 8 seqno 1062: conflict state 7 after post commit
@philip-galera

This comment has been minimized.

Copy link
Contributor Author

@philip-galera philip-galera commented Sep 2, 2015

To reproduce:

  1. Replace mysql-test/suite/galera/t/galera_as_slave.cnf with the following:
!include include/default_mysqld.cnf

[mysqld]
log-slave-updates
log-bin=mysqld-bin
binlog-format=row

[mysqld.1]
server-id=1

[mysqld.2]
server-id=2

wsrep_provider=@ENV.WSREP_PROVIDER
wsrep_cluster_address='gcomm://'
wsrep_provider_options='base_port=@mysqld.2.#galera_port;evs.install_timeout = PT15S; evs.max_install_timeouts=1;'

# enforce read-committed characteristics across the cluster
wsrep_causal_reads=ON
wsrep_sync_wait = 7

wsrep_node_address=127.0.0.1
wsrep_sst_receive_address=127.0.0.2:@mysqld.2.#sst_port
wsrep_node_incoming_address=127.0.0.1:@mysqld.2.port

# Required for Galera
innodb_autoinc_lock_mode=2

innodb_flush_log_at_trx_commit=2

[mysqld.3]
wsrep_provider=@ENV.WSREP_PROVIDER
wsrep_cluster_address='gcomm://127.0.0.1:@mysqld.2.#galera_port'
wsrep_provider_options='base_port=@mysqld.3.#galera_port;evs.install_timeout = PT15S; evs.max_install_timeouts = 1;'

# enforce read-committed characteristics across the cluster
wsrep_causal_reads=ON
wsrep_sync_wait = 7

wsrep_node_address=127.0.0.1
wsrep_sst_receive_address=127.0.0.2:@mysqld.3.#sst_port
wsrep_node_incoming_address=127.0.0.1:@mysqld.3.port

# Required for Galera
innodb_autoinc_lock_mode=2

innodb_flush_log_at_trx_commit=2

[mysqld.4]
wsrep_provider=@ENV.WSREP_PROVIDER
wsrep_cluster_address='gcomm://127.0.0.1:@mysqld.2.#galera_port'
wsrep_provider_options='base_port=@mysqld.4.#galera_port;evs.install_timeout = PT15S; evs.max_install_timeouts = 1;'

# enforce read-committed characteristics across the cluster
wsrep_causal_reads=ON
wsrep_sync_wait = 7

wsrep_node_address=127.0.0.1
wsrep_sst_receive_address=127.0.0.2:@mysqld.4.#sst_port
wsrep_node_incoming_address=127.0.0.1:@mysqld.4.port

# Required for Galera
innodb_autoinc_lock_mode=2

innodb_flush_log_at_trx_commit=2

[ENV]
NODE_MYPORT_1= @mysqld.1.port
NODE_MYSOCK_1= @mysqld.1.socket

NODE_MYPORT_2= @mysqld.2.port
NODE_MYSOCK_2= @mysqld.2.socket

NODE_MYPORT_3= @mysqld.3.port
NODE_MYSOCK_3= @mysqld.3.socket

NODE_MYPORT_3= @mysqld.4.port
NODE_MYSOCK_3= @mysqld.4.socket

NODE_GALERAPORT_2= @mysqld.2.#galera_port
NODE_GALERAPORT_3= @mysqld.3.#galera_port
NODE_GALERAPORT_4= @mysqld.4.#galera_port

NODE_SSTPORT_2= @mysqld.2.#sst_port
NODE_SSTPORT_3= @mysqld.3.#sst_port
NODE_SSTPORT_4= @mysqld.4.#sst_port
  1. Run:
 perl mysql-test-run.pl --suite galera galera_as_slave --start-and-exit

This will start 1 stand-alone server and 3 galera nodes

  1. Run:
mysql -uroot --protocol=tcp --port=13004 -e "CHANGE MASTER TO  MASTER_HOST='127.0.0.1', MASTER_PORT=13000"
mysql -uroot --protocol=tcp --port=13004 -e "START SLAVE USER='root'"

mysql -uroot --protocol=tcp --port=13000 test -e "CREATE TABLE IF NOT EXISTS t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT) ENGINE=InnoDB";
mysql -uroot --protocol=tcp --port=13000 test -e "CREATE TABLE IF NOT EXISTS t2 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT) ENGINE=InnoDB";


sh -c 'for i in {1..1000}; do mysql -uroot --protocol=tcp --port=13000 test -e "insert into t1 values (null)"; sleep 0.5; done ;' &
sh -c 'for i in {1..1000}; do mysql -uroot --protocol=tcp --port=13004 test -e "insert into t2 values (null)"; sleep 0.5; done ;' &


for i in {1..1000}; do mysql -uroot --protocol=tcp --port=13012 test -e "SET GLOBAL wsrep_provider_options='gmcast.isolate=1'";  sleep 5.5; mysql -uroot --protocol=tcp --port=13012 test -e "SET GLOBAL wsrep_provider_options='gmcast.isolate=0'"; sleep 5.5; done 

This will establish replication, start a workload and then disrupt the third node via gmcast.isolate .

  1. Observe the log on the async slave:
[philips@localhost mysql-test]$ cat var/log/mysqld.2.err  | grep conflict
...
2015-09-02 11:53:39 50794 [Warning] WSREP: thd 8 seqno 457: conflict state 7 after post commit
2015-09-02 11:53:40 50794 [Warning] WSREP: thd 8 seqno 458: conflict state 7 after post commit
2015-09-02 11:53:40 50794 [Warning] WSREP: thd 8 seqno 460: conflict state 7 after post commit
2015-09-02 11:53:41 50794 [Warning] WSREP: thd 8 seqno 463: conflict state 7 after post commit
2015-09-02 11:53:41 50794 [Warning] WSREP: thd 8 seqno 464: conflict state 7 after post commit
2015-09-02 11:53:47 50794 [Warning] WSREP: thd 8 seqno 466: conflict state 7 after post commit
2015-09-02 11:53:47 50794 [Warning] WSREP: thd 8 seqno 467: conflict state 7 after post commit
2015-09-02 11:53:47 50794 [Warning] WSREP: thd 8 seqno 469: conflict state 7 after post commit
2015-09-02 11:53:47 50794 [Warning] WSREP: thd 8 seqno 470: conflict state 7 after post commit
2015-09-02 11:53:47 50794 [Warning] WSREP: thd 8 seqno 471: conflict state 7 after post commit
2015-09-02 11:53:47 50794 [Warning] WSREP: thd 8 seqno 472: conflict state 7 after post commit
2015-09-02 11:53:47 50794 [Warning] WSREP: thd 8 seqno 473: conflict state 7 after post commit
2015-09-02 11:53:47 50794 [Warning] WSREP: thd 8 seqno 474: conflict state 7 after post commit
2015-09-02 11:53:47 50794 [Warning] WSREP: thd 8 seqno 475: conflict state 7 after post commit
2015-09-02 11:53:47 50794 [Warning] WSREP: thd 8 seqno 476: conflict state 7 after post commit
2015-09-02 11:53:47 50794 [Warning] WSREP: thd 8 seqno 478: conflict state 7 after post commit

The issue should show up after a few CC events, within 60 seconds.

@philip-galera

This comment has been minimized.

Copy link
Contributor Author

@philip-galera philip-galera commented Sep 2, 2015

If you convert the warning into an assertion I may be able to come up with a pure-MTR test case to be pushed into the regression test suite.

sjaakola added a commit that referenced this issue Sep 7, 2015
refs #181
- Galera related errors in mysql slave applying will now cause slave to abort
philip-galera added a commit that referenced this issue Sep 7, 2015
refs #181
philip-galera added a commit that referenced this issue Sep 8, 2015
refs #188, refs #181
@philip-galera

This comment has been minimized.

Copy link
Contributor Author

@philip-galera philip-galera commented Sep 8, 2015

Fixed and pushed. Commits
8c328d2
a566bcf
a35c12f

@philip-galera

This comment has been minimized.

Copy link
Contributor Author

@philip-galera philip-galera commented Sep 8, 2015

See also #188

nirbhayc pushed a commit to MariaDB/server that referenced this issue Sep 10, 2015
- Galera related errors in mysql slave applying will now cause slave to
abort
temeo pushed a commit that referenced this issue Jul 3, 2017
IN-12: fix SLES-based spec
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.