Cluster Installation Fails while starting MySQL Service

[amithkanand]

MySQL database services doesn't come up while building a new cluster. Initial investigation shows that there is change in the version of Percona software that is now being pulled from upstream repository. It looks like the new version requires some new configuraiton. Further investigation is needed to figure out which configuration now needs to go in to support the latest version of Percona.

[amithkanand]

During my testing I found out that the latest release of percona doesn't support wsrep_urls parameter. It need to be replaced with wsrep_cluster_address in wsrep.cnf. There are some more enhancement w.r.t to xtrabackup settings in the new release. I will provide a PR after testing is complete.

[cbaenziger]

For reference, the error one sees will be akin to the following on the slave MySQL instance's /var/lib/mysql/$(hostname).err:

2014-11-28 21:53:33 15077 [Note] WSREP: Service thread queue flushed.
2014-11-28 21:53:33 15077 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not ma
tch group state UUID (dff6f979-776f-11e4-8483-070d8ff82f5c): 1 (Operation not permitted)
         at galera/src/replicator_str.cpp:prepare_for_IST():456. IST will be unavailable.

...

2014-11-28 21:53:49 15077 [Note] WSREP: Member 0.0 (bcpc-vm1) synced with group.
WSREP_SST: [ERROR] xtrabackup process ended without creating '/var/lib/mysql//xtrabackup_galera_info' (20141128 21:53:49.483)
WSREP_SST: [INFO] Contents of datadir (20141128 21:53:49.490)
WSREP_SST: [INFO] -rw-rw---- 1 mysql mysql    8656 Nov 28 21:34 /var/lib/mysql//graphite/account_mygraph.frm

[pchandra]

I just made a bunch of commits to BCPC to migrate to using wsrep_cluster_address that might be helpful:

• bloomberg/chef-bcpc@ec6ec426b23dec87cc6d9480af700a43869803c3
• bloomberg/chef-bcpc@196862af8e66e7368b236747a259d070543c578b
• bloomberg/chef-bcpc@16243cd72683d8aaef654e936d7b2d705a07426c

[bijugs]

Changing wsrep_sst_method to use xtrabackup-v2 seem to bring up the mysql cluster. Raised PR #42 for that and noticed the (early morning!!) update from @pchandra which includes this change.

@cbaenziger , if we are cherry picking the commits from chef-bcpc, we can close the new PR or I can add the additional changes into it.

[amithkanand]

Thanks @pchandra for providing changes for this issue. I have merged your changes into PR #43 with a slight difference in the method name that we use to get all the MySQL nodes. I did test the replication on all MySQL instances and it works fine.