MySql Restart

It is normal that users need to bring down the chef-bach hadoop test cluster nodes since these cluster nodes can be VMs on a personal laptop and the laptops had to be rebooted. When the hypervisor host (laptop) gets rebooted and the cluster nodes are brought up most of the hadoop services start-up on node VM boot. But the mysql service which will not be brought up since the service expects one other instance to be running to start successfully. This is due to HA set-up of mysql service. Users need to bring up one instance of mysql service manually and following are the steps

The node on which you start the mysql service first should be the one on which mysql service was stopped last

• Stop mysql on all nodes
• Find out the node on which mysql was started last using the following command on each node

sudo find /var/lib/mysql -type f -a -name 'ib*' -printf '%T+ %p\n' | sort -r | head -1

• Logon to the node on which mysql was stopped last.
• As root user edit /etc/mysql/conf.d/wsrep.cnf
• Search for wsrep_urls

wsrep_urls="gcomm://10.0.100.11:4567,gcomm://10.0.100.12:4567,"

• Change wsrep_urls to

wsrep_urls="gcomm://"

• As root, start the mysql service on the node

service mysql start 

• Logon to bcpc-bootstrap node and change to chef-bcpc directory
• Run cluster-assign-roles.sh on the second head node where mysql service is not started manually

./cluster-assign-roles.sh Test-Laptop hadoop bcpc-vmX

• Once the chef client is run successfully on the first head node , repeat the same on the second head node

./cluster-assign-roles.sh Test-Laptop hadoop bcpc-vmY

• User can verify that the mysql is running fine by bringing up the Graphite URL https://10.0.100.5:8888