Bug30799097 MTR: ADD SUPPORT TO STOP AND START SECONDARY ENGINE SERVERS WITHIN TESTS

Issue
=====
If one or more secondary engine server nodes go down during
execution of a test, it is interpreted as a failure. However,
there may be cases where one actually wants to stop and possibly
restart one or more secondary engine servers intentionally.

Fix
===
A mechanism to let the system know if the test expects
secondary server(s) to terminate, and to either wait or have
it restarted immediately has been implemented.

Before the action that will stop the server is initiated, the
test case should write either "wait" or "restart" to the file
"$MYSQLTEST_VARDIR/tmp/<secondary_server_name>.<server_id>.expect".

If "restart" is written into this file, the server will be
immediately restarted. If "wait" is written instead, the server
will remain down, but can be restarted at a later time by
writing "restart" to the same file.

In addition, the patch also includes writing the PID of the
safe_process instance for each secondary engine server (i.e.,
the process which spawns and monitors the secondary engine
server's process) to a file in the MTR run directory i.e.,
"$MYSQLTEST_VARDIR/run/<secondary_server_name>.<server_id>.pid".
This can inturn be used to terminate an individual secondary
server process within tests by sending an appropriate signal.

Change-Id: I3c4d4a810f7389d06f8a3b3134fbc54a2bfae416

Author: Srikanth B R

mysql-test/mysql-test-run.pl

@@ -5553,7 +5553,7 @@ ($)
   return $res if $res;
 }
 
-# Loop through our list of processes and look for and entry with the
+# Loop through our list of processes and look for an entry with the
 # provided pid, if found check for the file indicating expected crash
 # and restart it.
 sub check_expected_crash_and_restart($$) {
@@ -5631,6 +5631,10 @@ ($$)
     }
   }
 
+  if ($tinfo->{'secondary-engine'}) {
+    return check_expected_secondary_server_crash_and_restart($proc, $tinfo);
+  }
+
   # Not an expected crash
   return 0;
 }