*: health check enhancement

[zhl003]

What type of PR is this?

/enhancement

Which issue(s) this PR fixes?

Fixes #739

What this PR does?

Summary:

1. Add a parameter to control the delay from the replica to primary, If the current latency exceeds the expected value ,the pod while be not ready

2. move all mysql pod health check and other checks to mysqlchecker in mysql container

Special notes for your reviewer?

how to verify

1. Use chaos to inject a disk delay error to create master-slave delay

kind: IOChaos
apiVersion: chaos-mesh.org/v1alpha1
metadata:
  namespace: default
  name: disk-latency
  annotations:
    experiment.chaos-mesh.org/pause: 'false'
spec:
  selector:
    namespaces:
      - default
    labelSelectors:
      app.kubernetes.io/instance: sample
    pods:
      default:
        - sample-mysql-1
  mode: all
  action: latency
  delay: 3000ms
  path: /var/lib/mysql
  percent: 100
  volumePath: /var/lib/mysql

2. sysbench load alarge amount data to cluster ,observe the lagency ,and the pod state

 kubectl exec -it sample-mysql-0 -c xenon -- xenoncli cluster mysql
+------------------------------------------+----------+-------+-----------+------------------------------+----------------+----------------+------------+
|                    ID                    |   Raft   | Mysql |  Option   |     Master_Log_File/Pos      | IO/SQL_Running | Seconds_Behind | Last_Error |
+------------------------------------------+----------+-------+-----------+------------------------------+----------------+----------------+------------+
| sample-mysql-0.sample-mysql.default:8801 | LEADER   | ALIVE | READWRITE | [mysql-bin.000005/406456643] | [true/true]    |              0 |            |
+------------------------------------------+----------+-------+-----------+------------------------------+----------------+----------------+------------+
| sample-mysql-1.sample-mysql.default:8801 | FOLLOWER | ALIVE | READONLY  | [mysql-bin.000005/406456643] | [true/true]    |             50 |            |
+------------------------------------------+----------+-------+-----------+------------------------------+----------------+----------------+------------+
| sample-mysql-2.sample-mysql.default:8801 | FOLLOWER | ALIVE | READONLY  | [mysql-bin.000005/406456643] | [true/true]    |              0 |            |
+------------------------------------------+----------+-------+-----------+------------------------------+----------------+----------------+------------+

kubectl get po -Lrole,healthy -l app.kubernetes.io/instance=sample
NAME             READY   STATUS    RESTARTS        AGE     ROLE       HEALTHY
sample-mysql-0   3/3     Running   0               2d23h   LEADER     yes
sample-mysql-2   3/3     Running   0               2d23h   FOLLOWER   yes
sample-mysql-1   2/3     Running   1 (2m38s ago)   2d23h   FOLLOWER   no

kubectl describe  po sample-mysql-1
time="2022-11-28T14:43:17+08:00" level=fatal msg="readiness failed: slave is too far behind master"
  Warning  Unhealthy  2m47s (x55 over 2d21h)  kubelet  (combined from similar events): Readiness probe failed: time="2022-11-28T14:45:17+08:00" level=info msg="output=[{\"state\":\"FOLLOWER\",\"leader\":\"sample-mysql-0.sample-mysql.default:8801\",\"nodes\":[\"sample-mysql-0.sample-mysql.default:8801\",\"sample-mysql-1.sample-mysql.default:8801\",\"sample-mysql-2.sample-mysql.default:8801\"]}]"
time="2022-11-28T14:45:17+08:00" level=info msg="stderr=[]"
time="2022-11-28T14:45:17+08:00" level=fatal msg="readiness failed: slave is too far behind master

[acekingke]

need rebuild the mysql image