Support require_secure_transport=ON

[masa213f]

What

A user wants to enforce encrypted communication within a k8s cluster using the require_secure_transport=ON option.
However, the current MOCO implementation does not suppose encrypted communication. Some updates are needed.

How

Describe how to address the issue.

Checklist

• Finish implementation of the issue
• Test all functions
• Have enough logs to trace activities
• Notify developers of necessary actions

[masa213f]

First, as @daichimukai pointed out in cybozu-go/moco-agent#83, the agent container gets an error.

1. Apply the follwoing manifests.

apiVersion: moco.cybozu.com/v1beta1
kind: MySQLCluster
metadata:
  name: test
spec:
  mysqlConfigMapName: test
  replicas: 3
  podTemplate:
    spec:
      containers:
      - name: mysqld
        image: quay.io/cybozu/mysql:8.0.30
  volumeClaimTemplates:
  - metadata:
      name: mysql-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: test
data:
  require_secure_transport: "ON"

2. Then the moco-agent does not start.

$ kubectl get pod
NAME          READY   STATUS             RESTARTS      AGE
moco-test-0   1/3     CrashLoopBackOff   4 (80s ago)   3m19s
moco-test-1   1/3     CrashLoopBackOff   4 (80s ago)   3m19s
moco-test-2   1/3     CrashLoopBackOff   4 (76s ago)   3m19s

$ kubectl logs -n sandbox moco-test-0 -c agent -p
Error: Error 3159: Connections using insecure transport are prohibited while --require_secure_transport=ON.
Usage:
  moco-agent [flags]

Flags:
      --address string                 Listening address and port for gRPC API. (default ":9080")
      --connection-timeout duration    Dial timeout (default 5s)
      --grpc-cert-dir string           gRPC certificate directory (default "/grpc-cert")
  -h, --help                           help for moco-agent
      --log-rotation-schedule string   Cron format schedule for MySQL log rotation (default "*/5 * * * *")
      --logfile string                 Log filename
      --logformat string               Log format [plain,logfmt,json]
      --loglevel string                Log level [critical,error,warning,info,debug]
      --max-delay duration             Acceptable max commit delay considering as ready; the zero value accepts any delay (default 1m0s)
      --max-idle-time duration         The maximum amount of time a connection may be idle (default 30s)
      --metrics-address string         Listening address and port for metrics. (default ":8080")
      --probe-address string           Listening address and port for mysqld health probes. (default ":9081")
      --read-timeout duration          I/O read timeout (default 30s)
      --socket-path string             Path of mysqld socket file. (default "/run/mysqld.sock")

Error 3159: Connections using insecure transport are prohibited while --require_secure_transport=ON.

[masa213f]

Next, the cluster-manager will cause errors.

{"level":"error","ts":"2023-07-24T07:33:49Z","logger":"cluster-manager.default/test","msg":"failed to get mysqld status","operationId":"op-m4ffp","error":"failed to get global variables: pod=moco-test-2, namespace=default: failed to get mysql global variables: Error 3159: Connections using insecure transport are prohibited while --require_secure_transport=ON.","stacktrace":"github.com/cybozu-go/moco/clustering.(*managerProcess).GatherStatus.func2\n\t/work/clustering/status.go:206"}

[masa213f]

Even if the above errors are resolved, replication will fail, and the MySQLCluster will not be HEALTHY.

$ kubectl get pod -l app.kubernetes.io/name=mysql
NAME          READY   STATUS    RESTARTS   AGE
moco-test-0   3/3     Running   0          4m1s
moco-test-1   2/3     Running   0          4m1s
moco-test-2   2/3     Running   0          4m1s

$ kubectl get mysqlcluster
NAME   AVAILABLE   HEALTHY   PRIMARY   SYNCED REPLICAS   ERRANT REPLICAS   LAST BACKUP
test   False       False     0         1                                   <no value>

Log of agent container in moco-test-1.

{"level":"info","ts":1690186048.8364568,"logger":"agent","caller":"server/mysqld_health.go:63","msg":"replication threads are stopped"}
2023-07-24T08:07:28.836730Z moco-test-1 moco-agent error: "well: access" http_host="10.244.2.14:9081" http_method="GET" http_status_code=503 http_user_agent="kube-probe/1.26" protocol="HTTP/1.1" remote_ipaddr="10.244.2.1" request_id="872f3224-6f0c-3d78-99c5-2650c609d452" request_size=0 response_size=31 response_time=0.003915732 type="access" url="/readyz"

Log of mysqld container in moco-test-1.

2023-07-24T08:10:28.830865Z 5238 [Note] [MY-010581] [Repl] Slave SQL thread for channel '' initialized, starting replication in log 'FIRST' at position 0, relay log './moco-test-1-relay-bin.000001' position: 4
2023-07-24T08:10:29.061344Z 5238 [Note] [MY-010596] [Repl] Error reading relay log event for channel '': slave SQL thread was killed
2023-07-24T08:10:29.061977Z 5238 [Note] [MY-010587] [Repl] Slave SQL thread for channel '' exiting, replication stopped in log 'FIRST' at position 0
2023-07-24T08:10:29.070286Z 5237 [Note] [MY-010563] [Repl] Slave I/O thread for channel '' killed while connecting to master
2023-07-24T08:10:29.070353Z 5237 [Note] [MY-010570] [Repl] Slave I/O thread exiting for channel '', read up to log 'FIRST', position 4
2023-07-24T08:10:29.077741Z 5244 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL '' executed'. Previous state master_host='moco-test-0.moco-test.default.svc', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='moco-test-0.moco-test.default.svc', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''.
2023-07-24T08:10:29.141562Z 5245 [Note] [MY-011179] [Repl] Slave I/O thread: Start semi-sync replication to master 'moco-repl@moco-test-0.moco-test.default.svc:3306' in log 'FIRST' at position 4.
2023-07-24T08:10:29.141620Z 5245 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
2023-07-24T08:10:29.143689Z 5245 [ERROR] [MY-010584] [Repl] Slave I/O for channel '': error connecting to master 'moco-repl@moco-test-0.moco-test.default.svc:3306' - retry-time: 60 retries: 1 message: Connections using insecure transport are prohibited while --require_secure_transport=ON. Error_code: MY-003159