mysql chart: crashloopbackoff after node restart

[rmorrise]

I installed the mysql chart in minikube (win64) with the following YAML settings:

#bitnami mysql:
service:
    type: NodePort
    port: 3306
master:
  persistence:
    enabled: false
slave:
  persistence:
    enabled: false

Chart starts up fine.

I stop and start minikube, and the chart doesn't run anymore. Both master and slave pods are getting a CrashLoopBackOff (Terminated: Error).

The master pod has the following in the log:

←[0m
←[0m←[1mWelcome to the Bitnami mysql container←[0m
←[0mSubscribe to project updates by watching ←[1mhttps://github.com/bitnami/bitnami-docker-mysql←[0m
←[0mSubmit issues and feature requests at ←[1mhttps://github.com/bitnami/bitnami-docker-mysql/issues←[0m
←[0m
nami    INFO  Initializing mysql
mysql   INFO
mysql   INFO  ########################################################################
mysql   INFO   Installation parameters for mysql:
mysql   INFO     Persisted data and properties have been restored.
mysql   INFO     Any input specified will not take effect.
mysql   INFO   This installation requires no credentials.
mysql   INFO  ########################################################################
mysql   INFO
nami    INFO  mysql successfully initialized
←[0m←[38;5;2mINFO ←[0m ==> Starting mysql...
←[0m←[38;5;2mINFO ←[0m ==> Starting mysqld_safe...
2018-08-03T13:27:55.114541Z mysqld_safe error: log-error set to '/opt/bitnami/mysql/logs/mysqld.log', however file don't exists. Create writable for user 'mysql'.

I think the root password is also different than on the previous run. I feel like this might be strange behavior? But I don't know if it's related to the crashing issue.

[yacut]

I have the same issue on k8s 1.10

[juan131]

Hi @yacut @rmorrise

Thanks for raising this up! I was able to reproduce the error. It seems that it's not setting the right permissions on the directory /opt/bitnami/mysql/logs/ when it starts MySQL container with persisted data (the one stored on the Persistent Volume claimed by the MySQL statefulset).

@javsalgar does it make sense to you?

[javsalgar]

It looks to me that it could be due to the minikube start up process that may change the permissions in the folders where the persistent volumes are stored. I would report this issue in minikube as they may have more information.

[rmorrise]

@javsalgar @juan131 I don't see any volume mount defined for the path that you specified. The statefulset has the following in describe. There are no PVCs, since persistence is disabled.

   Mounts:
     /bitnami/mysql from data (rw)
     /docker-entrypoint-initdb.d from custom-init-scripts (rw)
     /opt/bitnami/mysql/conf/my.cnf from config (rw)
 Volumes:
  config:
   Type:      ConfigMap (a volume populated by a ConfigMap)
   Name:      kbvaluedb-local-mysql-master
   Optional:  false
  custom-init-scripts:
   Type:      ConfigMap (a volume populated by a ConfigMap)
   Name:      kbvaluedb-local-mysql-master-init-scripts
   Optional:  false
  data:
   Type:       EmptyDir (a temporary directory that shares a pod's lifetime)

[rmorrise]

What information should I provide to the minikube team?

[juan131]

Hi @rmorrise as you mentioned it also happens when persistence is disabled....

I could workaround the issue by deleting manually the pods so the statefulset automatically creates new pods that seem to be working fine...

$ kubectl get pods
NAME                            READY     STATUS             RESTARTS   AGE
binging-monkey-mysql-master-0   0/1       CrashLoopBackOff   4          6m
binging-monkey-mysql-slave-0    0/1       CrashLoopBackOff   4          6m
$ kubectl delete pod binging-monkey-mysql-master-0
$ kubectl delete pod binging-monkey-mysql-slave-0
$ kubectl get pods
NAME                            READY     STATUS    RESTARTS   AGE
binging-monkey-mysql-master-0   1/1       Running   0          4m
binging-monkey-mysql-slave-0    1/1       Running   0          2m

That's pretty weird and I guess it must be related with some weird behaviour on Statefulsets. Do you have any clue @javsalgar ?

[stale]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[hoangleanhtu]

Unfortunately I still have this issue on Google Kubernetes v1.10.7-gke.11. Somehow master pod was restarted then it gets CrashLoopBackOff forever.
I try to delete master pod many times but it doesn't work.

Here is the log:

2018-11-26T13:59:55.966663Z 0 [Note] InnoDB: 5.7.24 started; log sequence number 6494052097
2018-11-26T13:59:55.967103Z 0 [Note] InnoDB: Loading buffer pool(s) from /bitnami/mysql/data/ib_buffer_pool
2018-11-26T13:59:55.967272Z 0 [Note] Plugin 'FEDERATED' is disabled.
2018-11-26T13:59:55.970225Z 0 [Note] InnoDB: Buffer pool(s) load completed at 181126 13:59:55
2018-11-26T13:59:55.985048Z 0 [Warning] Failed to set up SSL because of the following SSL library error: SSL context is not usable without certificate and private key
2018-11-26T13:59:55.985080Z 0 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
2018-11-26T13:59:55.985110Z 0 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
2018-11-26T13:59:55.985182Z 0 [Note] Server socket created on IP: '0.0.0.0'.
2018-11-26T13:59:55.989771Z 0 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-11-26T13:59:55.989954Z 0 [Warning] 'user' entry 'mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-11-26T13:59:55.989991Z 0 [Warning] 'db' entry 'performance_schema mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-11-26T13:59:55.990003Z 0 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-11-26T13:59:55.990018Z 0 [Warning] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
2018-11-26T13:59:55.991096Z 0 [Warning] 'tables_priv' entry 'user mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-11-26T13:59:55.991120Z 0 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-11-26T13:59:55.992566Z 0 [Note] Failed to start slave threads for channel ''
2018-11-26T13:59:55.997521Z 0 [Note] Event Scheduler: Loaded 0 events
2018-11-26T13:59:55.997808Z 0 [Note] /opt/bitnami/mysql/bin/mysqld: ready for connections.
Version: '5.7.24-log'  socket: '/opt/bitnami/mysql/tmp/mysql.sock'  port: 3306  MySQL Community Server (GPL)
2018-11-26T14:00:27.503043Z 2 [Note] Access denied for user 'root'@'localhost' (using password: YES)

[juan131]

Hi @hoangleanhtu

It could be related with some incomplete initialisation of MySQL..

Are you using persistence or did you disable it? Could you share the exact command you used to deploy the chart in order to reproduce it?

Could you share the initialisation logs too?

[hoangleanhtu]

Hi @juan131 ,

I just used Helm to install chart

helm install bitnami/mysql

I tried to delete master pod, Kubernetes nodes...but I can't reproduce this issue anymore.

Could you share the initialisation logs too?

I fixed CrashLoopBack issue by deleting PVC and install new one, so I can not access logs.

[juan131]

Hi @hoangleanhtu

It looks like a problem during the initialisation. Something went wrong and the data in the PVC was inconsistent.

I'm glad you could solved the issue by deleting the PVC! Please let me know if you need further help with it.

[stale]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[stale]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[HebeHH]

I'm also running into this issue; using mysql as a subchart with values.yaml:

mysql:
  inside: true
  fullnameOverride: my-mysql
  auth:
    database: mine
    username: me
    password: pwd
    rootPassword: root
  image:
    tag: 8.0.11
    registry: docker.io
    repository: bitnami/mysql

After I uninstall/reinstall a couple of times, I end up with the mysql stateful set pod in CrashLoopBackoff, with this error:

nami    INFO  Initializing mysql
mysql   INFO
mysql   INFO  ########################################################################
mysql   INFO   Installation parameters for mysql:
mysql   INFO     Persisted data and properties have been restored.
mysql   INFO     Any input specified will not take effect.
mysql   INFO   This installation requires no credentials.
mysql   INFO  ########################################################################
mysql   INFO
nami    INFO  mysql successfully initialized
INFO  ==> Starting mysql...
INFO  ==> Starting mysqld_safe...
2021-03-05T09:54:28.999663Z mysqld_safe error: log-error set to '/opt/bitnami/mysql/logs/mysqld.log', however file don't exists. Create writable for user 'mysql'.

[juan131]

Hi @HebeHH

Why did you overwrite the default image.tag? Is there any limitation that forces you to use 8.0.11?

As you can see in the "Notable Changes" section below:

• https://github.com/bitnami/bitnami-docker-mysql#5723-r52-and-8012-r34

The Bitnami MySQL image introduced structural changes in the 8.0.12-r34. The current chart is meant to work with the new images' structure, and it's likely that you find issues if you use older images.

[HebeHH]

Hi @juan131 ,

Thanks for the pointers! I was using 8.0.11 at direction from the dev team, but they should be able to deal with the upgrade. I'll change the image tag and hope that solves the problem. That'll teach me to skim that section of the docs.

I appreciate the help, it was severely puzzling me.

[juan131]

You're welcome @HebeHH

Please share with us your progress and let us know if you need anything else.