ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

[ijackcy]

I follow the steps of this page to install kubelfow:
https://www.kubeflow.org/docs/started/k8s/kfctl-k8s-istio/

but, i found some pods didn't work well.

pod/katib-db-75975d8dbd-989ln 0/1 CrashLoopBackOff 475 41h
pod/katib-manager-59bb84948f-jldjh 0/1 CrashLoopBackOff 483 41h
pod/metadata-db-7584d44b65-c2slb 0/1 CrashLoopBackOff 470 41h
pod/metadata-grpc-deployment-7cc5d84854-cbhmk 0/1 CrashLoopBackOff 431 41h
pod/ml-pipeline-persistenceagent-79ff896578-qfwpn 0/1 CrashLoopBackOff 332 41h
pod/tensorboard-5f685f9d79-dht6x 0/1 CrashLoopBackOff 477 41h

root@ubuntu-master:/home/xiaowei# kubectl logs katib-db-75975d8dbd-989ln -n kubeflow
2020-01-02 02:17:16+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.18-1debian9 started.
2020-01-02 02:17:17+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2020-01-02 02:17:17+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.18-1debian9 started.
2020-01-02T02:17:19.235713Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2020-01-02T02:17:19.236263Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.18) starting as process 1
2020-01-02T02:17:21.911796Z 1 [ERROR] [MY-011092] [Server] Upgrading the data dictionary from dictionary version '1' is not supported.
2020-01-02T02:17:21.913075Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2020-01-02T02:17:21.913569Z 0 [ERROR] [MY-010119] [Server] Aborting
2020-01-02T02:17:23.062667Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.18) MySQL Community Server - GPL.

root@ubuntu-master:/home/xiaowei# kubectl logs metadata-db-7584d44b65-c2slb -n kubeflow
2020-01-02T02:20:33.519364Z 0 [Note] Basedir set to /usr/
2020-01-02T02:20:33.533224Z 0 [Warning] The syntax '--symbolic-links/-s' is deprecated and will be removed in a future release
2020-01-02T02:20:33.534140Z 0 [Note] --secure-file-priv is set to NULL. Operations related to importing and exporting data are disabled
2020-01-02T02:20:33.534314Z 0 [Note] /usr/sbin/mysqld (mysqld 8.0.3-rc-log) starting as process 1 ...
2020-01-02T02:20:33.666629Z 0 [Warning] No argument was provided to --log-bin, and --log-bin-index was not used; so replication may break when this MySQL server acts as a master and has his hostname changed!! Please use '--log-bin=metadata-db-7584d44b65-c2slb-bin' to avoid this problem.
2020-01-02T02:20:33.689036Z 0 [Note] InnoDB: Using Linux native AIO
2020-01-02T02:20:33.711455Z 0 [Note] Plugin 'FEDERATED' is disabled.
2020-01-02T02:20:33.805915Z 1 [Note] InnoDB: PUNCH HOLE support available
2020-01-02T02:20:33.806187Z 1 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-01-02T02:20:33.806203Z 1 [Note] InnoDB: Uses event mutexes
2020-01-02T02:20:33.806218Z 1 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2020-01-02T02:20:33.806240Z 1 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-01-02T02:20:33.829806Z 1 [Note] InnoDB: Number of pools: 1
2020-01-02T02:20:33.831198Z 1 [Note] InnoDB: Not using CPU crc32 instructions
2020-01-02T02:20:33.919874Z 1 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2020-01-02T02:20:33.964436Z 1 [Note] InnoDB: Completed initialization of buffer pool
2020-01-02T02:20:33.972840Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-01-02T02:20:34.055602Z 1 [ERROR] InnoDB: Unsupported redo log format. The redo log was created with MySQL 8.0.18. Please follow the instructions at http://dev.mysql.com/doc/refman/8.0/en/upgrading-downgrading.html
2020-01-02T02:20:34.055914Z 1 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2020-01-02T02:20:34.689746Z 1 [ERROR] Failed to initialize DD Storage Engine
2020-01-02T02:20:34.715911Z 0 [ERROR] Data Dictionary initialization failed.
2020-01-02T02:20:34.716078Z 0 [ERROR] Aborting
2020-01-02T02:20:34.716247Z 0 [Note] Binlog end
2020-01-02T02:20:34.719114Z 0 [Note] Shutting down plugin 'InnoDB'
2020-01-02T02:20:34.772249Z 0 [Note] Shutting down plugin 'MyISAM'
2020-01-02T02:20:34.772737Z 0 [Note] Shutting down plugin 'CSV'
2020-01-02T02:20:34.781081Z 0 [Note] /usr/sbin/mysqld: Shutdown complete

I use NFS of node 2,and create PV. and i found that many files had been created.

I tried many times and had the same error. What should I do?

Environment:

Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard):
kfctl version: 0.70
Kubernetes version: 1.15.1
OS : ubuntu server 16.04

[issue-label-bot]

Issue Label Bot is not confident enough to auto-label this issue.
See dashboard for more details.

[jlewi]

@zhenghuiwang looks like an issue with metadata DB could you take a look and triage this please?

[zhenghuiwang]

The problem is likely to be MySql can't start due to incompatible versions. Based on the log, many pods from various services that depend on Mysql failed.

pod/katib-db-75975d8dbd-989ln 0/1 CrashLoopBackOff 475 41h
pod/katib-manager-59bb84948f-jldjh 0/1 CrashLoopBackOff 483 41h
pod/metadata-db-7584d44b65-c2slb 0/1 CrashLoopBackOff 470 41h
pod/metadata-grpc-deployment-7cc5d84854-cbhmk 0/1 CrashLoopBackOff 431 41h
pod/ml-pipeline-persistenceagent-79ff896578-qfwpn 0/1 CrashLoopBackOff 332 41h
pod/tensorboard-5f685f9d79-dht6x 0/1 CrashLoopBackOff 477 41h

Each of these services uses its own MySql server. So my guess is they are configured to share the same PV, which causes conflict during initialization.

@ijackcy what platform (or kfctl_v0.7.0_<platform>.tar.gz file) do you use? Do you install Kubeflow in an existing k8s cluster?

[ijackcy]

hi,thank you!
kfctl file is kfctl_v0.7.0_linux.tar.gz.
yaml file is kfctl_k8s_istio.0.7.1.yaml

i install Kubeflow in an existing k8s cluster, version is 1.15.1.

I recreated PV according to your idea，I use NFS of node 2(ubuntu-slaver02),and create PV. The relationship between four NFS folder paths and PV is as follows：

/home/xiaowei/nfs4/katibmysql -----mypv001
/home/xiaowei/nfs4/metadatamysql ---mypv002
/home/xiaowei/nfs4/miniomysql ---mypv003
/home/xiaowei/nfs4/mysqlmysql---mypv004

root@ubuntu-master:/home/xiaowei/kubeflow# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
mypv001 20Gi RWO Recycle Bound kubeflow/metadata-mysql
mypv002 20Gi RWO Recycle Bound kubeflow/katib-mysql
mypv003 20Gi RWO Recycle Bound kubeflow/minio-pv-claim
mypv004 20Gi RWO Recycle Bound kubeflow/mysql-pv-claim

now ,There are still a few bad pods
katib-db-75975d8dbd-gdnl2 0/1 Running 0 4m37s
katib-manager-59bb84948f-5j9w8 0/1 CrashLoopBackOff 1 113s
tensorboard-5f685f9d79-jdc6b 0/1 CrashLoopBackOff 138 11h

root@ubuntu-master:/home/xiaowei/kubeflow/mykubeflow# kubectl -n kubeflow logs katib-db-75975d8dbd-gdnl2
2020-01-06 03:08:59+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.18-1debian9 started.
2020-01-06 03:09:02+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2020-01-06 03:09:02+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.18-1debian9 started.
2020-01-06T03:09:06.081089Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2020-01-06T03:09:06.081676Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.18) starting as process 1
2020-01-06T03:09:14.913433Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2020-01-06T03:09:14.991914Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2020-01-06T03:09:15.112277Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.18' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server - GPL.
2020-01-06T03:09:15.516112Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/var/run/mysqld/mysqlx.sock' bind-address: '::' port: 33060
root@ubuntu-master:/home/xiaowei/kubeflow/mykubeflow# kubectl -n kubeflow logs katib-db-75975d8dbd-gdnl2
2020-01-06 03:08:59+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.18-1debian9 started.
2020-01-06 03:09:02+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2020-01-06 03:09:02+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.18-1debian9 started.
2020-01-06T03:09:06.081089Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2020-01-06T03:09:06.081676Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.18) starting as process 1
2020-01-06T03:09:14.913433Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2020-01-06T03:09:14.991914Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2020-01-06T03:09:15.112277Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.18' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server - GPL.
2020-01-06T03:09:15.516112Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/var/run/mysqld/mysqlx.sock' bind-address: '::' port: 33060

root@ubuntu-master:/home/xiaowei/kubeflow/mykubeflow# kubectl -n kubeflow logs katib-manager-59bb84948f-5j9w8
E0106 03:12:49.020808 1 mysql.go:57] Ping to Katib db failed: dial tcp 10.110.123.46:3306: connect: connection refused
E0106 03:12:54.017480 1 mysql.go:57] Ping to Katib db failed: dial tcp 10.110.123.46:3306: connect: connection refused
E0106 03:12:59.032935 1 mysql.go:57] Ping to Katib db failed: dial tcp 10.110.123.46:3306: connect: connection refused

I entered pod:
kubectl exec katib-db-75975d8dbd-gdnl2 -it -n kubeflow /bin/bash
found port 3306 and 33060 listen.
tcp6 0 0 :::33060 :::* LISTEN
tcp6 0 0 :::3306 :::* LISTEN

but on the node1 ,i telnet port 3306 and found not work well.10.110.123.46 is cluster ip.
root@ubuntu-slaver01:/home/xiaowei# telnet 10.110.123.46 33060
Trying 10.110.123.46...
telnet: Unable to connect to remote host: Connection timed out

[zhenghuiwang]

The problem seems to be isolated to Katib and Tensorboard.

@ijackcy can you help paste the logs of tensorboard? I actually don't find it use MySql.
https://github.com/kubeflow/manifests/blob/master/tensorboard/base/deployment.yaml

/cc @richardsliu @johnugeorge do you have any insight on the Katib pod failures?

[ijackcy]

sorry. Some pods still don't work properly and not ready.
NAME READY STATUS RESTARTS AGE
katib-db-75975d8dbd-qfj9w 0/1 Running 1 16h
katib-manager-59bb84948f-8t927 0/1 CrashLoopBackOff 200 16h
metadata-grpc-deployment-7cc5d84854-qr92k 0/1 CrashLoopBackOff 198 16h
tensorboard-5f685f9d79-brhfj 0/1 CrashLoopBackOff 198 16h

I view katib-db-75975d8dbd-qfj9w pod logs on k8s dashboard，
Readiness probe failed: mysql: [Warning] Using a password on the command line interface can be insecure. ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

[crazy-canux]

why closed?
it's exist on 1.0.2.