New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The latest tag has started failing on several of my projects some time between April 17 and April 20, 2018 #409

Closed
alberto56 opened this Issue Apr 20, 2018 · 15 comments

Comments

Projects
None yet
9 participants
@alberto56
Copy link

alberto56 commented Apr 20, 2018

On new Ubuntu 16 servers, using Docker compose, including on Digital Ocean and Circle CI, with configurations ranging from 4Gb to 8Gb RAM,

docker-compose ps
      Name                     Command                 State                         Ports                   
-------------------------------------------------------------------------------------------------------------
legacy_database_1   docker-entrypoint.sh mysqld      Restarting                                              
legacy_web_1        /opt/entrypoint.sh /usr/sb ...   Up           0.0.0.0:443->443/tcp, 0.0.0.0:32769->80/tcp

Changing the MySQL version from "latest" to "mysql:8.0.4" works fine.

Using v. mysql:5.5.59 of mysql instead of latest fixes the issue.

I have a project which has always been passing, and started failing today.

@alberto56

This comment has been minimized.

Copy link
Author

alberto56 commented Apr 20, 2018

It should be noted that Docker on Mac works fine; Ubuntu 16 is affected. Docker on mac is, in fact, affected; I was using a cached version which made me think it was not.

@ltangvald

This comment has been minimized.

Copy link
Collaborator

ltangvald commented Apr 21, 2018

To clarify, the latest tag fails, but only when Docker is running on Ubuntu 16, and 8.0.4 also fails?
MySQL 8.0.11 (first GA release of 8.0) was released just now, so the latest tag was changed from 5.7 to 8.0

If 8.0.4 also fails it's probably related from changing from 5.7 to 8.0, so using the 5.7 tag should work until we can figure out the problem. Have you run mysql_upgrade on the database?

@marceloverdijk

This comment has been minimized.

Copy link

marceloverdijk commented Apr 21, 2018

I think it's related to the latest now pointing to version 8.
I had the same problem that with Ubuntu (14) latest is not working anymore. Note that I also noticed no warning or error.
I changed to 5.7 and then it worked (again).

@ltangvald

This comment has been minimized.

Copy link
Collaborator

ltangvald commented Apr 21, 2018

Can you get the log of the mysql container?

@mds325

This comment has been minimized.

Copy link

mds325 commented Apr 21, 2018

I'm having the same issue. Version 8 (latest) of mysql is failing, I had to roll back to version 5.7 which was working nicely before.

It says root is being created with an empty password while the MYSQL_ROOT_PASSWORD env var is alredy set.

This is how my already working docker-compose.yaml: https://cl.ly/392E0J0L1w3s

This is the log for the version 8 of mysql:

Initializing database
2018-04-21T16:03:09.993544Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2018-04-21T16:03:09.994173Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.11) initializing of server in progress as process 28
mbind: Operation not permitted
mbind: Operation not permitted
2018-04-21T16:03:18.322733Z 5 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2018-04-21T16:03:22.721657Z 5 [Warning] [MY-010315] [Server] 'user' entry 'mysql.infoschema@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:22.721739Z 5 [Warning] [MY-010315] [Server] 'user' entry 'mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:22.721765Z 5 [Warning] [MY-010315] [Server] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:22.721785Z 5 [Warning] [MY-010315] [Server] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:22.721965Z 5 [Warning] [MY-010323] [Server] 'db' entry 'performance_schema mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:22.721989Z 5 [Warning] [MY-010323] [Server] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:22.722012Z 5 [Warning] [MY-010311] [Server] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:22.722580Z 5 [Warning] [MY-010330] [Server] 'tables_priv' entry 'user mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:22.722637Z 5 [Warning] [MY-010330] [Server] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:23.697400Z 0 [System] [MY-013170] [Server] /usr/sbin/mysqld (mysqld 8.0.11) initializing of server has completed
Database initialized
MySQL init process in progress...
2018-04-21T16:03:26.129085Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2018-04-21T16:03:26.130036Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.11) starting as process 79
mbind: Operation not permitted
mbind: Operation not permitted
MySQL init process in progress...
2018-04-21T16:03:27.143854Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2018-04-21T16:03:27.155798Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2018-04-21T16:03:27.175966Z 0 [Warning] [MY-010315] [Server] 'user' entry 'mysql.infoschema@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:27.176078Z 0 [Warning] [MY-010315] [Server] 'user' entry 'mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:27.176121Z 0 [Warning] [MY-010315] [Server] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:27.176577Z 0 [Warning] [MY-010315] [Server] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:27.176692Z 0 [Warning] [MY-010323] [Server] 'db' entry 'performance_schema mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:27.176719Z 0 [Warning] [MY-010323] [Server] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:27.176755Z 0 [Warning] [MY-010311] [Server] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:27.183442Z 0 [Warning] [MY-010330] [Server] 'tables_priv' entry 'user mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:27.183566Z 0 [Warning] [MY-010330] [Server] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:27.205616Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.11'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  MySQL Community Server - GPL.
Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/zone1970.tab' as time zone. Skipping it.
2018-04-21T16:03:36.589334Z 10 [Warning] [MY-010315] [Server] 'user' entry 'mysql.infoschema@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:36.589401Z 10 [Warning] [MY-010315] [Server] 'user' entry 'mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:36.589484Z 10 [Warning] [MY-010315] [Server] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:36.589510Z 10 [Warning] [MY-010315] [Server] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:36.589543Z 10 [Warning] [MY-010323] [Server] 'db' entry 'performance_schema mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:36.589586Z 10 [Warning] [MY-010323] [Server] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:36.589610Z 10 [Warning] [MY-010311] [Server] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:36.591384Z 10 [Warning] [MY-010330] [Server] 'tables_priv' entry 'user mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:03:36.591465Z 10 [Warning] [MY-010330] [Server] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
mysql: [Warning] Using a password on the command line interface can be insecure.

/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/ee_redonline.sql
mysql: [Warning] Using a password on the command line interface can be insecure.


2018-04-21T16:06:12.459062Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.11)  MySQL Community Server - GPL.

MySQL init process done. Ready for start up.

2018-04-21T16:06:12.938668Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2018-04-21T16:06:12.939477Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.11) starting as process 1
mbind: Operation not permitted
mbind: Operation not permitted
2018-04-21T16:06:16.286772Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2018-04-21T16:06:16.301056Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2018-04-21T16:06:16.326861Z 0 [Warning] [MY-010315] [Server] 'user' entry 'mysql.infoschema@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:06:16.326986Z 0 [Warning] [MY-010315] [Server] 'user' entry 'mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:06:16.327034Z 0 [Warning] [MY-010315] [Server] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:06:16.327058Z 0 [Warning] [MY-010315] [Server] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:06:16.327099Z 0 [Warning] [MY-010323] [Server] 'db' entry 'performance_schema mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:06:16.327153Z 0 [Warning] [MY-010323] [Server] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:06:16.327184Z 0 [Warning] [MY-010311] [Server] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:06:16.345425Z 0 [Warning] [MY-010330] [Server] 'tables_priv' entry 'user mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:06:16.345504Z 0 [Warning] [MY-010330] [Server] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-04-21T16:06:16.361073Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.11'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.
@imyxiao

This comment has been minimized.

Copy link

imyxiao commented Apr 21, 2018

Version 8 (latest) of mysql is failing too.

2018-04-21T22:07:21.090249Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2018-04-21T22:07:21.090434Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.11) starting as process 1
mbind: Operation not permitted
mbind: Operation not permitted
mbind: Operation not permitted
2018-04-21T22:07:25.575616Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2018-04-21T22:07:25.581632Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2018-04-21T22:07:25.583443Z 0 [ERROR] [MY-013129] [Server] A message intended for a client cannot be sent there as no client-session is attached. Therefore, we're sending the information to the error-log instead: MY-001146 - Table 'mysql.component' doesn't exist
2018-04-21T22:07:25.583542Z 0 [Warning] [MY-013129] [Server] A message intended for a client cannot be sent there as no client-session is attached. Therefore, we're sending the information to the error-log instead: MY-003543 - The mysql.component table is missing or has an incorrect definition.
2018-04-21T22:07:25.592438Z 0 [Warning] [MY-010929] [Server] Storage engine 'MyISAM' does not support system tables. [mysql.user].
2018-04-21T22:07:25.592466Z 0 [Warning] [MY-010929] [Server] Storage engine 'MyISAM' does not support system tables. [mysql.db].
2018-04-21T22:07:25.592473Z 0 [Warning] [MY-010929] [Server] Storage engine 'MyISAM' does not support system tables. [mysql.tables_priv].
2018-04-21T22:07:25.592503Z 0 [Warning] [MY-010929] [Server] Storage engine 'MyISAM' does not support system tables. [mysql.columns_priv].
2018-04-21T22:07:25.592513Z 0 [Warning] [MY-010929] [Server] Storage engine 'MyISAM' does not support system tables. [mysql.procs_priv].
2018-04-21T22:07:25.592519Z 0 [Warning] [MY-010929] [Server] Storage engine 'MyISAM' does not support system tables. [mysql.proxies_priv].
2018-04-21T22:07:25.593267Z 0 [ERROR] [MY-013143] [Server] Column count of mysql.user is wrong. Expected 49, found 45. The table is probably corrupted
2018-04-21T22:07:25.593327Z 0 [Warning] [MY-010966] [Server] ACL table mysql.role_edges missing. Some operations may fail.
2018-04-21T22:07:25.593340Z 0 [Warning] [MY-010966] [Server] ACL table mysql.default_roles missing. Some operations may fail.
2018-04-21T22:07:25.593345Z 0 [Warning] [MY-010966] [Server] ACL table mysql.global_grants missing. Some operations may fail.
2018-04-21T22:07:25.593351Z 0 [Warning] [MY-010966] [Server] ACL table mysql.password_history missing. Some operations may fail.
2018-04-21T22:07:25.594370Z 0 [Warning] [MY-010315] [Server] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
2018-04-21T22:07:25.594418Z 0 [Warning] [MY-010315] [Server] 'user' entry 'mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-04-21T22:07:25.594436Z 0 [Warning] [MY-010315] [Server] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-04-21T22:07:25.594830Z 0 [Warning] [MY-010323] [Server] 'db' entry 'performance_schema mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-04-21T22:07:25.594850Z 0 [Warning] [MY-010323] [Server] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-04-21T22:07:25.595235Z 0 [Warning] [MY-010311] [Server] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
2018-04-21T22:07:25.595299Z 0 [ERROR] [MY-010965] [Server] Missing system table mysql.global_grants; please run mysql_upgrade to create it.
2018-04-21T22:07:25.598629Z 0 [Warning] [MY-010330] [Server] 'tables_priv' entry 'user mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-04-21T22:07:25.598657Z 0 [Warning] [MY-010330] [Server] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-04-21T22:07:25.601203Z 0 [Warning] [MY-010727] [Server] System table 'func' is expected to be transactional.
2018-04-21T22:07:25.603526Z 0 [Warning] [MY-010405] [Repl] Info table is not ready to be used. Table 'mysql.slave_master_info' cannot be opened.
2018-04-21T22:07:25.603581Z 0 [ERROR] [MY-010422] [Repl] Error in checking mysql.slave_master_info repository info type of TABLE.
2018-04-21T22:07:25.603635Z 0 [ERROR] [MY-010415] [Repl] Error creating master info: Error checking repositories.
2018-04-21T22:07:25.603648Z 0 [ERROR] [MY-010426] [Repl] Slave: Failed to initialize the master info structure for channel ''; its record may still be present in 'mysql.slave_master_info' table, consider deleting it.
2018-04-21T22:07:25.603690Z 0 [ERROR] [MY-010529] [Repl] Failed to create or recover replication info repositories.
2018-04-21T22:07:25.605807Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.11'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.

@codeblooded

This comment has been minimized.

Copy link

codeblooded commented Apr 22, 2018

I'm on Mac and I'm experiencing this:

$ docker run mysql -p 3306:3306 -e MYSQL_ALLOW_EMPTY_PASSWORD MYSQL_ROOT_PASSWORD='' 

ERROR: mysqld failed while attempting to check config
command was: "mysqld -p 3306:3306 -e MYSQL_ALLOW_EMPTY_PASSWORD MYSQL_ROOT_PASSWORD= --verbose --help"

Enter password: mysqld: Can not perform keyring migration : Invalid --keyring-migration-source option.

@rwatts3

This comment has been minimized.

Copy link

rwatts3 commented Apr 22, 2018

I too am experiencing this. How could an update that is a minor release break so badly, now when I try to roll back to 5.7 The container dies within seconds of starting.

@tianon

This comment has been minimized.

Copy link
Member

tianon commented Apr 22, 2018

@rwatts3

This comment has been minimized.

Copy link

rwatts3 commented Apr 22, 2018

👍 noted I suppose the only thing I can try at this point is to boot the container, in 8, dump the data then hopefully restore with a fresh data directory.

@codeblooded

This comment has been minimized.

Copy link

codeblooded commented Apr 22, 2018

@rwatts3

This comment has been minimized.

Copy link

rwatts3 commented Apr 22, 2018

Solved
I was able to recover by following these steps.
Please confirm if this worked for you so that others can have the solution.

Note this works if you've mounted the volume of your mysql data

  1. docker exec into the container while it's running on mysql 8
  2. mysqldump the database to a file within the mounted directory
  3. exit the container and navigate to the directory where you've mounted on the host machine
  4. copy the file from mounted directory out of the mounted directory
  5. blitz / rm the mounted directory
  6. update docker-compose to use mysql:5.7
  7. docker-compose up and force recreate the container (it will recreate the mounted directory)
  8. after the mounted directory is created copy don't move but copy the .sql file back into the mounted directory
  9. docker exec into the mysql container and mysql restore the .sql file to the corresponding database
  10. exit the contaier
  11. never use mysql:latest tag again 😄

commands

mysqldump -u [uname] -p[pass] db_name > db_backup.sql

mysql -u username -p database_name < file.sql
@tianon

This comment has been minimized.

Copy link
Member

tianon commented Apr 23, 2018

Since this appears to be related to using mysql:latest (which is bad practice and should be avoided) and thus the switch from MySQL 5 to MySQL 8, I'm going to close.

There's a lot of good information here about how to upgrade from 5 to 8 properly. 👍

@tianon tianon closed this Apr 23, 2018

@BrodyHiggerson

This comment has been minimized.

Copy link

BrodyHiggerson commented Apr 30, 2018

For anyone following this - I had this problem as well (but not from using mysql:latest, just mysql for my Docker image - still ended up going from MySQL 5.x to 8.0.x). The above steps failed for me because of The user specified as a definer ('mysql.infoschema'@'localhost') does not exist. I couldn't dump out to an .sql file or really perform any operations at all. The solution was to run mysql-upgrade --force --all-databases. I tried with --force but still didn't end up with all of the required users. Note: this will obviously upgrade you to MySQL 8.0.x... hopefully you're okay with that. I guess you could use this to get into a state where the steps listed earlier in the thread will work for you.

Hope this helps if the above instructions don't quite work for you.

@ltangvald

This comment has been minimized.

Copy link
Collaborator

ltangvald commented Apr 30, 2018

When you don't specify a tag, you automatically use the "latest" tag, which as mentioned has moved from 5.7 to 8.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment