Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker mysql:latest has updated to 8.x and crashes database containers. #908

Closed
sk33lz opened this issue Apr 20, 2018 · 17 comments

Comments

Projects
None yet
@sk33lz
Copy link

commented Apr 20, 2018

Bug Report

Setup

OS: Mac 10.13.4
Docker: 18.03.0-ce-mac60
Lando: v3.0.0-beta.40

Problem

Running lando start on a new project ends up in a loop waiting for the database service to start. Running lando destroy and lando rebuild does not resolve the issue.

Rebuilding an existing Lando app on a previous version Lando (v3.0.0-beta.39) or installing Lando v3.0.0-beta.40 causes the mysql:latest docker tag to install MySQL 8.x. This causes lando mysql containers to fail to start with a repeating loop of the following message:

Waiting until database service is ready...

I did a docker ps -a to figure out what the container ID was for the mysql database container and ran docker start $CONTAINERID -a to see what the issue was. I ended up with a bunch of errors during the mysqld startup process at the docker-entrypoint.sh step.

Running command docker-entrypoint.sh mysqld
2018-04-20T19:56:07.344153Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2018-04-20T19:56:07.344245Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2018-04-20T19:56:07.344359Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.11) starting as process 103
Reading package lists...mbind: Operation not permitted
mbind: Operation not permitted
mysqld: Table 'mysql.plugin' doesn't exist...

The error continues on until mysqld shuts itself down.

I noticed in the output that mysql Ver 8.0.11 was installed. It turns out that the docker:latest tag was updated yesterday from 5.x to 8.x. See docker-library/mysql@ff62794.

Workaround

I was able to workaround this issue for now by adding database service with mysql:5.7 defined.

services:
  database:
    type: mysql:5.7

Proposed Solution

I feel that given the mysql:latest docker tag can change at any time, perhaps Lando should pin a specific mysql version as the default mysql version based on the current Debian release, or something clever so that the mysql version can't change behind the scenes.

@cboyden

This comment has been minimized.

Copy link

commented Apr 20, 2018

I just started evaluating Lando today, and I can confirm both the problem and the workaround. The proposed solution sounds fine to me.

@gugerard

This comment has been minimized.

Copy link

commented Apr 21, 2018

Same here, thanks for the workaround.

@dustinleblanc

This comment has been minimized.

Copy link
Member

commented Apr 22, 2018

@sk33lz I think you are right on. We've been letting people 'run with the devil' with their service versions for quite a while, specifically because Docker itself makes this so easy, and for a lot of services, breaking changes don't drop on us like this.

After the headache this has caused though, I think your suggestion is a good one. We'll definitely be taking a look at this.

@stefanospetrakis

This comment has been minimized.

Copy link

commented Apr 22, 2018

+1 many thanks for reporting this promptly and clearly!

@esteinborn

This comment has been minimized.

Copy link

commented Apr 23, 2018

Can additionally confirm this is working when using workaround.

@legovaer

This comment has been minimized.

Copy link

commented Apr 24, 2018

I'm actually using the workaround but when I do a lando rebuild I get an endless list of this:

Waiting until nginx service is ready...
Waiting until cache service is ready...
Waiting until database service is ready...
Waiting until pma service is ready...
Waiting until appserver service is ready...
Waiting until mailhog service is ready...
Waiting until nginx service is ready...
Waiting until database service is ready...
Waiting until nginx service is ready...
Waiting until database service is ready...
Waiting until nginx service is ready...
Waiting until database service is ready...
Waiting until nginx service is ready...
Waiting until nginx service is ready...
Waiting until nginx service is ready...
Waiting until nginx service is ready...
Waiting until nginx service is ready...
Waiting until nginx service is ready...

Which makes me feel like

(This was a good intended joke, I appreciate your hard work towards Lando!)

@dustinleblanc

This comment has been minimized.

Copy link
Member

commented Apr 24, 2018

@legovaer NICE!

You can probably destroy and then start if you have some cruft from the old database settings around. I intend to push a fix for this issue shortly

@uberhacker

This comment has been minimized.

Copy link
Contributor

commented Apr 24, 2018

I'm having this issue on ANY version of MySQL including 5.7 running beta.39.

@legovaer

This comment has been minimized.

Copy link

commented Apr 24, 2018

@dustinleblanc I tried:

$ lando destroy
? Are you sure you want to DESTROY? Yes
Stopping tqfsd_nginx_1     ... done
Stopping tqfsd_cache_1     ... done
Stopping tqfsd_database_1  ... done
Stopping tqfsd_pma_1       ... done
Stopping tqfsd_appserver_1 ... done
Stopping tqfsd_mailhog_1   ... done
Removing tqfsd_nginx_1     ... done
Removing tqfsd_cache_1     ... done
Removing tqfsd_database_1  ... done
Removing tqfsd_pma_1       ... done
Removing tqfsd_appserver_1 ... done
Removing tqfsd_mailhog_1   ... done
Removing network tqfsd_default
Removing volume tqfsd_data
Removing volume tqfsd_appserver
Removing volume tqfsd_data_database

Then:

$ lando start
landoproxyhyperion5000gandalfedition_proxy_1 is up-to-date
Creating network "tqfsd_default" with the default driver
Creating volume "tqfsd_data" with default driver
Creating volume "tqfsd_appserver" with default driver
Creating volume "tqfsd_data_database" with default driver
Creating tqfsd_pma_1       ... done
Creating tqfsd_database_1  ... done
Creating tqfsd_cache_1     ... done
Creating tqfsd_mailhog_1   ... done
Creating tqfsd_appserver_1 ... done
Creating tqfsd_nginx_1     ... done
Waiting until nginx service is ready...
Waiting until appserver service is ready...
Waiting until mailhog service is ready...
Waiting until database service is ready...
Waiting until pma service is ready...
Waiting until cache service is ready...
Waiting until nginx service is ready...
Waiting until database service is ready...
Waiting until nginx service is ready...
Waiting until database service is ready...
Waiting until nginx service is ready...
Waiting until database service is ready...
Waiting until nginx service is ready...
Waiting until database service is ready...
Waiting until nginx service is ready...

I'm using 3.0.0-beta.40

@dustinleblanc

This comment has been minimized.

Copy link
Member

commented Apr 24, 2018

I mean, you will see those until it finishes startup or times out, so I don't see an issue specifically there

@uberhacker

This comment has been minimized.

Copy link
Contributor

commented Apr 24, 2018

I have confirmed the proposed solution works on ANY MySQL version, not just the latest:

services:
  database:
    type: mysql:#.#

I guess the fix would/should be the config below either needs removed or updated so the services -> database section is not needed.

config:
  mysql: #.#
@jacine

This comment has been minimized.

Copy link

commented Apr 26, 2018

Is there a workaround for the Pantheon recipe? None of the above has worked for me. Thanks!

sherakama added a commit to SU-HSDO/suhumsci that referenced this issue Apr 26, 2018

@dustinleblanc

This comment has been minimized.

Copy link
Member

commented Apr 27, 2018

@jacine Mariadb is not affected by this issue, if a Pantheon recipe isn't booting, you probably have a different issue

@jacine

This comment has been minimized.

Copy link

commented Apr 27, 2018

Thank you @dustinleblanc! I wasn't sure, but this seemed like the best lead, after reading through the issues here, and trying a bunch of different things (including completely reinstalling Lando, Docker, and all config). My problem is actually with lando pull, which fails to import the DB with an SQL syntax error. Lando does boot and run properly, so I should probably open a separate issue. Thanks! 😄

@dustinleblanc

This comment has been minimized.

Copy link
Member

commented Apr 27, 2018

@jacine No problem 👍

We did have an issue on the platform the other day that was a mistake that was causing syntax errors like this on lando pulls. It was related to a patch we released the previous day, it got fixed yesterday though. Are things behaving normally for you now?

@jacine

This comment has been minimized.

Copy link

commented Apr 27, 2018

OHHHH, YES! It does work now. Thank you so much @dustinleblanc!

@nanophp

This comment has been minimized.

Copy link

commented Apr 16, 2019

8.0.15 still fail and waiting for the build. MySQL 5.7 totally works fine but I wanna upgrade MySQL version as 8+, anyone fixes the issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.