New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Database service is unhealthy after restart #1381

Closed
martin-klima opened this Issue Feb 2, 2019 · 17 comments

Comments

Projects
None yet
4 participants
@martin-klima
Copy link

martin-klima commented Feb 2, 2019

The database service starts only once just after rebuild and then database is unhealthy for all next starts in drupal7 recipe.

Tell us about your setup

Linux Mint 18.2
Linux 4.15.0-33-generic 36~16.04.1-Ubuntu SMP Wed Aug 15 17:21:05 UTC 2018 x86_64 GNU/Linux

Lando v3.0.0-rc.2

docker version
Client:
Version: 18.09.0
API version: 1.39

Server: Docker Engine - Community
Engine:
Version: 18.09.0
API version: 1.39 (minimum version 1.12)

Tell us about your .lando.yml

Very simple test site.

name: my-lando-app
recipe: drupal7
config:
  webroot: .

Tell us about the command you were running

first run lando start - all is OK.
lando restart - see error below

Tell us about the error you got

info: Waiting until database service is ready...
warn: Service database is **unhealthy**

lando mysql
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/opt/bitnami/mysql/tmp/mysql.sock' (111)

Tell us more

The problem is probably related to drupal7 recipe. The drupal8 recipe works normally. I tested lando poweroff, lando stop without effect. When I use lando rebuild, the database container starts normally, untill lando restart.

@martin-klima martin-klima added the bug label Feb 2, 2019

@martin-klima

This comment has been minimized.

Copy link
Author

martin-klima commented Feb 2, 2019

This is part of log, maybe it helps.

lando logs -s database | grep database

database_1   | Version: '5.7.25'  socket: '/opt/bitnami/mysql/tmp/mysql.sock'  port: 3306  MySQL Community Server (GPL)
database_1   | mkdir: cannot create directory '/etc/ssh': Permission denied
database_1   | Running command /entrypoint.sh /run.sh
database_1   | 
database_1   | Welcome to the Bitnami mysql container
database_1   | Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mysql
database_1   | Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mysql/issues
database_1   | Send us your feedback at containers@bitnami.com
database_1   | 
database_1   | INFO  ==> ** Starting MySQL setup **
database_1   | INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars..
database_1   | WARN  ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons, do not use this flag in a production environment.
database_1   | INFO  ==> Initializing mysql database...
database_1   | INFO  ==> Persisted data detected. Restoring...
database_1   | INFO  ==> Stopping mysql...
database_1   | Terminated
database_1   | mkdir: cannot create directory '/etc/ssh': Permission denied
database_1   | Running command /entrypoint.sh /run.sh
@pirog

This comment has been minimized.

Copy link
Member

pirog commented Feb 2, 2019

@martin-klima its something from our default drupal7 mysql conf file. If you remove that file then it restarts as expected.

@pirog pirog self-assigned this Feb 2, 2019

@pirog pirog added this to the 3.0.0-rc.3 milestone Feb 2, 2019

@pirog

This comment has been minimized.

Copy link
Member

pirog commented Feb 2, 2019

default-storage-engine = innodb seems to be the culprit. Our new bitnami images must set that as the default to begin with

@pirog

This comment has been minimized.

Copy link
Member

pirog commented Feb 2, 2019

Im pushing a fix for this to master. Thanks for reporting @martin-klima

@martin-klima

This comment has been minimized.

Copy link
Author

martin-klima commented Feb 2, 2019

@pirog I gladly helped.
I extended my tests and I can confirm the problem also appears in Drupal 8 recipe, so it is not related to only D7 recipe as I supposed, but more probably is related to database service (image or config).

The problem is dynamic and not all restarts failed. I tested these steps in sequence for simple D8 recipe:

lando destroy -y
lando restart -> all OK
lando restart -> all OK
lando restart -> all OK
lando restart -> warn: Service database is unhealthy
lando restart -> all OK
lando restart -> all OK
lando restart -> warn: Service database is unhealthy
lando restart -> warn: Service database is unhealthy
lando restart -> warn: Service database is unhealthy
@martin-klima

This comment has been minimized.

Copy link
Author

martin-klima commented Feb 2, 2019

@pirog It's hard to say, but unfortunately the problem seems to remain in RC3.

Step to reproduce:

  1. Use simple drupal7 or drupal8 recipe without any specialities.
  2. Use lando destroy -y; lando rebuild -y
  3. And repeat several times command lando restart

Randomly in approx. 50 % of cases you will see:

Waiting until database service is ready...
warn: Service database is unhealthy
warn: Run "lando logs -s database"

before
BOOMSHAKALAKA!!!

@martin-klima martin-klima changed the title Drupal7 recipe: database is unhealthy after restart Drupal recipes: database is unhealthy after restart Feb 2, 2019

@pirog

This comment has been minimized.

Copy link
Member

pirog commented Feb 2, 2019

@rabauss

This comment has been minimized.

Copy link

rabauss commented Feb 3, 2019

can confirm - that the problem randomly exists in RC3!

I use LAMP recipe with custom config for sql_mode:

services:
    database:
        config:
            database: .lando/mysql/my.cnf

Here is the ErrorLog:

image

finally again destroy + start started the app...

@martin-klima

This comment has been minimized.

Copy link
Author

martin-klima commented Feb 3, 2019

I continued with tests on another PC with Linux Mint and the pattern is unpredictible. 1 = OK, 0 = Fail. The 1st run is after lando destroy.

Test1: 1101100110100110011101110 (10/25 fail)
Test2: 1000110000000000000001100 (20/25 fail)
Test3: 1000000000110000110000000(20/25 fail)
Test4: 1011000011101000000110000 (16/25 fail)

The message in log

database_1   | mkdir: cannot create directory '/etc/ssh': Permission denied
database_1   | Running command /entrypoint.sh /run.sh

is always occurs regardless of the result. The only difference I found in log is:

database_1   | INFO  ==> ** Starting MySQL setup **
database_1   | INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars..
database_1   | WARN  ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons, do not use this flag in a production environment.
database_1   | INFO  ==> Initializing mysql database...
database_1   | INFO  ==> Persisted data detected. Restoring...
database_1   | INFO  ==> Stopping mysql...
database_1   | INFO  ==> ** MySQL setup finished! **
database_1   | 
database_1   | INFO  ==> ** Starting MySQL **

for started database and

database_1   | INFO  ==> ** Starting MySQL setup **
database_1   | INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars..
database_1   | WARN  ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons, do not use this flag in a production environment.
database_1   | INFO  ==> Initializing mysql database...
database_1   | INFO  ==> Persisted data detected. Restoring...
database_1   | INFO  ==> Stopping mysql...
database_1   | Terminated

for failed database.

There are also 2 opened issues look very similar to our problem on bitnami-docker-mysql:

@martin-klima martin-klima changed the title Drupal recipes: database is unhealthy after restart Database service is unhealthy after restart Feb 3, 2019

@martin-klima

This comment has been minimized.

Copy link
Author

martin-klima commented Feb 3, 2019

I renamed the issue title because the same problem is also in custom apps with services
type: mysql
type: mysql:8.0

e.g.:

name: testdbissue

services:
  appserver:
    type: mysql:8.0

and not related to Drupal recipes.

@pirog

This comment has been minimized.

Copy link
Member

pirog commented Feb 3, 2019

Ok yeah now i am definitely seeing this again. Investigating.

@pirog

This comment has been minimized.

Copy link
Member

pirog commented Feb 3, 2019

Looks like this also happens for the mariadb service but not postgres.

@pirog

This comment has been minimized.

Copy link
Member

pirog commented Feb 3, 2019

Alright, pretty sure we've got this locked down. Needed to delegate with a new process from the lando entrypoint to the db entrypoint. whew ;)

pirog added a commit that referenced this issue Feb 3, 2019

@pirog

This comment has been minimized.

Copy link
Member

pirog commented Feb 3, 2019

Alright, ive got this in pending tests. We will likely roll a bug fix release later tonight before the wave hits Monday morning! Thanks to everyone for the feedback here

@pirog

This comment has been minimized.

Copy link
Member

pirog commented Feb 4, 2019

OK this is in master and havent run into it in about 500 trials. I THINK WE ARE BACK IN BIZNIZZ!

@pirog pirog closed this Feb 4, 2019

@martin-klima

This comment has been minimized.

Copy link
Author

martin-klima commented Feb 4, 2019

@pirok I can confirm the problem was fixed. Thanks. It had to be a busy weekend.

@sarath49

This comment has been minimized.

Copy link

sarath49 commented Feb 5, 2019

I am facing this issue for postgres with rc5 release on again starting after shutdown on windows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment