Boot loop mysqld_safe Starting mysqld daemon with databases from /config/databases

[olidroide]

Hi! I'm using docker-compose at Raspberry Pi 3 to execute this arm image, suddenly I had this log error:

cloud.olidroide.es.db      | 180510 10:42:00 mysqld_safe Logging to syslog.
cloud.olidroide.es.db      | 180510 10:42:00 mysqld_safe Starting mysqld daemon with databases from /config/databases

And Im going to bash to execute mysqld_safe --log-error=er.log:

180510 10:51:13 mysqld_safe Can't log to error log and syslog at the same time.  Remove all --log-error configuration options for --syslog to take effect.
180510 10:51:13 mysqld_safe Logging to '/config/databases/er.log'.
180510 10:51:13 mysqld_safe Starting mysqld daemon with databases from /config/databases

the /config/databases/er.log contains:

root@1244ec814333:/# cat /config/databases/er.log 
180510 10:51:13 mysqld_safe Starting mysqld daemon with databases from /config/databases
180510 10:51:13 [Note] /usr/sbin/mysqld (mysqld 10.0.34-MariaDB-0ubuntu0.16.04.1) starting as process 4494 ...
180510 10:51:13 [Note] InnoDB: Using mutexes to ref count buffer pool pages
180510 10:51:13 [Note] InnoDB: The InnoDB memory heap is disabled
180510 10:51:13 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
180510 10:51:13 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
180510 10:51:13 [Note] InnoDB: Compressed tables use zlib 1.2.8
180510 10:51:13 [Note] InnoDB: Using Linux native AIO
180510 10:51:13 [Note] InnoDB: Not using CPU crc32 instructions
180510 10:51:13 [Note] InnoDB: Initializing buffer pool, size = 256.0M
180510 10:51:13 [Note] InnoDB: Completed initialization of buffer pool
180510 10:51:13 [Note] InnoDB: Highest supported file format is Barracuda.
InnoDB: Error: trying to access page number 4294911640 in space 0,
InnoDB: space name ./ibdata1,
InnoDB: which is outside the tablespace bounds.
InnoDB: Byte offset 0, len 16384, i/o type 10.
InnoDB: If you get this error at mysqld startup, please check that
InnoDB: your my.cnf matches the ibdata files that you have in the
InnoDB: MySQL server.
2018-05-10 10:51:14 76f1c000  InnoDB: Assertion failure in thread 1995554816 in file fil0fil.cc line 5596
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
180510 10:51:14 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.0.34-MariaDB-0ubuntu0.16.04.1
key_buffer_size=16777216
read_buffer_size=2097152
max_used_connections=0
max_threads=102
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 644359 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x30000
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
180510 10:51:14 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

I'm trying to remove and install again the image but always keep on boot load. What's the issue here?

Thanks!

[j0nnymoe]

Likely a corrupted database, to quote the logs:

InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.```

[olidroide]

Thanks! I'm stop the mariadb docker instance and I add to .cnf file:

[mysqld]
innodb_force_recovery = 6

after that run again the mariadb instance and I enter to bash to dump the databases with:

mysqldump -p database_name > database_name.sql

after that, stop again mariadb, and remove old databases files, remove innodb_force_recovery from .cnf file and I start from scratch to restore databases dumps with:

mysql -p database_name < database_name.sql

I feel is everything ok! thanks @j0nnymoe

[j0nnymoe]

Glad you got it working. I would highly recommend getting backups in place of your mariadb db files as I've seen cases where this doesn't fix the issue.

[olidroide]

Any suggestion to make backups from docker image? I have a volume than point to external hard drive.

Thanks again @j0nnymoe

[j0nnymoe]

Anything that can take snapshots of the files you have exposed on your host, rsync for example.

[olidroide]

Thanks @j0nnymoe

[aptalca]

I have a script that stops all containers, tars up their config folders and restarts them. I run it about once a week.

The backup tar file gets copied to a remote location

[olidroide]

Hi @aptalca do you have a Gist or repo with that script? Thanks :)

[aptalca]

Assuming the user is user and the config folders are all under /home/user/docker, then something like this would update all images, backup and recreate containers:

docker-compose pull --parallel
docker-compose down
sudo tar -C /home/user -cvzf /home/user/backup.tar.gz docker
sudo chown user:user /home/user/backup.tar.gz
docker-compose up -d
docker image prune -f

prune will remove all stale (untagged) images
sudo is needed for tar if there are any files in the config folders that your user cannot access

[olidroide]

Nice @aptalca I will create a shellscript to run in a cron with these lines :)