Error starting mariadb on every host reboot

[sprnza]

Hi there!
I've got a wierd issue I don't know how to debug. Every single reboot I get mariadb unable to start with this error:

2018-09-19  4:29:31 0 [Note] Plugin 'FEEDBACK' is disabled.
2018-09-19  4:29:31 0 [Note] Recovering after a crash using tc.log                                                               
2018-09-19  4:29:31 0 [ERROR] Bad magic header in tc log
2018-09-19  4:29:31 0 [ERROR] Crash recovery failed. Either correct the problem (if it's, for example, out of memory error) and restart, or delete tc log and start mysqld with --tc-heuristic-recover={commit|rollback}                                           
2018-09-19  4:29:31 0 [ERROR] Can't init tc log
2018-09-19  4:29:31 0 [ERROR] Aborting

my docker-compose looks like this

    mariadb:                                                                                                                      
        image: mariadb                                                                                                            
        restart: always                                                                                                           
        ports:                                                                                                                    
            - "3306:3306"                                                                                                         
        volumes:                                                                                                                  
            - /var/lib/mysql:/var/lib/mysql                                                                                       
        environment:                                                                                                              
            MYSQL_ROOT_PASSWORD: my+pass+here                                                                                          
        networks:                                                                                                                 
            - default                                                                                                             
        container_name: mariadb                                                                                                   
        labels:                                                                                                                   
            - "com.centurylinklabs.watchtower.enable=true"                                                                        
            - "traefik.enable=false"   

After I remove tc.log the container starts immediately.
What could be wrong? Not set uid? Very annoying

[tanji]

Are you using existing data or starting from blank installation?

[sprnza]

Do you mean was I migrated from standalone mariadb to dockerized one? Yes, I'm using the data I had before I started to adopt docker.

[tanji]

Did you use the exact same version of MariaDB in your standalone?
Because, in your compose file, you have this:
image: mariadb
This means in the absence of tag, that you're on the very latest version of mariadb published by docker, which would be 10.3.9 if I'm correct. If you were using a different version, serious issues might happen.

[sprnza]

I've disabled the standalone MariaDB once I migrated to docker and I don't remember what version it was (several updates occurred since then) but I suppose it must be the latest one as soon as I use ArchLinux. It works fine merely fails to start on the host machine reboot.

[tianon]

From the logs, it looks to me like your instance isn't shutting down cleanly when your system does -- when the system shuts down, is it doing so cleanly? Is Docker getting the appropriate shutdown signal which it can propagate down to the containers so they can clean up before getting killed? It might help to add a "stop timeout" on your container to give it a little more time to shut down gracefully if it's going too slow (the default is 10 seconds, I believe, which often isn't enough for MariaDB to close out cleanly if you've got slow disks).

[tianon]

Also, it looks like the latest in Arch is 10.1 (https://www.archlinux.org/packages/extra/x86_64/mariadb/), so you might need to run mysql_upgrade or something in order to upgrade from 10.1 to 10.3 (and you should probably look at getting off :latest and onto something slightly more specific like mariadb:10.3).

See https://mariadb.com/kb/en/library/upgrading-from-mariadb-101-to-mariadb-102/ and https://mariadb.com/kb/en/library/upgrading-from-mariadb-102-to-mariadb-103/ for more information on upgrades from 10.1 to 10.2 and 10.3.

[tanji]

It might work fine but innodb versions are really different between 10.1 and 10.3 (innodb 5.6 and 5.7) so that might cause an unclean shutdown. My advice is either:

• run mariadb as a 10.1 container, as @tianon noted Archlinux uses mariadb10.1, so use mariadb :10.1
• mysqldump your data, start a clean container and reimport it

[sprnza]

Thank you gents! mysql_upgrade solved the issue. MariaDB is shutting down gracefully now.