MDEV-25670 Run mysql_upgrade / mariadb-upgrade at startup

As this is quite an extensive change MARIADB_AUTO_UPGRADE=1 is
required to opt into the auto upgrade. If may eventually change if
this proves stable and based on user feedback and might change to
on by default.

Per the mysql_upgrade documentation
(https://mariadb.com/kb/en/mysql_upgrade/) we take a backup and leave
this in the datadir. This backup can be disabled with
MARIADB_DISABLE_UPGRADE_BACKUP=1.

The upgrade process requires the entrypoint to start the server
before being available to the use users (for reasons below).

The upgrade will only upgrade-system-tables as highighted in
 #350, a large set of user database can slow the update process
by hours. From feedback in
(https://jira.mariadb.org/browse/MDEV-27068?focusedCommentId=208660&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-208660)
it appears this will rarely if ever be required.

Because the root password may not be known to the entrypoint
(random or removed users), the temporary server is started in
--skip-grant-tables manner to avoid this contraint.

Background information:

MDEV-27068 showed the dangers of running two mysql_upgrades in
parallel.

MDEV-14622 and MDEV-27107 showed two deadlock cases with myql_upgrade
and the spider engine, and its still unknown if create/drop procedure
or create (udf) function will conflict if started from a user connection
instead of a plugin. As there are a few other user features that
depend on a stable set of system tables, a pre-start upgrade is
performed.

Some small hacks of ensuring the mysql_upgrade_info file is populated
is done in anticipation of upstream features being implementing
including:

With MDEV-27279, mariadb_upgrade added --check-if-upgrade-is-needed
we can test if an upgrade is needed. Unfortunately its not offline
yet (MDEV-27636) and requires the start of the server. As such for now
we do our own version checks based on mysql_upgrade_info.

MDEV-27607: mysql_install_db to install mysql_upgrade_info
should facilitate the avoiding of upgrades been needed on the first
installed. _mariadb_fake_upgrade_info function is anticipated to be
removed and likewise for the || true on mysql_upgrade.

The debian/ubuntu packaging component, MDEV-27068 I hope is going
to leave a set of config parameters for safe upgrades requiring a
small patch like:

--- a/docker-entrypoint.sh
+++ b/docker-entrypoint.sh
@@ -424,6 +424,10 @@ docker_mariadb_upgrade() {
                return 0
        fi
        mysql_note "Starting temporary server"
+       local mariadbd="$1"
+       shift
+       docker_temp_server_start "$mariadbd" --defaults-extra-file=/etc/mysql/upgrade.cnf "$@"
+       set -- "$mariadbd" "$@"
        docker_temp_server_start "$@"
        mysql_note "Temporary server started."

Closes #350

.test/run.sh

@@ -106,6 +106,11 @@ othertables=$(mariadbclient -u root --skip-column-names -Be "select group_concat
 
 otherusers=$(mariadbclient -u root --skip-column-names -Be "select user,host from mysql.user where (user,host) not in (('root', 'localhost'), ('root', '%'), ('mariadb.sys', 'localhost'))")
 [ "$otherusers" != '' ] && die "unexpected users $otherusers"
+
+	echo "Contents of /var/lib/mysql/mysql_upgrade_info:"
+	docker exec "$cid" cat /var/lib/mysql/mysql_upgrade_info || die "missing mysql_upgrade_info on install"
+	echo
+
 killoff
 
 	;&
@@ -380,6 +385,71 @@ else
 	echo -e "Test: jemalloc skipped - unknown arch '$architecture'\n"
 fi
 
+	;&
+	mariadbupgrade)
+	docker volume rm m57 || echo "m57 already cleaned"
+	docker volume create m57
+	docker pull docker.io/library/mysql:5.7
+	runandwait -v m57:/var/lib/mysql:Z -e MYSQL_INITDB_SKIP_TZINFO=1 -e MYSQL_ROOT_PASSWORD=bob docker.io/library/mysql:5.7
+	killoff
+
+	runandwait -e MARIADB_AUTO_UPGRADE=1 -v m57:/var/lib/mysql:Z "${image}"
+
+	version=$(mariadbclient --skip-column-names -B -u root -pbob -e "SELECT VERSION()")
+
+	docker exec "$cid" ls -la /var/lib/mysql/system_mysql_backup_unknown_version.sql.zst || die "hopeing for backup file"
+
+	echo "Did the upgrade run?"
+	docker logs "$cid" 2>&1 | grep -A 15 'Starting mariadb-upgrade' || die "missing upgrade message"
+	echo
+
+	docker exec "$cid" ls -la /var/lib/mysql/
+
+	echo "Final upgrade info reflects current version?"
+	docker exec "$cid" cat /var/lib/mysql/mysql_upgrade_info || die "missing mysql_upgrade_info on install"
+	echo
+
+	upgradeversion=$(docker exec "$cid" cat /var/lib/mysql/mysql_upgrade_info)
+	# note VERSION() is longer
+	[[ $version =~ ^${upgradeversion} ]] || die "upgrade version didn't match"
+
+	echo "fix version to 5.x"
+	docker exec "$cid" sed -i -e 's/[0-9]*\(.*\)/5\1/' /var/lib/mysql/mysql_upgrade_info
+	docker exec "$cid" cat /var/lib/mysql/mysql_upgrade_info
+	killoff
+
+	runandwait -e MARIADB_AUTO_UPGRADE=1 -v m57:/var/lib/mysql:Z "${image}"
+
+	echo "Did the upgrade run?"
+	docker logs "$cid" 2>&1 | grep -A 15 'Starting mariadb-upgrade' || die "missing upgrade from prev"
+	echo
+
+	echo "data dir"
+	docker exec "$cid" ls -la /var/lib/mysql/
+	echo
+
+	echo "Is the right backup file there?"
+	docker exec "$cid" ls -la /var/lib/mysql/system_mysql_backup_5."${upgradeversion#*.}".sql.zst || die "missing backup"
+	echo
+
+	echo "Final upgrade info reflects current version?"
+	docker exec "$cid" cat /var/lib/mysql/mysql_upgrade_info || die "missing mysql_upgrade_info on install"
+	echo
+
+	echo "Fixing back to 0 minor version"
+	docker exec "$cid" sed -i -e 's/[0-9]*-\(MariaDB\)/0-\1/' /var/lib/mysql/mysql_upgrade_info
+	upgradeversion=$(docker exec "$cid" cat /var/lib/mysql/mysql_upgrade_info)
+	killoff
+
+	runandwait -e MARIADB_AUTO_UPGRADE=1 -v m57:/var/lib/mysql:Z "${image}"
+	docker exec "$cid" cat /var/lib/mysql/mysql_upgrade_info
+	newupgradeversion=$(docker exec "$cid" cat /var/lib/mysql/mysql_upgrade_info)
+	[ "$upgradeversion" = "$newupgradeversion" ] || die "upgrade versions from mysql_upgrade_info should match"
+       	docker logs "$cid" 2>&1 | grep -C 5 'MariaDB upgrade not required' || die 'should not have upgraded'
+
+	killoff
+	docker volume rm m57
+
 # Insert new tests above by copying the comments below
 #	;&
 #	THE_TEST_NAME)

10.2/docker-entrypoint.sh

@@ -172,6 +172,18 @@ docker_create_db_directories() {
 	fi
 }
 
+_mariadb_version() {
+        local mariaVersion="${MARIADB_VERSION##*:}"
+        mariaVersion="${mariaVersion%%[-+~]*}"
+	echo -n "${mariaVersion}-MariaDB"
+}
+
+_mariadb_fake_upgrade_info() {
+	if [ ! -f "${DATADIR}"/mysql/mysql_upgrade_info ]; then
+		_mariadb_version > "${DATADIR}"/mysql_upgrade_info
+	fi
+}
+
 # initializes the database directory
 docker_init_database_dir() {
 	mysql_note "Initializing database files"
@@ -182,6 +194,7 @@ docker_init_database_dir() {
 	fi
 	# "Other options are passed to mysqld." (so we pass all "mysqld" arguments directly here)
 	mysql_install_db "${installArgs[@]}" "${@:2}" --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin
+	_mariadb_fake_upgrade_info
 	mysql_note "Database files initialized"
 }
 
@@ -320,6 +333,78 @@ docker_setup_db() {
 	fi
 }
 
+# backup the mysql database
+docker_mariadb_backup_system()
+{
+	if [ -n "$MARIADB_DISABLE_UPGRADE_BACKUP" ] \
+		&& [ "$MARIADB_DISABLE_UPGRADE_BACKUP" = 1 ]; then
+		mysql_note "MariaDB upgrade backup disabled due to \$MARIADB_DISABLE_UPGRADE_BACKUP=1 setting"
+		return
+	fi
+	local backup_db="system_mysql_backup_unknown_version.sql.zst"
+	local oldfullversion="unknown_version"
+	if [ -r "$DATADIR"/mysql_upgrade_info ]; then
+		read -r -d '' oldfullversion < "$DATADIR"/mysql_upgrade_info || true
+		if [ -n "$oldfullversion" ]; then
+			backup_db="system_mysql_backup_${oldfullversion}.sql.zst"
+		fi
+	fi
+
+	mysql_note "Backing up system database to $backup_db"
+	if ! mysqldump --skip-lock-tables --replace --databases mysql --socket="${SOCKET}" | zstd > "${DATADIR}/${backup_db}"; then
+		mysql_error "Unable backup system database for upgrade from $oldfullversion."
+	fi
+	mysql_note "Backing up complete"
+}
+
+# perform mariadb-upgrade
+# backup the mysql database if this is a major upgrade
+docker_mariadb_upgrade() {
+	if [ -z "$MARIADB_AUTO_UPGRADE" ] \
+		|| [ "$MARIADB_AUTO_UPGRADE" = 0 ]; then
+		mysql_note "MariaDB upgrade (mysql_upgrade) required, but skipped due to \$MARIADB_AUTO_UPGRADE setting"
+		return
+	fi
+	mysql_note "Starting temporary server"
+	docker_temp_server_start "$@" --skip-grant-tables
+	mysql_note "Temporary server started."
+
+	docker_mariadb_backup_system
+
+	mysql_note "Starting mariadb-upgrade"
+	mysql_upgrade --upgrade-system-tables || true # permission denied fixed in Jan 2022 release?
+	# _mariadb_fake_upgrade_info Possibly fixed by MDEV-27068
+        _mariadb_fake_upgrade_info
+	mysql_note "Finished mariadb-upgrade"
+
+	# docker_temp_server_stop needs authentication since
+	# upgrade ended in FLUSH PRIVILEGES
+	mysql_note "Stopping temporary server"
+	killall mysqld
+	while killall -0 mysqld ; do sleep 1; done
+	mysql_note "Temporary server stopped"
+}
+
+
+_check_if_upgrade_is_needed() {
+	if [ ! -f "$DATADIR"/mysql_upgrade_info ]; then
+		mysql_note "MariaDB upgrade information missing, assuming required"
+		return 0
+	fi
+	local mariadbVersion
+	mariadbVersion="$(_mariadb_version)"
+	IFS='.-' read -ra newversion <<<"$mariadbVersion"
+	IFS='.-' read -ra oldversion < "$DATADIR"/mysql_upgrade_info || true
+
+	if [[ ${#newversion[@]} -lt 2 ]] || [[ ${#oldversion[@]} -lt 2 ]] \
+		|| [[ ${oldversion[0]} -lt ${newversion[0]} ]] \
+		|| [[ ${oldversion[0]} -eq ${newversion[0]} && ${oldversion[1]} -lt ${newversion[1]} ]]; then
+		return 0
+	fi
+	mysql_note "MariaDB upgrade not required"
+	return 1
+}
+
 # check arguments for an option that would cause mysqld to stop
 # return true if there is one
 _mysql_want_help() {
@@ -378,6 +463,10 @@ _main() {
 			echo
 			mysql_note "MariaDB init process done. Ready for start up."
 			echo
+		# MDEV-27636 mariadb_upgrade --check-if-upgrade-is-needed cannot be run offline
+		#elif mysql_upgrade --check-if-upgrade-is-needed; then
+		elif _check_if_upgrade_is_needed; then
+			docker_mariadb_upgrade "$@"
 		fi
 	fi
 	exec "$@"

10.3/docker-entrypoint.sh

@@ -172,6 +172,18 @@ docker_create_db_directories() {
 	fi
 }
 
+_mariadb_version() {
+        local mariaVersion="${MARIADB_VERSION##*:}"
+        mariaVersion="${mariaVersion%%[-+~]*}"
+	echo -n "${mariaVersion}-MariaDB"
+}
+
+_mariadb_fake_upgrade_info() {
+	if [ ! -f "${DATADIR}"/mysql/mysql_upgrade_info ]; then
+		_mariadb_version > "${DATADIR}"/mysql_upgrade_info
+	fi
+}
+
 # initializes the database directory
 docker_init_database_dir() {
 	mysql_note "Initializing database files"
@@ -182,6 +194,7 @@ docker_init_database_dir() {
 	fi
 	# "Other options are passed to mysqld." (so we pass all "mysqld" arguments directly here)
 	mysql_install_db "${installArgs[@]}" "${@:2}" --default-time-zone=SYSTEM --enforce-storage-engine= --skip-log-bin
+	_mariadb_fake_upgrade_info
 	mysql_note "Database files initialized"
 }
 
@@ -320,6 +333,78 @@ docker_setup_db() {
 	fi
 }
 
+# backup the mysql database
+docker_mariadb_backup_system()
+{
+	if [ -n "$MARIADB_DISABLE_UPGRADE_BACKUP" ] \
+		&& [ "$MARIADB_DISABLE_UPGRADE_BACKUP" = 1 ]; then
+		mysql_note "MariaDB upgrade backup disabled due to \$MARIADB_DISABLE_UPGRADE_BACKUP=1 setting"
+		return
+	fi
+	local backup_db="system_mysql_backup_unknown_version.sql.zst"
+	local oldfullversion="unknown_version"
+	if [ -r "$DATADIR"/mysql_upgrade_info ]; then
+		read -r -d '' oldfullversion < "$DATADIR"/mysql_upgrade_info || true
+		if [ -n "$oldfullversion" ]; then
+			backup_db="system_mysql_backup_${oldfullversion}.sql.zst"
+		fi
+	fi
+
+	mysql_note "Backing up system database to $backup_db"
+	if ! mysqldump --skip-lock-tables --replace --databases mysql --socket="${SOCKET}" | zstd > "${DATADIR}/${backup_db}"; then
+		mysql_error "Unable backup system database for upgrade from $oldfullversion."
+	fi
+	mysql_note "Backing up complete"
+}
+
+# perform mariadb-upgrade
+# backup the mysql database if this is a major upgrade
+docker_mariadb_upgrade() {
+	if [ -z "$MARIADB_AUTO_UPGRADE" ] \
+		|| [ "$MARIADB_AUTO_UPGRADE" = 0 ]; then
+		mysql_note "MariaDB upgrade (mysql_upgrade) required, but skipped due to \$MARIADB_AUTO_UPGRADE setting"
+		return
+	fi
+	mysql_note "Starting temporary server"
+	docker_temp_server_start "$@" --skip-grant-tables
+	mysql_note "Temporary server started."
+
+	docker_mariadb_backup_system
+
+	mysql_note "Starting mariadb-upgrade"
+	mysql_upgrade --upgrade-system-tables || true # permission denied fixed in Jan 2022 release?
+	# _mariadb_fake_upgrade_info Possibly fixed by MDEV-27068
+        _mariadb_fake_upgrade_info
+	mysql_note "Finished mariadb-upgrade"
+
+	# docker_temp_server_stop needs authentication since
+	# upgrade ended in FLUSH PRIVILEGES
+	mysql_note "Stopping temporary server"
+	killall mysqld
+	while killall -0 mysqld ; do sleep 1; done
+	mysql_note "Temporary server stopped"
+}
+
+
+_check_if_upgrade_is_needed() {
+	if [ ! -f "$DATADIR"/mysql_upgrade_info ]; then
+		mysql_note "MariaDB upgrade information missing, assuming required"
+		return 0
+	fi
+	local mariadbVersion
+	mariadbVersion="$(_mariadb_version)"
+	IFS='.-' read -ra newversion <<<"$mariadbVersion"
+	IFS='.-' read -ra oldversion < "$DATADIR"/mysql_upgrade_info || true
+
+	if [[ ${#newversion[@]} -lt 2 ]] || [[ ${#oldversion[@]} -lt 2 ]] \
+		|| [[ ${oldversion[0]} -lt ${newversion[0]} ]] \
+		|| [[ ${oldversion[0]} -eq ${newversion[0]} && ${oldversion[1]} -lt ${newversion[1]} ]]; then
+		return 0
+	fi
+	mysql_note "MariaDB upgrade not required"
+	return 1
+}
+
 # check arguments for an option that would cause mysqld to stop
 # return true if there is one
 _mysql_want_help() {
@@ -378,6 +463,10 @@ _main() {
 			echo
 			mysql_note "MariaDB init process done. Ready for start up."
 			echo
+		# MDEV-27636 mariadb_upgrade --check-if-upgrade-is-needed cannot be run offline
+		#elif mysql_upgrade --check-if-upgrade-is-needed; then
+		elif _check_if_upgrade_is_needed; then
+			docker_mariadb_upgrade "$@"
 		fi
 	fi
 	exec "$@"