Misc Debian/Salsa-CI fixes (#2299)

* Deb: Handle codename 'n/a' from Debian Sid properly and autobake-deb cleanup This fixes autobake-deb.sh builds on Sid which was visible as 4 failing build steps on Salsa-CI. - In Sid the LSBNAME might evaluate to 'n/a', so accept it as 'Sid' to fix builds that failed with error: Error - unknown release codename n/a - Refactor list to have Ubuntu versions first, then Debian, and as last the special case of Debian Sid - Fix minor syntax issues detected by Shellcheck Also remove useless DEB_HOST_ARCH_CPU check from debian/rules: * It was never in effect as the 'sed' in autobake-deb.sh cleared it anyway * The variable name was wrong and always empty * If variable would have been correct, logic was still reversed - Define 3h timeout as the default 1h timeout on Gitlab.com (and others) is usually not enough for initial (uncached) MariaDB builds. - Replace Buster to Bookworm/Sid upgrade testing with upgrade inside Buster testing as direct upgrades from Stretch to Bullseye and Buster to Bookworm are no longer possible due to: Bug#993755: libcrypt.so.1: cannot open shared object file when upgrading from Stretch to Sid (https://bugs.debian.org/993755) - Stop ignoring MariaDB.org 10.6 to this version upgrade testing failures to reveal bug MDEV-28640. Originally this step was failing as the uring dependencies in upstream builders lagged behind and there was nothing that needed work, only time time to resolve. Now there is an actual bug in packaging that should be visible as a CI failure. - Stop testing for 'service mysql status' on systems that upgraded from MySQL 8.0 to MariaDB.org vended 10.6. Due to some unidentified debian/control changes in 10.6 on upstream the upgrade is no longer compatible in a way that would maintain the init.d script with name 'mysql'. - Fix typos where mergers had changed occurrences of 10.5 to 10.6 while they intentionally need to be exactly 10.5, otherwise the meaning changes. - Align autopkgtest code with downstream official Debian packaging one. This is change is safe on a stable branch because is only affects builds and testing, not any actual usage of MariaDB 10.6. - Standardize on using capitalized 'YES' in CMake build options (instead of 'yes' or mixed case) - Add some comments to better document debian/rules - Fix typo in Lintian overrides Ubuntu bug: https://bugs.launchpad.net/ubuntu/+source/mariadb-10.6/+bug/1970634 MariaDB ticket: https://jira.mariadb.org/browse/MDEV-25633 When built with LTO on Ubuntu, MariaDB does not catch an exception when the uring initialization fails due to a low RLIMIT_MEMLOCK value. This commit amends the commit 0609b34 to be identical to the one done downstream in Debian: https://salsa.debian.org/mariadb-team/mariadb-server/-/commit/8d20ca979cf422d3a507283b86c2547d78559179 This way both the inline comments and 'git blame' for this section will show properly why this is needed, and the fix is one that is fully tested on Debian and Ubuntu. Also having this section fully identical in upstream MariaDB and downstream Debian will make the packaging maintenance easier as 'diff` runs on this file will not flag this as a difference anymore. In MDEV-28640 the init script failed to stop/start the MariaDB server due to missing mysqladmin on the system. This was however very hard to spot from the console output. Add an explicit check for the binary the script depends on, and fail verbosely if the dependency is missing.
MariaDB · Nov 11, 2022 · f4adf35 · f4adf35
1 parent 67f4ba2
commit f4adf35
Show file tree

Hide file tree

Showing 6 changed files with 120 additions and 58 deletions.
diff --git a/debian/autobake-deb.sh b/debian/autobake-deb.sh
@@ -25,9 +25,9 @@ then
   sed '/Add support for verbose builds/,/^$/d' -i debian/rules
 elif [ -d storage/columnstore/columnstore/debian ]
 then
-  # ColumnStore is explicitly disabled in the native Debian build, so allow it
+  # ColumnStore is explicitly disabled in the native Debian build. Enable it
   # now when build is triggered by autobake-deb.sh (MariaDB.org) and when the
-  # build is not running on Travis or Gitlab-CI
+  # build is not running on Gitlab-CI.
   sed '/-DPLUGIN_COLUMNSTORE=NO/d' -i debian/rules
   # Take the files and part of control from MCS directory
   if [ ! -f debian/mariadb-plugin-columnstore.install ]
@@ -58,26 +58,45 @@ replace_uring_with_aio()
 {
   sed 's/liburing-dev/libaio-dev/g' -i debian/control
   sed -e '/-DIGNORE_AIO_CHECK=YES/d' \
-      -e '/-DWITH_URING=yes/d' -i debian/rules
+      -e '/-DWITH_URING=YES/d' -i debian/rules
 }
 
 disable_pmem()
 {
   sed '/libpmem-dev/d' -i debian/control
-  sed '/-DWITH_PMEM=yes/d' -i debian/rules
+  sed '/-DWITH_PMEM=YES/d' -i debian/rules
 }
 
 architecture=$(dpkg-architecture -q DEB_BUILD_ARCH)
 
+# Parse release name and number from Linux standard base release
+# Example:
+#   $ lsb_release -a
+#   No LSB modules are available.
+#   Distributor ID:	Debian
+#   Description:	Debian GNU/Linux bookworm/sid
+#   Release:	n/a
+#   Codename:	n/a
 LSBID="$(lsb_release -si  | tr '[:upper:]' '[:lower:]')"
 LSBVERSION="$(lsb_release -sr | sed -e "s#\.##g")"
 LSBNAME="$(lsb_release -sc)"
 
+# If 'n/a', assume 'sid'
+if [ "${LSBVERSION}" == "n/a" ] || [ "${LSBNAME}" == "n/a" ]
+then
+  LSBVERSION="sid"
+  LSBNAME="sid"
+fi
+
+# If not known, use 'unknown' in .deb version identifier
 if [ -z "${LSBID}" ]
 then
     LSBID="unknown"
 fi
-case "${LSBNAME}" in
+
+case "${LSBNAME}"
+in
+  # Debian
   stretch)
     # MDEV-16525 libzstd-dev-1.1.3 minimum version
     sed -e '/libzstd-dev/d' \
@@ -105,10 +124,10 @@ case "${LSBNAME}" in
     fi
     ;&
   sid)
-    # should always be empty here.
-    # need to match here to avoid the default Error however
+    # The default packaging should always target Debian Sid, so in this case
+    # there is intentionally no customizations whatsoever.
     ;;
-    # UBUNTU
+  # Ubuntu
   bionic)
     remove_rocksdb_tools
     [ "$architecture" != amd64 ] && disable_pmem
@@ -134,7 +153,7 @@ case "${LSBNAME}" in
     fi
     ;;
   *)
-    echo "Error - unknown release codename $LSBNAME" >&2
+    echo "Error: Unknown release '$LSBNAME'" >&2
     exit 1
 esac
 
@@ -153,7 +172,7 @@ LOGSTRING="MariaDB build"
 EPOCH="1:"
 VERSION="${EPOCH}${UPSTREAM}${PATCHLEVEL}~${LSBID:0:3}${LSBVERSION}"
 
-dch -b -D ${LSBNAME} -v "${VERSION}" "Automatic build with ${LOGSTRING}." --controlmaint
+dch -b -D "${LSBNAME}" -v "${VERSION}" "Automatic build with ${LOGSTRING}." --controlmaint
 
 echo "Creating package version ${VERSION} ... "
 

diff --git a/debian/mariadb-server-10.6.mariadb.init b/debian/mariadb-server-10.6.mariadb.init
@@ -21,9 +21,18 @@ test -x /usr/sbin/mariadbd || exit 0
 
 . /lib/lsb/init-functions
 
-SELF=$(cd $(dirname $0); pwd -P)/$(basename $0)
-
-MYADMIN="/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf"
+SELF=$(cd "$(dirname $0)"; pwd -P)/$(basename $0)
+
+if [ -f /usr/bin/mariadb-admin ]
+then
+  MYADMIN="/usr/bin/mariadb-admin --defaults-file=/etc/mysql/debian.cnf"
+elif [ -f /usr/bin/mysqladmin ]
+then
+  MYADMIN="/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf"
+else
+  log_failure_msg "Command mariadb-admin/mysqladmin not found! This SysV init script depends on it."
+  exit -1
+fi
 
 # priority can be overridden and "-s" adds output to stderr
 ERR_LOGGER="logger -p daemon.err -t /etc/init.d/mariadb -i"

diff --git a/debian/rules b/debian/rules
@@ -5,7 +5,13 @@ export DEB_BUILD_HARDENING=1
 
 # enable Debian Hardening
 # see: https://wiki.debian.org/Hardening
-export DEB_BUILD_MAINT_OPTIONS = hardening=+all optimize=-lto
+export DEB_BUILD_MAINT_OPTIONS = hardening=+all
+
+# Disable LTO on Ubuntu, see LP: #1970634 and https://jira.mariadb.org/browse/MDEV-25633
+ifeq ($(shell dpkg-vendor --derives-from Ubuntu && echo yes), yes)
+    export DEB_BUILD_MAINT_OPTIONS += optimize=-lto
+endif
+
 DPKG_EXPORT_BUILDFLAGS = 1
 # Include all defaults, including buildflags.mk
 include /usr/share/dpkg/default.mk
@@ -46,11 +52,6 @@ ifeq (32,$(DEB_HOST_ARCH_BITS))
     CMAKEFLAGS += -DPLUGIN_ROCKSDB=NO
 endif
 
-# ColumnStore can build only on amd64 and arm64
-ifneq (,$(filter $(DEB_HOST_ARCH_CPU),amd64 arm64))
-    CMAKEFLAGS += -DPLUGIN_COLUMNSTORE=NO
-endif
-
 # Cross building requires stack direction instruction
 ifneq ($(DEB_BUILD_ARCH),$(DEB_HOST_ARCH))
     ifneq (,$(filter $(DEB_HOST_ARCH_CPU),alpha amd64 arm arm64 i386 ia64 m68k mips64el mipsel powerpc ppc64 ppc64el riscv64 s390x sh4 sparc64))
@@ -64,7 +65,7 @@ endif
 # Only attempt to build with PMEM on archs that have package libpmem-dev available
 # See https://packages.debian.org/search?searchon=names&keywords=libpmem-dev
 ifneq (,$(filter $(DEB_HOST_ARCH_CPU),amd64 arm64 ppc64el riscv64))
-		CMAKEFLAGS += -DWITH_PMEM=yes
+    CMAKEFLAGS += -DWITH_PMEM=YES
 endif
 
 # Add support for verbose builds
@@ -105,9 +106,9 @@ endif
 	    -DBUILD_CONFIG=mysql_release \
 	    -DCONC_DEFAULT_CHARSET=utf8mb4 \
 	    -DPLUGIN_AWS_KEY_MANAGEMENT=NO \
-			-DPLUGIN_COLUMNSTORE=NO \
+	    -DPLUGIN_COLUMNSTORE=NO \
 	    -DIGNORE_AIO_CHECK=YES \
-	    -DWITH_URING=yes \
+	    -DWITH_URING=YES \
 	    -DDEB=$(DEB_VENDOR)
 
 # This is needed, otherwise 'make test' will run before binaries have been built
@@ -120,6 +121,7 @@ override_dh_auto_build:
 override_dh_auto_test:
 	@echo "RULES.$@"
 	dh_testdir
+	# Ensure at least an empty file exists
 	touch mysql-test/unstable-tests
 	[ ! -f debian/unstable-tests.$(DEB_HOST_ARCH) ] || cat debian/unstable-tests.$(DEB_HOST_ARCH) >> mysql-test/unstable-tests
 	# Run testsuite
@@ -203,6 +205,7 @@ override_dh_installinit-arch:
 	dh_installinit --name=mariadb --no-start -- defaults 19 21
 	dh_systemd_start --restart-after-upgrade
 
+# Use custom server version string variable
 override_dh_gencontrol:
 	dh_gencontrol -- -Tdebian/substvars
 

diff --git a/debian/salsa-ci.yml b/debian/salsa-ci.yml
@@ -16,12 +16,17 @@ variables:
   GIT_SUBMODULE_STRATEGY: recursive
   SALSA_CI_GBP_BUILDPACKAGE_ARGS: "--git-submodules" # did not apply to extract-sources
 
+# Extend Salsa-CI build jobs to have longer timeout as the default GitLab
+# timeout (1h) is often not enough
+.build-package:
+  timeout: 3h
+
 stages:
   - provisioning
   - build
   - test
   - upgrade in Sid
-  - upgrade from Bullseye/Buster
+  - upgrade from Bullseye
   - upgrade extras
   - test extras
   - publish # Stage referenced by Salsa-CI template aptly stanza, so must exist even though not used
@@ -134,6 +139,17 @@ blhc:
   apt-get update
   apt-get install -y apt
 
+.test-enable-buster-backports-repos: &test-enable-buster-backports-repos |
+  # Enable buster-backports (assumes environment already Debian Buster)
+  echo 'deb http://deb.debian.org/debian buster-backports main' > /etc/apt/sources.list.d/buster-backports.list
+  # Increase default backports priority policy from '100' to '500' so it can actually be used
+  cat << EOF > /etc/apt/preferences.d/enable-backports-to-satisfy-dependencies
+  Package: *
+  Pin: release n=buster-*
+  Pin-Priority: 500
+  EOF
+  apt-get update
+
 .test-install: &test-install |
   # Install MariaDB built in this commit
   apt-get install -y ./*.deb
@@ -213,7 +229,7 @@ fresh install:
   script:
     - *test-prepare-container
     - *test-install
-    - service mariadb status # There is no init.d/mysql in MariaDB 10.6
+    - service mariadb status # There is no init.d/mysql in MariaDB 10.5+
     - *test-verify-final
   variables:
     GIT_STRATEGY: none
@@ -234,7 +250,7 @@ mariadb-10.6 Sid upgrade:
   script:
     - *test-prepare-container
     - *test-install
-    - service mariadb status # There is no init.d/mysql in MariaDB 10.6
+    - service mariadb status # There is no init.d/mysql in MariaDB 10.5+
     - *test-verify-final
   variables:
     GIT_STRATEGY: none
@@ -243,7 +259,7 @@ mariadb-10.6 Sid upgrade:
       - $CI_COMMIT_TAG != null && $SALSA_CI_ENABLE_PIPELINE_ON_TAGS !~ /^(1|yes|true)$/
 
 mariadb-10.5 Bullseye to mariadb-10.6 upgrade:
-  stage: upgrade from Bullseye/Buster
+  stage: upgrade from Bullseye
   needs:
     - job: build
   image: debian:bullseye
@@ -268,10 +284,12 @@ mariadb-10.5 Bullseye to mariadb-10.6 upgrade:
     variables:
       - $CI_COMMIT_TAG != null && $SALSA_CI_ENABLE_PIPELINE_ON_TAGS !~ /^(1|yes|true)$/
 
-mariadb-10.3 Buster to mariadb-10.6 upgrade:
-  stage: upgrade from Bullseye/Buster
+# Upgrade of libcrypt.so.1 no longer possible from Buster to Sid,
+# so test upgrade only inside Buster (https://bugs.debian.org/993755)
+mariadb-10.3 to mariadb-10.6 upgrade in Buster:
+  stage: upgrade extras
   needs:
-    - job: build
+    - job: build buster-backports
   image: debian:buster
   artifacts:
     when: always
@@ -284,7 +302,7 @@ mariadb-10.3 Buster to mariadb-10.6 upgrade:
     - apt-get install -y 'default-mysql*' 'mariadb-*' 'libmariadb*'
     # Verify installation of MariaDB from Buster
     - *test-verify-initial
-    - *test-enable-sid-repos
+    - *test-enable-buster-backports-repos
     - *test-install
     - service mysql status
     - *test-verify-final
@@ -307,7 +325,7 @@ test basic features:
   script:
     - *test-prepare-container
     - *test-install
-    - service mariadb status # There is no init.d/mysql in MariaDB 10.6
+    - service mariadb status # There is no init.d/mysql in MariaDB 10.5+
     - *test-verify-final
     - |
       # Print info about server
@@ -452,7 +470,7 @@ default-libmysqlclient-dev Sid upgrade:
       - $CI_COMMIT_TAG != null && $SALSA_CI_ENABLE_PIPELINE_ON_TAGS !~ /^(1|yes|true)$/
 
 default-libmysqlclient-dev Bullseye upgrade:
-  stage: upgrade from Bullseye/Buster
+  stage: upgrade from Bullseye
   needs:
     - job: build
   image: debian:bullseye
@@ -472,10 +490,12 @@ default-libmysqlclient-dev Bullseye upgrade:
     variables:
       - $CI_COMMIT_TAG != null && $SALSA_CI_ENABLE_PIPELINE_ON_TAGS !~ /^(1|yes|true)$/
 
-default-libmysqlclient-dev Buster upgrade:
-  stage: upgrade from Bullseye/Buster
+# Upgrade of libcrypt.so.1 no longer possible from Buster to Sid,
+# so test upgrade only inside Buster (https://bugs.debian.org/993755)
+default-libmysqlclient-dev upgrade in Buster:
+  stage: upgrade extras
   needs:
-    - job: build
+    - job: build buster-backports
   image: debian:buster
   artifacts:
     when: always
@@ -486,7 +506,7 @@ default-libmysqlclient-dev Buster upgrade:
     - *test-prepare-container
     - apt-get install -y pkg-config default-libmysqlclient-dev
     - pkg-config --list-all
-    - *test-enable-sid-repos
+    - *test-enable-buster-backports-repos
     - *test-install-all-libs
     - *test-verify-libs
   except:
@@ -511,8 +531,20 @@ mysql-8.0 Sid to mariadb-10.6 upgrade:
     - apt-get install -y procps mysql-server 'libmysqlc*'
     - *test-verify-initial
     - *test-install
-    - service mysql status
+    # The Debian version of MariaDB 10.6 still maintains compatibility and there
+    # running 'service mysql status' in Salsa-CI job 'mysql-8.0 Sid to
+    # mariadb-10.6 upgrade' still works.
+    #
+    # However, due to debian/control changes, the upstream MariaDB 10.6 when
+    # installed on a system with a previous installation of MySQL 8.0 will first
+    # fully remove MySQL, including the /etc/init.d/mysql file, so previous
+    # techniques in mariadb-server-10.6.postinst to maintain backwards
+    # compatibility with 'service mysql status' after installing MariaDB on top
+    # MySQL no longer works, and thus the step to test it now intentionally has
+    # a fallback to use the service name 'mariadb' instead, and the fallback is
+    # always used.
     - sleep 5 # Give the mysql_upgrade a bit of time to complete before querying the server
+    - service mysql status || service mariadb status
     - *test-verify-final
   variables:
     GIT_STRATEGY: none
@@ -522,7 +554,7 @@ mysql-8.0 Sid to mariadb-10.6 upgrade:
 
 # Upgrading from MySQL 8.0 with datadir in place is not possible. Users need to do a data dump.
 # The Debian maintainer scripts detect this situation and simply moves old datadir aside and start fresh.
-mysql-8.0 Focal to mariadb-10.6 upgrade:
+mysql-8.0 Focal to mariadb-10.6 upgrade in Buster:
   stage: upgrade extras
   needs:
     - job: build buster-backports
@@ -582,18 +614,13 @@ mariadb.org-10.6 to mariadb-10.6 upgrade:
     # Verify installation of MariaDB built in this commit
     - dpkg -l | grep -iE 'maria|mysql|galera' || true # List installed
     - mariadb --version # Client version
-    - service mariadb status # There is no init.d/mysql in MariaDB 10.6
+    - service mariadb status # There is no init.d/mysql in MariaDB 10.5+
     - *test-verify-final
   variables:
     GIT_STRATEGY: none
   except:
     variables:
       - $CI_COMMIT_TAG != null && $SALSA_CI_ENABLE_PIPELINE_ON_TAGS !~ /^(1|yes|true)$/
-  allow_failure: true
-  # Installation on Sid fails on missing liburing1 because upstream 10.6
-  # MariaDB.org buildbot has not run 'apt upgrade' for a long time.
-  # Remove this allow_failure once buildbot has built a new 10.6
-  # release using latest liburing-dev in Debian Sid.
 
 mariadb.org-10.5 to mariadb-10.6 upgrade:
   stage: upgrade extras
@@ -617,7 +644,7 @@ mariadb.org-10.5 to mariadb-10.6 upgrade:
     # Verify installation of MariaDB built in this commit
     - dpkg -l | grep -iE 'maria|mysql|galera' || true # List installed
     - mariadb --version # Client version
-    - service mariadb status # There is no init.d/mysql in MariaDB 10.5
+    - service mariadb status # There is no init.d/mysql in MariaDB 10.5+
     - *test-verify-final
   variables:
     GIT_STRATEGY: none
@@ -728,7 +755,7 @@ mariadb.org-10.2 to mariadb-10.6 upgrade:
     variables:
       - $CI_COMMIT_TAG != null && $SALSA_CI_ENABLE_PIPELINE_ON_TAGS !~ /^(1|yes|true)$/
 
-mysql.com-5.7 to mariadb-10.6 upgrade:
+mysql.com-5.7 to mariadb-10.6 upgrade in Buster:
   stage: upgrade extras
   needs:
     - job: build buster-backports
@@ -759,7 +786,7 @@ mysql.com-5.7 to mariadb-10.6 upgrade:
     variables:
       - $CI_COMMIT_TAG != null && $SALSA_CI_ENABLE_PIPELINE_ON_TAGS !~ /^(1|yes|true)$/
 
-percona-xtradb-5.7 to mariadb-10.6 upgrade (MDEV-22679):
+percona-xtradb-5.7 to mariadb-10.6 upgrade in Buster (MDEV-22679):
   stage: upgrade extras
   needs:
     - job: build buster-backports

diff --git a/debian/source/lintian-overrides b/debian/source/lintian-overrides
@@ -48,7 +48,7 @@ very-long-line-length-in-source-file mysql-test/std_data/init_file_longline_3816
 very-long-line-length-in-source-file scripts/fill_help_tables.sql *
 very-long-line-length-in-source-file scripts/mysql_system_tables.sql *
 very-long-line-length-in-source-file scripts/mysql_test_data_timezone.sql *
-# Machine formated HTML
+# Machine formatted HTML
 very-long-line-length-in-source-file sql/share/charsets/languages.html *
 very-long-line-length-in-source-file sql/share/errmsg-utf8.txt *
 # Very long test string