feat: add support for MariaDB 11.4 LTS, fixes #6061 #6243

hussainweb · 2024-05-24T18:42:27Z

The Issue

MariaDB 11.4 is LTS. We can add support for that here.

MariaDB 11.x removed mysql* symlinks from the Docker images:

latest tag 1:11.0.2+maria~ubu2204 - mysql command is gone MariaDB/mariadb-docker#512

And added warning messages:

https://mariadb.com/kb/en/suppress-mariadb-11-deprecation-warnings/

How This PR Solves The Issue

This PR adds MariaDB 11.4 to Makefile for the Docker image and to the allowed options list.
Adds mysql* symlinks back for mariadb:11.4 Docker image.
Removes deprecation messages for mysql and mysqldump from ddev debug migrate-database, ddev export-db and ddev import-db:

mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead
mysqldump: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb-dump' instead

Manual Testing Instructions

On a DDEV project, run ddev config --database=mariadb:11.4 and make sure it works.
ddev debug migrate-database mariadb:10.11 and ddev debug migrate-database mariadb:11.4
ddev export-db --file=db.sql.gz
ddev import-db --file=db.sql.gz

Automated Testing Overview

I mainly tried to copy the PR #4902 and I couldn't find any tests to change here.

Release/Deployment Notes

rfay · 2024-05-24T19:01:21Z

Wow, so ambitious! Congrats! It will require images to be pushed and the tag to be updated.

hussainweb · 2024-05-24T19:48:25Z

@rfay, I see. I think you have to push the images, right? I see where I have to update the tag but what should I set it as? Should I match 20240524_mariadb_11_4 as per the other PR?

rfay · 2024-05-24T20:36:11Z

I almost always use the branch name as tag, yes.

If you want to push it into your own hub.docker.com account you'll be able to test it... Otherwise, we'll get to it an push it. But you always want to be able to test your own PR yourself right ? :)

It takes some time to push, may be frustrating. It's easier for us to do with the automated workflows.

rfay · 2024-05-24T23:36:41Z

For some reason they don't yet have a mariadb:11.4 docker image out there at this point, see https://hub.docker.com/_/mariadb/tags?page=&page_size=&ordering=&name=11.4

As a result... we won't have one either.

I'm going to close this for now, but when you are able to

cd containers/ddev-dbserver
make mariadb_11.4

Please come back and comment and we'll re-open it.

There may be problems that have to be resolved with the upstream jammy image.

hussainweb · 2024-05-25T11:52:34Z

Thanks. I didn't realize that it needed that base image (I was in a different impression from a different task long back). I'll reopen this when the Docker image is available.

For then, can you point me to instructions on how I would test this locally? It was not obvious to me from this README. I suppose that since there are no other changes to DDEV code except allowed options, I can just use this as base image in my config.yaml. Is that good enough?

hussainweb · 2024-05-25T11:56:37Z

Also, to be able to push to my own Docker Hub, I suppose I should modify this line in build_image.sh. Am I missing something obvious?

rfay · 2024-05-25T12:35:04Z

`make mariadb_11.4_both VERSION=<your-branch-tag``

Change the tag in versionconstants.go.

Test.

You don't have to push it to test. If you did want to push it, `make PUSH=true VERSION= mariadb_11.4_both

More in https://ddev.readthedocs.io/en/stable/developers/release-management/#pushing-docker-images-manually

stasadev · 2024-06-07T13:31:56Z

I tested this PR, Docker images for mariadb:11.4 are ready.

There are compatibility breaking changes in MariaDB 11.

The image cannot be built:

 > [14/22] RUN /create_base_db.sh:
0.371 Initializing mysql
0.371 /create_base_db.sh: line 20: mysqld: command not found
------
Dockerfile:60

There is a new compatibility package mariadb-client-compat:

$ docker run --rm -it mariadb:11.4 bash -c "apt update && apt install mariadb-client-compat && dpkg -L mariadb-client-compat"
/.
/usr
/usr/bin
/usr/share
/usr/share/doc
/usr/share/doc/mariadb-client-compat
/usr/share/doc/mariadb-client-compat/changelog.gz
/usr/share/doc/mariadb-client-compat/copyright
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/mysql.1.gz
/usr/share/man/man1/mysql_convert_table_format.1.gz
/usr/share/man/man1/mysql_find_rows.1.gz
/usr/share/man/man1/mysql_fix_extensions.1.gz
/usr/share/man/man1/mysql_plugin.1.gz
/usr/share/man/man1/mysql_secure_installation.1.gz
/usr/share/man/man1/mysql_setpermission.1.gz
/usr/share/man/man1/mysql_tzinfo_to_sql.1.gz
/usr/share/man/man1/mysql_waitpid.1.gz
/usr/share/man/man1/mysqlaccess.1.gz
/usr/share/man/man1/mysqladmin.1.gz
/usr/share/man/man1/mysqlbinlog.1.gz
/usr/share/man/man1/mysqlcheck.1.gz
/usr/share/man/man1/mysqldump.1.gz
/usr/share/man/man1/mysqldumpslow.1.gz
/usr/share/man/man1/mysqlhotcopy.1.gz
/usr/share/man/man1/mysqlimport.1.gz
/usr/share/man/man1/mysqlshow.1.gz
/usr/share/man/man1/mysqlslap.1.gz
/usr/bin/mysql
/usr/bin/mysql_convert_table_format
/usr/bin/mysql_find_rows
/usr/bin/mysql_fix_extensions
/usr/bin/mysql_plugin
/usr/bin/mysql_secure_installation
/usr/bin/mysql_setpermission
/usr/bin/mysql_tzinfo_to_sql
/usr/bin/mysql_waitpid
/usr/bin/mysqlaccess
/usr/bin/mysqladmin
/usr/bin/mysqlanalyze
/usr/bin/mysqlbinlog
/usr/bin/mysqlcheck
/usr/bin/mysqldump
/usr/bin/mysqldumpslow
/usr/bin/mysqlhotcopy
/usr/bin/mysqlimport
/usr/bin/mysqloptimize
/usr/bin/mysqlrepair
/usr/bin/mysqlreport
/usr/bin/mysqlshow
/usr/bin/mysqlslap
/usr/share/man/man1/mysqlanalyze.1.gz
/usr/share/man/man1/mysqloptimize.1.gz
/usr/share/man/man1/mysqlrepair.1.gz
/usr/share/man/man1/mysqlreport.1.gz

But it doesn't have mysqld inside.

Edit: there is another package for server symlinks, mariadb-server-compat:

$ docker run --rm -it mariadb:11.4 bash -c "apt update && apt install mariadb-server-compat && dpkg -L mariadb-server-compat"
/.
/usr
/usr/bin
/usr/sbin
/usr/share
/usr/share/doc
/usr/share/doc/mariadb-server-compat
/usr/share/doc/mariadb-server-compat/changelog.gz
/usr/share/doc/mariadb-server-compat/copyright
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/mysql_install_db.1.gz
/usr/share/man/man1/mysql_upgrade.1.gz
/usr/share/man/man1/mysqld_multi.1.gz
/usr/share/man/man1/mysqld_safe.1.gz
/usr/share/man/man1/mysqld_safe_helper.1.gz
/usr/share/man/man8
/usr/share/man/man8/mysqld.8.gz
/usr/bin/mysql_install_db
/usr/bin/mysql_upgrade
/usr/bin/mysqld_multi
/usr/bin/mysqld_safe
/usr/bin/mysqld_safe_helper
/usr/sbin/mysqld

rfay · 2024-06-07T13:44:02Z

They're working on naming changes. mysqld becomes mariadbd. We should be able to accommodate that OK.

rfay · 2024-06-07T13:45:45Z

Perhaps the simplest thing is to add a symlink, ln -s /usr/bin/mariadbd /usr/bin/mysqld. Otherwise we can fiddle with the logic so it knows about mariadbd or tries it if mysqld is not found.

stasadev · 2024-06-07T14:19:55Z

It's hard-coded to show deprecation for each of the programs, that are called by symlinks:

$ docker run --rm -it mariadb:11.4 bash -c "apt update && apt install mariadb-client-compat && mariadb-dump --version && mysqldump --version"

mariadb-dump from 11.4.2-MariaDB, client 10.19 for debian-linux-gnu (x86_64)

mysqldump: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb-dump' instead
mysqldump from 11.4.2-MariaDB, client 10.19 for debian-linux-gnu (x86_64)

$ docker run --rm -it mariadb:11.4 bash -c "ln -s /usr/sbin/mariadbd /usr/sbin/mysqld && mariadbd --version && mysqld --version"

mariadbd  Ver 11.4.2-MariaDB-ubu2404 for debian-linux-gnu on x86_64 (mariadb.org binary distribution)

mysqld: Deprecated program name. It will be removed in a future release, use '/usr/sbin/mariadbd' instead
mysqld  Ver 11.4.2-MariaDB-ubu2404 for debian-linux-gnu on x86_64 (mariadb.org binary distribution)

Therefore, it is probably better to use the original names.

rfay · 2024-06-07T14:28:42Z

Therefore, it is probably better to use the original names

I assume you mean the new name, mariadbd ?

stasadev · 2024-06-07T14:38:08Z

I assume you mean the new name, mariadbd ?

Yes, I think we can add a check: if there is mysqld command available, then we can use all mysql* commands, otherwise use mariadb*

So this will continue to work for older Docker mariadb images with mysqld available, but the new mariadb:11.4 will introduce a breaking change - and if people want to migrate, they will know about it.

Of course, it requires more attention, but it's worth it in the long run, I don't think people will be happy when they switch to mariadb:11.4 and see deprecation messages for every command they run.

stasadev · 2024-06-07T14:40:58Z

But I'm not sure about all the other things that have mysql, like /var/lib/mysql, /var/tmp/mysql.sock, and our .ddev/mysql directory.

rfay · 2024-06-07T14:51:48Z

Each of those should be driven by configuration, so should not be affected by their name changing. However, the default /etc/mysql.conf could definitely be affected.

stasadev · 2024-06-10T14:51:22Z

Rebased to resolve conflicts.

rfay · 2024-06-14T19:47:47Z

Yes, I think we can add a check: if there is mysqld command available, then we can use all mysql* commands, otherwise use mariadb*

I think we can and should continue using the client and server compat packages and keep things the same for now.

libmariadb-dev-compat/unknown 1:11.4.2+maria~ubu2404 arm64
  MariaDB Connector/C, compatibility symlinks
libmariadb3-compat/unknown 1:11.4.2+maria~ubu2404 arm64
  MariaDB database client library MySQL compat package
mariadb-client-compat/unknown 1:11.4.2+maria~ubu2404 all
  MySQL compatibility links to mariadb-client binaries/scripts.
mariadb-server-compat/unknown 1:11.4.2+maria~ubu2404 all
  MySQL compatibility links to mariadb-server binaries/scripts.

stasadev · 2024-06-17T19:02:56Z

Rebased.

I added mariadb-server-compat and mariadb-client-compat to the MariaDB 11.4 Docker image, and pushed Docker images.

One drawback is that the build has become slower for MariaDB 11.4 (9 minutes vs 2.5 minutes for everything else):
https://github.com/stasadev/ddev/actions/runs/9553101676/job/26331149740

rfay

Looks good, works perfectly.

I did a ddev debug migrate-database mariadb:11.4 and it didn't have any hiccups at all.

I don't like the output (as you don't):

All project containers are now ready.
mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead
mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead

We might try to capture that in a future PR. I think it would be the same as capturing the mysql 8.x complaint about using password on command line.

$ ddev describe
┌────────────────────────────────────────────────────────────────────────────┐
│ Project: d10 ~/workspace/d10 https://d10.ddev.site                         │
│ Docker platform: orbstack                                                  │
│ Router: traefik                                                            │
├──────────┬──────┬─────────────────────────────────────┬────────────────────┤
│ SERVICE  │ STAT │ URL/PORT                            │ INFO               │
├──────────┼──────┼─────────────────────────────────────┼────────────────────┤
│ web      │ OK   │ https://d10.ddev.site               │ drupal PHP8.2      │
│          │      │ InDocker: web:8025,443,80           │ nginx-fpm          │
│          │      │ Host: 127.0.0.1:33103,33102         │ docroot:'web'      │
│          │      │                                     │ Perf mode: mutagen │
│          │      │                                     │ NodeJS:20          │
├──────────┼──────┼─────────────────────────────────────┼────────────────────┤
│ db       │ OK   │ InDocker: db:3306                   │ mariadb:11.4       │
│          │      │ Host: 127.0.0.1:33104               │ User/Pass: 'db/db' │
│          │      │                                     │ or 'root/root'     │
├──────────┼──────┼─────────────────────────────────────┼────────────────────┤
│ Mailpit  │      │ Mailpit: https://d10.ddev.site:8026 │                    │
│          │      │ Launch: ddev mailpit                │                    │
├──────────┼──────┼─────────────────────────────────────┼────────────────────┤
│ All URLs │      │ https://d10.ddev.site,              │                    │
│          │      │ https://127.0.0.1:33102,            │                    │
│          │      │ http://d10.ddev.site,               │                    │
│          │      │ http://127.0.0.1:33103              │                    │
└──────────┴──────┴─────────────────────────────────────┴────────────────────┘

github-actions · 2024-06-17T19:23:53Z

Download the artifacts for this pull request:

See Testing a PR.

fixes #6061

rfay · 2024-06-19T21:43:18Z

pkg/nodeps/values.go

+// MySQLRemoveDeprecatedMessage is used to remove the deprecation from stderr when using mysql tools with MariaDB 11.x and later.
+// mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead
+// mysqldump: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb-dump' instead
+const MySQLRemoveDeprecatedMessage = " 2> >(grep -v 'Deprecated program name.' >&2) "


Very fancy. For all my years of bash and shell I've never seen this notation before! I had to ask chatgpt to explain it to me.

I found it here https://stackoverflow.com/a/52575087/8097891

stasadev · 2024-06-19T21:43:30Z

I removed the deprecation warnings from ddev import-db and ddev export-db.

At first I thought I could wrap mysql and mysqldump in a bash script, but it's difficult because when you pass arguments in quotes, the quotes are removed:

#!/bin/bash

mariadb "$@"

This command

mysql -uroot -proot -e "SELECT @@character_set_database, @@collation_database;"

transforms into this

mysql -uroot -proot -e SELECT @@character_set_database, @@collation_database;

In the end I just filtered the stderr output.

rfay · 2024-06-19T21:45:29Z

We don't have to do it here, but the output from mysql:8.0 about using credentials on command-line is the same class of annoyance.

stasadev · 2024-06-19T21:52:48Z

Yes, I saw how it was fixed for mysql:8.0 in a different way here:

ci: make github action build dbserver images, fixes #3748 #5329

(Edit: it's not fixed mysql: [Warning] Using a password on the command line interface can be insecure.)

Unfortunately there is no workaround for MariaDB (at least for now):

https://mariadb.com/kb/en/suppress-mariadb-11-deprecation-warnings/

* docs: Warn about Codespaces (ddev#6321) [skip ci] Co-authored-by: Matthias Andrasch <777278+mandrasch@users.noreply.github.com> * docs: Improve ddev debug test by offering suggestions early [skip ci] (ddev#6323) * feat: add support for MariaDB 11.4 LTS, fixes ddev#6061 (ddev#6243) Co-authored-by: Stanislav Zhuk <stasadev@gmail.com> * docs: pacify textlint on mysql/https (ddev#6336) [skip ci] * docs: revisit WSL Docker Desktop setup instructions (ddev#6330) * fix: autodownload yarn from corepack, and set cache-folder, fixes ddev#6332 (ddev#6333) * docs: windows-wsl2-dd needs apt install for ngrok (ddev#6341) [skip ci] * build: check docker-compose 2.28.1 (ddev#6340) [skip ci] * fix: use real path to global config location, fixes ddev#6328 (ddev#6329) * test: stop running nightly tests [skip ci] (ddev#6347) * test: Improve reliability of TestProcessHooks (intermittent failures), fixes ddev#6313 (ddev#6314) * fix: Drupal 10 and Drupal 11 settings have migrated, set state_cache (ddev#6346) [skip ci] Co-authored-by: Stanislav Zhuk <stasadev@gmail.com> * build: use IPv4 inside a container, mariadb-client 11.4 for the webserver (ddev#6334) [skip ci] * fix: pass all valid options to nested commands in `ddev composer create`, fixes ddev#6300, fixes ddev#6246, for ddev#5058 (ddev#6303) Co-authored-by: Randy Fay <randy@randyfay.com> Co-authored-by: Ralf Koller <1665422+rpkoller@users.noreply.github.com> * docs: fix up textlint complaints * Turn off defaultTerms * Use exclude properly with textlint for VS Code * update for additional textlint complaints * remove redundant empty exclude section * Try to get textlint/reviewdog to report --------- Co-authored-by: Matthias Andrasch <777278+mandrasch@users.noreply.github.com> Co-authored-by: hussainweb <hussainweb@gmail.com> Co-authored-by: Stanislav Zhuk <stasadev@gmail.com> Co-authored-by: Ralf Koller <1665422+rpkoller@users.noreply.github.com>

hussainweb requested review from a team as code owners May 24, 2024 18:42

github-actions bot added enhancement labels May 24, 2024

rfay marked this pull request as draft May 24, 2024 19:01

rfay closed this May 24, 2024

hussainweb mentioned this pull request May 27, 2024

ddev import-db fails with "Error: Unknown command '\-'" because of new directive in mariadb-dump output #6249

Closed

stasadev reopened this Jun 7, 2024

stasadev force-pushed the mariadb-114 branch from d5b1657 to 541859f Compare June 7, 2024 13:33

stasadev self-assigned this Jun 7, 2024

stasadev force-pushed the mariadb-114 branch from 541859f to 8753f2e Compare June 10, 2024 14:51

stasadev force-pushed the mariadb-114 branch from 8753f2e to ec39ba3 Compare June 17, 2024 19:02

rfay approved these changes Jun 17, 2024

View reviewed changes

github-actions bot added the dependencies Pull requests that update a dependency file label Jun 17, 2024

hussainweb and others added 3 commits June 20, 2024 00:10

feat: add support for MariaDB 11.4 LTS [skip ci]

11707d4

fixes #6061

feat: add MySQL symlinks, bump images

8ece977

fix: hide "Deprecated program name" warnings

515f42c

stasadev force-pushed the mariadb-114 branch from ec39ba3 to 515f42c Compare June 19, 2024 21:36

rfay reviewed Jun 19, 2024

View reviewed changes

stasadev marked this pull request as ready for review June 19, 2024 21:45

This comment was marked as off-topic.

Sign in to view

stasadev mentioned this pull request Jun 20, 2024

build: use IPv4 inside a container, mariadb-client 11.4 for the webserver #6334

Merged

rfay merged commit 38a31a0 into ddev:master Jun 21, 2024
30 checks passed

hussainweb deleted the mariadb-114 branch June 21, 2024 13:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add support for MariaDB 11.4 LTS, fixes #6061 #6243

feat: add support for MariaDB 11.4 LTS, fixes #6061 #6243

hussainweb commented May 24, 2024 •

edited by stasadev

Loading

rfay commented May 24, 2024

hussainweb commented May 24, 2024

rfay commented May 24, 2024

rfay commented May 24, 2024

hussainweb commented May 25, 2024

hussainweb commented May 25, 2024

rfay commented May 25, 2024

stasadev commented Jun 7, 2024 •

edited

Loading

rfay commented Jun 7, 2024 •

edited

Loading

rfay commented Jun 7, 2024

stasadev commented Jun 7, 2024

rfay commented Jun 7, 2024

stasadev commented Jun 7, 2024

stasadev commented Jun 7, 2024

rfay commented Jun 7, 2024

stasadev commented Jun 10, 2024

rfay commented Jun 14, 2024

stasadev commented Jun 17, 2024

rfay left a comment

github-actions bot commented Jun 17, 2024 •

edited

Loading

rfay Jun 19, 2024

stasadev Jun 19, 2024

stasadev commented Jun 19, 2024

rfay commented Jun 19, 2024

stasadev commented Jun 19, 2024 •

edited

Loading

This comment was marked as off-topic.

This comment was marked as off-topic.

feat: add support for MariaDB 11.4 LTS, fixes #6061 #6243

feat: add support for MariaDB 11.4 LTS, fixes #6061 #6243

Conversation

hussainweb commented May 24, 2024 • edited by stasadev Loading

The Issue

How This PR Solves The Issue

Manual Testing Instructions

Automated Testing Overview

Related Issue Link(s)

Release/Deployment Notes

rfay commented May 24, 2024

hussainweb commented May 24, 2024

rfay commented May 24, 2024

rfay commented May 24, 2024

hussainweb commented May 25, 2024

hussainweb commented May 25, 2024

rfay commented May 25, 2024

stasadev commented Jun 7, 2024 • edited Loading

rfay commented Jun 7, 2024 • edited Loading

rfay commented Jun 7, 2024

stasadev commented Jun 7, 2024

rfay commented Jun 7, 2024

stasadev commented Jun 7, 2024

stasadev commented Jun 7, 2024

rfay commented Jun 7, 2024

stasadev commented Jun 10, 2024

rfay commented Jun 14, 2024

stasadev commented Jun 17, 2024

rfay left a comment

Choose a reason for hiding this comment

github-actions bot commented Jun 17, 2024 • edited Loading

rfay Jun 19, 2024

Choose a reason for hiding this comment

stasadev Jun 19, 2024

Choose a reason for hiding this comment

stasadev commented Jun 19, 2024

rfay commented Jun 19, 2024

stasadev commented Jun 19, 2024 • edited Loading

This comment was marked as off-topic.

This comment was marked as off-topic.

hussainweb commented May 24, 2024 •

edited by stasadev

Loading

stasadev commented Jun 7, 2024 •

edited

Loading

rfay commented Jun 7, 2024 •

edited

Loading

github-actions bot commented Jun 17, 2024 •

edited

Loading

stasadev commented Jun 19, 2024 •

edited

Loading