latest image has no connection #1048

hanna-eismant · 2024-05-01T14:36:55Z

I have an CI/CD that is configured to run integration tests and it uses mysql docker image for it. Previously it run without problem, but today all tests are failed cause cannot connect to database.

For database startup I have follow configuration. As you can see it is not specified with image tag so it uses latest.

version: '3.9'

services:
  db:
    image: mysql
    networks:
      - doky-test-net
    ports:
      - "3306:3306"
    restart: unless-stopped
    command: '--default-authentication-plugin=mysql_native_password'
    cap_add:
      - SYS_NICE
    environment:
      MYSQL_ROOT_PASSWORD: root
      MYSQL_DATABASE: doky-test
      MYSQL_USER: doky-test
      MYSQL_PASSWORD: doky-test

networks:
  doky-test-net:

After investigation I found out that success builds uses mysql:8.3.0 (sha256:9de9d54fecee6253130e65154b930978b1fcc336bcc86dfd06e89b72a2588ebe) and failed uses mysql:8.4.0 (sha256:f7a8e140a7d6d1e6e0c99eeb0489c50a186ee4ac44ff55323a176529b9a43d33).
I changes my test-db.yml to use image mysql:8.3.0 and it works.

The service is started without error. There are logs from CI/CD:

Step 1/3: Start Database (Docker Compose)
16:38:23   Starting docker-compose for teamcity/test-db.yml
16:38:23   Starting: /bin/sh -c docker-compose  -f "teamcity/test-db.yml" up -d
16:38:23   in directory: /mnt/agent/work/bc4e634d0c6d276e
16:38:24   Creating network "teamcity_doky-test-net" with the default driver
16:38:24   Pulling db (mysql:)...
16:38:24   latest: Pulling from library/mysql
16:38:37   Digest: sha256:f7a8e140a7d6d1e6e0c99eeb0489c50a186ee4ac44ff55323a176529b9a43d33
16:38:37   Status: Downloaded newer image for mysql:latest
16:38:37   Creating teamcity_db_1 ...
16:38:41   Creating teamcity_db_1 ... done
16:38:41   Process exited with code 0
16:38:42   docker-compose has started successfully
16:38:43   Registering environment variable TEAMCITY_DOCKER_NETWORK=teamcity_doky-test-net

And corresponding messages in tests:

Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. 
The driver has not received any packets from the server.
Caused by: java.net.ConnectException: Connection refused

The text was updated successfully, but these errors were encountered:

amsterreynolds-123 · 2024-05-01T15:34:43Z

I'm having the same issue following upgrade to mysql:8.4.0, however rolling back to mysql:8.3.0 doesn't work (only permitted between patch releases.)

Container logs:

2024-05-01T15:18:35.265911Z 0 [System] [MY-015015] [Server] MySQL Server - start.
2024-05-01T15:18:35.665802Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.3.0) starting as process 1
2024-05-01T15:18:35.681111Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-05-01T15:18:36.340873Z 1 [ERROR] [MY-014061] [InnoDB] Invalid MySQL server downgrade: Cannot downgrade from 80400 to 80300. Downgrade is only permitted between patch releases.
mysqld: Can't open file: 'mysql.ibd' (errno: 0 - )
2024-05-01T15:18:36.655654Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2024-05-01T15:18:36.656539Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2024-05-01T15:18:36.656732Z 0 [ERROR] [MY-010119] [Server] Aborting
2024-05-01T15:18:36.659590Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.3.0) MySQL Community Server - GPL.
2024-05-01T15:18:36.659632Z 0 [System] [MY-015016] [Server] MySQL Server - end.
2024-05-01 15:18:39+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.3.0-1.el8 started.
2024-05-01 15:18:39+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-05-01 15:18:39+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.3.0-1.el8 started.
'/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock'
2024-05-01T15:18:40.027563Z 0 [System] [MY-015015] [Server] MySQL Server - start.
2024-05-01T15:18:40.412605Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.3.0) starting as process 1
2024-05-01T15:18:40.426081Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-05-01T15:18:40.979824Z 1 [ERROR] [MY-014061] [InnoDB] Invalid MySQL server downgrade: Cannot downgrade from 80400 to 80300. Downgrade is only permitted between patch releases.
mysqld: Can't open file: 'mysql.ibd' (errno: 0 - )
2024-05-01T15:18:41.308533Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2024-05-01T15:18:41.308969Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2024-05-01T15:18:41.309086Z 0 [ERROR] [MY-010119] [Server] Aborting
2024-05-01T15:18:41.311853Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.3.0) MySQL Community Server - GPL.
2024-05-01T15:18:41.311941Z 0 [System] [MY-015016] [Server] MySQL Server - end.

Any assistance will be appreciated as I'm at a dead end.

yonran · 2024-05-01T16:51:38Z

mysql:8.4.0 gives this message with --default-authentication-plugin=mysql_native_password before exiting: [Server] unknown variable 'default-authentication-plugin=mysql_native_password'.:

docker run --name=my-mysql --rm -e MYSQL_ROOT_USER=root -e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=test  mysql:8.4.0 --default-authentication-plugin=mysql_native_password
…
2024-05-01T16:48:44.890653Z 0 [ERROR] [MY-000067] [Server] unknown variable 'default-authentication-plugin=mysql_native_password'.
2024-05-01T16:48:44.891100Z 0 [ERROR] [MY-013236] [Server] The designated data directory /var/lib/mysql/ is unusable. You can remove all files that the server added to it.
2024-05-01T16:48:44.891137Z 0 [ERROR] [MY-010119] [Server] Aborting
2024-05-01T16:48:46.079987Z 0 [System] [MY-015018] [Server] MySQL Server Initialization - end.

zarevskaya · 2024-05-01T17:02:33Z

mysql:8.4.0 gives this message with --default-authentication-plugin=mysql_native_password before exiting: [Server] unknown variable 'default-authentication-plugin=mysql_native_password'.:

docker run --name=my-mysql --rm -e MYSQL_ROOT_USER=root -e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=test  mysql:8.4.0 --default-authentication-plugin=mysql_native_password
…
2024-05-01T16:48:44.890653Z 0 [ERROR] [MY-000067] [Server] unknown variable 'default-authentication-plugin=mysql_native_password'.
2024-05-01T16:48:44.891100Z 0 [ERROR] [MY-013236] [Server] The designated data directory /var/lib/mysql/ is unusable. You can remove all files that the server added to it.
2024-05-01T16:48:44.891137Z 0 [ERROR] [MY-010119] [Server] Aborting
2024-05-01T16:48:46.079987Z 0 [System] [MY-015018] [Server] MySQL Server Initialization - end.

Yes, the same here. A real disaster... You are not alone. ;)

LaurentGoderre · 2024-05-01T17:05:41Z

Sort of related to #1039

amsterreynolds-123 · 2024-05-01T17:10:04Z

mysql:8.4.0 gives this message with --default-authentication-plugin=mysql_native_password before exiting: [Server] unknown variable 'default-authentication-plugin=mysql_native_password'.:

docker run --name=my-mysql --rm -e MYSQL_ROOT_USER=root -e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=test  mysql:8.4.0 --default-authentication-plugin=mysql_native_password
…
2024-05-01T16:48:44.890653Z 0 [ERROR] [MY-000067] [Server] unknown variable 'default-authentication-plugin=mysql_native_password'.
2024-05-01T16:48:44.891100Z 0 [ERROR] [MY-013236] [Server] The designated data directory /var/lib/mysql/ is unusable. You can remove all files that the server added to it.
2024-05-01T16:48:44.891137Z 0 [ERROR] [MY-010119] [Server] Aborting
2024-05-01T16:48:46.079987Z 0 [System] [MY-015018] [Server] MySQL Server Initialization - end.

Awesome - thanks!

yosifkit · 2024-05-01T17:49:47Z

Deprecation and Removal Notes
Important Change: The deprecated mysql_native_password authentication plugin is now disabled by default. It can be enabled by starting MySQL with the new --mysql-native-password=ON server option, or by adding mysql_native_password=ON to the [mysqld] section of your MySQL configuration file.

For more information, see Native Pluggable Authentication. (Bug #36337893)

- https://dev.mysql.com/doc/relnotes/mysql/8.4/en/news-8-4-0.html

yosifkit · 2024-05-01T18:05:50Z

I would recommend reading at least the full Deprecation and Removal Notes section of those release notes in case you are using now deprecated or removed features, but all of it might contain important information relevant to your particular deployment.

https://dev.mysql.com/blog-archive/introducing-mysql-innovation-and-long-term-support-lts-versions/ would also be good to read about when updates or breaking changes happen.

TL;DR: add --mysql-native-password=ON to your args ~~along with~~ and remove the --default-authentication-plugin=mysql_native_password that you already have or use mysql:8.3 (the image won't be updated anymore). If your database files already upgraded, then you'll have to dump from 8.4 and load on your target.

I'd also recommend moving away from mysql:latest or mysql (implied :latest) and instead picking an actively supported version that suits your needs like mysql:8.0 or mysql:8.4 (as the two versions currently supported).

hanna-eismant · 2024-05-02T08:40:59Z

@yosifkit Thanks for clarification.
My mistake is that I used latest and was not notified that version was upgraded without additional preparations.

zarevskaya · 2024-05-02T09:05:59Z

@yosifkit Thanks for clarification.
My mistake is that I used latest and was not notified that version was upgraded without additional preparations.

Yep, the same here with Ghost (CMS), what nightmare.
I'm guilty.

thebigbone · 2024-05-02T14:50:27Z

I would recommend reading at least the full Deprecation and Removal Notes section of those release notes in case you are using now deprecated or removed features, but all of it might contain important information relevant to your particular deployment.

https://dev.mysql.com/blog-archive/introducing-mysql-innovation-and-long-term-support-lts-versions/ would also be good to read about when updates or breaking changes happen.

TL;DR: add --mysql-native-_password=ON to your args along with the --default-authentication-plugin=mysql_native_password that you already have or use mysql:8.3 (the image won't be updated anymore). If your database files already upgraded, then you'll have to dump from 8.4 and load on your target.

I'd also recommend moving away from mysql:latest or mysql (implied :latest) and instead picking an actively supported version that suits your needs like mysql:8.0 or mysql:8.4 (as the two versions currently supported).

A little correction here. I was facing the same issue on the latest image and just by adding --mysql-native-password=ON to the command (you have a typo there) I was able to start mysql. If you add the deprecated command along with it, it will spit out the same error and refuse to start.

TLDR: Remove the --default-authentication-plugin=mysql_native_password and add --mysql-native-password=ON command.

tianon · 2024-05-02T17:42:50Z

I believe the upstream intent is to completely remove the functionality, so don't be surprised when you find that can again after kicking it down the road (again). 😅

yonran · 2024-05-02T18:19:56Z

I'd also recommend moving away from mysql:latest or mysql (implied :latest) and instead picking an actively supported version that suits your needs like mysql:8.0 or mysql:8.4 (as the two versions currently supported).

fyi the the documentation at https://hub.docker.com/_/mysql currently has an example showing mysql:latest with --default-authentication-plugin=mysql_native_password:

# Use root/example as user/password credentials
version: '3.1'

services:

  db:
    image: mysql
    # NOTE: use of "mysql_native_password" is not recommended: https://dev.mysql.com/doc/refman/8.0/en/upgrading-from-previous-series.html#upgrade-caching-sha2-password
    # (this is just an example, not intended to be a production configuration)
    command: --default-authentication-plugin=mysql_native_password
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: example

  adminer:
    image: adminer
    restart: always
    ports:
      - 8080:8080

tianon · 2024-05-02T18:34:23Z

Good callout, thank you! docker-library/docs#2443

yosifkit closed this as completed May 1, 2024

LaurentGoderre mentioned this issue May 2, 2024

mysql:oracle stopped working after recent pull #1051

Closed

rdimaio mentioned this issue May 2, 2024

MySQL8 autotests are failing due to "Temporary failure in name resolution" rucio/rucio#6747

Closed

cvharris mentioned this issue May 2, 2024

Latest MySQL image deprecated --default-authentication-plugin, database won't start on latest image Automattic/vip-cli#1816

Open

RedPine404 mentioned this issue May 7, 2024

Fixed mysql authentication option causing db fail to start jackyzy823/fxa-selfhosting#26

Closed

This was referenced May 7, 2024

Environment broken (container restarting) in last version #1056

Open

Cannot configure mysql auth_plugin to use when using MYSQL_PASSWORD environment variable #1039

Closed

Unable to start mysql with --mysql-native-password=ON #1059

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

latest image has no connection #1048

latest image has no connection #1048

hanna-eismant commented May 1, 2024

amsterreynolds-123 commented May 1, 2024 •

edited

yonran commented May 1, 2024

zarevskaya commented May 1, 2024

LaurentGoderre commented May 1, 2024

amsterreynolds-123 commented May 1, 2024

yosifkit commented May 1, 2024 •

edited

yosifkit commented May 1, 2024 •

edited

hanna-eismant commented May 2, 2024

zarevskaya commented May 2, 2024 •

edited

thebigbone commented May 2, 2024 •

edited

tianon commented May 2, 2024

yonran commented May 2, 2024

tianon commented May 2, 2024 via email

latest image has no connection #1048

latest image has no connection #1048

Comments

hanna-eismant commented May 1, 2024

amsterreynolds-123 commented May 1, 2024 • edited

yonran commented May 1, 2024

zarevskaya commented May 1, 2024

LaurentGoderre commented May 1, 2024

amsterreynolds-123 commented May 1, 2024

yosifkit commented May 1, 2024 • edited

yosifkit commented May 1, 2024 • edited

hanna-eismant commented May 2, 2024

zarevskaya commented May 2, 2024 • edited

thebigbone commented May 2, 2024 • edited

tianon commented May 2, 2024

yonran commented May 2, 2024

tianon commented May 2, 2024 via email

amsterreynolds-123 commented May 1, 2024 •

edited

yosifkit commented May 1, 2024 •

edited

yosifkit commented May 1, 2024 •

edited

zarevskaya commented May 2, 2024 •

edited

thebigbone commented May 2, 2024 •

edited