MySQL client cannot connect to database when using SNI routing with TLS

[NoOverflow]

Welcome!

• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

For a while Traefik could not route TCP (w/ TLS) requests for MySQL as both the client and server did not support SNI. The only option was to use a wildcard router ('HostSNI(*') which works fine, until you need enable TLS to host multiple MySQL instances using the same entrypoint on Traefik.

This has since been addressed by MySQL and SNI support has been added as of version 8.1.0 (2023/07)

What did you see instead?

Enabling TLS on the router somehow confuse the mysql client (and also anything relying on the mysql library) which ends up hanging, waiting for a specific packet (?)

Example (see docker-compose.yml provided) with TLS enabled on the router, tested with passthrough on, and off:

docker run --rm -it --network host mysql:8.3.0 mysql -u root -h 127.0.0.1 --port 3306 --ssl-mode=REQUIRED 
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0

We can confirm Traefik routes the traffic to the right container using ncat, implying that this may just be a protocol issue between the mysql client and traefik (I'm nowhere near qualified enough in either mysql or traefik to say for sure) :

➜  traefik-l4 ncat --ssl 127.0.0.1 3306 -v
Ncat: Version 7.94SVN ( https://nmap.org/ncat )
Ncat: Subject: CN=TRAEFIK DEFAULT CERT
Ncat: Issuer: CN=TRAEFIK DEFAULT CERT
Ncat: SHA-1 fingerprint: 0E49 C6BF 4852 D091 EDB7 2B6F E29A 42EC 64C5 795E
Ncat: Certificate verification failed (self-signed certificate).
Ncat: SSL connection to 127.0.0.1:3306.
Ncat: SHA-1 fingerprint: 0E49 C6BF 4852 D091 EDB7 2B6F E29A 42EC 64C5 795E
I
8.3.0   :;
          :I�����O
                  7M89Me Bcaching_sha2_password

And with TLS disabled on the router, it works fine:

docker run --rm -it  --network host mysql:8.3.0 mysql -h 127.0.0.1 --port 3306 --ssl-mode=REQUIRED -u root
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
...

Maybe this is a misunderstanding on my part, and that there is additional configuration required. But I guess there may be some work needed on Traefik to improve compatibility with MySQL as there was a similar PR done for PostgreSQL and I would be happy to help diagnose / open a PR if that's the case.

A community forum discussion exists for this issue with people experiencing the same behaviour: https://community.traefik.io/t/multiple-mysql-communication-through-tcp-with-tls-based-on-sni/18936

What version of Traefik are you using?

Version: 3.0.0-rc1
Codename: cheddar
Go version: go1.22.0
Built: 2024-02-13T13:41:20Z
OS/Arch: linux/amd64

What is your environment & configuration?

Minimal reproduction example:

version: '2.1'

services:
  traefik:
    image: "traefik:v3.0.0-rc1"
    container_name: "traefik"
    command:
      - "--log.level=DEBUG"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.mysql.address=:3306"
    ports:
      - "3306:3306"
      - "8089:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
  db:
    image: 'mysql:8.3.0'
    labels:
      - "traefik.enable=true"
      - "traefik.tcp.routers.db.rule=HostSNI(`*`)"
      - "traefik.tcp.services.db.loadbalancer.server.port=3306"
      - "traefik.tcp.routers.db.entrypoints=mysql"
      - "traefik.tcp.routers.db.tls=true"
    environment:
      - MYSQL_ROOT_PASSWORD=magento2
      - MYSQL_DATABASE=magento2
      - MYSQL_USER=magento2
      - MYSQL_PASSWORD=magento2

(Obviously, to get SNI routing working, this docker would run "behind" a domain name, with proper certificates etc, this is only meant to show that enabling TLS on a TCP router doesn't work with MySQL)

If applicable, please paste the log output in DEBUG level

No response

[ylazaar]

We are experiencing the same thing with a proxysql cluster hosted in kuberenetes.

[NoOverflow]

So, after firing up Wireshark and modifying Traefik I managed to get a TLS Client hello from the MySQL client by jamming a valid Protocol::HandshakeV10 packet (https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase_packets_protocol_handshake_v10.html).

Now I need to dig more into the MySQL protocol and figure out how to make it work (properly) with Traefik.

Unfortunately I found no proper way to fingerprint a MySQL connection since it's a server-first protocol (the client waits for a Handshake packet after socket creation), and someone will need to figure that out (a quick dirty-fix would be to force the packet if the port is 3306 but... yeah...). Moreover, the initial packet contains a lot of information about the server (auth plugin, capabilities flags...), while Traefik could "fake" those values to at least kickstart the TLS negotiation, I'm not sure if that is a viable solution.

I've got .pcap files for mysql client <-> mysql server w/TLS and mysql client <-> Traefik if anyone is up to take a look. Will open source the fork I made to work on it once It's at least a bit functional.

[NoOverflow]

Fork is at: https://github.com/NoOverflow/traefik/tree/feat-mysql-support

[rtribotte]

Hello @NoOverflow,

Thanks for sharing this!

Are you still iterating on the implementation? Are you planning to open a PR?
Please feel free to open a pull request to draw attention to this.

[NoOverflow]

Hi @rtribotte,
I'll gladly open a PR to open a discussion concerning this issue.

Time is a bit of a constraint right now for me but I'll find some to keep working on it 😄

[NoOverflow]

Hey @rtribotte, what's the PR flow for Traefik ? Should I open the PR right now as draft to kickstart a discussion ? Or should I wait to have a first working PoC ?

[umeshkhatiwada1]

@NoOverflow Does this fork version(Link) work with hostSNI with tls ?