Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gmysql backend fails when server sends "Authentication Switch Request" #10340

Closed
brianredbeard opened this issue Apr 30, 2021 · 15 comments
Closed

Comments

@brianredbeard
Copy link

  • Program: Authoritative
  • Issue type: Bug report

Short description

Attempting to use the gmysql backend with MariaDB fails when the server sends an Authentication Switch Request. This has been validated using the ed25519 authentication plugin.

Environment

  • Operating system: RHEL 8
  • Software version: pdns 4.4.x, pdns 4.5.x
  • Software source: pdns repository
  • Database: MariaDB 10.4.13; 10.3.11
  • PDNS configuration:
[root@ip-172-28-9-114 ~]# grep -v -e "^#" -e "^$" /etc/pdns/pdns.conf
launch=gmysql
gmysql-user=powerdns
gmysql-dbname=powerdns
gmysql-password=powerdns
gmysql-host=powerdns.xxxxxxxxxxxx.us-west-2.rds.amazonaws.com
gmysql-group=pdns
gmysql-dnssec=yes
8bit-dns=yes
allow-axfr-ips=127.0.0.0/8,::1
also-notify=<redacted>
api=yes
default-soa-content=<redacted> hostmaster.@ 0 900 180 3600000 60
expand-alias=yes
master=yes
setgid=pdns
setuid=pdns
webserver=yes
  • MariaDB client configuration:
[root@ip-172-28-9-114 ~]# cat /etc/my.cnf
#
# This group is read both both by the client and the server
# use it for options that affect everything
#
[client-server]

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

[root@ip-172-28-9-114 ~]# cat /etc/my.cnf.d/rds.cnf 
# /etc/my.cnf.d/rds.cnf 
[client]
ssl-ca=/etc/pki/tls/certs/rds-combined-ca-bundle.pem
ssl
ssl-verify-server-cert

[pdns]
ssl-ca=/etc/pki/tls/certs/rds-combined-ca-bundle.pem
ssl
ssl-verify-server-cert

Steps to reproduce

  1. Install MariaDB (tested with 10.3.x & 10.4.x)
  2. Configure the database to use the ed25519 authentication plugin as per the documentation above.
  3. Validate correct configuration by logging in with the mysql command line client and check the plugin used for the user:
(admin) [(none)]> select host,user,plugin from mysql.user ;
+------------------------+-------------+-----------------------+
| Host                   | User        | plugin                |
+------------------------+-------------+-----------------------+
| localhost              | mariadb.sys | mysql_native_password |
| localhost              | rdsadmin    | mysql_native_password |
| %                      | admin       | mysql_native_password |
| 172.28.0.0/255.255.0.0 | powerdns    | ed25519               |
+------------------------+-------------+-----------------------+
4 rows in set (0.001 sec)
  1. Start a wireshark capture to introspect the connection handling (tshark -i eth0 -w /tmp/mysql.pcap tcp port 3306)
  2. Attempt to start pdns

Expected behaviour

Actual behaviour

It depends on whether SSL is enabled or not. If SSL is enabled directly in pdns.conf (e.g. gmysql-ssl=yes) then pdns fails to start the backend, throwing the error Caught an exception instantiating a backend: Unable to launch gmysql connection: Unable to connect to database: ERROR 1043 (08S01): Bad handshake:

Apr 30 00:07:03 ip-172-28-9-114.us-west-2.compute.internal systemd[1]: Starting PowerDNS Authoritative Server...
Apr 30 00:07:03 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6453]: Loading '/usr/lib64/pdns/libgmysqlbackend.so'
Apr 30 00:07:03 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6453]: This is a standalone pdns
Apr 30 00:07:03 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6453]: Listening on controlsocket in '/run/pdns/pdns.controlsocket'
Apr 30 00:07:03 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6453]: UDP server bound to 0.0.0.0:53
Apr 30 00:07:03 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6453]: UDP server bound to [::]:53
Apr 30 00:07:03 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6453]: TCP server bound to 0.0.0.0:53
Apr 30 00:07:03 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6453]: TCP server bound to [::]:53
Apr 30 00:07:03 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6453]: PowerDNS Authoritative Server 4.5.0-alpha0.1260.master.g65eb2c4ec (C) 2001-2021 PowerDNS.COM BV
Apr 30 00:07:03 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6453]: Using 64-bits mode. Built using gcc 8.3.1 20191121 (Red Hat 8.3.1-5) on Apr 28 2021 10:55:53 by root@8a4a141a6c98.
Apr 30 00:07:03 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6453]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
Apr 30 00:07:03 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6453]: [webserver] Listening for HTTP requests on 127.0.0.1:8081
Apr 30 00:07:03 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6453]: Not validating response for security status update, this is a non-release version
Apr 30 00:07:03 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6453]: Creating backend connection for TCP
Apr 30 00:07:03 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6453]: Primary/secondary communicator launching
Apr 30 00:07:23 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6453]: gmysql Connection failed: Unable to connect to database: ERROR 1043 (08S01): Bad handshake
Apr 30 00:07:23 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6453]: Caught an exception instantiating a backend: Unable to launch gmysql connection: Unable to connect to database: ERROR 1043 (08S01): Bad handshake
Apr 30 00:07:23 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6453]: Cleaning up
Apr 30 00:07:23 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6453]: TCP server is unable to launch backends - will try again when questions come in: Unable to launch gmysql connection: Unable to connect to database: ERROR 1043 (08S01): Bad handshake
Apr 30 00:07:23 ip-172-28-9-114.us-west-2.compute.internal systemd[1]: Started PowerDNS Authoritative Server.
Apr 30 00:07:23 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6453]: About to create 3 backend threads for UDP
Apr 30 00:07:23 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6453]: Done launching threads, ready to distribute questions

If SSL is configured through the system my.cnf through the use of the group mechanism or if SSL is unused pdns crashes outright:

Apr 29 23:52:52 ip-172-28-9-114.us-west-2.compute.internal systemd[1]: Starting PowerDNS Authoritative Server...
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: Loading '/usr/lib64/pdns/libgmysqlbackend.so'
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: This is a standalone pdns
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: Listening on controlsocket in '/run/pdns/pdns.controlsocket'
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: UDP server bound to 0.0.0.0:53
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: UDP server bound to [::]:53
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: TCP server bound to 0.0.0.0:53
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: TCP server bound to [::]:53
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: PowerDNS Authoritative Server 4.5.0-alpha0.1260.master.g65eb2c4ec (C) 2001-2021 PowerDNS.COM BV
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: Using 64-bits mode. Built using gcc 8.3.1 20191121 (Red Hat 8.3.1-5) on Apr 28 2021 10:55:53 by root@8a4a141a6c98.
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: [webserver] Listening for HTTP requests on 127.0.0.1:8081
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: Not validating response for security status update, this is a non-release version
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: Creating backend connection for TCP
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: Primary/secondary communicator launching
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: Got a signal 11, attempting to print trace:
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: /usr/sbin/pdns_server(+0x31991a) [0x55a6ff77491a]
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: /lib64/libc.so.6(+0x37880) [0x7fe8584d2880]
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: /lib64/libc.so.6(+0x15dfa5) [0x7fe8585f8fa5]
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: /lib64/libsodium.so.23(crypto_sign_ed25519+0x2c) [0x7fe859c92f7c]
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: /usr/lib64/mariadb/plugin/client_ed25519.so(+0x21ff) [0x7fe84dd2d1ff]
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: /lib64/libmariadb.so.3(+0x18375) [0x7fe8528ca375]
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: /lib64/libmariadb.so.3(+0x22372) [0x7fe8528d4372]
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: /lib64/libmariadb.so.3(mysql_real_connect+0x27b) [0x7fe8528d0eab]
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: /usr/lib64/pdns/libgmysqlbackend.so(_ZN6SMySQL7connectEv+0x155) [0x7fe852b18605]
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: /usr/lib64/pdns/libgmysqlbackend.so(_ZN6SMySQLC1ENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES5_tS5_S5_S5_S5_bjbb+0x231) [0x7fe852b18ab1]
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: /usr/lib64/pdns/libgmysqlbackend.so(_ZN13gMySQLBackend9reconnectEv+0x40c) [0x7fe852b0fd9c]
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: /usr/lib64/pdns/libgmysqlbackend.so(_ZN13gMySQLBackendC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_+0x4a) [0x7fe852b101aa]
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: /usr/lib64/pdns/libgmysqlbackend.so(_ZN13gMySQLFactory4makeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x2a) [0x7fe852b1398a]
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: /usr/sbin/pdns_server(_ZN17BackendMakerClass3allEb+0x341) [0x55a6ff624311]
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: /usr/sbin/pdns_server(_ZN12UeberBackendC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x1bb) [0x55a6ff7de1eb]
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: /usr/sbin/pdns_server(_ZN13PacketHandlerC1Ev+0x96) [0x55a6ff762436]
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: /usr/sbin/pdns_server(_ZN13TCPNameserver2goEv+0x92) [0x55a6ff7c9c62]
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: /usr/sbin/pdns_server(_Z10mainthreadv+0xc82) [0x55a6ff5f8c22]
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: /usr/sbin/pdns_server(main+0x28a5) [0x55a6ff5ab225]
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal pdns_server[6260]: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7fe8584be7b3]
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal systemd[1]: pdns.service: Main process exited, code=dumped, status=6/ABRT
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal systemd[1]: pdns.service: Failed with result 'core-dump'.
Apr 29 23:52:53 ip-172-28-9-114.us-west-2.compute.internal systemd[1]: Failed to start PowerDNS Authoritative Server.

Other information

Additionally, I am curious as to why the behavior of pdns is further mutated by the presence of gmysql-ssl=yes in pdns.conf.

The packet trace of a vanilla connection with the configuration files supplied above (and no SSL so as to allow introspection of the protocol stream) appears as follows, with successful authentication occurring on lines 6-10:

$ tshark -r mysql2.pcap
    1 0.000000000 172.28.9.114 → 172.28.15.230 TCP 74 55244 → 3306 [SYN] Seq=1173260165 Win=26883 Len=0 MSS=8961 SACK_PERM=1 TSval=1420931651 TSecr=0 WS=128 0.000000000 26883 0 74
    2 0.000735293 172.28.15.230 → 172.28.9.114 TCP 74 3306 → 55244 [SYN, ACK] Seq=3179053976 Ack=1173260166 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=150387060 TSecr=1420931651 WS=128 0.000735293 28960 0 74
    3 0.000749337 172.28.9.114 → 172.28.15.230 TCP 66 55244 → 3306 [ACK] Seq=1173260166 Ack=3179053977 Win=27008 Len=0 TSval=1420931652 TSecr=150387060 0.000014044 211 0 66
    4 0.001463521 172.28.15.230 → 172.28.9.114 MySQL 163 Server Greeting proto=10 version=10.4.13-MariaDB-log 0.000714184 227 0 163
    5 0.001471725 172.28.9.114 → 172.28.15.230 TCP 66 55244 → 3306 [ACK] Seq=1173260166 Ack=3179054074 Win=27008 Len=0 TSval=1420931653 TSecr=150387060 0.000008204 211 0 66
    6 0.001501909 172.28.9.114 → 172.28.15.230 MySQL 335 Login Request user=powerdns db=powerdns 0.000030184 211 0 335
    7 0.002097047 172.28.15.230 → 172.28.9.114 TCP 66 3306 → 55244 [ACK] Seq=3179054074 Ack=1173260435 Win=30080 Len=0 TSval=150387061 TSecr=1420931653 0.000595138 235 0 66
    8 0.002128963 172.28.15.230 → 172.28.9.114 MySQL 118 Auth Switch Request 0.000031916 235 0 118
    9 0.002363169 172.28.9.114 → 172.28.15.230 MySQL 134 Auth Switch Response 0.000234206 211 0 134
   10 0.003136554 172.28.15.230 → 172.28.9.114 MySQL 92 Response OK 0.000773385 235 0 92
   11 0.004928724 172.28.9.114 → 172.28.15.230 MySQL 85 Request Query 0.001792170 211 0 85
   12 0.005765317 172.28.15.230 → 172.28.9.114 MySQL 206 Response 0.000836593 235 0 206
   13 0.005813650 172.28.9.114 → 172.28.15.230 MySQL 82 Request Query 0.000048333 219 0 82
   14 0.006523149 172.28.15.230 → 172.28.9.114 MySQL 286 Response 0.000709499 235 0 286
   15 0.006616769 172.28.9.114 → 172.28.15.230 MySQL 80 Request Show Fields 0.000093620 227 0 80
   16 0.007263653 172.28.15.230 → 172.28.9.114 MySQL 537 Response 0.000646884 235 0 537
   17 0.007360011 172.28.9.114 → 172.28.15.230 MySQL 82 Request Show Fields 0.000096358 236 0 82
   18 0.007983953 172.28.15.230 → 172.28.9.114 MySQL 497 Response 0.000623942 235 0 497
   19 0.008062412 172.28.9.114 → 172.28.15.230 MySQL 86 Request Show Fields 0.000078459 244 0 86
   20 0.008697302 172.28.15.230 → 172.28.9.114 MySQL 381 Response 0.000634890 235 0 381
   21 0.008755440 172.28.9.114 → 172.28.15.230 MySQL 79 Request Show Fields 0.000058138 252 0 79
   22 0.009367058 172.28.15.230 → 172.28.9.114 MySQL 529 Response 0.000611618 235 0 529
   23 0.009460022 172.28.9.114 → 172.28.15.230 MySQL 79 Request Show Fields 0.000092964 261 0 79
   24 0.010114480 172.28.15.230 → 172.28.9.114 MySQL 696 Response 0.000654458 235 0 696
   25 0.010212504 172.28.9.114 → 172.28.15.230 MySQL 84 Request Show Fields 0.000098024 271 0 84
   26 0.010830448 172.28.15.230 → 172.28.9.114 MySQL 296 Response 0.000617944 235 0 296
   27 0.010877724 172.28.9.114 → 172.28.15.230 MySQL 80 Request Show Fields 0.000047276 280 0 80
   28 0.011492312 172.28.15.230 → 172.28.9.114 MySQL 330 Response 0.000614588 235 0 330
   29 0.011686661 172.28.9.114 → 172.28.15.230 MySQL 103 Request Query 0.000194349 290 0 103
   30 0.012371724 172.28.15.230 → 172.28.9.114 MySQL 156 Response 0.000685063 235 0 156
   31 0.012502883 172.28.9.114 → 172.28.15.230 MySQL 84 Request Query 0.000131159 290 0 84
   32 0.013147163 172.28.15.230 → 172.28.9.114 MySQL 149 Response 0.000644280 235 0 149
   33 0.053627691 172.28.9.114 → 172.28.15.230 TCP 66 55244 → 3306 [ACK] Seq=1173260701 Ack=3179057489 Win=37120 Len=0 TSval=1420931705 TSecr=150387063 0.040480528 290 0 66
   34 1.371474999 172.28.9.114 → 172.28.15.230 MySQL 71 Request Quit 1.317847308 290 0 71
   35 1.371493021 172.28.9.114 → 172.28.15.230 TCP 66 55244 → 3306 [FIN, ACK] Seq=1173260706 Ack=3179057489 Win=37120 Len=0 TSval=1420933023 TSecr=150387063 0.000018022 290 0 66
   36 1.375614053 172.28.9.114 → 172.28.15.230 TCP 66 [TCP Retransmission] 55244 → 3306 [FIN, ACK] Seq=1173260706 Ack=3179057489 Win=37120 Len=0 TSval=1420933027 TSecr=150387063 0.004121032 290 0 66
   37 1.376113570 172.28.15.230 → 172.28.9.114 TCP 66 3306 → 55244 [FIN, ACK] Seq=3179057489 Ack=1173260707 Win=30080 Len=0 TSval=150387404 TSecr=1420933023 0.000499517 235 0 66
   38 1.376128062 172.28.9.114 → 172.28.15.230 TCP 66 55244 → 3306 [ACK] Seq=1173260707 Ack=3179057490 Win=37120 Len=0 TSval=1420933027 TSecr=150387404 0.000014492 290 0 66
   39 1.377076861 172.28.15.230 → 172.28.9.114 TCP 78 [TCP Dup ACK 37#1] 3306 → 55244 [ACK] Seq=3179057490 Ack=1173260707 Win=30080 Len=0 TSval=150387404 TSecr=1420933027 SLE=1173260706 SRE=1173260707 0.000948799 235 0 78

Conversely the packet trace of pdns attempting to login appears as follows:

$ tshark -r mysql5.pcap
    1 0.000000000 172.28.9.114 → 172.28.15.230 TCP 74 55260 → 3306 [SYN] Seq=922255759 Win=26883 Len=0 MSS=8961 SACK_PERM=1 TSval=1421350668 TSecr=0 WS=128 0.000000000 26883 0 74
    2 0.000731947 172.28.15.230 → 172.28.9.114 TCP 74 3306 → 55260 [SYN, ACK] Seq=2608666981 Ack=922255760 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=150491814 TSecr=1421350668 WS=128 0.000731947 28960 0 74
    3 0.000746091 172.28.9.114 → 172.28.15.230 TCP 66 55260 → 3306 [ACK] Seq=922255760 Ack=2608666982 Win=27008 Len=0 TSval=1421350669 TSecr=150491814 0.000014144 211 0 66
    4 0.001506146 172.28.15.230 → 172.28.9.114 MySQL 163 Server Greeting proto=10 version=10.4.13-MariaDB-log 0.000760055 227 0 163
    5 0.001513943 172.28.9.114 → 172.28.15.230 TCP 66 55260 → 3306 [ACK] Seq=922255760 Ack=2608667079 Win=27008 Len=0 TSval=1421350670 TSecr=150491814 0.000007797 211 0 66
    6 0.001547182 172.28.9.114 → 172.28.15.230 MySQL 314 Login Request user=powerdns db=powerdns 0.000033239 211 0 314
    7 0.002148769 172.28.15.230 → 172.28.9.114 TCP 66 3306 → 55260 [ACK] Seq=2608667079 Ack=922256008 Win=30080 Len=0 TSval=150491814 TSecr=1421350670 0.000601587 235 0 66
    8 0.002930479 172.28.15.230 → 172.28.9.114 MySQL 118 Auth Switch Request 0.000781710 235 0 118
    9 0.043110838 172.28.9.114 → 172.28.15.230 TCP 66 55260 → 3306 [ACK] Seq=922256008 Ack=2608667131 Win=27008 Len=0 TSval=1421350711 TSecr=150491814 0.040180359 211 0 66
   10 0.398611611 172.28.9.114 → 172.28.15.230 TCP 66 55260 → 3306 [FIN, ACK] Seq=922256008 Ack=2608667131 Win=27008 Len=0 TSval=1421351067 TSecr=150491814 0.355500773 211 0 66
   11 0.399377319 172.28.15.230 → 172.28.9.114 TCP 66 3306 → 55260 [FIN, ACK] Seq=2608667131 Ack=922256009 Win=30080 Len=0 TSval=150491914 TSecr=1421351067 0.000765708 235 0 66
   12 0.399390201 172.28.9.114 → 172.28.15.230 TCP 66 55260 → 3306 [ACK] Seq=922256009 Ack=2608667132 Win=27008 Len=0 TSval=1421351068 TSecr=150491914 0.000012882 211 0 66
   13 1.638543843 172.28.9.114 → 172.28.15.230 TCP 74 55262 → 3306 [SYN] Seq=4089836164 Win=26883 Len=0 MSS=8961 SACK_PERM=1 TSval=1421352307 TSecr=0 WS=128 1.239153642 26883 1 74
   14 1.639206022 172.28.15.230 → 172.28.9.114 TCP 74 3306 → 55262 [SYN, ACK] Seq=3970098767 Ack=4089836165 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=150492223 TSecr=1421352307 WS=128 0.000662179 28960 1 74
   15 1.639221657 172.28.9.114 → 172.28.15.230 TCP 66 55262 → 3306 [ACK] Seq=4089836165 Ack=3970098768 Win=27008 Len=0 TSval=1421352307 TSecr=150492223 0.000015635 211 1 66
   16 1.639953312 172.28.15.230 → 172.28.9.114 MySQL 163 Server Greeting proto=10 version=10.4.13-MariaDB-log 0.000731655 227 1 163
   17 1.639961150 172.28.9.114 → 172.28.15.230 TCP 66 55262 → 3306 [ACK] Seq=4089836165 Ack=3970098865 Win=27008 Len=0 TSval=1421352308 TSecr=150492224 0.000007838 211 1 66
   18 1.639992671 172.28.9.114 → 172.28.15.230 MySQL 314 Login Request user=powerdns db=powerdns 0.000031521 211 1 314
   19 1.640601571 172.28.15.230 → 172.28.9.114 TCP 66 3306 → 55262 [ACK] Seq=3970098865 Ack=4089836413 Win=30080 Len=0 TSval=150492224 TSecr=1421352308 0.000608900 235 1 66
   20 1.640606203 172.28.15.230 → 172.28.9.114 MySQL 118 Auth Switch Request 0.000004632 235 1 118
   21 1.681112509 172.28.9.114 → 172.28.15.230 TCP 66 55262 → 3306 [ACK] Seq=4089836413 Ack=3970098917 Win=27008 Len=0 TSval=1421352349 TSecr=150492224 0.040506306 211 1 66
   22 2.009527283 172.28.9.114 → 172.28.15.230 TCP 66 55262 → 3306 [FIN, ACK] Seq=4089836413 Ack=3970098917 Win=27008 Len=0 TSval=1421352678 TSecr=150492224 0.328414774 211 1 66
   23 2.010268240 172.28.15.230 → 172.28.9.114 TCP 66 3306 → 55262 [FIN, ACK] Seq=3970098917 Ack=4089836414 Win=30080 Len=0 TSval=150492316 TSecr=1421352678 0.000740957 235 1 66
   24 2.010281462 172.28.9.114 → 172.28.15.230 TCP 66 55262 → 3306 [ACK] Seq=4089836414 Ack=3970098918 Win=27008 Len=0 TSval=1421352679 TSecr=150492316 0.000013222 211 1 66
   25 3.138012084 172.28.9.114 → 172.28.15.230 TCP 74 55264 → 3306 [SYN] Seq=3279534912 Win=26883 Len=0 MSS=8961 SACK_PERM=1 TSval=1421353806 TSecr=0 WS=128 1.127730622 26883 2 74
   26 3.138718624 172.28.15.230 → 172.28.9.114 TCP 74 3306 → 55264 [SYN, ACK] Seq=3016317066 Ack=3279534913 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=150492598 TSecr=1421353806 WS=128 0.000706540 28960 2 74
   27 3.138734409 172.28.9.114 → 172.28.15.230 TCP 66 55264 → 3306 [ACK] Seq=3279534913 Ack=3016317067 Win=27008 Len=0 TSval=1421353807 TSecr=150492598 0.000015785 211 2 66
   28 3.139498690 172.28.15.230 → 172.28.9.114 MySQL 163 Server Greeting proto=10 version=10.4.13-MariaDB-log 0.000764281 227 2 163
   29 3.139507062 172.28.9.114 → 172.28.15.230 TCP 66 55264 → 3306 [ACK] Seq=3279534913 Ack=3016317164 Win=27008 Len=0 TSval=1421353808 TSecr=150492599 0.000008372 211 2 66
   30 3.139542919 172.28.9.114 → 172.28.15.230 MySQL 314 Login Request user=powerdns db=powerdns 0.000035857 211 2 314
   31 3.140148917 172.28.15.230 → 172.28.9.114 TCP 66 3306 → 55264 [ACK] Seq=3016317164 Ack=3279535161 Win=30080 Len=0 TSval=150492599 TSecr=1421353808 0.000605998 235 2 66
   32 3.140165864 172.28.15.230 → 172.28.9.114 MySQL 118 Auth Switch Request 0.000016947 235 2 118
   33 3.181113380 172.28.9.114 → 172.28.15.230 TCP 66 55264 → 3306 [ACK] Seq=3279535161 Ack=3016317216 Win=27008 Len=0 TSval=1421353849 TSecr=150492599 0.040947516 211 2 66
   34 3.509382424 172.28.9.114 → 172.28.15.230 TCP 66 55264 → 3306 [FIN, ACK] Seq=3279535161 Ack=3016317216 Win=27008 Len=0 TSval=1421354178 TSecr=150492599 0.328269044 211 2 66
   35 3.510156911 172.28.15.230 → 172.28.9.114 TCP 66 3306 → 55264 [FIN, ACK] Seq=3016317216 Ack=3279535162 Win=30080 Len=0 TSval=150492691 TSecr=1421354178 0.000774487 235 2 66
   36 3.510170546 172.28.9.114 → 172.28.15.230 TCP 66 55264 → 3306 [ACK] Seq=3279535162 Ack=3016317217 Win=27008 Len=0 TSval=1421354178 TSecr=150492691 0.000013635 211 2 66
   37 4.636926422 172.28.9.114 → 172.28.15.230 TCP 74 55266 → 3306 [SYN] Seq=2677458632 Win=26883 Len=0 MSS=8961 SACK_PERM=1 TSval=1421355305 TSecr=0 WS=128 1.126755876 26883 3 74
   38 4.637642311 172.28.15.230 → 172.28.9.114 TCP 74 3306 → 55266 [SYN, ACK] Seq=3592082758 Ack=2677458633 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=150492973 TSecr=1421355305 WS=128 0.000715889 28960 3 74
   39 4.637658933 172.28.9.114 → 172.28.15.230 TCP 66 55266 → 3306 [ACK] Seq=2677458633 Ack=3592082759 Win=27008 Len=0 TSval=1421355306 TSecr=150492973 0.000016622 211 3 66
   40 4.638405324 172.28.15.230 → 172.28.9.114 MySQL 163 Server Greeting proto=10 version=10.4.13-MariaDB-log 0.000746391 227 3 163
   41 4.638413601 172.28.9.114 → 172.28.15.230 TCP 66 55266 → 3306 [ACK] Seq=2677458633 Ack=3592082856 Win=27008 Len=0 TSval=1421355307 TSecr=150492973 0.000008277 211 3 66
   42 4.638449772 172.28.9.114 → 172.28.15.230 MySQL 314 Login Request user=powerdns db=powerdns 0.000036171 211 3 314
   43 4.639109074 172.28.15.230 → 172.28.9.114 TCP 66 3306 → 55266 [ACK] Seq=3592082856 Ack=2677458881 Win=30080 Len=0 TSval=150492973 TSecr=1421355307 0.000659302 235 3 66
   44 4.639113314 172.28.15.230 → 172.28.9.114 MySQL 118 Auth Switch Request 0.000004240 235 3 118
   45 4.680111961 172.28.9.114 → 172.28.15.230 TCP 66 55266 → 3306 [ACK] Seq=2677458881 Ack=3592082908 Win=27008 Len=0 TSval=1421355348 TSecr=150492973 0.040998647 211 3 66
   46 5.016199758 172.28.9.114 → 172.28.15.230 TCP 66 55266 → 3306 [FIN, ACK] Seq=2677458881 Ack=3592082908 Win=27008 Len=0 TSval=1421355684 TSecr=150492973 0.336087797 211 3 66
   47 5.016937434 172.28.15.230 → 172.28.9.114 TCP 66 3306 → 55266 [FIN, ACK] Seq=3592082908 Ack=2677458882 Win=30080 Len=0 TSval=150493068 TSecr=1421355684 0.000737676 235 3 66
   48 5.016951471 172.28.9.114 → 172.28.15.230 TCP 66 55266 → 3306 [ACK] Seq=2677458882 Ack=3592082909 Win=27008 Len=0 TSval=1421355685 TSecr=150493068 0.000014037 211 3 66
   49 6.160362998 172.28.9.114 → 172.28.15.230 TCP 74 55268 → 3306 [SYN] Seq=1399073288 Win=26883 Len=0 MSS=8961 SACK_PERM=1 TSval=1421356829 TSecr=0 WS=128 1.143411527 26883 4 74
   50 6.161130739 172.28.15.230 → 172.28.9.114 TCP 74 3306 → 55268 [SYN, ACK] Seq=2707774809 Ack=1399073289 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=150493354 TSecr=1421356829 WS=128 0.000767741 28960 4 74
   51 6.161146982 172.28.9.114 → 172.28.15.230 TCP 66 55268 → 3306 [ACK] Seq=1399073289 Ack=2707774810 Win=27008 Len=0 TSval=1421356829 TSecr=150493354 0.000016243 211 4 66
   52 6.161875007 172.28.15.230 → 172.28.9.114 MySQL 163 Server Greeting proto=10 version=10.4.13-MariaDB-log 0.000728025 227 4 163
   53 6.161885189 172.28.9.114 → 172.28.15.230 TCP 66 55268 → 3306 [ACK] Seq=1399073289 Ack=2707774907 Win=27008 Len=0 TSval=1421356830 TSecr=150493354 0.000010182 211 4 66
   54 6.161921475 172.28.9.114 → 172.28.15.230 MySQL 314 Login Request user=powerdns db=powerdns 0.000036286 211 4 314
   55 6.162621544 172.28.15.230 → 172.28.9.114 TCP 66 3306 → 55268 [ACK] Seq=2707774907 Ack=1399073537 Win=30080 Len=0 TSval=150493354 TSecr=1421356830 0.000700069 235 4 66
   56 6.162626266 172.28.15.230 → 172.28.9.114 MySQL 118 Auth Switch Request 0.000004722 235 4 118
   57 6.203111342 172.28.9.114 → 172.28.15.230 TCP 66 55268 → 3306 [ACK] Seq=1399073537 Ack=2707774959 Win=27008 Len=0 TSval=1421356871 TSecr=150493354 0.040485076 211 4 66
   58 6.534159124 172.28.9.114 → 172.28.15.230 TCP 66 55268 → 3306 [FIN, ACK] Seq=1399073537 Ack=2707774959 Win=27008 Len=0 TSval=1421357202 TSecr=150493354 0.331047782 211 4 66
   59 6.534973808 172.28.15.230 → 172.28.9.114 TCP 66 3306 → 55268 [FIN, ACK] Seq=2707774959 Ack=1399073538 Win=30080 Len=0 TSval=150493447 TSecr=1421357202 0.000814684 235 4 66
   60 6.534987743 172.28.9.114 → 172.28.15.230 TCP 66 55268 → 3306 [ACK] Seq=1399073538 Ack=2707774960 Win=27008 Len=0 TSval=1421357203 TSecr=150493447 0.000013935 211 4 66
   61 7.637762935 172.28.9.114 → 172.28.15.230 TCP 74 55270 → 3306 [SYN] Seq=1624086946 Win=26883 Len=0 MSS=8961 SACK_PERM=1 TSval=1421358306 TSecr=0 WS=128 1.102775192 26883 5 74
   62 7.638544247 172.28.15.230 → 172.28.9.114 TCP 74 3306 → 55270 [SYN, ACK] Seq=3711996320 Ack=1624086947 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=150493723 TSecr=1421358306 WS=128 0.000781312 28960 5 74
   63 7.638559679 172.28.9.114 → 172.28.15.230 TCP 66 55270 → 3306 [ACK] Seq=1624086947 Ack=3711996321 Win=27008 Len=0 TSval=1421358307 TSecr=150493723 0.000015432 211 5 66
   64 7.639301826 172.28.15.230 → 172.28.9.114 MySQL 163 Server Greeting proto=10 version=10.4.13-MariaDB-log 0.000742147 227 5 163
   65 7.639310111 172.28.9.114 → 172.28.15.230 TCP 66 55270 → 3306 [ACK] Seq=1624086947 Ack=3711996418 Win=27008 Len=0 TSval=1421358308 TSecr=150493724 0.000008285 211 5 66
   66 7.639349332 172.28.9.114 → 172.28.15.230 MySQL 314 Login Request user=powerdns db=powerdns 0.000039221 211 5 314
   67 7.640023387 172.28.15.230 → 172.28.9.114 TCP 66 3306 → 55270 [ACK] Seq=3711996418 Ack=1624087195 Win=30080 Len=0 TSval=150493724 TSecr=1421358308 0.000674055 235 5 66
   68 7.640061477 172.28.15.230 → 172.28.9.114 MySQL 118 Auth Switch Request 0.000038090 235 5 118
   69 7.681111357 172.28.9.114 → 172.28.15.230 TCP 66 55270 → 3306 [ACK] Seq=1624087195 Ack=3711996470 Win=27008 Len=0 TSval=1421358349 TSecr=150493724 0.041049880 211 5 66
   70 8.021457160 172.28.9.114 → 172.28.15.230 TCP 66 55270 → 3306 [FIN, ACK] Seq=1624087195 Ack=3711996470 Win=27008 Len=0 TSval=1421358690 TSecr=150493724 0.340345803 211 5 66
   71 8.022196688 172.28.15.230 → 172.28.9.114 TCP 66 3306 → 55270 [FIN, ACK] Seq=3711996470 Ack=1624087196 Win=30080 Len=0 TSval=150493819 TSecr=1421358690 0.000739528 235 5 66
   72 8.022210475 172.28.9.114 → 172.28.15.230 TCP 66 55270 → 3306 [ACK] Seq=1624087196 Ack=3711996471 Win=27008 Len=0 TSval=1421358691 TSecr=150493819 0.000013787 211 5 66
@Habbie
Copy link
Member

Habbie commented Apr 30, 2021

This look like an excellent piece of investigation, thank you!

@Habbie Habbie added this to the auth-4.5.0 milestone Apr 30, 2021
@brianredbeard
Copy link
Author

/me breathes a sigh of relief.

Happy to dodge the dreaded "this seems like a support request" comment. 😄

@Habbie
Copy link
Member

Habbie commented Apr 30, 2021

Right now, the only opportunity for us saying that is when this turns out to be a bug in the mysql/maria client lib. We'll see once we investigate more. We have also been no strangers to reporting bugs elsewhere as a result of our investigations :)

@Habbie
Copy link
Member

Habbie commented Apr 30, 2021

I have reproduced the segfault (with zero SSL involved).

@Habbie
Copy link
Member

Habbie commented Apr 30, 2021

Valgrind reports:

==11186== Invalid read of size 8
==11186==    at 0x483C924: memmove (vg_replace_strmem.c:1270)
==11186==    by 0x55FEA57: crypto_sign_ed25519 (in /usr/lib/x86_64-linux-gnu/libsodium.so.23.2.0)
==11186==    by 0x4846498: auth_ed25519_client (ed25519.c:111)
==11186==    by 0x4B37873: run_plugin_auth (my_auth.c:610)
==11186==    by 0x4B417D0: mthd_my_real_connect (mariadb_lib.c:1646)
==11186==    by 0x4B3E2AF: mysql_real_connect (mariadb_lib.c:1295)
==11186==    by 0xC10610: SMySQL::connect() (smysql.cc:506)
==11186==    by 0xC109A3: SMySQL::SMySQL(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned short, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, unsigned int, bool, bool) (smysql.cc:532)
==11186==    by 0xC09891: gMySQLBackend::reconnect() (gmysqlbackend.cc:65)
==11186==    by 0xC08EF1: gMySQLBackend::gMySQLBackend(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (gmysqlbackend.cc:43)
==11186==    by 0xC0FFC9: gMySQLFactory::make(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (gmysqlbackend.cc:168)
==11186==    by 0xBCC045: BackendFactory::makeMetadataOnly(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (dnsbackend.hh:411)
==11186==  Address 0x20 is not stack'd, malloc'd or (recently) free'd
==11186== 
==11186== 
==11186== Process terminating with default action of signal 11 (SIGSEGV)
==11186==  Access not within mapped region at address 0x20
==11186==    at 0x483C924: memmove (vg_replace_strmem.c:1270)
==11186==    by 0x55FEA57: crypto_sign_ed25519 (in /usr/lib/x86_64-linux-gnu/libsodium.so.23.2.0)
==11186==    by 0x4846498: auth_ed25519_client (ed25519.c:111)
==11186==    by 0x4B37873: run_plugin_auth (my_auth.c:610)
==11186==    by 0x4B417D0: mthd_my_real_connect (mariadb_lib.c:1646)
==11186==    by 0x4B3E2AF: mysql_real_connect (mariadb_lib.c:1295)
==11186==    by 0xC10610: SMySQL::connect() (smysql.cc:506)
==11186==    by 0xC109A3: SMySQL::SMySQL(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned short, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, unsigned int, bool, bool) (smysql.cc:532)
==11186==    by 0xC09891: gMySQLBackend::reconnect() (gmysqlbackend.cc:65)
==11186==    by 0xC08EF1: gMySQLBackend::gMySQLBackend(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (gmysqlbackend.cc:43)
==11186==    by 0xC0FFC9: gMySQLFactory::make(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (gmysqlbackend.cc:168)
==11186==    by 0xBCC045: BackendFactory::makeMetadataOnly(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (dnsbackend.hh:411)
==11186==  If you believe this happened as a result of a stack
==11186==  overflow in your program's main thread (unlikely but
==11186==  possible), you can try to increase the size of the
==11186==  main thread stack using the --main-stacksize= flag.
==11186==  The main thread stack size used in this run was 8388608.

relevant part of our call stack:

#5  0x00007ffff7cbe2b0 in mysql_real_connect (mysql=0x5555565fb908, host=0x5555565fbe30 "127.0.0.1", 
    user=0x5555565fbe70 "pdnstest", passwd=0x5555565fbe90 "pdnstest", db=0x5555565fbe10 "pdnstest", 
    port=3306, unix_socket=0x0, client_flag=131072) at ./libmariadb/libmariadb/mariadb_lib.c:1295
1295	in ./libmariadb/libmariadb/mariadb_lib.c
(gdb) 
#6  0x000055555605c611 in SMySQL::connect (this=0x5555565fb900) at smysql.cc:506
506	    if (!mysql_real_connect(&d_db, d_host.empty() ? nullptr : d_host.c_str(),

so, this actually does look like a libmariadb bug.

@Habbie
Copy link
Member

Habbie commented May 1, 2021

  1. https://github.com/mariadb-corporation/mariadb-connector-c/blob/3.1/plugins/auth/ed25519.c#L110-L111 calls crypto_sign(signature, packet, NONCE_BYTES, (unsigned char*)mysql->passwd, strlen(mysql->passwd));
  2. it wants to call https://github.com/mariadb-corporation/mariadb-connector-c/blob/3.1/plugins/auth/ref10/crypto_sign.h#L5-L9 (int crypto_sign( unsigned char *sm, const unsigned char *m, unsigned long long len, const unsigned char *pw, unsigned long long pwlen);
  3. but because pdns links libsodium, it calls int crypto_sign(unsigned char *sm, unsigned long long *smlen_p, const unsigned char *m, unsigned long long mlen, const unsigned char *sk) __attribute__ ((nonnull(1, 3, 5))); (you see libsodium in your logs too - libmariadb did not want to call that at all!)

Ironically, these two functions are very related. They just are not compatible.

I did not investigate why SSL settings matter; perhaps something about lib loading order, symbol resolution, waves hands?

In any case, this really is a MariaDB bug. I haven't been able to think of a workaround yet (other than your various combinations of SSL settings). I also did not check if there's a ticket at MariaDB about this.

@Habbie
Copy link
Member

Habbie commented May 1, 2021

I haven't been able to think of a workaround yet (other than your various combinations of SSL settings).

Sorry, I misread that. You had no combination that actually worked. That resolves one mystery.

@Habbie
Copy link
Member

Habbie commented May 1, 2021

(I confirmed that a pdns built without libsodium works fine.)

@Habbie
Copy link
Member

Habbie commented May 1, 2021

If SSL is enabled directly in pdns.conf (e.g. gmysql-ssl=yes) then pdns fails to start the backend, throwing the error Caught an exception instantiating a backend: Unable to launch gmysql connection: Unable to connect to database: ERROR 1043 (08S01): Bad handshake:

I am aware I have ignored this part so far; I suspect it is unrelated to the crashes, but I did not look into it yet.

@Habbie
Copy link
Member

Habbie commented May 1, 2021

I also did not check if there's a ticket at MariaDB about this.

There does not appear to be one yet.

@Habbie
Copy link
Member

Habbie commented May 1, 2021

LD_PRELOAD=/usr/lib/x86_64-linux-gnu/mariadb19/plugin/client_ed25519.so pdns_server (and pdnsutil) works! (But if it breaks you get to keep both halves of the stack trace.)

@Habbie
Copy link
Member

Habbie commented May 2, 2021

Reported to MariaDB as https://jira.mariadb.org/browse/CONC-548

@Habbie
Copy link
Member

Habbie commented May 3, 2021

@brianredbeard
Copy link
Author

Thanks so much!

In parallel I'm coordinating figuring out how to get these changes pulled back into RHEL / CentOS through the "Modularity*"/"AppStream" mechanism.

*Modularity is the "circuit breaker" for working around some of the versioning statements I made in #9929, but is narrowly scoped for specific user-space applications.

As the challenges with SSL are a totally separate issue, I'm closing this as resolved. I'll explore that more and open an issue once I can craft a more meaningful bug report.

@Habbie
Copy link
Member

Habbie commented May 3, 2021

In parallel I'm coordinating figuring out how to get these changes pulled back into RHEL / CentOS through the "Modularity*"/"AppStream" mechanism.

Please let us know how that goes. Normally we discourage commenting on closed tickets but in this case I'd say this ticket is a fine place for that.

As the challenges with SSL are a totally separate issue, I'm closing this as resolved. I'll explore that more and open an issue once I can craft a more meaningful bug report.

Perfect. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants