Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Quarkus version change from 2.10.1.Final to 2.11.1.Final has breaking changes with Percona SqlProxy #27386

Closed
harpermj66 opened this issue Aug 19, 2022 · 41 comments
Closed

Quarkus version change from 2.10.1.Final to 2.11.1.Final has breaking changes with Percona SqlProxy #27386

harpermj66 opened this issue Aug 19, 2022 · 41 comments
Assignees
@tsegismont
Labels
area/hibernate-reactive kind/bug Something isn't working
Milestone
2.15.0.CR1

Comments

@harpermj66
Copy link

harpermj66 commented Aug 19, 2022 •
edited

Describe the bug

Quarkus with hibernate reactive and mysql database configured with a version to 2.11.1 version of quarkus allowed data retrieval from Percona SqlProxy.
It worked very well in fact.
As of 2.11.1 I'm not able to retrieve data.

Expected behavior

Executing queries writen using hibernate reactive with panache should continue to function as they did in version 2.10.1

Actual behavior

a) on the client Failed to execute statement [select terminalen0_.TERMINAL_ID as terminal1_22_, terminalen0_.COMPANY_ID as company_2_22_, terminalen0_.NAME as name3_22_, terminalen0_.UUID as uuid4_22_ from TERMINAL terminalen0_ where TERMINAL_ID=?]: could not execute query: java.util.concurrent.CompletionException: io.vertx.core.impl.NoStackTraceThrowable: Pending requests failed to be sent due to connection has been closed.

b) on SqlProxy itself

2022-08-19 15:20:29 MySQL_Session.cpp:3364:handler___status_NONE_or_default(): [ERROR] Unexpected packet from client 127.0.0.1 . Session_status: 21 , client_status: 0 Disconnecting it
2022-08-19 15:20:29 MySQL_Session.cpp:149:kill_query_thread(): [WARNING] KILL CONNECTION 1254813 on 10.2.5.21:3306

How to Reproduce?

  1. configure a mysql cluster with a Percona SqlProxy
  2. configure a quarkus test to retrieve data via reactive hibernate panache
  3. use verstion 2.10.1 of quarkus to verify the data is returned.
  4. upgrade to version 2.11.1 and observer the test will error as above.

Output of uname -a or ver

Linux locus-dev01 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Output of java -version

openjdk version "11.0.16" 2022-07-19 OpenJDK Runtime Environment (build 11.0.16+8-post-Ubuntu-0ubuntu120.04) OpenJDK 64-Bit Server VM (build 11.0.16+8-post-Ubuntu-0ubuntu120.04, mixed mode, sharing)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.11.1

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

Only breaks when going through the percona SqlProxy
@harpermj66 harpermj66 added the kind/bug Something isn't working label Aug 19, 2022
@quarkus-bot
Copy link

quarkus-bot bot commented Aug 19, 2022

/cc @DavideD, @Sanne, @gavinking

@gastaldi
Copy link
Contributor

Can you provide a test project reproducing the error? Also, as I have never used Percona SqlProxy, maybe a Docker compose file can help in setting up the environment to reproduce this issue

@harpermj66
Copy link
Author

harpermj66 commented Aug 22, 2022 •
edited

I would be extremely difficult for me to provide a test project that would have 3 mysql servers in a cluster and another 3 percona sql proxies. The setup of a proxy itself is no walk in the park.

What about taking the what did we change between versions route? I have feeling it may just be a configuration default that we may be able to test reverting.

@Sanne
Copy link
Member

Sanne commented Aug 22, 2022

We use the Reactive Client drivers from Vert.x - the most likely "change" was a version upgrade in these drivers.

Do you think you could test by overriding the vert.x version for client? Unfortunately the coupling Quarkus/Vert.x is quite high so it might not be possible to get a functioning application with a different driver, but provided you're only changing some micro versions it might work well enough to narrow this problem down.

cc/ @tsegismont

@harpermj66
Copy link
Author

harpermj66 commented Aug 22, 2022 via email

@Sanne
Copy link
Member

Sanne commented Aug 22, 2022

In your dependencyManagement section, add the vert.x client version above the import for the Quarkus BOM, so that it takes precedence:

  
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>io.vertx</groupId>
                <artifactId>vertx-mysql-client</artifactId>
                <version>something-something</version>
            </dependency>
            <dependency>
            <groupId>io.quarkus</groupId>
                <artifactId>quarkus-bom</artifactId>
                <version>${quarkus.platform.version}</version>
                <scope>import</scope>
                <type>pom</type>
            </dependency>

@harpermj66
Copy link
Author

harpermj66 commented Aug 22, 2022 via email

@tsegismont
Copy link
Contributor

This might be related to pipelining (eclipse-vertx/vertx-sql-client#1168).

I am trying to reproduce atm

@tsegismont
Copy link
Contributor

I was able to reproduce. I captured network traffic and, indeed, pipelining seems involved:

  • with Quarkus 2.12.2, the client sends execute and close packets immediately after prepare response packet is received
  • with Quarkus 2.10.2, the client waits for response ok packet before sending close packet

@harpermj66
Copy link
Author

harpermj66 commented Aug 23, 2022 via email

@tsegismont
Copy link
Contributor

tsegismont commented Aug 24, 2022 via email

@harpermj66
Copy link
Author

harpermj66 commented Aug 24, 2022 via email

@tsegismont
Copy link
Contributor

tsegismont commented Aug 24, 2022 via email

@geoand
Copy link
Contributor

geoand commented Sep 1, 2022

Should this be closed as the actual issue is tracked in Vert.x?

@tsegismont
Copy link
Contributor

I believe we can keep it open until the fixed version is integrated in Quarkus.

@geoand
Copy link
Contributor

geoand commented Sep 1, 2022

Okay, no problem

@rasvi24
Copy link

rasvi24 commented Sep 3, 2022 •
edited

Great, this saved me. Thank you. This error gave me no stacktrace even in TRACE level. After trying hours with onFailure just after the query execution, found this issue here just before creating a new one. For anybody trying to use proxysql with the latest 2.12.final version Just downgrade the vertx mysql client as mentioned above. Below version is working.

        <dependency>
            <groupId>io.vertx</groupId>
            <artifactId>vertx-mysql-client</artifactId>
            <version>4.2.7</version>
        </dependency>

This was referenced Sep 16, 2022
Closed
Merged
@tsegismont
Copy link
Contributor

Fixed upstream in eclipse-vertx/vertx-sql-client#1236 (Vert.x 4.3.4)

@tsegismont tsegismont self-assigned this Sep 27, 2022
@geoand
Copy link
Contributor

geoand commented Sep 28, 2022

Thanks for the update @tsegismont.

Just out of curiosity, when is Vert.x 4.3.4 planned?

@harpermj66
Copy link
Author

harpermj66 commented Sep 28, 2022 via email

@tsegismont
Copy link
Contributor

@geoand very soon, beginning of next week maybe, right @vietj?

@harpermj66 you're welcome. Any news from your contacts at Percona? Are they still willing to talk about this?

@harpermj66
Copy link
Author

Embarressingly not willing to engage for reasons which I don't really go along with.
Is there something they should be doing for future proofing?

@tsegismont
Copy link
Contributor

No worries, it is fine to not support pipelining. I can think of reasons why ProxySQL doesn't (shared connections).

@harpermj66
Copy link
Author

harpermj66 commented Oct 11, 2022 via email

@tsegismont
Copy link
Contributor

This should be fixed in 2.13.1.Final. Can you have a try? Thank you

@harpermj66
Copy link
Author

I don't know where that comment came from 4 hours ago :-)
I've already been using 2.13.1.Final and had no issues but we'll do a bit more testing.

@diegoherrojo
Copy link

diegoherrojo commented Oct 20, 2022 •
edited

Hi everyone!

We are developing a Quarkus based app (with reactive Panache stack), and we cannot make it work with version 2.13.3.Final (or with any version equals or greater than 2.11.0) with a database that sits behind a SQL proxy. It works perfectly when we connect with a DB without a proxy in the middle, or when we use up to 2.10.4 version.
We can override vertx mysql driver to version 4.2.7, but from 2.13.0 that override produces runtime errors.

The error we get when we execute transactions against a DB behind a SQL proxy is:

(vert.x-eventloop-thread-0) Unexpected error: io.smallrye.mutiny.CompositeException: Multiple exceptions caught:
	[Exception 0] io.vertx.core.impl.NoStackTraceThrowable: Fail to read any response from the server, the underlying connection might get lost unexpectedly.
	[Exception 1] io.vertx.core.impl.NoStackTraceThrowable: Connection is not active now, current status: CLOSED

Is there any workaround to this problem?

Thanks!

@tsegismont
Copy link
Contributor

@diegoherrojo thanks for reporting.

Can you share a simple reproducer? We have a test upstream which verifies connectivity through ProxySQL, so I'm confused.

Can you verify with mvn dependency:tree if your project uses vertx-mysql-client 4.3.4 ?

@diegoherrojo
Copy link

Thanks for your quick response @tsegismont!

Yes, I verified that we are using vertx-mysql-client 4.3.4.

Let me create a simple test to share here.

@diegoherrojo
Copy link

Hi everyone!

@tsegismont I created a small app reproducing the error with Quarkus 2.13.2.Final.

In order to run it, please unzip test.zip and then execute run.sh.
Docker compose, maven and jdk 17 are needed for the test.

The test spins up a MySQL and a ProxySQL and then executes 2 times a POST to the app API (which creates some entities in DB):

  • Once connecting directly to MySQL (successful).
  • Second time connecting through ProxySQL (fails).

Thanks!

@tsegismont
Copy link
Contributor

Thank you, I'll try this out asap and come back to you @diegoherrojo

@diegoherrojo
Copy link

Thank you @tsegismont !

@tsegismont
Copy link
Contributor

@diegoherrojo I was able to reproduce today and will come back to you shortly

@diegoherrojo
Copy link

Thanks @tsegismont !

@tsegismont
Copy link
Contributor

This is another with pipelining which I had not identified in the previous fix.

I've filed eclipse-vertx/vertx-sql-client#1254

@diegoherrojo
Copy link

diegoherrojo commented Nov 9, 2022 •
edited

Oh great @tsegismont! Thanks for you help!

@tsegismont
Copy link
Contributor

@diegoherrojo the fix has been merged in eclipse-vertx/vertx-sql-client#1257

I tried a snapshot version with your reproducer and the problem goes away.

Vert.x 4.3.5 will be released very soon. When Quarkus upgrades to this version we can close this issue.

@diegoherrojo
Copy link

Thanks @tsegismont for all your effort and support! 👌

@tsegismont tsegismont added this to the 2.15 - main milestone Nov 24, 2022
@tsegismont
Copy link
Contributor

Closed by #29246

@tsegismont
Copy link
Contributor

@diegoherrojo if you still experience issues with a snasphot version or 2.15 (when it is released) please reopen.

@diegoherrojo
Copy link

Thank you @tsegismont !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tsegismont tsegismont

Labels
area/hibernate-reactive kind/bug Something isn't working
Projects
None yet
Milestone
2.15.0.CR1
Development

No branches or pull requests
7 participants
@Sanne @gastaldi @tsegismont @geoand @rasvi24 @harpermj66 @diegoherrojo