Performance degradation after migrating from the mariadb driver

[jubui]

I am seeing 25-50% performance degradation against aurora mysql instances when using the aurora mysql driver when compared to the mariadb driver. I am using the driver through hibernate and making thousands of calls to average out the performance and doing this for about 50 different operations and almost across the board aurora mysql driver performs worse.

Will provide a repro when I have time, but I wanted to make this issue visible.

[davecramer]

Are you connecting on each call ? There is a considerable initial connection overhead.

[jubui]

No, but even if we were, we are averaging thousands of operations so any network anomalies would be averaged out anyway.

Are there any performance regression tests?

[davecramer]

There are: https://github.com/awslabs/aws-mysql-jdbc/tree/main/src/test/java/testsuite/perf

I'm curious is the MariaDB driver faster than the MySQL driver ?

[adibaranga]

We tried to replace the maria db driver with aws driver and I can confirm, right after the switch, we've seen a decrease in our QPS (daily performance tests) approximately between 30%-50%

[davecramer]

@adibaranga Can you share the performance tests ?

[jubui]

@davecramer I have a somewhat recent update that will provide more context.

Originally back around Jan 10, I was comparing maria driver v2.7.2 against the aurora driver. The tests are a bit application specific so you won't understand exactly what they mean, but a simplified explanation is that we have numerous tests around deletion, insertion, querying, and updating. For each of those, we have tests for doing the operation for single objects or multiple objects. On top of that, we do multi-threaded, single threaded, batch, and nested object testing. Each test runs 100-1000s of runs and averages out the result to try to account for network latencies, network anomalies, caching, etc. Yes, in that test, I found that the averaged-out results showed that the maria driver v2.7.2 was much faster (25-50%) than the test using the aurora driver. Do keep in mind that many of the tests only showed a single-digit millisecond difference between the maria and aurora driver, so in the grand scheme of things, it still isn't a huge deal.

After that, as I learned more about how the maria driver works with aurora and how the maria driver v2.7.x line is no longer compatible with Aurora (see https://jira.mariadb.org/browse/CONJ-1033) I decided to test the latter maria driver versions (in this case v2.7.8) with the connection properties needed to make maria driver work against aurora (disabling usePipelineAuth as well as useBatchMultiSend).

With these properties disabled (documentation indicates disabling these will have performance implications) to get the maria driver to even connect to the aurora instance, I ran the tests again. This time I saw that aurora actually performed better than the maria driver for a lot of the tests. Whether the performance is due to some other factor in the experiments (e.g., network, or latencies associated with the particular aurora instance I was using that day) or if it was due the connection parameters (I am inclined to believe this is the case, since we run a lot of tests and have more faith in them), I can say that the new fairer comparison shows that the aurora driver is more on par with the maria performance. A lot of tests showed that the aurora driver was faster for a particular operation, though with some tests the maria driver was faster. In any case, the numbers looked way better than when I compared it to the maria driver v2.7.2 without the connection parameters disabled.

In short, I'm no longer concerned about the aurora performance when compared to the maria driver.

[davecramer]

@jubui thanks for your detailed response!

[adibaranga]

@davecramer unfortunately I can't share the tests with you, but they are performance and loading tests (using jmeter) against a REST API. The application essentially is doing a few reads and inserts from/to database, but also compute pre-signed URLs from S3 bucket.
I updated the driver, replacing mariadb driver (2.7.4) with aws jdbc driver (1.1.4). Just to clarify, this was the only change done on the application level. We kept this new setup for about one week and the QPS decrease in average from 8 to 6.
I can not prove 100% percent that the driver was the main culprit (because there are many pre-signed urls computed for each request) but once I reverted the driver, the QPS got back to 8.

Anyway, I didn't revert it because of this performance downgrade and also no further investigation was performed, but because of the lack of support for Aurora Global Database (MySQL) and more specific, support for read replica write forwarding.

[davecramer]

@adibaranga Thanks for the information.
Just to clarify your queries per second on average went from 8 to 6 ? This seems like a relatively small number. I'm guessing there are times where the QPS is much higher? It would be useful for us to understand the degradation for the higher number.

As for Aurora Global Database what kind of support are you looking for ?

[adibaranga]

8 QPS vs 6 QPS is actually referring to the number of API calls per second (within each API call there are a few selects + one insert in the database, along with some S3 pre-signed urls computation).

As for the support I am looking for, please check: #389

[jubui]

@adibaranga You're on 2.7.4 which was before the maria devs put out a change that required people to use usePipelineAuth and useMultiBatchSend. It isn't super clear to me what the problem was, but it sounds like the Aurora server/proxy was always broken and to not disable those parameters would be incorrect behavior or something (see https://jira.mariadb.org/browse/CONJ-1033 and comments). Without disabling those parameters, it may appear to work and maybe even faster than when you disable the parameters, but it sounds like the problem is with the server. In other words, I'm not sure the comparison you're doing is fair.