-
-
Notifications
You must be signed in to change notification settings - Fork 9.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Messenger] Bad performances with the Doctrine/MySQL transport #47828
Comments
Non-support for |
@stof I meant that there is no method in
|
We have this issue as well. I added the The delete statement introduced in the Merge Request (this one) is causing the discussed SELECT to wait regardless of The combination of these 2 changes makes it so we're able to process a large queue w/o any issues with ~10 of consumers. |
@Wouter0100 Did you publicly publish on GitHub a fork of the package with the setup that works for you? If anyone is interested, I made this fork https://github.com/sylfabre/doctrine-messenger/tree/5.4 for Symfony 5.4 which adds the |
Unfortunately, I don't have that available. Sorry. :( We hot patched it for now, as we needed to get it working on the very short term - mainly because we're migrating to an alternative queuing backend as we speak. |
Skipping lock isn't dangerous when there are multiple consumers ? |
Hey, thanks for your report! |
Friendly ping? Should this still be open? I will close if I don't hear anything. |
This issue appears every now and then in one of our projects.. |
Hello @davidroth We are planning to use RabbitMQ as a backend as it is more appropriate So I don't plan to create a PR here: feel free to use my fork and submit one if you want to! |
@sylfabre Thx for the update. Yes we we also switched to RabbitMQ for the high volume message workloads. But for a transactional outbox the table based queue is still needed. |
@davidroth @sylfabre @Wouter0100 |
Symfony version(s) affected
5.4.13 (but I guess it happens with 6.* as well as the code is the same)
Description
We have the following setup
READ-COMMITTED
as transaction isolation levelphp bin/console messenger:consume async_email --limit=100 --time-limit=300 --memory-limit=480M
We experienced a lag of up to ~80k messages in the queue because workers spend most of their time waiting to lock a message to process.
A colleague tried to increase the number of processes from 8 to 80 but it made things worst.
Our processing tops at 3 or 4 consumed messages per seconds on this specific queue
Here are my findings so far:
The MySQL command
SHOW FULL PROCESSLIST
shows a lot of theSELECT ... FOR UPDATE
requests are in theSending data
state.The
SELECT ... FOR UPDATE
uses the index to find and lock a message:IDX_DC4C2A40E3BD61CE is on the
available_at
columnBecause it uses an index and we are using
READ-COMMITTED
as the transaction isolation level, then the following happens:Source: https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-isolation-levels.html
So the bottleneck in my case happens when workers search for a message to lock because the query is blocking.
Once locked, the message is very very quick to handle.
How to reproduce
I guess it's quite difficult to reproduce as I think there is a balance between:
Possible Solution
Solution 1 - Remove the indexes
As suggested by #42868 removing the indexes actually removes the blockings issue as per MySQL documentation because without indexes:
However, when there are a lot of messages, it leads to slow queries because MySQL uses a "filesort" because of the
ORDER BY available_at
clause.I think this clause could be removed if losing the FIFO-like strategy is acceptable
Solution 2 - Use SKIP LOCKED
MySQL has a
SKIP LOCKED
clause that could be used here: https://dev.mysql.com/doc/refman/8.0/en/innodb-locking-reads.htmlI measured these performances for the SELECT query:
FOR UPDATE
&SKIP LOCKED
: about 0.002 seconds to return a rowFOR UPDATE
& withoutSKIP LOCKED
: about 2 seconds to return a rowFOR UPDATE
& withSKIP LOCKED
: about 0.2 seconds to return a row=> x10 faster with the
SKIP LOCKED
optionAs far as I know,
doctrine/dbal
doesn't support theSKIP LOCKED
option.An issue has been around for years about it with
doctrine/orm
: doctrine/orm#7746I suggest here that the Doctrine transport "manually" adds the
SKIP LOCKED
option when the database platform is MySQL 8.0 or later (maybe others that support it)This is my preferred solution
Additional Context
No response
The text was updated successfully, but these errors were encountered: