JAMES-3204 Push limit to Cassandra backend when reading messages #3428
Conversation
|
The impact of other usages of |
|
test this please |
...ssandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdDAOTest.java
Outdated
Show resolved
Hide resolved
|
Let's try that again test this please |
...sandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageMapperTest.java
Outdated
Show resolved
Hide resolved
...ds-common/cassandra/src/test/java/org/apache/james/backends/cassandra/StatementRecorder.java
Outdated
Show resolved
Hide resolved
...sandra/src/test/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageMapperTest.java
Outdated
Show resolved
Hide resolved
Of course. |
mbaechler
added
the
requires-load-testing
|
I set the requires-load-testing flag |
This enables to test executed statements and allows enabling assertions on query count reduction, for performance purpose.
This class ends up doing reads to messageV2 for entries that are later on discarded by the soft limit.
|
Force pushed to solve a rebase conflict |
Let me argue that it don't as I prove to reduce the query count by a x25 factor for some basic JMAP operations. I will be waiting for feedback about this decision but I would like that change to be part of the next James release for OpenPaaS as it solves some of our customer concerns. |
|
merged |
Well, that would not be the first time in my work life that things are not going as expected. Checking a property we expect is usual practice. Why should we avoid that? |
I tried but couldn't run IMAP scenari on the charge CI... Anyway, there still is a task about evaluating the performance impact of this change. |
We noticed some
BusyPoolexceptions filling up the driver queue upon IMAP query (FETCH flags for all the messages in the mailbox).The MessageManager do batch those reads (by default by 200 for metadata)., wich then call the MessageMapper with this limit.
Some unit tests performed at the Cassandra mailbox level proved the soft filtering did badly applies, and that we were performing uneeded extra reads for the full batch read from the database.
One good mitigation strategy is to push the limit to the Cassandra query, and ensures filtering happens before the extra reads are performed.
https://projectreactor.io/docs/core/release/api/reactor/core/publisher/Flux.html#take-long- documents that it just don't propagate downstream request once a given amount is reached however it do not ensure any form of backpressure. We might want to further audit our code, looking for similar take mis-usages.