MongoWaitQueueFullException in Webflux + Reactive Mongo

[cbornet]

Affects: 5.1.2.RELEASE

---

The latest TechEmpower's benchmark shows a lot of errors for Webflux + Spring data mongo.
I digged and found out that under load, we get a lot of
com.mongodb.MongoWaitQueueFullException: Too many threads are already waiting for a connection. Max number of threads (maxWaitQueueSize) of 500 has been exceeded.
exceptions.

From our discussion on gitter, @mp911de thinks that "WebFlux could have a notion of connection backpressure and should limit subscription rate/demand rate to provide a sense of protection for downstream components within the application".

Code of the benchmark here

Related issue : #20338

[rstoyanchev]

One easy change would be to replace Flux.merge here with Flux.concat. That would match to what Dropwizard and Vert.x are effectively doing as far as I can see.

[cbornet]

@rstoyanchev thanks for your answer. Unfortunately that doesn't work. Still getting MongoWaitQueueFullException ...

[rstoyanchev]

From the test summary:

All tests are run at 512 concurrency.

So that will put it over the 500 limit indeed even if each request is (should be) using at most 1 connection to Mongo at any time. May the Mongo config could be updated to accommodate the expected max concurrency?

WebFlux could have a notion of connection backpressure and should limit subscription rate/demand rate to provide a sense of protection for downstream components within the application

I don't see a way for WebFlux to influence the number of HTTP connections. We could try and limit the number of concurrently handled HTTP requests, and that might be a useful feature, but how would it help in this case. In other words I don't think we'd want to restrict the number of concurrent HTTP requests to 500, would we?

[cbornet]

May the Mongo config could be updated to accommodate the expected max concurrency?

I guess we could increase the queue size to make this test pass but that wouldn't eliminate the root issue that a large number of requests can overflow the driver.

In other words I don't think we'd want to restrict the number of concurrent HTTP requests to 500, would we?

Indeed. But how should we solve this issue ? In Webflux ? In Spring data ? In the mongo driver ? One of the most advertised feature (if not the most) of reactive streams is the stability and reliability brought by backpressure. But here it looks like backpressure doesn't operate. WebFlux + Reactive Mongo will be one of the most current use case until we have something like R2DBC or ADBC for SQL so I think it's very important to show that it works at scale.

[mp911de]

The actual underlying issue is an overload of a particular resource. That can happen IMO with any kind of resource, regardless of the technology that is used as a data source.

WebFlux has no knowledge of downstream resource capacity.
The Reactive Streams protocol looks at backpressure from a subscriber perspective to not overwhelm a subscriber. It's not considering the the producer perspective, whether a producing Publisher has sufficient resources to perform its work.

/cc @smaldini @simonbasle @bsideup

[schauder]

I stumbled over this some time ago and cobbled together something that creates backpressure for resources that don't do so themselves:

https://github.com/schauder/reactive-threads/tree/master/src/main/java/de/schauder/reactivethreads/limited

The implementation is probably really bad, but I think we (possibly Reactor) should have something like this for resources that don't provide backpressure themselves.

[simonbasle]

Yes, and I believe that would also be resolved by a reactive DB Connection pooled capped at 500 connections.

[cbornet]

The connection pool is currently managed by the mongo async/rx-streams driver. Does it mean that it should be modified to support backpressure ?

[sdeleuze]

I think this would deserve an issue in https://jira.mongodb.org/projects/JAVARS/ indeed. See also that one: Driver uses only one thread to process incoming data.

[mp911de]

Not quite sure what problem we're trying to solve right now. The MongoDB driver respects backpressure. It starts allocating resources as soon as there is demand (calling Subscription.request(n)) and not earlier.

In fact, the driver gracefully queues database commands until reaching a limitation threshold. Once the request queue hits the limit, the driver rejects additional work.

What is the behavior you would like to see? (Note this part is probably rather data-specific than Spring Framework-specific)

[mp911de]

I checked the benchmark implementation and second @rstoyanchev regarding Flux.concat(…).

Applying this fix allows scaling using a higher request concurrency while a high query level argument does not exhaust the MongoDB connection pool immediately. Serializing queries aligns with the behavior exposed in other benchmarks such as vert.x.

[cbornet]

What is the behavior you would like to see?

Shouldn't the driver pull commands into the queue only when there's room available in it ? The current behavior looks like publishers are directly pushing there commands into the queue.

Applying this fix allows scaling using a higher request concurrency while a high query level argument does not exhaust the MongoDB connection pool immediately. Serializing queries aligns with the behavior exposed in other benchmarks such as vert.x.

Yes, I agree. But that doesn't make the exception disappear...