The best experience to use Couchbase with webflux?

[adar-v]

spring-boot-starter-data-couchbase-reactive:2.6.9
com.couchbase.client: java-client: 3.3.1

Sometimes I get a lot of Couchbase UnambiguouslyTimeoutException errors and I find that the Couchbase cb-io thread is occupied by function code.
Then I found out that when using
repository.findById
Or after other couchbase operations, the thread will switch to the couchbase cb-io thread. If I have some time-consuming functions after the cousebase operation, the cb-io thread will be occupied until the function ends, resulting in fewer threads in the couchbase thread pool. , couchbase operations in a large number of queues wait for a timeout.
So my best practice is to publishOn/subscribeOn it to other threads immediately after the couchbase request ends?
such as this

repository.findById(id)
                 .publishOn(Schedulers.boundedElastic())
                 .flatMap(user -> {
                 // logic code
                 });

Is this best practice? Or any other suggestions? Or is there any global setting that controls the early release of database operation threads

[mikereiche]

eventLoopThreadCount is configurable

IoEnvironment.Builder	eventLoopThreadCount​(int eventLoopThreadCount)

IoEnvironment.Builder eventLoopThreadCount​(int eventLoopThreadCount)

[adar-v]

eventLoopThreadCount is configurable

IoEnvironment.Builder eventLoopThreadCount​(int eventLoopThreadCount)
IoEnvironment.Builder eventLoopThreadCount​(int eventLoopThreadCount)

I currently configure eventLoopThreadCount to be the same number of cores as my CPU.
When I increased eventLoopThreadCount, I found that the occurrence of timeout was effectively reduced, but the problem of timeout could not be completely solved.
During peak business hours, I still find that the operation waiting in the couchbase queue times out because the function code occupies the cb-io thread.

    @Override
    protected void configureEnvironment(final ClusterEnvironment.Builder builder) {
        builder.ioConfig(IoConfig.builder().numKvConnections(4))
                .ioEnvironment(IoEnvironment.builder().eventLoopThreadCount(6))
                .timeoutConfig(TimeoutConfig.connectTimeout(Duration.ofSeconds(3)).kvTimeout(Duration.ofSeconds(10)))
                .retryStrategy(BestEffortRetryStrategy.withExponentialBackoff(Duration.ofMillis(8), Duration.ofMillis(100), 2))
                .compressionConfig(CompressionConfig.builder().minSize(2048))
                .orphanReporterConfig(OrphanReporterConfig.builder().emitInterval(Duration.ofHours(1)))
                .thresholdRequestTracerConfig().sampleSize(20);
    }

Please help me to confirm if there is any problem with my configuration, or which configuration can be adjusted to solve the timeout problem?

error message

  {timestamp=2023-06-07T13:03:03.722+0000, path=/user/openOrder, status=500, error=Internal Server Error, requestId=2403f4b3-10009923, trace=com.couchbase.client.core.error.UnambiguousTimeoutException: SubdocGetRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0x1c4ff57400000001","idempotent":true,"lastChannelId":"1C4FF57400000001/0000000018887A02","lastDispatchedFrom":"10.1.131.1:47768","lastDispatchedTo":"10.1.138.182:11210","reason":"TIMEOUT","requestId":8730422,"requestType":"SubdocGetRequest","retried":0,"service":{"bucket":"default","collection":"_default","documentId":"10023123-user","opaque":"0x85394a","scope":"_default","type":"kv","vbucket":1006},"timeoutMs":10000,"timings":{"totalMicros":10009312}}

[mikereiche]

publishOn(Schedulers.boundedElastic()) seems like the correct solution.

  .ioEnvironment(IoEnvironment.builder().eventLoopThreadCount(6))

You could try a higher value like 120 ( Schedulers.boundedElastic() is cores X 10). See https://stackoverflow.com/questions/61304762/difference-between-boundedelastic-vs-parallel-scheduler

[adar-v]

publishOn(Schedulers.boundedElastic()) seems like the correct solution.

  .ioEnvironment(IoEnvironment.builder().eventLoopThreadCount(6))

You could try a higher value like 120 ( Schedulers.boundedElastic() is cores X 10). See https://stackoverflow.com/questions/61304762/difference-between-boundedelastic-vs-parallel-scheduler

Do you mean to increase the number of eventLoopThreadCount? For example, to 120?
The number of eventLoopThreadCount has a default value of fairThreadCount(). This value is calculated to be a maximum of 8. The following is the source code

  private static int fairThreadCount() {
    int cores = Runtime.getRuntime().availableProcessors() / 2;
    cores = Math.max(cores, 2);
    cores = Math.min(cores, 8);
    return cores;
  }

Seems weird to raise eventLoopThreadCount directly to 120?

[mikereiche]

Yes, increase it to 120. Schedulers.boundedElastic() is cores X 10 according to the stackoverflow post. If you have six cores, that would be 60. So 120 threads would be just twice that (cores x 20). Increasing the threadCount is exactly what should resolve your primary complaint is that there are not enough threads.

[adar-v]

Yes, increase it to 120. Schedulers.boundedElastic() is cores X 10 according to the stackoverflow post. If you have six cores, that would be 60. So 120 threads would be just twice that (cores x 20). Increasing the threadCount is exactly what should resolve your primary complaint is that there are not enough threads.

If I have increased the eventLoopThreadCount to a large value, then I should no longer need to publish to a Schedulers.boundedElastic() after the Couchbase operation is over, and the number of threads feels sufficient.