Performance CouchDB v2 versus CouchDB v3

[H--o-l]

Hey!

This is a question about a performance loss when updating from CouchDB 2.3.1 to 3.1.1.

We use CouchDB in a cluster configuration, with q=2 and n=3, max_dbs_open = 20000, update_lru_on_read = false and use_kill_all = true.
Every day, we run a script that iterates over more than 70,000 databases.
For half of these bases it does a POST /_all_docs?include_docs=true -d {"keys": [1, 2, 3]} of three documents. For the other half it does only one GET /document.
Requests are performed in parallel, using 16 processes (we've also tested 32 ... 512).

After we upgraded to 3.1.1, the script became 4× slower. On average: 8 min with CouchDB 2, 25 min with CouchDB 3.

To solve that, we tried:

• Using 16, 32, 64, 128, 512 processes (in our script) to perform requests.
  With CouchDB 2, the more processes, the faster the script runs, however it could more easily crash CouchDB (error 500).
  With CouchDB 3, the script time is constant, CouchDB doesn't use more CPU, and doesn't crash.
• Change [ioq] configuration (high concurrency + ratio / and the opposite) but without success.
• Use a higher HTTP backlog with server_options = [{backlog, 128}, {acceptor_pool_size, 16}, {max, 4096}].

What we consider trying:

• Changing stream_limit value on one node only, but we are not sure whether it's dangerous or not.
• Using POST /_all_docs with the stale option, but we don't think it will change anything.

My question is if someone can give us clues of parameters we could change to get back the performance we lost? Is there some kind of new restrain/concurrency limit in CouchDB 3?

[wohali]

First step: check ken and smoosh.

ken keeps all views warm by pre-calculating them. This didn't occur in CouchDB 2.x, you had to manually access each view to keep it warm. Do you see views being built in GET /_active_tasks after POSTing a few documents? Does top on the nodes show couchjs eating significant CPU?

smoosh keeps databases compacted much more effectively than the 2.x compactor. Do you see compactions being built in GET /_active_tasks during the slowness?

Both can be adjusted to be less aggressive, or if desired, disabled completely.

[H--o-l]

Hi @wohali, thanks for the quick feedback.

I've conducted an experiment to measure whether ken and smoosh could be the culprits.

While our script was running, I sampled GET /_active_tasks every second.
82% of the time the returned list was empty, 18% of the time there was one compaction running, 0% of the time there was ken running

I can also add that, when our script is running, the CPU activity on the machine is lower with CouchDB 3 than with CouchDB 2, from 60% to 30% now, and the disk read activity seems to be higher with CouchDB 3 (+ 14%)

Do you think there could be some kind of new restrain/concurrency limit in CouchDB 3?

[H--o-l]

Hi @wohali !

Do you have any more lead we could try?
We are happy to do any test you think is worth doing.

🙏

[H--o-l]

Hi!

I have two new elements to add to this issue.

First we tried to POST /base/_all_docs?update=false -d {"keys": ...} instead of POST /base/_all_docs -d {"keys": ...} but it did not change the time the script takes to run (it's still 4× slower than with CouchDB 2)

Secondly, we made a small chart of the number of databases we can read per second. We do a POST /base/_all_docs and a GET /base/document on each database:

Three gaps, at 1:15:00, 1:25:00, and 1:35:00 are due to the limitation of the logger and should be ignored.

With this chart, we can see the performance is constant over time.

So we still don't know why there is this 4× performance decrease between CouchDB 2 and 3.
Next steps:

1. We are considering another test (changing stream_limit value from 5 to 10) but we must avoid downtime. Do you think the test is worth doing? Can it be done on one CouchDB node at a time (in a cluster) without risk?
2. Is there some kind of new restrain/concurrency limit in CouchDB 3?
3. Do you have any idea of how to investigate further (we've already read the whole changelog multiple times).

[kocolosk]

Changing the stream_limit is safe to do without incurring downtime, but it will not have any effect at all on your individual document requests nor on requests to retrieve specific keys from _all_docs. It only comes into play when streaming contiguous ranges of indexes.

Are you tracking individual request latencies? You're increasing concurrency without seeing any throughput improvement in your tests against 3.1.1, so I'd expect to see latencies rising as a function of concurrency. Do you happen to see any difference for the individual document requests vs. the ones requesting multiple keys? Just curious.

This behavior of CouchDB running stably, not using all its CPU and not delivering great throughput sounds like an aggressive IOQ to me. Can you share the server configuration you're using, or at least the [ioq] and [ioq.bypass] settings? The latter block is particularly relevant as the settings in the default.ini are not replicated in the codebase; i.e. if you don't pull in default.ini in your build you'll have a different (and more restrictive) configuration. 3.1.1 is supposed to open up the throttle more than 2.3.1 in its default configuration, so I'm curious to explore this one.

[kocolosk]

One other investigation tip -- the _system endpoint provides quite a lot of detailed internal server metrics:

https://docs.couchdb.org/en/stable/api/server/common.html?highlight=_system#node-node-name-system

In particular for IOQ, the value of message_queues.ioq is noteworthy. Generally this should be 0; if it's above 10 then you're seeing significant throttling.

[H--o-l]

Hi @kocolosk, thanks a lot for helping us on the subject.

Changing the stream_limit is safe to do without incurring downtime, but it will not have any effect at all on your individual document requests nor on requests to retrieve specific keys from _all_docs. It only comes into play when streaming contiguous ranges of indexes.

Ok, good to know, thanks!

Are you tracking individual request latencies? You're increasing concurrency without seeing any throughput improvement in your tests against 3.1.1, so I'd expect to see latencies rising as a function of concurrency. Do you happen to see any difference for the individual document requests vs. the ones requesting multiple keys? Just curious.

We do not track individual request latencies.
But we established that the latency is spread over all databases we have to go through (it's the graph of my previous post).
We do not know if there are differences between the individual document requests vs. the ones requesting multiple keys, but I suppose that the one that is done first will probably take more time (due to disk access). Or not, it's just a guess.

This behavior of CouchDB running stably, not using all its CPU and not delivering great throughput sounds like an aggressive IOQ to me. Can you share the server configuration you're using, or at least the [ioq] and [ioq.bypass] settings? The latter block is particularly relevant as the settings in the default.ini are not replicated in the codebase; i.e. if you don't pull in default.ini in your build you'll have a different (and more restrictive) configuration. 3.1.1 is supposed to open up the throttle more than 2.3.1 in its default configuration, so I'm curious to explore this one.

Thanks for the suggestion!

We indeed pull the default.ini file, and we are using ioq configuration default values.
Just to be sure, for a few runs, we completly bypassed ioq with the following:

[ioq.bypass]
os_process = true
read = true
write = true
view_update = true
shard_sync = true
compaction = true

But, as expected, it had no impact.

We think like you that the key issue is that the CPU is under-used compared to what we had with CouchDB 2.
Do you see any other cause that would explain the strong decrease of performance the day we switched to CouchDB 3 ?

[H--o-l]

Hi!

We finally understood what was happening, I’m posting the explanation here in the hope it can help other users using the same hardware as we do.

Our disks are AWS EBS volumes (SSD called io1), they are “IOPS provisioned” with a user-defined maximum limit of 300 disk I/O operations per second. AWS documentation explains how IOPS are counted:

I/O operations that are smaller than 256K each count as 1 consumed IOPS. I/O operations that are larger than 256K are counted in 256K capacity units. For example, a 1024K I/O would count as 4 consumed IOPS.

It appeared that our IOPS value of 300 IOPS was the bottleneck. After raising the IOPS limit, CouchDB 3 now performs very well. For our usage, it is as fast as CouchDB v2, possibly better.

@wohali, @kocolosk, and anyone that has taken the time to read this, thanks for the help.

We think that CouchDB 3 performs more disk operations than CouchDB 2, is it possible?

[kocolosk]

Hi, glad you tracked it down! It was definitely puzzling me, I appreciate you circling back and sharing the findings.

I don't personally have any reason to expect that v3 consumes more IOPS for a given workload than v2. I skimmed the release notes and nothing jumped out at me either. But it could be the case; I don't think we have too many benchmarks that specifically study this over time. My team's deplyoments are usually running on local SSDs that don't have explicit IOPS throttling so this wouldn't stand out so clearly either.

[wohali]

Most likely is more compaction - the default compaction settings in 3.x are more aggressive than the old 2.x compaction daemon.

[H--o-l]

@kocolosk My pleasure! I hope it can help others. Thanks for the feedback.

@wohali Thanks for the idea! However, I am not convinced because I played with compaction parameters a fair bit this year, for another issue. What I mean is that I am starting to have experience with using the compaction daemon (I hope, but still not an expert). The thing is, at the time the workload described in this issue is executed, there are almost no users or services connected to the DB, it's during the night for us, and thus there is almost no compaction going on (no compaction returned by GET /_active_tasks and no compaction trace in the log like ratio_dbs: Starting compaction for ...). I'm happy to discuss it further, don't hesitate to ask me for details or to contradict me!