Skip to content

Commit

Permalink
Enforce docs ids _changes filter optimization limit
Browse files Browse the repository at this point in the history
It turns out `changes_doc_ids_optimization_threshold` limit has never been
applied for the clustered changes feeds. So it was effectively unlimited. This
commit enables it, and also adds tests to ensure the limit works.

Since we didn't have a good Erlang integration test suite for clustered changes
feeds, which allowed this case to slip through the cracks, add a few more tests
along the way to test the majority of parameter combinations which might
interact: sharding (single shards vs multiple), continuous vs normal, reverse,
row limits etc.

The previous limit was 100, but since it was never actually applied it's
equivalent not having one at all, so let's pick a new one. I chose 1000
noticing that at Cloudant, with close to 3000 we had fabric timeouts on a busy
cluster, so that seemed too high. And 1000 seemed about the ballpark of the
what size of _bulk_get batch might be. Adding a benchmarking eunit test
https://gist.github.com/nickva/a21ef04b7e4bdbed5fdeb708f1d613b5 showed about
50-75 msec to query batches of 1000 random (uuid) doc_ids for Q values 1
through 8.
  • Loading branch information
nickva committed Jan 24, 2023
1 parent 3181d92 commit 262e4d1
Show file tree
Hide file tree
Showing 4 changed files with 683 additions and 39 deletions.
5 changes: 4 additions & 1 deletion rel/overlay/etc/default.ini
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,10 @@ view_index_dir = {{view_index_dir}}
; The speed of processing the _changes feed with doc_ids filter can be
; influenced directly with this setting - increase for faster processing at the
; expense of more memory usage.
;changes_doc_ids_optimization_threshold = 100
;
; NOTE: Up until version 3.3.1 this setting did not actually take effect for
; clustered _changes operations. It was applied to node local _changes feeds only.
;changes_doc_ids_optimization_threshold = 1000

; Maximum document ID length. Can be set to an integer or 'infinity'.
;max_document_id_length = infinity
Expand Down

0 comments on commit 262e4d1

Please sign in to comment.