Skip to content

2.29.0.0-b132

2.29.0.0-b132: [#27601] docdb: Limit number of vector index compactions per tserver
@arybochkin arybochkin tagged this 03 Nov 16:53 
Summary:
Based on perf runs, vector index background compactions should have a way to be limited.
The following options have been added to limit vector index background compactions:
1. Vector index compactions upper limit for number of chunks taken for a compaction. Controlled by a runtime gflags `vector_index_compaction_size_amp_max_merge_width` and `vector_index_compaction_size_ratio_max_merge_width` correspondingly with default values set to `0` meaning no upper limit for the number of chunks take for the vector index compaction.
2. Leveraging of a priority thread pool token to limit the number of running vector index compactions per a tserver. Controlled by a runtime gflag `vector_index_num_compactions_limit` with a default value of `1` meaning only one vector index compaction will run at a time. Priority thread pool token has a parameter `max concurrency` which controls number of tasks run in parallel; each task is added in a deferred state, but is being scheduled for the execution in accordance with its timestamp.

Additionally, two gflags were renamed to be aligned with the current naming pattern:
`vector_index_max_size_amplification_percent` => `vector_index_compaction_size_amp_max_percent`
`vector_index_compaction_min_merge_width` => `vector_index_compaction_size_ratio_min_merge_width`
Jira: DB-17183

Test Plan:
Jenkins

Manual:
1. Create cluster with adjusted flags to test tasks are being deferred by the token and having compaction limit set to 1:
```
$ yb-ctl start --rf=1 --tserver_flags 'priority_thread_pool_size=3,vector_index_num_compactions_limit=1,vector_index_disable_compactions=false,"vmodule=vector_lsm=4,priority_thread_pool=4'
```
2. Create vector extension, table with 3 tablets and index
```
# CREATE EXTENSION vector;
# CREATE TABLE test (id bigserial PRIMARY KEY, embedding vector(1)) split into 3 tablets;
# CREATE INDEX ON test USING ybhnsw (embedding vector_l2_ops);
```
3. Insert 20 rows
```
# INSERT INTO test SELECT i, ARRAY[i] FROM generate_series(0, 19) as i;
```
4. Flush vector index
```
$ yb-admin -master_addresses 127.0.0.1:7100 flush_table ysql.yugabyte test_embedding_idx
```
5. Repeat steps 3-4 four more time to have 5 chunk files to trigger background compaction
```
# INSERT INTO test SELECT i, ARRAY[i] FROM generate_series(20,39) as i;
$ yb-admin -master_addresses 127.0.0.1:7100 flush_table ysql.yugabyte test_embedding_idx
# INSERT INTO test SELECT i, ARRAY[i] FROM generate_series(40,59) as i;
$ yb-admin -master_addresses 127.0.0.1:7100 flush_table ysql.yugabyte test_embedding_idx
# INSERT INTO test SELECT i, ARRAY[i] FROM generate_series(60,79) as i;
$ yb-admin -master_addresses 127.0.0.1:7100 flush_table ysql.yugabyte test_embedding_idx
# INSERT INTO test SELECT i, ARRAY[i] FROM generate_series(80,99) as i;
$ yb-admin -master_addresses 127.0.0.1:7100 flush_table ysql.yugabyte test_embedding_idx
```
6. Inspect the logs and make sure vector index background compactions are limited in parallelism and happen one after another.
7. Rerun steps 1-6 having vector_index_num_compactions_limit set to 2.
8. Inspect the logs and make sure at most two vector index background compactions happen in parallel.
9. Rerun steps 1-6 having vector_index_num_compactions_limit set to 0 to allow max parallelism.
10. Inspect the logs and make sure vector index background compactions happen in parallel.

Reviewers: sergei, rthallam

Reviewed By: sergei, rthallam

Subscribers: ybase

Differential Revision: https://phorge.dev.yugabyte.com/D47941
Assets 2
Loading