[Help] What's the reliable way to evaluate milvus performance ?

[HackToday]

I am asking for help, since I find Milvus some performance related issue.

The configuration is as following:
1> use milvus 2.6.9 rpm (rpm install, use default configuration) to run as standalone. use numactl to make sure it is not effected by cross numa.
numactl -N 1 -m 1 /usr/bin/milvus run standalone

2> run test with vectordb-bench as following:
case_type: "Performance768D1M"
dataset_with_size_type: "Medium Cohere (768dim, 1M)"
db_label: "milvus_hnsw_test"
k: **
num_concurrency: "160"
concurrency_duration: 300
m: **
ef_construction: **
ef_search: **
num_shards: 1
replica_number: 1

What's the Results： I found that vectordbbench milvushnsw would have much difference results(with same vectordb-bench parameters) each time, for example, QPS varies significantly from 400 to 2000, it is not stable.

I checked the milvus official website, it seems not have any guide for the proper standalone performance evaluation.
It seems not like sysbench bench against MySQL way.(results would quite stable for same test parameters).

Question: Is the proper milvus behavior expected ? Or Could any milvus expert offer the right guide link for how to evaluate standalone instance?

Since if the QPS varies significantly, I could not know what's the reliable performance for the platform.
Does any milvus expert know what's the expected Milvus behavior？

[yhmo]

num_concurrency: "160" --------------- this value means the vectordbbench will create 160 sub-processes to do parallel query/search. If the vdbbench and mivlus standalone are running on the same machine, they will compete CPU resources.

Our bench results: vdbbench is deployed on machine A with at least 8 cores CPU, milvus standalone is deployed on machine B with sufficient cpu/memory resources. So there is no cpu resources competation

You can deploy them on two different machines and try again. It is recommended to use the latest version of vdbbench.

[HackToday]

hi @yhmo thanks for you reply.

Right now, the vdbench run with different NUMA(and memory node), CPU resource would not compete with milvus (another NUMA and its memory node)

I searched some from web, it seems it is suggested to run following: (each time, follow this way to evaluate)
1.clean all milvus data
2.run with milvus vdbbench one time with data loaded
3. and run following tests with -skip-drop-old --skip-load --skip-search-serial several times. ( I assume this is very important here ?)

It seems the result the more stable than before.
What do you think ?

[yhmo]

Data is inserted batch by batch, maybe 1000 rows per batch. Datanode will persist the data into s3 when accumulated data exceeds a threshold(100MB+ by default), or 10 minutes with no changes. So, the number of generated segments is a runtime behavior. 1000 rows per batch and 10K rows per batch might get different numbers of segments. One batch per minute and two batches per minute might get different numbers of segments.

Ideally, fewer segments will get better search performance. The reason is: for each search request, each segment is computed to output a topk result, and N segments get N topk results, then N topk results are merged into one topk final result. More segments will require more network transfer between nodes and more effort to merge results.

In the latest version of v2.6.x, we have compact(target_size: int) interface to force combine segments into a few segments(or into one segment). With the latest version of milvus 2.6.17 and latest version of vectordbbench, the 768D 1M data is merged into one segment so that you can get the best performance. In milvus 2.6.9, the compact(target_size: int) interface doesn't work properly.

[HackToday]

Thanks @yhmo
Let me try latest version and I will update it here when new data come.

[HackToday]

Hello @yhmo

I tried with latest milvus 2.6.18(latest) and vectorDBBench with 1.0.22(latest)

The Performance768D1M bench run still shows that

I am not sure what's the way to make it one segment ? It seems the latest version not work for that.

[yhmo]

Could you double-check your pymilvus version? (the latest version of pymilvus is 2.6.15)
By this command: pip show pymilvus
The vectorDBBench uses pymilvus to interact with milvus, if the version of pymilvus is too old, some parameters will be ignored without any error.

In the latest source code of vectorDBBench, it calls compact() interface, waits for the index done, and finally calls refresh_load() to reload the collection: https://github.com/zilliztech/VectorDBBench/blob/c6f96f7b9829b462d392db71cccbc55202ce3c27/vectordb_bench/backend/clients/milvus/milvus.py#L234
If your pymilvus is an old version, the parameter "target_size=MILVUS_FORCE_MERGE_TARGET_SIZE_MB" might be ignored.

With the latest version of pymilvus, you can also manually compact all segments into one with the following snippet:

client.optimize(collection_name=COLLECTION_NAME, target_size="10gb")

The "target_size" is the expected size of the compacted segment, the max value of target_size is internally limited by 1/8 of the memory capacity. For example, your machine has 32GB mem, the max target_size is internally limited to 4GB, even if you set target_size=8GB.

For 768dim 1M dataset, the data size is 3GB, so you need a machine with 24GB of memory to merge all segments into one segment.

[HackToday]

@yhmo Thanks for you reply. It seems the milvus docker image which is 2.6.18 can not compact correctly as expected:
I used this: https://milvus.io/docs/v2.6.x/install_standalone-docker.md （image tag is 2.6.18)

run with: (vectordbbench 1.0.22, pymilvus is 2.6.15)
vectordbbench milvushnsw
--config-file ${CONFIG_FILE}
--concurrency-duration 120

As it could find many errors like

 [error="segment is compacting: compaction plan conflict"] [errorVerbose="segment is compacting: compaction plan conflict\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/v2/util/merr.WrapErrCompactionPlanConflict\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:1262\n  | github.com/milvus-io/milvus/internal/datacoord.(*compactionInspector).createCompactTask\n  | \t/go/src/github.com/milvus-io/milvus/internal/datacoord/compaction_inspector.go:598\n  | github.com/milvus-io/milvus/internal/datacoord.(*compactionInspector).enqueueCompaction\n  | \t/go/src/github.com/milvus-io/milvus/internal/datacoord/compaction_inspector.go:553\n  | github.com/milvus-io/milvus/internal/datacoord.(*CompactionTriggerManager).SubmitSingleViewToScheduler\n  | \t/go/src/github.com/milvus-io/milvus/internal/datacoord/compaction_trigger_v2.go:514\n  | github.com/milvus-io/milvus/internal/datacoord.(*CompactionTriggerManager).notify\n  | \t/go/src/github.com/milvus-io/milvus/internal/datacoord/compaction_trigger_v2.go:340\n  | github.com/milvus-io/milvus/internal/datacoord.(*CompactionTriggerManager).loop\n  | \t/go/src/github.com/milvus-io/milvus/internal/datacoord/compaction_trigger_v2.go:238\n  | github.com/milvus-io/milvus/internal/datacoord.(*CompactionTriggerManager).Start.func1\n  | \t/go/src/github.com/milvus-io/milvus/internal/datacoord/compaction_trigger_v2.go:158\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1693\nWraps: (2) segment is compacting\nWraps: (3) compaction plan conflict\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]

And the second way you suggested "client.optimize(collection_name=COLLECTION_NAME, target_size="10gb") can work. I will bench with this way. Will keep update the final results for this way.