enhance: Add a config item for partition name as regexp feature and disable it by default

[congqixia]

See also #29177
Add a config item for partition name as regexp feature and disable it by default

[mergify]

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[congqixia]

/run-cpu-e2e

[mergify]

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[congqixia]

/run-cpu-e2e

[mergify]

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[congqixia]

/run-cpu-e2e

[codecov]

Codecov Report

Merging #29154 (c969891) into master (cb75e73) will decrease coverage by 0.06%.
Report is 10 commits behind head on master.
The diff coverage is 91.66%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #29154      +/-   ##
==========================================
- Coverage   82.02%   81.96%   -0.06%     
==========================================
  Files         862      861       -1     
  Lines      121926   122118     +192     
==========================================
+ Hits       100004   100091      +87     
- Misses      18600    18683      +83     
- Partials     3322     3344      +22     

Files	Coverage Δ
pkg/util/paramtable/component_param.go	98.18% <100.00%> (+<0.01%)	⬆️
internal/proxy/task_search.go	79.51% <89.28%> (+0.28%)	⬆️

... and 28 files with indirect coverage changes

[mergify]

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify]

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: congqixia

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• internal/proxy/OWNERS [congqixia]
• pkg/util/OWNERS [congqixia]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mergify]

@congqixia Please associate the related issue to the body of your Pull Request. (eg. “issue: #”)

[yah01]

/lgtm

[NiftyliuS]

In the release notes you have:
To reduce resource consumption, regular expression searches in partitions have been discontinued. However, this feature can be re-enabled through configuration (see #29154 for details).

Can you please add an explanation of how to re-enable this feature? I cant update to 2.3.4 because this breaks a filter i have on a partition key field...

PS: this disables automatic partitions expr

[xiaofan-luan]

In the release notes you have: To reduce resource consumption, regular expression searches in partitions have been discontinued. However, this feature can be re-enabled through configuration (see #29154 for details).

Can you please add an explanation of how to re-enable this feature? I cant update to 2.3.4 because this breaks a filter i have on a partition key field...

PS: this disables automatic partitions expr

you need to set proxy.partitionNameRegexp to true

[xiaofan-luan]

In fact we don't recommend to enable this features since this might a serious performnace problem

[NiftyliuS]

"you need to set proxy.partitionNameRegexp to true" - you mean in the mivlus.yaml? is there an env param to override this?

Yes i get it that its not recommended,
I did read the notes on performance, the problem is that currently i have milvus setup using automatic partition feature.
Disabling thing means i cant filter by document_id at all, for any value.

And since i initially setup the partitions automatically using is_partition_key it means i have to redo the code and migrate the collection to a new one entirely...

For me this is a breaking change that came in a patch version...

Speaking of - what is the correct way to address document_id in my case? i intend to have 60.000 + books vectorized and i do want to be able to search all of them at once ( standard search ) as well as a search in a specific book only.
I cant create partition per book, the limit is 10k partitions right?

Should i just set 128 partitions and hash the document_id to one of them and use that for search?
help is much appreciated

[xiaofan-luan]

"you need to set proxy.partitionNameRegexp to true" - you mean in the mivlus.yaml? is there an env param to override this?

Yes i get it that its not recommended, I did read the notes on performance, the problem is that currently i have milvus setup using automatic partition feature. Disabling thing means i cant filter by document_id at all, for any value.

And since i initially setup the partitions automatically using is_partition_key it means i have to redo the code and migrate the collection to a new one entirely...

For me this is a breaking change that came in a patch version...

Speaking of - what is the correct way to address document_id in my case? i intend to have 60.000 + books vectorized and i do want to be able to search all of them at once ( standard search ) as well as a search in a specific book only. I cant create partition per book, the limit is 10k partitions right?

Should i just set 128 partitions and hash the document_id to one of them and use that for search? help is much appreciated

you have to config this in milvus.yaml

[xiaofan-luan]

can if you use partition keys, then each time you should just search with one partition key right?
you can directly specify which partition key you are gonna to use and there is no need to use a regular expression

[NiftyliuS]

In the auto partition key mode the new flag disables searching by document_id entirely.
It creates 64 partitions named _default0-63

when not specifying partitions document_id is ignored and i get 0 results.
when specifying partitions i get an error manually setting partitions is not allowed when using partition mode
(i will update with the exact message once i get to the laptop)

I think you are right, setting up a new collection with manual partitions might be the way to go...