what is dml channel ?

[wookiist]

I found that vchannel refers shard of a collection and pchannel is for a topic.
But what does dml channel means?

[yhmo]

pchannel is "physical channel"
vchannel is "virtual channel"

Basically, a pchannel is bound to a topic of Pulsa/Kafka.
A pchannel is like a network cable, a vchannel is like a TCP connection between two hosts.

dml is Data Manipulation Language for traditional database.
dml operation is insert, delete, or upsert.
In source code, a dml channel is equal to a vchannel.

[wookiist]

@yhmo Thank you for your great answers every time. You always seem to make things clear. 👍🏼
BTW, I have an additional question.
In version 2.2, I think there were two topics: by-dev-rootcoord-delta and by-dev-rootcoord-dml,
In 2.3, it seems that only by-dev-replicate-msg and by-dev-rootcoord-dml are left. Is it possible that something changed between versions?
Also, I'm curious about the by-dev-replicate-msg topic.

Thanks in advance.

[yhmo]

So far as I know, in 2.3, the by-dev-rootcoord-delta is removed, only by-dev-rootcoord-dml is left.
In 2.2, the delta channel is for delete() requests. In 2.3 both delete() and insert() requests are passed to dml channel.

The by-dev-replicate-msg is for this tool: https://github.com/zilliztech/milvus-cdc
Milvus-CDC is a change data capture tool for Milvus. It can capture the changes of upstream Milvus collections and sink them to downstream Milvus.
To support this tool, we created this feature: #22272
We added a replicate topic to copy the dml messages so that they can be consumed by the milvus-cdc tool: https://github.com/milvus-io/milvus/pull/27240/files
This topic doesn't affect milvus internal logic.

[tianshihan818]

@yhmo Hello! I'm new here. Could you introduce the relationships between the rootCoord dml channel and the collection shards? It seems the default value of the rootCoord.dmlChannelNum is 256 and the default value of the collections shards amount is 1. What if I want modify the values of these two settings after inserting a certain amount of data, how could I do that?

And for billion-level 768-dim data, I saw the sizing tool suggesting 1 datanode, some docs online said that the shard amount of the collection should be equal to the datanode replicas number. But if my use case is inserting data continuously over a period of time, I am not sure what is the better setting.

[xiaofan-luan]

You can not change this two value for now.

rootCoord.dmlChannelNum is set when cluster created. collection shard is fixed when collection created.

ideally, each 100m - 200m share 1 shards. If you have 1 billion data, I would recommend to use 4-8 shards.

each shards will assign to one datanode, so with 4-8 shards, 2-4 datanodes is recommend