[Enhancement]: Support Log Service in Milvus

[chyezh]

Is there an existing issue for this?

• I have searched the existing issues

What would you like to be added?

Milvus is now using proxy (maybe multi node) to interactive with message queue producer.
There's no any roles in Milvus cluster to achieve �consistency consensus at write path.
All things is done by outside message queue.
So any optimization feature at write path is a distributed problem, and hard to apply on current architecture.

We can introduce a log-based role to solve those problem. There's two new role in our new milvus architecture.

• LogCoord, like other coord node such as querycoord, it hold the global assignment relation of underlying lognode and pchannel.
• LogNode, a role that to do producing and consuming, any pchannel can be and only be assign to unique lognode at same time. All write and read operation need to done by it.

RoadMap or goals:

• new message interface, provide fast filtering by using properties of message. Avoid too much unserialized operation applied to milvus lognode.
  • Properties for filtering and sorted: timetick, vchannel name, message type.
• timestamp assignment is moved into lognode.
  • Continuous timetick message (no message between two time tick message) can be optimized as no-persistent message.
  • Timestamp sync up can be more shorter (200ms by default now), we can optimize it into 10ms level without two much overhead.
• VChannel level consuming and producing api.
  • Consuming operation support filtering at lognode server side.
• Remove mqwrapper，we need a wal but not a message queue in new architecture. we didn't rely on message queue's consuming api, so there's no stateful message queue side consumer on new architecture. Like pulsar, we use reader api but not consumer api.
  • Pulsar
  • Rocksmq
  • Natsmq
  • Kafka
• Write ahead cache at lognode server side. We can no longer always consuming every message from underlying message queue, many message can be consumed from local storage of lognode by cached writing log entry.

Why is this needed?

No response

Anything else?

No response

[xiaofan-luan]

what about name it as streaming service?