Storing Structured Data in Milvus

[gem7008]

I plan to use Milvus as my vector database to store chat interaction between the user and my LLM so that I can give my LLM a "long-term memory". The thing is, I want to store some metadata (timestamp, username, original message before embedding, etc) along the chat message.

Here are the two options I thought of so far:

Option 1 - Use Milvus along with a structured Database

Use Milvus and another traditional structured database (like Postgresql) simultaneously. Store the chat message embedding (vectors) in Milvus with an "entity ID" to uniquely identify the chat message, and store all the metadata (timestamp, username, etc) in the structured database.

CONS: It's slower and less efficient because the app has to sequentially first query the search at Milvus and then select by ID in the structured database.

Option 2 - Only use Milvus

Only use Milvus and to store all the data, including metadata such as the message timestamp, username, original message, etc. Store all these information into fields.

CONS: Milvus is not designed to store structured data (to my knowledge) and the field are not intended for just storing data in a structured manner (I think). Furthermore, I believe I cannot easily query by an attribute like timestamp.

---

Are any of my statements wrong/incorrect? What option should I go with and why? Or perhaps am I missing a 3rd option?

[xiaofan-luan]

I plan to use Milvus as my vector database to store chat interaction between the user and my LLM so that I can give my LLM a "long-term memory". The thing is, I want to store some metadata (timestamp, username, original message before embedding, etc) along the chat message.

Here are the two options I thought of so far:

Option 1 - Use Milvus along with a structured Database

Use Milvus and another traditional structured database (like Postgresql) simultaneously. Store the chat message embedding (vectors) in Milvus with an "entity ID" to uniquely identify the chat message, and store all the metadata (timestamp, username, etc) in the structured database.

CONS: It's slower and less efficient because the app has to sequentially first query the search at Milvus and then select by ID in the structured database.

Option 2 - Only use Milvus

Only use Milvus and to store all the data, including metadata such as the message timestamp, username, original message, etc. Store all these information into fields.

CONS: Milvus is not designed to store structured data (to my knowledge) and the field are not intended for just storing data in a structured manner (I think). Furthermore, I believe I cannot easily query by an attribute like timestamp.

Are any of my statements wrong/incorrect? What option should I go with and why? Or perhaps am I missing a 3rd option?

Hi gem,
I think it's highly depend on how large is your structured data.
If you have short text, I would suggest store store it along with milvus because just as you said it would be much cheaper and easier to maintain.

On the other side, if you have relatively long document and many fields, store in traditional database, especially mongo db is highly recommended.

[jamesjohnson1025]

@gem7008 Hi, Just out of curiosity, which did approach did you decide to take. I have a similar usecase.

[gem7008]

Hi, @jamesjohnson1025. Based on @xiaofan-luan answer, I have decided to go with the 2 database approach, since I plan to store a lot of metadata. Another reason for using a traditional (structured) database along Milvus is because I am experienced and used to these conventional databases, thus lowering any unforeseen risks.

[KarthikRaju391]

I already have structured data stored in mongodb. I am planning to incorporate vector search in the application and to do so, I need to convert the structured data to vector embeddings.
Basically, I want to know how I can use MongoDB as the main database but use Milvus only for vector search. Is there any way I can sync the data between MongoDB and Milvus consistently?

[xiaofan-luan]

if mongodb has any cdc functionionality. you can subscribe mongodb log and convert into milvus insertions

[KarthikRaju391]

yeah, I found that mongodb has this change stream api at db level and collection level. So, I am thinking of using this to manage insertions into milvus when there are updates in mongodb.