Milvus collections/data lost - PVCs corrupted and emptied but S3 data exists

[RoshiniAshokkumarQB]

Hello,
We have had an issue with our Milvus cluster setup wherein our PVCs were destroyed which resulted in all our collections disappearing. We do however notice that the S3 data still exists in the bucket, we did not do an explicit backup however. Is there a way to dump this data from S3 back to the cluster then?
The milvus backup/restore utility mentions that it uses bulk insert here: https://github.com/zilliztech/milvus-backup?tab=readme-ov-file#restore - is there a way to use this to restore the collections on the cluster?

@vishwabasak @prateekkhera92

[yhmo]

Milvus depends on sime third-parties services:

• Pulsar/Kafka acts as a message queue to store WAL(write-ahead-log) for insert/delete requests, and time-tick messages for data consistency. All the inserted data will be stored in Pulsar/Kafka before consumed by datanode/querynode.
• MinIO/S3 is the major data storage of Milvus, all the original data and index data is stored here.
• Etcd stores the system metadata of Milvus, including schema of collections, runtime address of each component, S3 paths of data/index of each segment, etc.

When milvus starts, it first reads the metadata from Etcd, then knows where to find the data files from S3. The metadata of Etcd must be consistent with the data of S3, otherwise milvus will run into trouble.
If the data of Etcd is lost, Milvus will act as "all collections are disappeared".
Milvus cannot restore without the data of Etcd.

[prateekkhera92]

Thanks @yhmo - is pulsar data also required for the restoration?

[yhmo]

Pulsar/Kafka mainly stores write-ahead-log messages.
If you don't have intensive/continually insert actions, the messages will be quickly consumed by Milvus in a few seconds. Ideally, if all the messages have been consumed by Milvus, no need to restore it.

[prateekkhera92]

@yhmo - we had some old backups of etcd which we restored. However, we managed to load around 50% of collections only. We are seeing below error in query nodes. Does that mean, our etcd is outdated and is not in sync with S3? Any work around here?

│ milvus-querynode-6584658c99-j4p6k [2024/06/05 06:38:37.180 +00:00] [WARN] [segments/segment_loader.go:442] ["load remote segment bloom filter failed"] [traceID=a56a0202 │
│ fed15977b95ff00ff0707c22] [collectionID=445184838699371855] [segmentIDs="[445184838699260399]"] [partitionID=445184838700622941] [segmentID=445184838699260399] [error=" │
│ failed to read milvus/stats_log/445184838699371855/445184838700622941/445184838699260399/104/445184838699260934: NoSuchKey(key=milvus/stats_log/445184838699371855/44518 │
│ 4838700622941/445184838699260399/104/445184838699260934)"]

[yhmo]

The metadata in Etcd and data in S3 must be strongly consistent. It is easy to encounter this error if switching the etcd service or storage of a milvus.
Sometimes bugs of old version 2.2/2.3 also lead to this error.

[prateekkhera92]

@yhmo - any workaround? We are on 2.3.

[xiaofan-luan]

when sayting PVCs were destroyed, did etcd PVC destroyed as well?

[xiaofan-luan]

if you get the collection schema, then you can read out all the file and reimport, that's the easiest way to recover. but this requires some code