Google Cloud Storage Support

[JornEngelbart]

Please state your issue using the following template and, most importantly, in English.

Is your feature request related to a problem? Please describe.
We are running Milvus on GKE. We store on PV's, however, these run out of space from time to time. I saw there is support for S3, is there also a plan to support Google Cloud Storage as a persistent layer?

Describe the solution you'd like
Persistence support for GCS

Describe alternatives you've considered

Additional context

[suvashishtha]

@JornEngelbart - Are you running milvus in mishards mode?

[JornEngelbart]

@suvashishtha No, we are not running in mishards mode

[JinHai-CN]

Thank you for your suggestion, we are trying to provide a framework to support various of storage systems.

[bhushanbrb]

@JinHai-CN just checking to know if there are any updates on this? Is there any article for Milvus set up on GKE... thanks for any further information

[yhmo]

@bhushanbrb
Now we are focusing on developing a new architect milvus. Although we will continue maintain the 0.10.x versions, we could not make big/middle features on 0.10.x due to resource shortage. On the new milvus, the ability to store data in different storage systems will be designed as kv interface, so that milvus can use minio/S3, and even GCS. The new milvus is in developing, when it pass main function test, we will public its documents and source code.

[bhushanbrb]

@yhmo thanks for the reply, if at all, is there any roadmap or approximate timeline that you have on this

[yhmo]

@yhmo thanks for the reply, if at all, is there any roadmap or approximate timeline that you have on this

First stage we will change 0.10.x/master branch to 1.0 branch, and publish announcement for v1.0. this will happens in 1~2 weeks.
The second stage we will publish some articles to explain the reason why we re-design milvus. And then merge the new milvus source code to master branch, including its documentations.
Ideally, these things would be finished in the March.

[hadim]

I would also be interested in using GCS as a storage backend. Is the integration planned for 2.0? I guess if S3 is already implemented adding GCS should not be that complex right?

[yhmo]

The 2.0 has implement the minio storage which is compatible with S3: https://github.com/milvus-io/milvus/tree/master/internal/kv/minio

But GCS is not implemented yet, in 2.0 all storage interface is treated as KV service, so it is not a complex thing to implement GCS storage component. I just discuss with 2.0 team leader, the GCS storage priority is middle, currently there still some other high priority features not yet implemented, if you can help develop it would be appreciate.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[hadim]

Is GCS still a middle priority here?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.