Loading and releasing collections

[davidlis]

Hello,

I need to perform searches in multiple collections at the same time, as well as add vectors to those collections.

For performance reasons, I believe it would be best to load() all collections in parallel (within same script) and then run the searches/additions.
Is this good practice? Is it possible to load an arbitrary number of collections in parallel and for months, or do I need to release collections to avoid issues/errors?

Thank you

[yhmo]

Typically, you only need to call load() one time for a collection, milvus will automatically load incremental data into memory when you call insert().
If you have several collections, the first script can be:

if __name__ == '__main__':
    c_1 = create_collection_1()
    c_1.load()

    c_2 = create_collection_2()
    c_2.load()

    c_3 = create_collection_3()
    c_3.load()

   ......

Then you can use different clients to call insert() parallelly. But make sure the milvus server has enough CPU memory capacity to load the incremental data into memory.

[drassokhin]

@yhmo @davidlis I have a related question: do the load and unload operations on a collection maintain any type of server-side counting? For example, after the following snippet is executed and assuming that the collection initially was not loaded

c_1 = create_collection_1()
c_1.load()
c_1.load()
c_1.unload()

is the collection going to be left in the 'loaded' or 'unloaded' state? Also, if multiple clients (running in parallel in the same process, or in different processes or even on different hosts) are accessing the same collection in the same instance of Milvus at the same time, and one client calls 'unload' while some other client calls 'load' and then tries to run a search, what is going to happen?

[yhmo]

No, milvus doesn't maintain load/release counter on the server side.
All the load()/release() operations will be processed in a task list, if the collection has been loaded, then the next load operation will do nothing.
So, if you use multiple clients to call load/release/search, it is like a "Schrödinger's cat" ---- sometimes the search() can return results, sometimes the search() returns error "the collection is not loaded".

[nileshsironja]

I've a similar question where I need to do search based on multiple collections..

If collection is loaded while creation as mentioned by @yhmo , what happens when the node gets restart/rebooted. In this case we might lose the loaded collection right?
And while starting the service again, we need to load all the collections?

[yhmo]

The load state of each collection is recorded in etcd.
If the server is restarted, it reads the load state from etcd. For "loaded" collections, the server automatically loads them into memory. "not-loaded" collections are ignored.

Note: it might take a long time to restart if the "loaded" collection is huge.