Is indexing server going to support multi tenancy?

[oronsh]

I've noticed that tantivy holds a thread pool per IndexWriter so I suppose each indexing server will open NUMBER_OF_INDEXES * NUM_THREADS_PER_INDEX threads?
It sounds like a lot of threads to me, am I missing something?

Thanks a lot! :)

[fulmicoton]

Note those threads are most of the time parked have very little impact on performance.

If think we should not have trouble handling 1000s of tenants.

Right now we are addressing the multitenant case in a different manner.
A single index can host hundreds of tenants...

As a user, you would then group your tenants in pools that are fed in different sources.
Quickwit will then produce splits that are tagged with the tenants it contains.

When a query arrives, quickwit will only search within the splits containing the tenant.
We even have some hidden feature to improve the pruning power of this tag by taking 10
matures splits and producing 10 splits that tries to separate tenants as much as possible.