performance with large number of collections

[gut4]

For example I have tens of thousands collections with thousands docs each.
How will this impact on
– index performance?
– search performance?
– memory usage?

How I can benchmark this?

[kishorenc]

@gut4 As far as the underlying design goes, thousands of collections is not a problem as the in-memory overhead associated with each collection is minimal. All documents (across all collections) are stored in a single RocksDB store on disk, so disk performance is also the same as storing all documents in a single collection. We also only do O(1) per-collection lookups during indexing and searching so no impact there as well.

As for benchmarking, you can create a few thousand collections and index them with the kind of data you would be using in your actual production application. That would give you a sense of how the indexing performs. To test the search performance, you can run a benchmark using a tool like siege.