Out of memory error running bench3 with 500000 records

[meddulla]

Hi,

I was looking into tiedot and was running the benchmark with UID's with 500000 as benchsize (./tiedot --mode=bench3 --benchsize=500000). I get this error:

runtime: memory allocated by OS (0xb0514000) not in usable range [0x11407000,0x91407000)
fatal error: out of memory

Is this a bug or a known limitation due to the use of memory mapped files?

[HouzuoGuo]

Hello Meddulla.

Benchmark 1 and 3 are designed to find performance regression in individual document operation. To make benchmark result more accurate, all documents are prepared up-front before being inserted into collection.

Therefore, there are benchsize deserialized JSON structures in memory before the real benchmark begins.

Deserialized 500k JSON documents should use approx. 3 - 3.5 GB of memory. Please make sure your system has enough free memory before proceeding.

On another hand, Benchmark 2 (-mode=bench2) is designed to find concurrency performance regressions, and require very little available memory to run even with a very large benchsize.

If your system has at least 1.5 GB of free memory, it should be able to handle benchmark 2 with more than 2 million records easily.

Some more information can be found at (slightly outdated) page "Performance Tuning and Benchmark"(https://github.com/HouzuoGuo/tiedot/wiki/Performance-tuning-and-benchmarks),

In normal operation, tiedot requires little memory to run. Experiments have been conducted to find that tiedot HTTP service still runs well in under 90 MB of available system memory.

[HouzuoGuo]

Hey Meddulla. Has the situation improved?

[meddulla]

Hi

Sorry for the delay. Yes, running the benchmark2 with half a million as benchsize works out perfectly - the wiki page you pointed to helped clear up some confusion. We were looking into tiedot but ended up going a different route for our use case (it involved the concept of 'live queries', ie. queries that stayed as live entities emitting events like itemEnteredLiveQuery and itemLeft as items were updated). Although I did develop some more operators besides equal, like contains, not equal, greater than, etc. Would you be interested in a pull request containing these changes?

[HouzuoGuo]

Hey Meddulla.

Sounds good, please make a pull request.

And please let me know how I can make tiedot better for your use case... if there was a chance.

[intel352]

I can see how the additional operators would be handy, as otherwise you have to get fancier with union/intersect, etc.

Really, the same probably could have been achieved (to some extent) using the Regexp search, but I wonder what the performance impact of Regexp vs a standard operator comparison would be...

[meddulla]

Yes, i implemented them on par to the equal operator, all going to the V2Lookup function, instead of going to the more optimized functions like V2IntRange so they were not optimized at all, eg. greater than/less than did not use the possible indexes defined and start running through the collection from there. So I don't think it's a worthy trade-off - better off using the current api, with regex a whole lot can be done.

[HouzuoGuo]

All right.

By the way a proper range index is coming alone... let's wait for skip list.