-
Notifications
You must be signed in to change notification settings - Fork 223
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
tune for performance #80
Comments
...or vice-versa. I ended using bitmapContainer operations by default for most things in the runContainer binary operations, since they sped up the high cardinality test cases so much. For low cardinality however the non-bitmapContainer ops may be a win. Requires measurement. |
@lemire are there existing benchmarks (in the Java version?) that should be ported to Go; or do the Go benchmarks already suffice? I'd like to do at least one pass of perf tuning in order to catch glaring issues while the code is still fresh in mind. To that end, I'd like to have benchmarks that include the runContainer16 as a part of their testing. If nothing is available I can generate random ones easily, but I think it would be nice to compare speed directly against the Java or C Roaring implementations if possible. |
here is what I get from the java repo, if someone could explain the columns, this might be a good place to start....
|
@glycerine We have Go wrappers around C code, so that's a good reference point... that is, Go's roaring should aim to be at least as good as gocroaring... |
I have a shallow benchmark... |
The easiest reference point might be this C/C++ benchmark... https://github.com/RoaringBitmap/CBitmapCompetition The README.md file there goes into the details of what each and every column means. This is not Go code, of course... it is C and C++... but the idea is the same. |
great! thanks @lemire |
As for your specific question: bitspervalue: bits of storage per value stored on average... should be self-explanatory... should be better when using run compression for obvious reasons nanotimefor2by2and : take bitmaps two-by-two, compute intersection, report the time nanotimefor2by2or : same with union nanotimeforwideor : compute the "wide" (complete) union of all bitmaps nanotimeforcontains : we compute the universe size (all integers in [0,N)) then for each bitmap we check whether the values N/4, N/2 and 3N/4 are present in teach bitmap... this stresses random access... |
looking at the java test data, it is about 120MB; I'm going to make a separate repo for it so as to not duplicate too much in the Go implementation. |
Actually, yes, I think it is nice to have it as a secondary repository. |
isolated and duplicated here: https://github.com/RoaringBitmap/real-roaring-datasets |
The benchmark against the C version of Roaring has been extended considerably: see https://github.com/lemire/gobitmapbenchmark |
often times in the run container operations it appeared faster to convert to bitmaps and the do the operations there. Meaure and see if any of the remaining operations could benefit from this kind of tuning.
The text was updated successfully, but these errors were encountered: