In place comparison of serializable values.

[andrii0lomakin]

Now when we compare keys in indexes we do following:

1. deserialize key.
2. compare it with other keys in bucket.

String keys can be relatively big but to compare them we often need very small piece of them. So following is proposed:

1. Serialize key passed in to compare.
2. Provide compare method for serializers so we will compare serialized binary key and one stored in direct memory it will save as sensible amount of operations.

[lvca]

This idea is VERY VERY cool!

[andrii0lomakin]

Forgot to mention if we read bytes one by one to compare from direct memory it will not be fast but we can use memcmp which is standard C library which exist on all platforms like tens years already, this approach will provide good speed. The main bottleneck now for indexes is serialization speed. It should be as fast as possible.

[lvca]

Is memcmp available as JNA API?

[andrii0lomakin]

It is one line of code.

On Sat, Nov 16, 2013 at 9:49 AM, Luca Garulli notifications@github.comwrote:

Is memcmp available as JNA API?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1826#issuecomment-28621758
.

Best regards,
Andrey Lomakin.

Orient Technologies
the Company behind OrientDB

[andrii0lomakin]

Actually 4 lines.

public static native int memcmp(long str1, long str2, long len);

static {
Native.register(Platform.C_LIBRARY_NAME);
}

[lvca]

Cool

[andrii0lomakin]

Sorry guys no luck here call overhead eats everything. Actually I have doubts when created this issue too (spent several days thinking whether to create this issue or not ))) ) May be will come up with more nice idea about in place comparison later. Good that I did benchmark on prototype.

[lvca]

Si it's cheaper to call the Java comparison than C ? What's the cost? However you could avoid serialization and do the same using Java, couldn't it?.

[andrii0lomakin]

It is very slow if you compare data byte by byte it is 1/4 slower, if you do in place comparison we need to copy array do native call and as result we have the same performance.

[andrii0lomakin]

Actually you are right, binary comparison in Java may be faster. I will reopen it and try after #856 . It may be faster. Need to try it.

[andrii0lomakin]

It matters of 3 hours to test it. not a lot of time.

[lvca]

Good!

[lvca]

We supported this in new binary serialization. Where was you referring in the code for this issue?

[andrii0lomakin]

do not think that we refer to the same approach, do you mean that comparison string lets say "adf" in binary form with "sdf" will have return the same comparison result , as if they are compared in lexicographical order, do not you ?

[lvca]

When you need to unmarshall a field, it's unmarshalled the field name and passed to the compare, but it does byte-per-byte comparison avoiding to unmarshall the entire key before the comparison.

[andrii0lomakin]

It is related to indexes, but any way, it can be reused.
Does your answer mean that any serialized primitive value can be compared
byte by byte and we can use this comparison result to sort serialized
values in lexicographical order ?

On Fri, Aug 29, 2014 at 3:15 PM, Luca Garulli notifications@github.com
wrote:

When you need to unmarshall a field, it's unmarshalled the field name and
passed to the compare, but it does byte-per-byte comparison avoiding to
unmarshall the entire key before the comparison.

—
Reply to this email directly or view it on GitHub
#1826 (comment)
.

Best regards,
Andrey Lomakin.

Orient Technologies
the Company behind OrientDB

[lvca]

@tglman started this, he's better to answer.

[tglman]

All dipends on the way data is serialized, so depends also on the types!
in the binary serialization for example Strings are serialized as UTF-8, and as far as i know the UTF-8 encoding support the byte comparison, i think also the Numeric value written as VarInt can be compared in a byte to byte way, we should check that for all the types, for implement this!

If you know already some issue for this let's take note :)

[andrii0lomakin]

Optional for 3.0

[andrii0lomakin]

Not actual any more