-
Notifications
You must be signed in to change notification settings - Fork 157
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add comparable bytes to row decorator #1750
Conversation
@@ -29,4 +30,6 @@ | |||
* same order that queries iterate / paginate over the Cassandra data ring. | |||
*/ | |||
<T extends Comparable<T>> ComparableKey<T> decoratePartitionKey(Row row); | |||
|
|||
Stream<Byte> getComparableBytes(Object... rawKeyValues); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I decided to use Stream here because ByteSource
(the class that comparable bytes returns) is best represented as an iterable of bytes with no known length, rather than a list or array - so the consumer of this method would be responsible for either consuming the whole stream and creating a list, or passing the stream along (maybe gRPC supports this?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But isn't Stream<Byte>
very inefficient? Byte
is wrapper type and that seems ... like a very inefficient abstraction, and at least wrt performance worse than ByteSource
.
Even plain old byte[]
seems better unless I am missing something obvious (byte array can be wrapped as ByteBuffer
etc)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ah I should use ByteBuffer 🤦 idk what I was thinking
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah that's a solid choice. Np! :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep good points @tatu-at-datastax
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, but test needs to be better..
persistence-dse-6.8/src/main/java/io/stargate/db/dse/impl/RowDecoratorImpl.java
Outdated
Show resolved
Hide resolved
persistence-dse-6.8/src/main/java/io/stargate/db/dse/impl/RowDecoratorImpl.java
Outdated
Show resolved
Hide resolved
restapi/src/test/java/io/stargate/db/datastore/AlphabeticalOrderPartitionKeyDecorator.java
Outdated
Show resolved
Hide resolved
ResultSet rs1 = dataStore.execute(selectAll).get(); | ||
RowDecorator dec1 = rs1.makeRowDecorator(); | ||
ByteBuffer src = dec1.getComparableBytes(rs1.one()); | ||
if (backend.isDse()) { | ||
assertThat(src.array().length).isGreaterThan(0); | ||
} else { | ||
assertThat(src.array().length).isEqualTo(0); | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is not enough, we need the same test as in the testRowDecorator
, just confirming that bytes are created is not enough, we need to confirm that comapring is done correctly.. should be one-to-one with the testRowDecorator
and then test comparing using a unsigned bytes comparator .. And sure only active for DSE now..
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added a better test for this - but in order to do so I had to have ByteComparable
in scope and so had to add a DSE dependency to the persistence-test module...hope that's ok.
Maybe it can be reconfigured to only import that package
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tests still not looking good, new dependency is definitely not a good choice.
persistence-test/pom.xml
Outdated
@@ -8,6 +8,23 @@ | |||
</parent> | |||
<groupId>io.stargate.db</groupId> | |||
<artifactId>persistence-test</artifactId> | |||
<properties> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is definitely not needed.. Will comment on the test more..
.map(ByteComparable::fixedLength) | ||
.map(x -> x.byteComparableAsString(ByteComparable.Version.DSE68)) | ||
.collect(Collectors.toList()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You shouldn't depend on these DSE related interfaces? What's the issue with comparing bytes in unsigned order directly?
I know there is UnsignedBytes
in Guava, this should have something we could use.. Or use something like this (take from PureJavaOperations
class):
@Override
public int compare(ByteBuffer buffer1, ByteBuffer buffer2) {
int end1 = buffer1.limit();
int end2 = buffer2.limit();
for (int i = buffer1.position(), j = buffer2.position(); i < end1 && j < end2; i++, j++) {
int a = (buffer1.get(i) & 0xff);
int b = (buffer2.get(j) & 0xff);
if (a != b) {
return a - b;
}
}
return buffer1.remaining() - buffer2.remaining();
}
Collections.sort(allComparableByteStrings); | ||
for (int idx = 0; idx < allComparableByteStrings.size(); idx++) { | ||
String cb1 = allComparableByteStrings.get(idx); | ||
String cb2 = | ||
ByteComparable.fixedLength(dec1.getComparableBytes(rows.get(idx))) | ||
.byteComparableAsString(ByteComparable.Version.DSE68); | ||
assertThat(cb1.compareTo(cb2)).isEqualTo(0); | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure what's going on here, why simply not using and adapting the code from the above method:
while (it1.hasNext()) {
assertThat(it2.hasNext()).isTrue();
Row r1 = it1.next();
Row r2 = it2.next();
first = first == null ? r1 : first;
last = r1;
assertEq(r1, r2, dec1, dec2);
if (p1 == null) {
p1 = r1;
}
assertGtEq(r1, p1, dec1, dec2);
}
assertThat(it2.hasNext()).isFalse();
assertGt(last, first, dec1, dec2);
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yup that's it..
What this PR does:
Which issue(s) this PR fixes:
Fixes #1716
Checklist