Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add comparable bytes to row decorator #1750

Merged
merged 8 commits into from
Apr 4, 2022
Merged

Add comparable bytes to row decorator #1750

merged 8 commits into from
Apr 4, 2022

Conversation

EricBorczuk
Copy link
Collaborator

What this PR does:

Which issue(s) this PR fixes:
Fixes #1716

Checklist

  • Changes manually tested
  • Automated Tests added/updated
  • Documentation added/updated
  • CLA Signed: DataStax CLA

@@ -29,4 +30,6 @@
* same order that queries iterate / paginate over the Cassandra data ring.
*/
<T extends Comparable<T>> ComparableKey<T> decoratePartitionKey(Row row);

Stream<Byte> getComparableBytes(Object... rawKeyValues);
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I decided to use Stream here because ByteSource (the class that comparable bytes returns) is best represented as an iterable of bytes with no known length, rather than a list or array - so the consumer of this method would be responsible for either consuming the whole stream and creating a list, or passing the stream along (maybe gRPC supports this?)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But isn't Stream<Byte> very inefficient? Byte is wrapper type and that seems ... like a very inefficient abstraction, and at least wrt performance worse than ByteSource.
Even plain old byte[] seems better unless I am missing something obvious (byte array can be wrapped as ByteBuffer etc)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah I should use ByteBuffer 🤦 idk what I was thinking

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that's a solid choice. Np! :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep good points @tatu-at-datastax

Copy link
Contributor

@ivansenic ivansenic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, but test needs to be better..

Comment on lines 1982 to 1990
ResultSet rs1 = dataStore.execute(selectAll).get();
RowDecorator dec1 = rs1.makeRowDecorator();
ByteBuffer src = dec1.getComparableBytes(rs1.one());
if (backend.isDse()) {
assertThat(src.array().length).isGreaterThan(0);
} else {
assertThat(src.array().length).isEqualTo(0);
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not enough, we need the same test as in the testRowDecorator, just confirming that bytes are created is not enough, we need to confirm that comapring is done correctly.. should be one-to-one with the testRowDecorator and then test comparing using a unsigned bytes comparator .. And sure only active for DSE now..

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a better test for this - but in order to do so I had to have ByteComparable in scope and so had to add a DSE dependency to the persistence-test module...hope that's ok.

Maybe it can be reconfigured to only import that package

Copy link
Contributor

@ivansenic ivansenic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests still not looking good, new dependency is definitely not a good choice.

@@ -8,6 +8,23 @@
</parent>
<groupId>io.stargate.db</groupId>
<artifactId>persistence-test</artifactId>
<properties>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is definitely not needed.. Will comment on the test more..

Comment on lines 1991 to 1993
.map(ByteComparable::fixedLength)
.map(x -> x.byteComparableAsString(ByteComparable.Version.DSE68))
.collect(Collectors.toList());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You shouldn't depend on these DSE related interfaces? What's the issue with comparing bytes in unsigned order directly?

I know there is UnsignedBytes in Guava, this should have something we could use.. Or use something like this (take from PureJavaOperations class):

    @Override
    public int compare(ByteBuffer buffer1, ByteBuffer buffer2) {
      int end1 = buffer1.limit();
      int end2 = buffer2.limit();
      for (int i = buffer1.position(), j = buffer2.position(); i < end1 && j < end2; i++, j++) {
        int a = (buffer1.get(i) & 0xff);
        int b = (buffer2.get(j) & 0xff);
        if (a != b) {
          return a - b;
        }
      }
      return buffer1.remaining() - buffer2.remaining();
    }

Comment on lines 1994 to 2013
Collections.sort(allComparableByteStrings);
for (int idx = 0; idx < allComparableByteStrings.size(); idx++) {
String cb1 = allComparableByteStrings.get(idx);
String cb2 =
ByteComparable.fixedLength(dec1.getComparableBytes(rows.get(idx)))
.byteComparableAsString(ByteComparable.Version.DSE68);
assertThat(cb1.compareTo(cb2)).isEqualTo(0);
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure what's going on here, why simply not using and adapting the code from the above method:

while (it1.hasNext()) {
      assertThat(it2.hasNext()).isTrue();
      Row r1 = it1.next();
      Row r2 = it2.next();
      first = first == null ? r1 : first;
      last = r1;
      assertEq(r1, r2, dec1, dec2);
      if (p1 == null) {
        p1 = r1;
      }
      assertGtEq(r1, p1, dec1, dec2);
    }
    assertThat(it2.hasNext()).isFalse();
    assertGt(last, first, dec1, dec2);
}    

Copy link
Contributor

@ivansenic ivansenic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup that's it..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Extend RowDecorator interface to return comparable bytes for a row
3 participants