Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add comparable bytes to row decorator #1750

Merged
merged 8 commits into from
Apr 4, 2022
Merged

Add comparable bytes to row decorator #1750

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
caef986
Add in comparableBytes
EricBorczuk Mar 30, 2022
013012f
Add in a test
EricBorczuk Mar 30, 2022
53817e1
Fix dse-related formatting
EricBorczuk Mar 30, 2022
1012593
Fix compile issues in dse code
EricBorczuk Mar 30, 2022
3b689a2
This should be working now, I think - fix dse build
EricBorczuk Mar 30, 2022
801ea58
Change to ByteBuffer
EricBorczuk Mar 30, 2022
17b74dc
Test the comparable bytes coming out in sorted order
EricBorczuk Mar 31, 2022
8158933
Use FastByteOperations instead
EricBorczuk Apr 4, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions persistence-api/src/main/java/io/stargate/db/AbstractRowDecorator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,8 @@ protected AbstractRowDecorator(TableName table, List<String> partitionKeyColumnN

protected abstract ComparableKey<?> decoratePrimaryKey(Object... rawKeyValues);

@Override
public <T extends Comparable<T>> ComparableKey<T> decoratePartitionKey(Row row) {

protected Object[] primaryKeyValues(Row row) {
Object[] pkValues = new Object[partitionKeyColumnNames.size()];

int idx = 0;
for (String columnName : partitionKeyColumnNames) {
ByteBuffer value = row.getBytesUnsafe(columnName);
Expand All @@ -51,6 +48,13 @@ public <T extends Comparable<T>> ComparableKey<T> decoratePartitionKey(Row row)
pkValues[idx++] = value;
}

return pkValues;
}

@Override
public <T extends Comparable<T>> ComparableKey<T> decoratePartitionKey(Row row) {
Object[] pkValues = primaryKeyValues(row);

//noinspection unchecked
return (ComparableKey<T>) decoratePrimaryKey(pkValues);
}
Expand Down
3 changes: 3 additions & 0 deletions persistence-api/src/main/java/io/stargate/db/RowDecorator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@

import io.stargate.db.datastore.ResultSet;
import io.stargate.db.datastore.Row;
import java.nio.ByteBuffer;

/**
* A table-specific interface for extracting key column values from a {@link ResultSet} {@link Row}
Expand All @@ -29,4 +30,6 @@ public interface RowDecorator {
* same order that queries iterate / paginate over the Cassandra data ring.
*/
<T extends Comparable<T>> ComparableKey<T> decoratePartitionKey(Row row);

ByteBuffer getComparableBytes(Row row);
EricBorczuk marked this conversation as resolved.
Show resolved Hide resolved
}
8 changes: 8 additions & 0 deletions persistence-cassandra-3.11/src/main/java/io/stargate/db/cassandra/impl/RowDecoratorImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@

import io.stargate.db.AbstractRowDecorator;
import io.stargate.db.ComparableKey;
import io.stargate.db.datastore.Row;
import io.stargate.db.schema.TableName;
import java.nio.ByteBuffer;
import java.util.stream.Collectors;
Expand Down Expand Up @@ -49,4 +50,11 @@ protected ComparableKey<?> decoratePrimaryKey(Object... pkValues) {
DecoratedKey decoratedKey = metadata.partitioner.decorateKey(serializedKey);
return new ComparableKey<>(PartitionPosition.class, decoratedKey);
}

@Override
public ByteBuffer getComparableBytes(Row row) {
// TODO replace this with the relevant row's byte-comparable value when
// https://github.com/apache/cassandra/pull/1294 is ready
return ByteBuffer.wrap(new byte[] {});
}
}
9 changes: 9 additions & 0 deletions persistence-cassandra-4.0/src/main/java/io/stargate/db/cassandra/impl/RowDecoratorImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,9 @@

import io.stargate.db.AbstractRowDecorator;
import io.stargate.db.ComparableKey;
import io.stargate.db.datastore.Row;
import io.stargate.db.schema.TableName;
import java.nio.ByteBuffer;
import java.util.stream.Collectors;
import org.apache.cassandra.db.Clustering;
import org.apache.cassandra.db.DecoratedKey;
Expand Down Expand Up @@ -47,4 +49,11 @@ protected ComparableKey<?> decoratePrimaryKey(Object... pkValues) {
DecoratedKey decoratedKey = metadata.partitioner.decorateKey(key.serializeAsPartitionKey());
return new ComparableKey<>(PartitionPosition.class, decoratedKey);
}

@Override
public ByteBuffer getComparableBytes(Row row) {
// TODO replace this with the relevant row's byte-comparable value when
// https://github.com/apache/cassandra/pull/1294 is ready
return ByteBuffer.wrap(new byte[] {});
}
}
19 changes: 19 additions & 0 deletions persistence-dse-6.8/src/main/java/io/stargate/db/dse/impl/RowDecoratorImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,21 @@
*/
package io.stargate.db.dse.impl;

import static org.apache.cassandra.utils.ByteSource.END_OF_STREAM;

import io.stargate.db.AbstractRowDecorator;
import io.stargate.db.ComparableKey;
import io.stargate.db.datastore.Row;
import io.stargate.db.schema.TableName;
import java.io.ByteArrayOutputStream;
import java.nio.ByteBuffer;
import java.util.stream.Collectors;
import org.apache.cassandra.db.Clustering;
import org.apache.cassandra.db.DecoratedKey;
import org.apache.cassandra.db.PartitionPosition;
import org.apache.cassandra.schema.TableMetadata;
import org.apache.cassandra.utils.ByteComparable;
import org.apache.cassandra.utils.ByteSource;

public class RowDecoratorImpl extends AbstractRowDecorator {

Expand All @@ -43,4 +50,16 @@ protected ComparableKey<?> decoratePrimaryKey(Object... rawKeyValues) {
DecoratedKey decoratedKey = metadata.partitioner.decorateKey(key.serializeAsPartitionKey());
return new ComparableKey<>(PartitionPosition.class, decoratedKey);
}

@Override
public ByteBuffer getComparableBytes(Row row) {
Clustering key = metadata.partitionKeyAsClusteringComparator().make(primaryKeyValues(row));
DecoratedKey decoratedKey = metadata.partitioner.decorateKey(key.serializeAsPartitionKey());
EricBorczuk marked this conversation as resolved.
Show resolved Hide resolved
ByteSource src = decoratedKey.asComparableBytes(ByteComparable.Version.DSE68);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
for (int i = src.next(); i != END_OF_STREAM; i = src.next()) {
baos.write(i);
}
EricBorczuk marked this conversation as resolved.
Show resolved Hide resolved
return ByteBuffer.wrap(baos.toByteArray());
}
}
25 changes: 25 additions & 0 deletions persistence-test/src/main/java/io/stargate/it/PersistenceTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1964,6 +1964,31 @@ public void testRowDecorator() throws ExecutionException, InterruptedException {
assertGt(last, first, dec1, dec2);
}

@Test
public void testRowDecoratorComparableBytes() throws ExecutionException, InterruptedException {
setupCustomPagingData();

// Obtain partition keys in "ring" order
AbstractBound<?> selectAll =
dataStore
.queryBuilder()
.select()
.column("pk")
.column("val")
.from(keyspace, table)
.build()
.bind();

ResultSet rs1 = dataStore.execute(selectAll).get();
RowDecorator dec1 = rs1.makeRowDecorator();
ByteBuffer src = dec1.getComparableBytes(rs1.one());
if (backend.isDse()) {
assertThat(src.array().length).isGreaterThan(0);
} else {
assertThat(src.array().length).isEqualTo(0);
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not enough, we need the same test as in the testRowDecorator, just confirming that bytes are created is not enough, we need to confirm that comapring is done correctly.. should be one-to-one with the testRowDecorator and then test comparing using a unsigned bytes comparator .. And sure only active for DSE now..

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a better test for this - but in order to do so I had to have ByteComparable in scope and so had to add a DSE dependency to the persistence-test module...hope that's ok.

Maybe it can be reconfigured to only import that package


private boolean isCassandra4() {
return !backend.isDse()
&& Version.parse(backend.clusterVersion()).nextStable().compareTo(Version.V4_0_0) >= 0;
Expand Down
8 changes: 8 additions & 0 deletions restapi/src/test/java/io/stargate/db/datastore/AlphabeticalOrderPartitionKeyDecorator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
import io.stargate.db.ComparableKey;
import io.stargate.db.RowDecorator;
import io.stargate.db.schema.Column;
import java.nio.ByteBuffer;
import java.util.List;
import java.util.stream.Collectors;

Expand All @@ -43,4 +44,11 @@ public <T extends Comparable<T>> ComparableKey<T> decoratePartitionKey(Row row)
//noinspection unchecked
return (ComparableKey<T>) new ComparableKey<>(String.class, decorated);
}

@Override
public ByteBuffer getComparableBytes(Row row) {
// TODO replace this with the relevant row's byte-comparable value when
// https://github.com/apache/cassandra/pull/1294 is ready
return ByteBuffer.wrap(new byte[] {});
EricBorczuk marked this conversation as resolved.
Show resolved Hide resolved
}
}