-
Notifications
You must be signed in to change notification settings - Fork 68
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Initial Qdrant driver implementation #1942
Merged
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
6545156
minor updates to milvus driver files
msmygit 053dbec
Initial draft skeleton for Qdrant driver adapter
msmygit d05d663
Delete Collection request
msmygit c81968b
Improve ParsedOp error message
msmygit 34177a0
Fixed a trivial typo in logging
msmygit 696081c
Qdrant run configs
msmygit 73c09b9
Create collection fully working
msmygit e9b652a
Create collection clean-ups
msmygit c74a2ac
Jacoco maven plugin update
msmygit 7aacbb0
misc async->sync operations
msmygit 05a5322
Initial version of count points impl
msmygit c64fd45
[enhancement] - to honor .yaml and .txt file clean-up with --log-max …
msmygit bbdd641
upsert points operation enhancements
msmygit 309fd6f
working code
msmygit e253412
cleanup dead code
msmygit File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
<component name="ProjectRunConfigurationManager"> | ||
<configuration default="false" name="nb --list-drivers" type="JarApplication" folderName="Common"> | ||
<extension name="software.aws.toolkits.jetbrains.core.execution.JavaAwsConnectionExtension"> | ||
<option name="credential" /> | ||
<option name="region" /> | ||
<option name="useCurrentConnection" value="false" /> | ||
</extension> | ||
<option name="JAR_PATH" value="$PROJECT_DIR$/nb5/target/nb5-5.21.1-SNAPSHOT-jar-with-dependencies.jar" /> | ||
<option name="PROGRAM_PARAMETERS" value="--list-drivers --show-stacktraces" /> | ||
<option name="WORKING_DIRECTORY" value="$PROJECT_DIR$/local/" /> | ||
<option name="ALTERNATIVE_JRE_PATH" /> | ||
<method v="2" /> | ||
</configuration> | ||
</component> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
<component name="ProjectRunConfigurationManager"> | ||
<configuration default="false" name="nb --list-scenarios" type="JarApplication" folderName="Common"> | ||
<extension name="software.aws.toolkits.jetbrains.core.execution.JavaAwsConnectionExtension"> | ||
<option name="credential" /> | ||
<option name="region" /> | ||
<option name="useCurrentConnection" value="false" /> | ||
</extension> | ||
<option name="JAR_PATH" value="$PROJECT_DIR$/nb5/target/nb5-5.21.1-SNAPSHOT-jar-with-dependencies.jar" /> | ||
<option name="PROGRAM_PARAMETERS" value="--list-scenarios --show-stacktraces" /> | ||
<option name="WORKING_DIRECTORY" value="$PROJECT_DIR$/local/" /> | ||
<option name="ALTERNATIVE_JRE_PATH" /> | ||
<method v="2" /> | ||
</configuration> | ||
</component> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
<component name="ProjectRunConfigurationManager"> | ||
<configuration default="false" name="qdrant_delete_collection_glove_25" type="JarApplication" folderName="Qdrant"> | ||
<extension name="software.aws.toolkits.jetbrains.core.execution.JavaAwsConnectionExtension"> | ||
<option name="credential" /> | ||
<option name="region" /> | ||
<option name="useCurrentConnection" value="false" /> | ||
</extension> | ||
<option name="JAR_PATH" value="$PROJECT_DIR$/nb5/target/nb5.jar" /> | ||
<option name="PROGRAM_PARAMETERS" value="qdrant_vectors_live qdrant_vectors.delete_collection dimensions=25 testsize=10000 trainsize=1183514 dataset=glove-25-angular filetype=hdf5 collection=glove_25 similarity_function=1 qdranthost=ded78a51-8370-47d8-adb0-6147f0fcbba2.us-east4-0.gcp.cloud.qdrant.io token_file=./apikey grpc_port=6334 --progress console:1s -v --add-labels "dimensions:25,dataset=glove-25" --add-labels="target:qdrant,instance:vectors,vendor:qdrant_v191" --report-prompush-to https://vector-perf.feat.apps.paas.datastax.com:8427/api/v1/import/prometheus/metrics/job/nosqlbench/instance/vectors --annotators "[{'type':'log','level':'info'},{'type':'grafana','baseurl':'https://vector-perf.feat.apps.paas.datastax.com/'}]" --report-interval 10 --show-stacktraces --logs-max 5" /> | ||
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$/local/qdrant" /> | ||
<option name="ALTERNATIVE_JRE_PATH" value="jdk21" /> | ||
<method v="2" /> | ||
</configuration> | ||
</component> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
<component name="ProjectRunConfigurationManager"> | ||
<configuration default="false" name="qdrant_schema_collection_glove_25" type="JarApplication" folderName="Qdrant"> | ||
<extension name="software.aws.toolkits.jetbrains.core.execution.JavaAwsConnectionExtension"> | ||
<option name="credential" /> | ||
<option name="region" /> | ||
<option name="useCurrentConnection" value="false" /> | ||
</extension> | ||
<option name="JAR_PATH" value="$PROJECT_DIR$/nb5/target/nb5.jar" /> | ||
<option name="PROGRAM_PARAMETERS" value="qdrant_vectors_live qdrant_vectors.schema_collection dimensions=25 testsize=10000 trainsize=1183514 dataset=glove-25-angular filetype=hdf5 collection=glove_25 similarity_function=1 qdranthost=ded78a51-8370-47d8-adb0-6147f0fcbba2.us-east4-0.gcp.cloud.qdrant.io token_file=./apikey grpc_port=6334 --progress console:1s -v --add-labels "dimensions:25,dataset=glove-25" --add-labels="target:qdrant,instance:vectors,vendor:qdrant_v191" --report-prompush-to https://vector-perf.feat.apps.paas.datastax.com:8427/api/v1/import/prometheus/metrics/job/nosqlbench/instance/vectors --annotators "[{'type':'log','level':'info'},{'type':'grafana','baseurl':'https://vector-perf.feat.apps.paas.datastax.com/'}]" --report-interval 10 --show-stacktraces --logs-max 5" /> | ||
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$/local/qdrant" /> | ||
<option name="ALTERNATIVE_JRE_PATH" value="jdk21" /> | ||
<method v="2" /> | ||
</configuration> | ||
</component> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
<component name="ProjectRunConfigurationManager"> | ||
<configuration default="false" name="qdrant_search_points_glove_25" type="JarApplication" folderName="Qdrant"> | ||
<extension name="software.aws.toolkits.jetbrains.core.execution.JavaAwsConnectionExtension"> | ||
<option name="credential" /> | ||
<option name="region" /> | ||
<option name="useCurrentConnection" value="false" /> | ||
</extension> | ||
<option name="JAR_PATH" value="$PROJECT_DIR$/nb5/target/nb5.jar" /> | ||
<option name="PROGRAM_PARAMETERS" value="qdrant_vectors_live qdrant_vectors.search_points dimensions=25 testsize=10000 trainsize=1183514 dataset=glove-25-angular filetype=hdf5 collection=glove_25 similarity_function=1 qdranthost=ded78a51-8370-47d8-adb0-6147f0fcbba2.us-east4-0.gcp.cloud.qdrant.io token_file=./apikey grpc_port=6334 --progress console:1s -v --add-labels "dimensions:25,dataset=glove-25" --add-labels="target:qdrant,instance:vectors,vendor:qdrant_v191" --report-prompush-to https://vector-perf.feat.apps.paas.datastax.com:8427/api/v1/import/prometheus/metrics/job/nosqlbench/instance/vectors --annotators "[{'type':'log','level':'info'},{'type':'grafana','baseurl':'https://vector-perf.feat.apps.paas.datastax.com/'}]" --report-interval 10 --report-csv-to metrics --show-stacktraces --logs-max 5" /> | ||
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$/local/qdrant" /> | ||
<option name="ALTERNATIVE_JRE_PATH" value="jdk21" /> | ||
<method v="2" /> | ||
</configuration> | ||
</component> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
<component name="ProjectRunConfigurationManager"> | ||
<configuration default="false" name="qdrant_upsert_points_glove_25" type="JarApplication" folderName="Qdrant"> | ||
<extension name="software.aws.toolkits.jetbrains.core.execution.JavaAwsConnectionExtension"> | ||
<option name="credential" /> | ||
<option name="region" /> | ||
<option name="useCurrentConnection" value="false" /> | ||
</extension> | ||
<option name="JAR_PATH" value="$PROJECT_DIR$/nb5/target/nb5.jar" /> | ||
<option name="PROGRAM_PARAMETERS" value="qdrant_vectors_live qdrant_vectors.rampup dimensions=25 testsize=10000 trainsize=1183514 dataset=glove-25-angular filetype=hdf5 collection=glove_25 similarity_function=1 qdranthost=ded78a51-8370-47d8-adb0-6147f0fcbba2.us-east4-0.gcp.cloud.qdrant.io token_file=./apikey grpc_port=6334 --progress console:1s -v --add-labels "dimensions:25,dataset=glove-25" --add-labels="target:qdrant,instance:vectors,vendor:qdrant_v191" --report-prompush-to https://vector-perf.feat.apps.paas.datastax.com:8427/api/v1/import/prometheus/metrics/job/nosqlbench/instance/vectors --annotators "[{'type':'log','level':'info'},{'type':'grafana','baseurl':'https://vector-perf.feat.apps.paas.datastax.com/'}]" --report-interval 10 --show-stacktraces --logs-max 5" /> | ||
<option name="WORKING_DIRECTORY" value="$ProjectFileDir$/local/qdrant" /> | ||
<option name="ALTERNATIVE_JRE_PATH" value="jdk21" /> | ||
<method v="2" /> | ||
</configuration> | ||
</component> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,62 @@ | ||
<?xml version="1.0" encoding="UTF-8"?> | ||
<project xmlns="http://maven.apache.org/POM/4.0.0" | ||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | ||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> | ||
<modelVersion>4.0.0</modelVersion> | ||
|
||
<artifactId>adapter-qdrant</artifactId> | ||
<packaging>jar</packaging> | ||
|
||
<parent> | ||
<artifactId>mvn-defaults</artifactId> | ||
<groupId>io.nosqlbench</groupId> | ||
<version>${revision}</version> | ||
<relativePath>../../mvn-defaults</relativePath> | ||
</parent> | ||
|
||
<name>${project.artifactId}</name> | ||
<description> | ||
An nosqlbench adapter driver module for the Qdrant database. | ||
</description> | ||
|
||
<dependencies> | ||
<dependency> | ||
<groupId>io.nosqlbench</groupId> | ||
<artifactId>nb-annotations</artifactId> | ||
<version>${revision}</version> | ||
<scope>compile</scope> | ||
</dependency> | ||
<dependency> | ||
<groupId>io.nosqlbench</groupId> | ||
<artifactId>adapters-api</artifactId> | ||
<version>${revision}</version> | ||
<scope>compile</scope> | ||
</dependency> | ||
<dependency> | ||
<groupId>io.grpc</groupId> | ||
<artifactId>grpc-protobuf</artifactId> | ||
<!-- <version>1.63.0</version> --> | ||
<!-- Trying to match https://github.com/qdrant/java-client/blob/v1.9.0/build.gradle#L80 --> | ||
<version>1.59.0</version> | ||
</dependency> | ||
<dependency> | ||
<groupId>com.google.protobuf</groupId> | ||
<artifactId>protobuf-java-util</artifactId> | ||
<!--<version>3.25.3</version>--> | ||
<!-- Trying to match https://github.com/qdrant/java-client/blob/master/build.gradle#L81 --> | ||
<version>3.24.0</version> | ||
</dependency> | ||
<dependency> | ||
<groupId>com.google.guava</groupId> | ||
msmygit marked this conversation as resolved.
Show resolved
Hide resolved
|
||
<artifactId>guava</artifactId> | ||
<!--<version>33.1.0-jre</version>--> | ||
<!-- Trying to match https://github.com/qdrant/java-client/blob/master/build.gradle#L93 --> | ||
<version>30.1-jre</version> | ||
</dependency> | ||
<dependency> | ||
<groupId>io.qdrant</groupId> | ||
<artifactId>client</artifactId> | ||
<version>1.9.0</version> | ||
</dependency> | ||
</dependencies> | ||
</project> |
67 changes: 67 additions & 0 deletions
67
...dapters/adapter-qdrant/src/main/java/io/nosqlbench/adapter/qdrant/QdrantAdapterUtils.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,67 @@ | ||
/* | ||
* Copyright (c) 2020-2024 nosqlbench | ||
* | ||
* Licensed under the Apache License, Version 2.0 (the "License"); | ||
* you may not use this file except in compliance with the License. | ||
* You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
package io.nosqlbench.adapter.qdrant; | ||
|
||
import io.qdrant.client.grpc.Points.ScoredPoint; | ||
import org.apache.commons.lang3.StringUtils; | ||
|
||
import java.util.Arrays; | ||
import java.util.List; | ||
|
||
public class QdrantAdapterUtils { | ||
|
||
public static final String QDRANT = "qdrant"; | ||
|
||
public static List<String> splitNames(String input) { | ||
assert StringUtils.isNotBlank(input) && StringUtils.isNotEmpty(input); | ||
return Arrays.stream(input.split("( +| *, *)")) | ||
.filter(StringUtils::isNotBlank) | ||
.toList(); | ||
} | ||
|
||
public static List<Long> splitLongs(String input) { | ||
assert StringUtils.isNotBlank(input) && StringUtils.isNotEmpty(input); | ||
return Arrays.stream(input.split("( +| *, *)")) | ||
.filter(StringUtils::isNotBlank) | ||
.map(Long::parseLong) | ||
.toList(); | ||
} | ||
|
||
/** | ||
* Mask the digits in the given string with '*' | ||
* | ||
* @param unmasked The string to mask | ||
* @return The masked string | ||
*/ | ||
protected static String maskDigits(String unmasked) { | ||
assert StringUtils.isNotBlank(unmasked) && StringUtils.isNotEmpty(unmasked); | ||
int inputLength = unmasked.length(); | ||
StringBuilder masked = new StringBuilder(inputLength); | ||
for (char ch : unmasked.toCharArray()) { | ||
if (Character.isDigit(ch)) { | ||
masked.append("*"); | ||
} else { | ||
masked.append(ch); | ||
} | ||
} | ||
return masked.toString(); | ||
} | ||
|
||
public static int[] searchPointsResponseIdNumToIntArray(List<ScoredPoint> response) { | ||
return response.stream().mapToInt(r -> ((Number) r.getId().getNum()).intValue()).toArray(); | ||
} | ||
} |
54 changes: 54 additions & 0 deletions
54
...apters/adapter-qdrant/src/main/java/io/nosqlbench/adapter/qdrant/QdrantDriverAdapter.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,54 @@ | ||
/* | ||
* Copyright (c) 2020-2024 nosqlbench | ||
* | ||
* Licensed under the Apache License, Version 2.0 (the "License"); | ||
* you may not use this file except in compliance with the License. | ||
* You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
package io.nosqlbench.adapter.qdrant; | ||
|
||
import io.nosqlbench.adapter.qdrant.ops.QdrantBaseOp; | ||
import io.nosqlbench.adapters.api.activityimpl.OpMapper; | ||
import io.nosqlbench.adapters.api.activityimpl.uniform.BaseDriverAdapter; | ||
import io.nosqlbench.adapters.api.activityimpl.uniform.DriverAdapter; | ||
import io.nosqlbench.nb.annotations.Service; | ||
import io.nosqlbench.nb.api.components.core.NBComponent; | ||
import io.nosqlbench.nb.api.config.standard.NBConfigModel; | ||
import io.nosqlbench.nb.api.config.standard.NBConfiguration; | ||
import io.nosqlbench.nb.api.labels.NBLabels; | ||
|
||
import java.util.function.Function; | ||
|
||
import static io.nosqlbench.adapter.qdrant.QdrantAdapterUtils.QDRANT; | ||
|
||
@Service(value = DriverAdapter.class, selector = QDRANT) | ||
public class QdrantDriverAdapter extends BaseDriverAdapter<QdrantBaseOp<?>, QdrantSpace> { | ||
|
||
public QdrantDriverAdapter(NBComponent parentComponent, NBLabels labels) { | ||
super(parentComponent, labels); | ||
} | ||
|
||
@Override | ||
public OpMapper<QdrantBaseOp<?>> getOpMapper() { | ||
return new QdrantOpMapper(this); | ||
} | ||
|
||
@Override | ||
public Function<String, ? extends QdrantSpace> getSpaceInitializer(NBConfiguration cfg) { | ||
return (s) -> new QdrantSpace(s, cfg); | ||
} | ||
|
||
@Override | ||
public NBConfigModel getConfigModel() { | ||
return super.getConfigModel().add(QdrantSpace.getConfigModel()); | ||
} | ||
} |
32 changes: 32 additions & 0 deletions
32
.../adapter-qdrant/src/main/java/io/nosqlbench/adapter/qdrant/QdrantDriverAdapterLoader.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
/* | ||
* Copyright (c) 2020-2024 nosqlbench | ||
* | ||
* Licensed under the Apache License, Version 2.0 (the "License"); | ||
* you may not use this file except in compliance with the License. | ||
* You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
package io.nosqlbench.adapter.qdrant; | ||
|
||
import io.nosqlbench.adapter.diag.DriverAdapterLoader; | ||
import io.nosqlbench.nb.annotations.Service; | ||
import io.nosqlbench.nb.api.components.core.NBComponent; | ||
import io.nosqlbench.nb.api.labels.NBLabels; | ||
|
||
import static io.nosqlbench.adapter.qdrant.QdrantAdapterUtils.QDRANT; | ||
|
||
@Service(value = DriverAdapterLoader.class, selector = QDRANT) | ||
public class QdrantDriverAdapterLoader implements DriverAdapterLoader { | ||
@Override | ||
public QdrantDriverAdapter load(NBComponent parent, NBLabels childLabels) { | ||
return new QdrantDriverAdapter(parent, childLabels); | ||
} | ||
} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We've moved away from indirecting these version numbers into properties, since it makes live editing/analysis harder, and there is already a way to centralize version numbers into one place using maven dependency management.