-
Notifications
You must be signed in to change notification settings - Fork 642
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SOLR-15880: K Nearest Neighbors Search #476
SOLR-15880: K Nearest Neighbors Search #476
Conversation
- indexing knn vector field - custom query parser for knn vector search
solr/core/src/java/org/apache/solr/schema/DenseVectorField.java
Outdated
Show resolved
Hide resolved
The Pull Request is now complete and ready for review. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is great! Very excited to see this getting in to Solr. My main comments are about the de/serialization format. I wonder also if you have a plan for providing access to the index-time hyperparameters, like graph fanout (maxConns) etc. It is pretty important, for best performance, to be able to optimize these for a particular set of vectors embeddings.
} | ||
|
||
private IndexableField createStoredField(SchemaField field, Object value) { | ||
return new StoredField(field.getName(), value.toString()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This doesn't seem like something very useful for numeric vector fields? You'd end up with strings like [0.3485723998437523,...]
which would be a very inefficient way to store the information
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should have been resolved with latest commit
|
||
@Override | ||
public void write(TextResponseWriter writer, String name, IndexableField f) throws IOException { | ||
writer.writeVal(name, f.stringValue()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure what stringValue will return here - will it be the Arrays.toString()
of the vector? Something else?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should have been resolved with latest commit
/** | ||
* Parses a String vector. | ||
* | ||
* @param value with format: [f1, f2, f3, f4...fn] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have you considered an alternate more compressed string format? Maybe b64-encode the bytes of the float array?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should have been resolved with latest commit
thanks @msokolov for your initial feedback! |
I should have addressed most of the comments regarding the documentation. Only a few questions remain, that I don't think are really critical, so I'll wait for some reply on them before committing. In regards to the "Neural Search" name, as I expressed in a comment before, I think it's fine, but I'll raise the matter to the wider community through slack for a quick feedback! Cheers and thanks again! |
I refactored the documentation contribution, changing the name of the section to "Dense Vector Search". If no blockers, I'll proceed tomorrow with the merge! |
This contribution introduces dense vector indexing and searching
This contribution introduces dense vector indexing and searching
Hi guys, thanks for working on this feature! I would like to index vectors with a dimension > 1024 (embeddings come from a ResNet), do you know how this would be possible 🤔 Many thanks in advance ❤️ |
We put in a limit of 1024 since we need some limit in order to help people not to shoot themselves in the foot and then blame the software... What dimension in fact are your vectors? Maybe consider adding some dimensionality reduction step? Sorry I don't know what a ResNet is :) |
I'm currently using 2048, but I can use a smaller model that outputs 1024 :) |
@msokolov : larger embeddings are definitely useful. Oxford's "VGG-Face" outputs a vector of 2622 elements. Support for larger embeddings would enable Solr features like "boost the results of word docs containing an image with a similar face to George Washington" using such models . |
Trying to query for topK greater than 10 but cannot get more than 10 documents in response.
This is from the film examples in solr docs, the query works fine on the UI with
Is there something in the documentation regarding this? |
https://issues.apache.org/jira/browse/SOLR-15880
Description
This is the first milestone in integrating neural search in Apache Solr.
We are reviewing the Pull Request internally and finishing the documentation.
Other than that, the code is ready for review.
It includes a DenseVectorField to wrap the Lucene implementation using Hierarchical Navigable Small World to index the dense vector.
It includes a Knn query parser to run K Nearest Neighbors query.
Solution
A new Apache Solr field Type has been added.
A new Apache Solr Query Parser has been added.
Tests
Dense vector fields and the query parser have tests.
Checklist
Please review the following and check all that apply:
main
branch../gradlew check
.