Discussion: Add _featureProjection (optimized for visualization) to GraphQL and REST API #1139
Comments
|
Depending on how urgently this is needed and whether a dimensionality reduction algo is available out-of-the box in Go, a temporary solution could also be to add this to the python-based weaviate client and and do the calculations on top of Weaviate until we have ported such an algorithm to go. Note: The above is based on the assumption that no such algorithm exists in Go and it's sufficiently complex (i.e. more than a day or two of effort) to write it from scratch. If both assumptions prove untrue, this temporary workaround wouldn't add any value. |
It is not "blocker" urgent but I would like to do this from the Weaviate perspective from the get-go. |
|
cc: @graphileon |
|
Here is the result from a first experiment with https://github.com/danaugrs/go-tsne to plot the 15NN to the word "car" and "wine" according to their contextionary position. With these 30 results, the dimensionality reduction takes about 14ms which should absolutely be viable for a regular request.
|
|
@bobvanluijt @etiennedi Am I correct understanding this as the 15 best results from submitting a query car or wine ? To avoid a lot of clutter, and having in mind our own d3 based viz, I imagine that the semantic path to a specific result is visible when the user selects that result. |
The above was meant as an experiment to see if we can reduce our n-dimensional vectors to 2d (or 3d) coordinates which seem intuitive to humans. The graph shows two clusters; one is the nearest neighbors (in the contextionary) of the concept
This is entirely open at the moment, but definitely an option.
In the end we would just list a set of 2d or 3d coordinates and it'd be up to you on how you use them for a visualization. Please note that the semantic path (as outlined in #1126 ) is something very early stage that we're experimenting with at the moment. We have not yet finalized how a path is constructed - or the elements it contains, so it's too early for me to predict how that would translate to a visualization. A first step would just be to visualize the search query and all search results. Ideally that would show that the first result is closer to the query than the last result. |
etiennedi
changed the title
etiennedi
changed the title
|
Closing this analysis issue in favor of #1178 which is used for implementation. |
Some end-users use Weaviate to visualize data and results. Normalizing the data to 2 and 3 dimensions is helpful for those types of users.
Questions:
__metaresults as proposed here?Example query and results
Query
{ Get { Things { Article( explore: { concepts: ["beatles"] } limit: 1 ) { title __meta { dimensions( algorithm: foobar, # which algo should be used to determine the positions? fullVectorSpace: boolean # are the results relative to the complete vector space or only the results? ) { 2D 3D fullD } } } } } }Result
Example in Python
The text was updated successfully, but these errors were encountered: