Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add unified filter to marqo #41

Merged
merged 3 commits into from
Dec 24, 2023
Merged

Conversation

sky-2002
Copy link
Contributor

Description: This PR intends to add a unified filtering interface to neum ai, which can be mapped to filters of respective sinks. To test this, I have made changes to MarqoSink and tested them. Addresses this issue.
Approach: To start with, I have assumed that the user provides the filter as a string like "field1 <= value1, field2 != value2" which is then mapped to a class called FilterCondition. This is subject to change as per discussions and review.

Example usage:

from SinkConnectors.MarqoSink import MarqoSink

ms = MarqoSink(url="http://0.0.0.0:8882", index_name="neum_index", api_key=None)
ms.store(vectors_to_store=neum_vecs)

# Both produce same result
# ms.search(vector=[1.0, 0.4], number_of_results=3, filter={"position": "x-axis", "power":4})
ms.search(vector=[1.0, 0.4], number_of_results=3, filter="position = x-axis, power=4")

Note: Benefit of this unified filter over dictionary is that we can now add range based queries.

@sky-2002
Copy link
Contributor Author

Hi @ddematheu , I have made some changes as per discussion here, looking for review.

@ddematheu
Copy link
Contributor

Looks good, will implement changes back into other sinks.

@ddematheu ddematheu merged commit d4a4339 into NeumTry:main Dec 24, 2023
@sky-2002 sky-2002 deleted the feature/unified-filter branch December 31, 2023 09:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants