Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: Add support for _split_overlap meta to Pinecone and dict metadata in general to Weaviate #4805

Merged
merged 8 commits into from
May 5, 2023
Merged

fix: Add support for _split_overlap meta to Pinecone and dict metadata in general to Weaviate #4805

merged 8 commits into from
May 5, 2023

Conversation

bogdankostic
Copy link
Contributor

Related Issues

Proposed Changes:

This PR adds support for the _split_overlap meta field to the PineconeDocumentStore and adds support for any metadata of type dict to WeaviateDocumentStore.

How did you test it?

I added unit tests.

Notes for the reviewer

Weaviate and Pinecone don't support dict type natively, so the workaround is to convert dict metadata fields to JSON strings at index time and convert the JSON strings back to dicts at query time. I haven't found a way to support dict in general for Pinecone, so I restricted the implementation to at least support the _split_overlap meta field.

Checklist

  • I have read the contributors guidelines and the code of conduct
  • I have updated the related issue with new insights and changes
  • I added tests that demonstrate the correct behavior of the change
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
  • I documented my code
  • I ran pre-commit hooks and fixed any issue

@bogdankostic bogdankostic requested a review from a team as a code owner May 3, 2023 21:13
@bogdankostic bogdankostic requested review from vblagoje and removed request for a team May 3, 2023 21:13
@bogdankostic bogdankostic changed the title fix: Add support _split_overlap meta to Pinecone and dict metadata in general to Weaviate fix: Add support for _split_overlap meta to Pinecone and dict metadata in general to Weaviate May 3, 2023
@coveralls
Copy link
Collaborator

coveralls commented May 3, 2023 •
edited
Loading

Coverage Status

Coverage: 36.537% (+0.8%) from 35.749% when pulling 53d04e7 on fix_weaviate_pinecone_dict_meta into d8dc0d7 on main.

vblagoje
vblagoje requested changes May 4, 2023
View reviewed changes
Copy link
Member

@vblagoje vblagoje left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor changes would be helpful. See comments inline.

@bogdankostic bogdankostic requested a review from vblagoje May 5, 2023 09:16
vblagoje
vblagoje approved these changes May 5, 2023
View reviewed changes
Copy link
Member

@vblagoje vblagoje left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bogdankostic bogdankostic merged commit 43509c8 into main May 5, 2023
56 checks passed
@bogdankostic bogdankostic deleted the fix_weaviate_pinecone_dict_meta branch May 5, 2023 09:20
@masci masci mentioned this pull request May 17, 2023
Merged
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vblagoje vblagoje vblagoje approved these changes

Assignees
No one assigned
Labels
topic:document_store topic:pinecone topic:tests topic:weaviate type:documentation Improvements on the docs
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Split overlap metadata array can't be added to Pinecone and Weaviate
3 participants
@bogdankostic @coveralls @vblagoje