Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add Faiss as VectorStore provider #685

Merged
merged 47 commits into from
May 18, 2023
Merged

feat: add Faiss as VectorStore provider #685

merged 47 commits into from
May 18, 2023

Conversation

ewfian
Copy link
Contributor

@ewfian ewfian commented Apr 9, 2023

Add support for Faiss and also implemented a loadFromPython method, which can read VectorStore saved from Python's implementation.

TODO:

  • Improve the pickle parser and fully implement Pickle protocol version 4. Extract pickle.js as an NPM package and then import it. The package will be here pickleparser.
  • Port id to uuid mapping from Python implementation.
  • Save mapping into JSON file.
  • Add examples.
    • Create a new index from texts.
    • Create a new index from a loader.
    • Save an index to file and load it again.
    • Load the saved file from Python.
  • Add test cases.
    • Unit Test: fromTexts + addVectors.
    • Unit Test: fromDocuments + addVectors.
    • Unit Test: exceptions.
    • Integration Test: save and load.
    • Integration Test: Load from Python.
    • Integration Test: Add documents after loaded from Python.
    • Integration Test: Save and re-load after loaded from Python.
  • Add documentation.
    • Create pages.
    • Add basic information.
    • Add example for how to read from Python.

@vercel
Copy link

vercel bot commented Apr 9, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Updated (UTC)
langchainjs-docs ✅ Ready (Inspect) Visit Preview May 18, 2023 4:53pm

@nfcampos
Copy link
Collaborator

Thanks a lot for this @ewfian this is a great PR, sorry for the delay in getting this in

@nfcampos nfcampos added the lgtm PRs that are ready to be merged as-is label May 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lgtm PRs that are ready to be merged as-is
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants