Add LanceDB vectorstore #1597

gsilvestrin · 2023-06-09T00:50:53Z

Adds support to LanceDB, an embedded vector database.

It should behave as the Python Integration

I believe the Vercel failure is unrelated to my change, is caused by langchainplus:

[INFO] [en] Creating an optimized production build...
../langchain/src/client/langchainplus.ts:1:46 - error TS2307: Cannot find module 'langchainplus-sdk' or its corresponding type declarations.
1 import { Example, LangChainPlusClient } from "langchainplus-sdk";
                                               ~~~~~~~~~~~~~~~~~~~
../langchain/src/client/langchainplus.ts:175:25 - error TS7006: Parameter 'example' implicitly has an 'any' type.
175     examples.map(async (example) => {
                            ~~~~~~~
../langchain/src/client/tests/langchainplus.int.test.ts:3:37 - error TS2307: Cannot find module 'langchainplus-sdk' or its corresponding type declarations.
3 import { LangChainPlusClient } from "langchainplus-sdk";
                                      ~~~~~~~~~~~~~~~~~~~
../langchain/src/client/tests/langchainplus.int.test.ts:17:22 - error TS7006: Parameter 'd' implicitly has an 'any' type.
17   if (!datasets.map((d) => d.name).includes(datasetName)) {
                        ~
../langchain/src/client/tests/langchainplus.int.test.ts:51:22 - error TS7006: Parameter 'd' implicitly has an 'any' type.
51   if (!datasets.map((d) => d.name).includes(datasetName)) {
                        ~
../langchain/src/client/tests/langchainplus.int.test.ts:77:24 - error TS7006: Parameter 's' implicitly has an 'any' type.
77   expect(sessions.map((s) => s.name)).toContain(sessionName);
                          ~
../langchain/src/client/tests/langchainplus.int.test.ts:102:22 - error TS7006: Parameter 'd' implicitly has an 'any' type.
102   if (!datasets.map((d) => d.name).includes(fileName)) {
                         ~
../langchain/src/client/tests/langchainplus.int.test.ts:143:22 - error TS7006: Parameter 'd' implicitly has an 'any' type.
143   if (!datasets.map((d) => d.name).includes(datasetName)) {

vercel · 2023-06-09T00:50:56Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
langchainjs-docs	❌ Failed (Inspect)		Jun 14, 2023 4:22pm

dqbd

Thank you for your contribution and sorry for the delay! Just minor nitpicks here and there, but lgtm otherwise :)

dqbd · 2023-06-21T15:21:52Z

langchain/package.json

    "ts-jest": "^29.1.0",
    "typeorm": "^0.3.12",
    "typescript": "^5.0.0",
+    "vectordb": "^0.1.4",
    "typesense": "^1.5.3",


It does seem like Yarn reorders the dependencies and creates a new lockfile, would it be possible to run yarn once again?

dqbd · 2023-06-21T15:22:55Z

langchain/scripts/create-entrypoints.js

@@ -57,6 +57,7 @@ const entrypoints = {
  "vectorstores/hnswlib": "vectorstores/hnswlib",
  "vectorstores/faiss": "vectorstores/faiss",
  "vectorstores/weaviate": "vectorstores/weaviate",
+  "vectorstores/lancedb": "vectorstores/lancedb",


It does seem like node langchain/scripts/create-entrypoints.js has not been executed yet. Would it be possible to run these commands?

dqbd · 2023-06-21T15:32:52Z

langchain/src/vectorstores/lancedb.ts

+    const data: Array<Record<string, unknown>> = [];
+    for (let i = 0; i < documents.length; i += 1) {
+      // eslint-disable-next-line @typescript-eslint/no-explicit-any
+      const record: Record<string, any> = {};


nit: preferably use Record<string, unknown> instead.

dqbd · 2023-06-21T15:33:38Z

langchain/src/vectorstores/lancedb.ts

+    const docsAndScore: [Document, number][] = [];
+    results.forEach((item) => {
+      // eslint-disable-next-line @typescript-eslint/no-explicit-any
+      const metadata: Record<string, any> = {};


nit: preferably, use unknown instead of any whenever possible.

dqbd · 2023-06-21T15:42:55Z

langchain/src/vectorstores/tests/lancedb.test.ts

+  test("constructor works", async () => {
+    const lancedb = new LanceDB(new FakeEmbeddings(), {
+      // eslint-disable-next-line @typescript-eslint/no-explicit-any
+      table: new Table(null as any, "vectors"),


nit: afaict first argument of Table is already any?

dqbd · 2023-06-21T15:44:38Z

langchain/src/vectorstores/tests/lancedb.test.ts

+
+  test("should add vectors to the table", async () => {
+    const embeddings = new FakeEmbeddings();
+    const dir = await track().mkdir("lancedb");


Do we need to import track as an another dev-dependency? Could we just remove the temp directory after the test completes?

Thanks for the review @dqbd! I can refactor the code and remove the track dependency. I saw that the unit test was not included in #1787, were there any problems other the two comments raised here? I can clean up and submit it in another PR.

It did not pass our CI, build seemed to fail on old Macs and Windows. It's all good to have it as an integration test IMO.

jacoblee93 · 2023-06-27T23:42:32Z

Thanks @gsilvestrin and @dqbd - I've made a few fixes and taken the comments into account. Will aim to have the extended PR #1787 merged as part of tomorrow's release!

jacoblee93 · 2023-06-28T04:48:40Z

Merged in #1787

Add LanceDB vectorstore

f82aaa3

vercel bot had a problem deploying to Preview June 9, 2023 00:51 Failure

gsilvestrin mentioned this pull request Jun 9, 2023

[Integration] Langchain JS integration lancedb/lancedb#148

Closed

Merge branch 'main' into gsilvestrin/lancedb_langchainjs

6c17a62

vercel bot had a problem deploying to Preview June 13, 2023 15:55 Failure

Merge branch 'main' into gsilvestrin/lancedb_langchainjs

3fcdfae

vercel bot had a problem deploying to Preview June 13, 2023 17:59 Failure

Merge branch 'main' into gsilvestrin/lancedb_langchainjs

15fbee6

vercel bot had a problem deploying to Preview June 14, 2023 16:22 Failure

dqbd self-requested a review June 14, 2023 16:47

dqbd requested changes Jun 21, 2023

View reviewed changes

jacoblee93 mentioned this pull request Jun 27, 2023

Add LanceDB vectorstore #1787

Merged

jacoblee93 closed this Jun 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add LanceDB vectorstore #1597

Add LanceDB vectorstore #1597

gsilvestrin commented Jun 9, 2023 •

edited

vercel bot commented Jun 9, 2023 •

edited

dqbd left a comment

dqbd Jun 21, 2023

dqbd Jun 21, 2023

dqbd Jun 21, 2023

dqbd Jun 21, 2023

dqbd Jun 21, 2023

dqbd Jun 21, 2023

gsilvestrin Jul 3, 2023

jacoblee93 Jul 3, 2023

jacoblee93 commented Jun 27, 2023

jacoblee93 commented Jun 28, 2023

Add LanceDB vectorstore #1597

Add LanceDB vectorstore #1597

Conversation

gsilvestrin commented Jun 9, 2023 • edited

vercel bot commented Jun 9, 2023 • edited

dqbd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jacoblee93 commented Jun 27, 2023

jacoblee93 commented Jun 28, 2023

gsilvestrin commented Jun 9, 2023 •

edited

vercel bot commented Jun 9, 2023 •

edited