[Feature][VectorStore] Support StarRocks as vector db #6119

dirtysalt · 2023-06-13T21:35:01Z

Fixes # (issue)

Before submitting

Here are some examples to use StarRocks as vectordb

from langchain.vectorstores import StarRocks
from langchain.vectorstores.starrocks import StarRocksSettings

embeddings = OpenAIEmbeddings()

# conifgure starrocks settings
settings = StarRocksSettings()
settings.port = 41003
settings.host = '127.0.0.1'
settings.username = 'root'
settings.password = ''
settings.database = 'zya'

# to fill new embeddings
docsearch = StarRocks.from_documents(split_docs, embeddings, config = settings)   


# or to use already-built embeddings in database.
docsearch = StarRocks(embeddings, settings)

Who can review?

Tag maintainers/contributors who might be interested:

@dev2049

hwchase17

an example notebook would be very helpful!

hwchase17 · 2023-06-18T17:51:32Z

langchain/vectorstores/starrocks.py

+            config: Optional[StarRocksSettings] = None,
+            **kwargs: Any,
+    ) -> None:
+        """StarRocks Wrapper to LangChain


better docstring would be nice

I've updated docstring and added a example notebook to use starrocks as vectordb.

dirtysalt · 2023-06-19T01:37:42Z

an example notebook would be very helpful!

OK. I'll fix that.

vercel · 2023-06-19T05:57:53Z

@dirtysalt is attempting to deploy a commit to the LangChain Team on Vercel.

A member of the Team first needs to authorize it.

rlancemartin · 2023-06-19T06:07:15Z

an example notebook would be very helpful!

OK. I'll fix that.

Thanks! I'll have a look at the notebook tomorrow, and help get this merged.

dirtysalt · 2023-06-19T08:09:15Z

an example notebook would be very helpful!

OK. I'll fix that.

Thanks! I'll have a look at the notebook tomorrow, and help get this merged.

Thanks. I've added a example notebook to use starrocks as vectordb.

Add [`langchain`](langchain-ai/langchain#6119) extension you can test the function with the following SQLs ``` create table t1 (id int, data array<float>) engine = olap distributed by hash(id) properties ("replication_num" = "1"); insert into t1 values(1, array<float>[0.1, 0.2, 0.3]), (2, array<float>[0.2, 0.1, 0.3]), (3, array<float>[0.3, 0.2, 0.1]); select cosine_similarity(array<float>[0.1, 0.2, 0.3], data) as dist, id from t1; ``` Signed-off-by: yanz <dirtysalt1987@gmail.com>

rlancemartin · 2023-06-20T15:56:12Z

@dirtysalt thanks!

(1) In the notebook, please add a header # Starrocks and a short description with context on the DB. It will help with adoption. E.g., see example for other vectorDBs here.

(2) Please run make format to format and also have a look at the lint errors in the tests:

langchain/vectorstores/starrocks.py:26: error: Function is missing a type annotation  [no-untyped-def]
langchain/vectorstores/starrocks.py:32: error: Function is missing a type annotation  [no-untyped-def]
langchain/vectorstores/starrocks.py:33: error: Library stubs not installed for "pymysql.cursors"  [import]
langchain/vectorstores/starrocks.py:33: note: Hint: "python3 -m pip install types-PyMySQL"
langchain/vectorstores/starrocks.py:33: note: (or run "mypy --install-types" to install all missing stub packages)
langchain/vectorstores/starrocks.py:33: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
langchain/vectorstores/starrocks.py:33: error: Library stubs not installed for "pymysql"  [import]
langchain/vectorstores/starrocks.py:34: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
langchain/vectorstores/starrocks.py:[19](https://github.com/hwchase17/langchain/actions/runs/5309392803/jobs/9644405342?pr=6119#step:6:20)3: error: "Iterable[str]" has no attribute "index"  [attr-defined]
langchain/vectorstores/starrocks.py:335: error: Need type annotation for "settings_strs" (hint: "settings_strs: List[<type>] = ...")  [var-annotated]
langchain/vectorstores/starrocks.py:449: error: "StarRocks" has no attribute "client"  [attr-defined]

vercel · 2023-06-21T02:38:56Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)			Jun 21, 2023 11:44am

dirtysalt · 2023-06-21T03:43:50Z

@rlancemartin I'm quite a newbie to this community, and thanks for pointing out problems.

I've fixed lint and format problems.

And I've updated the notebook and added some descriptions about the background of StarRocks.

Really appreciate your review and time.

rlancemartin · 2023-06-21T04:49:28Z

@rlancemartin I'm quite a newbie to this community, and thanks for pointing out problems.

I've fixed lint and format problems.

And I've updated the notebook and added some descriptions about the background of StarRocks.

Really appreciate your review and time.

Thanks! No problem. Lint errors are always a bit annoying :) I kicked off tests and will look again tomorrow. I can resolve any remaining ones quickly and get this in.

dirtysalt · 2023-06-21T07:22:37Z

OK, Thanks.

I've checked lint error. Looks like you have to install a python package in lint python env.

Because I've used pymysql this package, and there is no pymysql stub for type annotation checking.

python3 -m pip install types-PyMySQL

dev2049 · 2023-06-21T07:59:32Z

langchain/vectorstores/starrocks.py

+    def __init__(
+        self,
+        embedding: Embeddings,
+        config: Optional[StarRocksSettings] = None,


why make config optional and give it default None if we're gonna assert that it's non-null later?

The assert statement is right after handling the null case

if config is not None: self.config = config else: self.config = StarRocksSettings() assert self.config

if config is None, we give it a default setting.

So in theory, we can remove assert self.config this stmt, because it will always true.

rlancemartin · 2023-06-21T16:01:22Z

OK, Thanks.

I've checked lint error. Looks like you have to install a python package in lint python env.

Because I've used pymysql this package, and there is no pymysql stub for type annotation checking.

python3 -m pip install types-PyMySQL

Looks like checks are now passing. Good work.

* master: (28 commits) [Feature][VectorStore] Support StarRocks as vector db (langchain-ai#6119) Relax string input mapper check (langchain-ai#6544) bump to ver 208 (langchain-ai#6540) Harrison/multi tool (langchain-ai#6518) Infino integration for simplified logs, metrics & search across LLM data & token usage (langchain-ai#6218) Update model token mappings/cost to include 0613 models (langchain-ai#6122) Fix issue with non-list `To` header in GmailSendMessage Tool (langchain-ai#6242) Integrate Rockset as Vectorstore (langchain-ai#6216) Feat: Add a prompt template parameter to qa with structure chains (langchain-ai#6495) Add async support for HuggingFaceTextGenInference (langchain-ai#6507) Be able to use Codey models on Vertex AI (langchain-ai#6354) Add KuzuQAChain (langchain-ai#6454) Update index.mdx (langchain-ai#6326) Export trajectory eval fn (langchain-ai#6509) typo(llamacpp.ipynb): 'condiser' -> 'consider' (langchain-ai#6474) Fix typo in docstring of format_tool_to_openai_function (langchain-ai#6479) Make streamlit import optional (langchain-ai#6510) Fixed: 'readible' -> readable (langchain-ai#6492) Documentation Fix: Correct the example code output in the prompt templates doc (langchain-ai#6496) Fix link (langchain-ai#6501) ...

Add [`langchain`](langchain-ai/langchain#6119) extension you can test the function with the following SQLs ``` create table t1 (id int, data array<float>) engine = olap distributed by hash(id) properties ("replication_num" = "1"); insert into t1 values(1, array<float>[0.1, 0.2, 0.3]), (2, array<float>[0.2, 0.1, 0.3]), (3, array<float>[0.3, 0.2, 0.1]); select cosine_similarity(array<float>[0.1, 0.2, 0.3], data) as dist, id from t1; ``` Signed-off-by: yanz <dirtysalt1987@gmail.com> (cherry picked from commit 4253167)

Add [`langchain`](langchain-ai/langchain#6119) extension you can test the function with the following SQLs ``` create table t1 (id int, data array<float>) engine = olap distributed by hash(id) properties ("replication_num" = "1"); insert into t1 values(1, array<float>[0.1, 0.2, 0.3]), (2, array<float>[0.2, 0.1, 0.3]), (3, array<float>[0.3, 0.2, 0.1]); select cosine_similarity(array<float>[0.1, 0.2, 0.3], data) as dist, id from t1; ``` Signed-off-by: yanz <dirtysalt1987@gmail.com> (cherry picked from commit 4253167) # Conflicts: # be/src/exprs/vectorized/math_functions.cpp # be/src/exprs/vectorized/math_functions.h # gensrc/script/vectorized/vectorized_functions.py

Add [`langchain`](langchain-ai/langchain#6119) extension you can test the function with the following SQLs ``` create table t1 (id int, data array<float>) engine = olap distributed by hash(id) properties ("replication_num" = "1"); insert into t1 values(1, array<float>[0.1, 0.2, 0.3]), (2, array<float>[0.2, 0.1, 0.3]), (3, array<float>[0.3, 0.2, 0.1]); select cosine_similarity(array<float>[0.1, 0.2, 0.3], data) as dist, id from t1; ``` Signed-off-by: yanz <dirtysalt1987@gmail.com> (cherry picked from commit 4253167) Signed-off-by: dirtysalt <dirtysalt1987@gmail.com>

Add [`langchain`](langchain-ai/langchain#6119) extension you can test the function with the following SQLs ``` create table t1 (id int, data array<float>) engine = olap distributed by hash(id) properties ("replication_num" = "1"); insert into t1 values(1, array<float>[0.1, 0.2, 0.3]), (2, array<float>[0.2, 0.1, 0.3]), (3, array<float>[0.3, 0.2, 0.1]); select cosine_similarity(array<float>[0.1, 0.2, 0.3], data) as dist, id from t1; ``` Signed-off-by: yanz <dirtysalt1987@gmail.com> (cherry picked from commit 4253167)

dirtysalt mentioned this pull request Jun 13, 2023

[Feature] add cosine_similarity function StarRocks/starrocks#25199

Merged

16 tasks

kangkaisen approved these changes Jun 13, 2023

View reviewed changes

ss892714028 approved these changes Jun 13, 2023

View reviewed changes

dirtysalt changed the title ~~Support StarRocks as vector db~~ [Feature][VectorStore] Support StarRocks as vector db Jun 14, 2023

hwchase17 reviewed Jun 18, 2023

View reviewed changes

hwchase17 assigned rlancemartin Jun 18, 2023

hwchase17 added the 03 enhancement Enhancement of existing functionality label Jun 18, 2023

dirtysalt added 2 commits June 19, 2023 13:57

fix conflict

ebd10b3

use normalized version cosine_similarity_norm

a3ea08f

dirtysalt force-pushed the starrocks-vectordb branch from dc1bb62 to a3ea08f Compare June 19, 2023 05:57

dirtysalt added 3 commits June 19, 2023 14:15

add more comment

7bb425d

add example of using starrocks as vectordb

a7f8eac

update example notebook

c3bddd7

dirtysalt added 2 commits June 21, 2023 10:16

to format python code

a677c44

fix for lint

9719a49

dirtysalt added 4 commits June 21, 2023 11:21

update notebook and add some comments

51c6d70

update notebook and add some comments

e84450d

update notebook and add some comments

b18f4aa

update notebook and add some comments

68dc8a4

update notebook and add some comments

27b339c

dev2049 added the Ɑ: vector store Related to vector store module label Jun 21, 2023

ignore

9aa8338

dev2049 reviewed Jun 21, 2023

View reviewed changes

dev2049 and others added 3 commits June 21, 2023 01:02

mv nb

78509f1

nb nit

a187121

fix typo

61a8475

rlancemartin approved these changes Jun 21, 2023

View reviewed changes

rlancemartin merged commit 57cc3d1 into langchain-ai:master Jun 21, 2023

This was referenced Jun 25, 2023

Zep Authentication #6725

Closed

Zep Authentication #6728

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][VectorStore] Support StarRocks as vector db #6119

[Feature][VectorStore] Support StarRocks as vector db #6119

dirtysalt commented Jun 13, 2023 •

edited

Loading

hwchase17 left a comment

hwchase17 Jun 18, 2023

dirtysalt Jun 19, 2023

dirtysalt commented Jun 19, 2023

vercel bot commented Jun 19, 2023

rlancemartin commented Jun 19, 2023

dirtysalt commented Jun 19, 2023

rlancemartin commented Jun 20, 2023

vercel bot commented Jun 21, 2023 •

edited

Loading

dirtysalt commented Jun 21, 2023

rlancemartin commented Jun 21, 2023

dirtysalt commented Jun 21, 2023 •

edited

Loading

dev2049 Jun 21, 2023

dirtysalt Jun 21, 2023

rlancemartin commented Jun 21, 2023

[Feature][VectorStore] Support StarRocks as vector db #6119

[Feature][VectorStore] Support StarRocks as vector db #6119

Conversation

dirtysalt commented Jun 13, 2023 • edited Loading

Before submitting

Who can review?

hwchase17 left a comment

Choose a reason for hiding this comment

hwchase17 Jun 18, 2023

Choose a reason for hiding this comment

dirtysalt Jun 19, 2023

Choose a reason for hiding this comment

dirtysalt commented Jun 19, 2023

vercel bot commented Jun 19, 2023

rlancemartin commented Jun 19, 2023

dirtysalt commented Jun 19, 2023

rlancemartin commented Jun 20, 2023

vercel bot commented Jun 21, 2023 • edited Loading

dirtysalt commented Jun 21, 2023

rlancemartin commented Jun 21, 2023

dirtysalt commented Jun 21, 2023 • edited Loading

dev2049 Jun 21, 2023

Choose a reason for hiding this comment

dirtysalt Jun 21, 2023

Choose a reason for hiding this comment

rlancemartin commented Jun 21, 2023

dirtysalt commented Jun 13, 2023 •

edited

Loading

vercel bot commented Jun 21, 2023 •

edited

Loading

dirtysalt commented Jun 21, 2023 •

edited

Loading