New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Vector DBs: Upgrade CDK #31329
Vector DBs: Upgrade CDK #31329
Conversation
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
Before Merging a Connector Pull RequestWow! What a great pull request you have here! 🎉 To merge this PR, ensure the following has been done/considered for each connector added or updated:
If the checklist is complete, but the CI check is failing,
|
…ithub.com:airbytehq/airbyte into flash1293/allow-openai-compatible-embedding-modes
…nai-compatible-embedding-modes
destination-pinecone test report (commit
|
Step | Result |
---|---|
Build destination-pinecone docker image for platform(s) linux/x86_64 | ✅ |
Unit tests | ✅ |
Integration tests | ✅ |
Acceptance tests | ✅ |
Code format checks | ✅ |
Validate metadata for destination-pinecone | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-pinecone test
destination-weaviate test report (commit
|
Step | Result |
---|---|
Build destination-weaviate docker image for platform(s) linux/x86_64 | ✅ |
Unit tests | ✅ |
Integration tests | ✅ |
Acceptance tests | ✅ |
Code format checks | ✅ |
Validate metadata for destination-weaviate | ❌ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-weaviate test
destination-milvus test report (commit
|
Step | Result |
---|---|
Build destination-milvus docker image for platform(s) linux/x86_64 | ✅ |
Unit tests | ✅ |
Integration tests | ✅ |
Acceptance tests | ✅ |
Code format checks | ✅ |
Validate metadata for destination-milvus | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-milvus test
destination-qdrant test report (commit
|
Step | Result |
---|---|
Build destination-qdrant docker image for platform(s) linux/x86_64 | ✅ |
Unit tests | ✅ |
Code format checks | ✅ |
Validate metadata for destination-qdrant | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-qdrant test
destination-milvus test report (commit
|
Step | Result |
---|---|
Build destination-milvus docker image for platform(s) linux/x86_64 | ✅ |
Unit tests | ✅ |
Integration tests | ✅ |
Acceptance tests | ✅ |
Code format checks | ✅ |
Validate metadata for destination-milvus | ❌ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-milvus test
destination-weaviate test report (commit
|
Step | Result |
---|---|
Build destination-weaviate docker image for platform(s) linux/x86_64 | ✅ |
Unit tests | ✅ |
Integration tests | ✅ |
Acceptance tests | ✅ |
Code format checks | ✅ |
Validate metadata for destination-weaviate | ❌ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-weaviate test
destination-pinecone test report (commit
|
Step | Result |
---|---|
Build destination-pinecone docker image for platform(s) linux/x86_64 | ✅ |
Unit tests | ✅ |
Integration tests | ✅ |
Acceptance tests | ✅ |
Code format checks | ✅ |
Validate metadata for destination-pinecone | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-pinecone test
destination-chroma test report (commit
|
Step | Result |
---|---|
Build destination-chroma docker image for platform(s) linux/x86_64 | ✅ |
Unit tests | ✅ |
Code format checks | ✅ |
Validate metadata for destination-chroma | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-chroma test
destination-qdrant test report (commit
|
Step | Result |
---|---|
Build destination-qdrant docker image for platform(s) linux/x86_64 | ✅ |
Unit tests | ✅ |
Code format checks | ✅ |
Validate metadata for destination-qdrant | ❌ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-qdrant test
destination-chroma test report (commit
|
Step | Result |
---|---|
Build destination-chroma docker image for platform(s) linux/x86_64 | ✅ |
Unit tests | ✅ |
Code format checks | ✅ |
Validate metadata for destination-chroma | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-chroma test
destination-pinecone test report (commit
|
Step | Result |
---|---|
Build destination-pinecone docker image for platform(s) linux/x86_64 | ✅ |
Unit tests | ✅ |
Integration tests | ✅ |
Acceptance tests | ✅ |
Code format checks | ✅ |
Validate metadata for destination-pinecone | ❌ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-pinecone test
destination-milvus test report (commit
|
Step | Result |
---|---|
Build destination-milvus docker image for platform(s) linux/x86_64 | ✅ |
Unit tests | ✅ |
Integration tests | ✅ |
Acceptance tests | ✅ |
Code format checks | ✅ |
Validate metadata for destination-milvus | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-milvus test
destination-weaviate test report (commit
|
Step | Result |
---|---|
Build destination-weaviate docker image for platform(s) linux/x86_64 | ✅ |
Unit tests | ✅ |
Integration tests | ✅ |
Acceptance tests | ✅ |
Code format checks | ✅ |
Validate metadata for destination-weaviate | ❌ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-weaviate test
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a nice set of improvements and cleanup with the version bump. 👍
LGTM! 🚢
self.embedder = ( | ||
create_from_config(config.embedding, config.processing) | ||
if config.embedding.mode != "no_embedding" | ||
else NoEmbedder(config.embedding) | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could this logic be added into the create_from_config()
decision matrix? Not a blocker, just want to get your thoughts.
…nai-compatible-embedding-modes
destination-qdrant test report (commit
|
Step | Result |
---|---|
Build destination-qdrant docker image for platform(s) linux/x86_64 | ✅ |
Unit tests | ✅ |
Code format checks | ✅ |
Validate metadata for destination-qdrant | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-qdrant test
destination-pinecone test report (commit
|
Step | Result |
---|---|
Build destination-pinecone docker image for platform(s) linux/x86_64 | ✅ |
Unit tests | ✅ |
Integration tests | ✅ |
Acceptance tests | ✅ |
Code format checks | ✅ |
Validate metadata for destination-pinecone | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-pinecone test
destination-weaviate test report (commit
|
Step | Result |
---|---|
Build destination-weaviate docker image for platform(s) linux/x86_64 | ✅ |
Unit tests | ✅ |
Integration tests | ✅ |
Acceptance tests | ✅ |
Code format checks | ✅ |
Validate metadata for destination-weaviate | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-weaviate test
destination-milvus test report (commit
|
Step | Result |
---|---|
Build destination-milvus docker image for platform(s) linux/x86_64 | ✅ |
Unit tests | ✅ |
Integration tests | ✅ |
Acceptance tests | ✅ |
Code format checks | ✅ |
Validate metadata for destination-milvus | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-milvus test
destination-chroma test report (commit
|
Step | Result |
---|---|
Build destination-chroma docker image for platform(s) linux/x86_64 | ✅ |
Unit tests | ✅ |
Code format checks | ✅ |
Validate metadata for destination-chroma | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-chroma test
Co-authored-by: flash1293 <flash1293@users.noreply.github.com> Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: flash1293 <flash1293@users.noreply.github.com> Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
What
Two important bits:
Side note:
create_from_config
to simplify embedder instance generationHow to test
How to run with LocalAI:
git clone https://github.com/go-skynet/LocalAI
cd LocalAI
wget https://huggingface.co/skeskinen/ggml/resolve/main/all-MiniLM-L6-v2/ggml-model-q4_0.bin -O models/bert
models/embeddings.yml
:docker-compose up -d --pull always
Then configure the destination:
http://host.docker.internal:8080
384