Milvus, Chroma, Qdrant, Pinecone: Update CDK #30820

flash1293 · 2023-09-28T11:00:02Z

What

Updates CDK for Milvus, Chroma, Qdrant and Pinecone connectors to get fix in for Azure embedders.

vercel · 2023-09-28T11:00:07Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 1, 2023 2:12pm

github-actions · 2023-09-28T11:00:22Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

airbyte-oss-build-runner · 2023-09-28T11:08:38Z

destination-qdrant test report (commit `118dfae4e2`) - ✅

⏲️ Total pipeline duration: 03mn46s

Step	Result
Connector package install	✅
Build destination-qdrant docker image for platform linux/x86_64	✅
Unit tests	✅
Code format checks	✅
Validate metadata for destination-qdrant	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-qdrant test

airbyte-oss-build-runner · 2023-09-28T11:08:48Z

destination-chroma test report (commit `118dfae4e2`) - ❌

⏲️ Total pipeline duration: 04mn04s

Step	Result
Connector package install	✅
Build destination-chroma docker image for platform linux/x86_64	✅
Unit tests	✅
Code format checks	❌
Validate metadata for destination-chroma	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-chroma test

airbyte-oss-build-runner · 2023-09-28T11:08:52Z

destination-pinecone test report (commit `118dfae4e2`) - ✅

⏲️ Total pipeline duration: 04mn06s

Step	Result
Connector package install	✅
Build destination-pinecone docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for destination-pinecone	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-pinecone test

airbyte-oss-build-runner · 2023-09-28T11:08:53Z

destination-milvus test report (commit `118dfae4e2`) - ✅

⏲️ Total pipeline duration: 04mn08s

Step	Result
Connector package install	✅
Build destination-milvus docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for destination-milvus	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-milvus test

airbyte-oss-build-runner · 2023-09-28T11:21:23Z

destination-qdrant test report (commit `40af6af121`) - ✅

⏲️ Total pipeline duration: 01mn45s

Step	Result
Connector package install	✅
Build destination-qdrant docker image for platform linux/x86_64	✅
Unit tests	✅
Code format checks	✅
Validate metadata for destination-qdrant	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-qdrant test

airbyte-oss-build-runner · 2023-09-28T11:21:40Z

destination-milvus test report (commit `40af6af121`) - ✅

⏲️ Total pipeline duration: 02mn54s

Step	Result
Connector package install	✅
Build destination-milvus docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for destination-milvus	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-milvus test

airbyte-oss-build-runner · 2023-09-28T11:21:41Z

destination-pinecone test report (commit `40af6af121`) - ✅

⏲️ Total pipeline duration: 02mn54s

Step	Result
Connector package install	✅
Build destination-pinecone docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for destination-pinecone	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-pinecone test

airbyte-oss-build-runner · 2023-09-28T11:21:53Z

destination-chroma test report (commit `40af6af121`) - ✅

⏲️ Total pipeline duration: 03mn09s

Step	Result
Connector package install	✅
Build destination-chroma docker image for platform linux/x86_64	✅
Unit tests	✅
Code format checks	✅
Validate metadata for destination-chroma	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-chroma test

pedroslopez

nits/Qs, but LGTM!

pedroslopez · 2023-09-28T20:47:25Z

airbyte-integrations/connectors/destination-chroma/destination_chroma/destination.py

+        if config.embedding.mode == "azure_openai" or config.embedding.mode == "openai":
+            self.embedder = embedder_map[config.embedding.mode](config.embedding, config.processing.chunk_size)
+        else:
+            self.embedder = embedder_map[config.embedding.mode](config.embedding)


could you share more about what's going on here and why only chroma needs this?

oh - are we just now adding the embedder for chroma but others have it already?

yeah, exactly, this is aligning with the others.

pedroslopez · 2023-09-28T20:49:03Z

airbyte-integrations/connectors/destination-chroma/setup.py

@@ -6,7 +6,7 @@
 from setuptools import find_packages, setup

 MAIN_REQUIREMENTS = [
-    "airbyte-cdk[vector-db-based]",
+    "airbyte-cdk[vector-db-based]==0.51.22",


Do we need to pin exact? If we want to at least include this version but go up to 0.51.x we could use ~=

As the vector db CDK currently changes dramatically even in patch versions, I would like to keep it like this for now until things are more stable-

airbyte-oss-build-runner · 2023-10-01T14:16:55Z

destination-pinecone test report (commit `1c2c5848b6`) - ✅

⏲️ Total pipeline duration: 03mn46s

Step	Result
Connector package install	✅
Build destination-pinecone docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for destination-pinecone	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-pinecone test

airbyte-oss-build-runner · 2023-10-01T14:17:12Z

destination-qdrant test report (commit `1c2c5848b6`) - ✅

⏲️ Total pipeline duration: 04mn43s

Step	Result
Connector package install	✅
Build destination-qdrant docker image for platform linux/x86_64	✅
Unit tests	✅
Code format checks	✅
Validate metadata for destination-qdrant	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-qdrant test

airbyte-oss-build-runner · 2023-10-01T14:17:13Z

destination-chroma test report (commit `1c2c5848b6`) - ✅

⏲️ Total pipeline duration: 04mn44s

Step	Result
Connector package install	✅
Build destination-chroma docker image for platform linux/x86_64	✅
Unit tests	✅
Code format checks	✅
Validate metadata for destination-chroma	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-chroma test

airbyte-oss-build-runner · 2023-10-01T14:17:33Z

destination-milvus test report (commit `1c2c5848b6`) - ✅

⏲️ Total pipeline duration: 05mn06s

Step	Result
Connector package install	✅
Build destination-milvus docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for destination-milvus	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-milvus test

Co-authored-by: flash1293 <flash1293@users.noreply.github.com>

update cdk

8aa2d3c

octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/destination/milvus connectors/destination/pinecone connectors/destination/qdrant connectors/destination/chroma labels Sep 28, 2023

update pr number

118dfae

flash1293 requested review from pedroslopez and aaronsteers September 28, 2023 11:03

vercel bot deployed to Preview September 28, 2023 11:04 View deployment

Automated Commit - Formatting Changes

40af6af

pedroslopez approved these changes Sep 28, 2023

View reviewed changes

flash1293 enabled auto-merge (squash) October 1, 2023 14:02

Merge branch 'master' into flash1293/update-connectors

1c2c584

vercel bot deployed to Preview October 1, 2023 14:12 View deployment

flash1293 merged commit 14bfc11 into master Oct 1, 2023
23 checks passed

flash1293 deleted the flash1293/update-connectors branch October 1, 2023 14:28

girarda pushed a commit that referenced this pull request Oct 10, 2023

Milvus, Chroma, Qdrant, Pinecone: Update CDK (#30820)

c2e0122

Co-authored-by: flash1293 <flash1293@users.noreply.github.com>

girarda pushed a commit that referenced this pull request Oct 10, 2023

Milvus, Chroma, Qdrant, Pinecone: Update CDK (#30820)

8e4edbf

Co-authored-by: flash1293 <flash1293@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Milvus, Chroma, Qdrant, Pinecone: Update CDK #30820

Milvus, Chroma, Qdrant, Pinecone: Update CDK #30820

flash1293 commented Sep 28, 2023

vercel bot commented Sep 28, 2023 •

edited

github-actions bot commented Sep 28, 2023 •

edited by flash1293

airbyte-oss-build-runner commented Sep 28, 2023

airbyte-oss-build-runner commented Sep 28, 2023

airbyte-oss-build-runner commented Sep 28, 2023

airbyte-oss-build-runner commented Sep 28, 2023

airbyte-oss-build-runner commented Sep 28, 2023

airbyte-oss-build-runner commented Sep 28, 2023

airbyte-oss-build-runner commented Sep 28, 2023

airbyte-oss-build-runner commented Sep 28, 2023

pedroslopez left a comment

pedroslopez Sep 28, 2023

pedroslopez Sep 28, 2023

flash1293 Oct 1, 2023

pedroslopez Sep 28, 2023

flash1293 Oct 1, 2023

airbyte-oss-build-runner commented Oct 1, 2023

airbyte-oss-build-runner commented Oct 1, 2023

airbyte-oss-build-runner commented Oct 1, 2023

airbyte-oss-build-runner commented Oct 1, 2023

Milvus, Chroma, Qdrant, Pinecone: Update CDK #30820

Milvus, Chroma, Qdrant, Pinecone: Update CDK #30820

Conversation

flash1293 commented Sep 28, 2023

What

vercel bot commented Sep 28, 2023 • edited

github-actions bot commented Sep 28, 2023 • edited by flash1293

Before Merging a Connector Pull Request

airbyte-oss-build-runner commented Sep 28, 2023

destination-qdrant test report (commit 118dfae4e2) - ✅

airbyte-oss-build-runner commented Sep 28, 2023

destination-chroma test report (commit 118dfae4e2) - ❌

airbyte-oss-build-runner commented Sep 28, 2023

destination-pinecone test report (commit 118dfae4e2) - ✅

airbyte-oss-build-runner commented Sep 28, 2023

destination-milvus test report (commit 118dfae4e2) - ✅

airbyte-oss-build-runner commented Sep 28, 2023

destination-qdrant test report (commit 40af6af121) - ✅

airbyte-oss-build-runner commented Sep 28, 2023

destination-milvus test report (commit 40af6af121) - ✅

airbyte-oss-build-runner commented Sep 28, 2023

destination-pinecone test report (commit 40af6af121) - ✅

airbyte-oss-build-runner commented Sep 28, 2023

destination-chroma test report (commit 40af6af121) - ✅

pedroslopez left a comment

Choose a reason for hiding this comment

pedroslopez Sep 28, 2023

Choose a reason for hiding this comment

pedroslopez Sep 28, 2023

Choose a reason for hiding this comment

flash1293 Oct 1, 2023

Choose a reason for hiding this comment

pedroslopez Sep 28, 2023

Choose a reason for hiding this comment

flash1293 Oct 1, 2023

Choose a reason for hiding this comment

airbyte-oss-build-runner commented Oct 1, 2023

destination-pinecone test report (commit 1c2c5848b6) - ✅

airbyte-oss-build-runner commented Oct 1, 2023

destination-qdrant test report (commit 1c2c5848b6) - ✅

airbyte-oss-build-runner commented Oct 1, 2023

destination-chroma test report (commit 1c2c5848b6) - ✅

airbyte-oss-build-runner commented Oct 1, 2023

destination-milvus test report (commit 1c2c5848b6) - ✅

vercel bot commented Sep 28, 2023 •

edited

github-actions bot commented Sep 28, 2023 •

edited by flash1293

destination-qdrant test report (commit `118dfae4e2`) - ✅

destination-chroma test report (commit `118dfae4e2`) - ❌

destination-pinecone test report (commit `118dfae4e2`) - ✅

destination-milvus test report (commit `118dfae4e2`) - ✅

destination-qdrant test report (commit `40af6af121`) - ✅

destination-milvus test report (commit `40af6af121`) - ✅

destination-pinecone test report (commit `40af6af121`) - ✅

destination-chroma test report (commit `40af6af121`) - ✅

destination-pinecone test report (commit `1c2c5848b6`) - ✅

destination-qdrant test report (commit `1c2c5848b6`) - ✅

destination-chroma test report (commit `1c2c5848b6`) - ✅

destination-milvus test report (commit `1c2c5848b6`) - ✅