Milvus / Qdrant / Pinecone: Update CDK #30689

flash1293 · 2023-09-22T13:06:13Z

What

Based off #30718

This PR upgrades the CDK for Milvus and Qdrant desinations to allow processing of large records as well as adding support for Azure OpenAI embeddings and adding more text splitting options.

It also sets allowed hosts correctly.

vercel · 2023-09-22T13:06:19Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 26, 2023 9:46am

airbyte-oss-build-runner · 2023-09-22T13:12:53Z

destination-qdrant test report (commit `feef5222a6`) - ✅

⏲️ Total pipeline duration: 03mn22s

Step	Result
Connector package install	✅
Build destination-qdrant docker image for platform linux/x86_64	✅
Unit tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/destination-qdrant/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-qdrant test

airbyte-oss-build-runner · 2023-09-22T13:13:02Z

destination-milvus test report (commit `feef5222a6`) - ❌

⏲️ Total pipeline duration: 03mn32s

Step	Result
Connector package install	✅
Build destination-milvus docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌
Code format checks	✅
Validate airbyte-integrations/connectors/destination-milvus/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-milvus test

airbyte-oss-build-runner · 2023-09-25T17:01:32Z

destination-qdrant test report (commit `25fc12e857`) - ✅

⏲️ Total pipeline duration: 01mn21s

Step	Result
Connector package install	✅
Build destination-qdrant docker image for platform linux/x86_64	✅
Unit tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/destination-qdrant/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-qdrant test

airbyte-oss-build-runner · 2023-09-25T17:01:48Z

destination-pinecone test report (commit `25fc12e857`) - ✅

⏲️ Total pipeline duration: 02mn24s

Step	Result
Connector package install	✅
Build destination-pinecone docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/destination-pinecone/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-pinecone test

airbyte-oss-build-runner · 2023-09-25T17:02:03Z

destination-milvus test report (commit `25fc12e857`) - ❌

⏲️ Total pipeline duration: 02mn40s

Step	Result
Connector package install	✅
Build destination-milvus docker image for platform linux/x86_64	✅
Unit tests	❌
Code format checks	✅
Validate airbyte-integrations/connectors/destination-milvus/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-milvus test

…update

…e-fix

airbyte-oss-build-runner · 2023-09-26T08:24:56Z

destination-milvus test report (commit `97fdd834b7`) - ❌

⏲️ Total pipeline duration: 03mn56s

Step	Result
Connector package install	✅
Build destination-milvus docker image for platform linux/x86_64	✅
Unit tests	❌
Code format checks	✅
Validate airbyte-integrations/connectors/destination-milvus/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-milvus test

airbyte-oss-build-runner · 2023-09-26T08:24:57Z

destination-pinecone test report (commit `97fdd834b7`) - ✅

⏲️ Total pipeline duration: 04mn09s

Step	Result
Connector package install	✅
Build destination-pinecone docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/destination-pinecone/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-pinecone test

airbyte-oss-build-runner · 2023-09-26T08:24:58Z

destination-qdrant test report (commit `97fdd834b7`) - ✅

⏲️ Total pipeline duration: 04mn07s

Step	Result
Connector package install	✅
Build destination-qdrant docker image for platform linux/x86_64	✅
Unit tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/destination-qdrant/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-qdrant test

aaronsteers

I performed a full code readthrough. A couple questions but no blockers. 👍

aaronsteers · 2023-09-26T08:30:33Z

airbyte-integrations/connectors/destination-qdrant/metadata.yaml

+      - "${indexing.url}"
+      - api.openai.com
+      - api.cohere.ai
+      - "${embedding.api_base}"


Just for my own understanding, I'm curious how these dynamic refs are parsed. I see embedding.api_base matches the config setting name but I'm not sure about indexing.url.

It's referencing this config value:

airbyte/airbyte-integrations/connectors/destination-qdrant/destination_qdrant/config.py

Lines 30 to 31 in 97fdd83

aaronsteers · 2023-09-26T08:38:28Z

airbyte-integrations/connectors/destination-milvus/destination_milvus/destination.py

    def _init_indexer(self, config: ConfigModel):
-        self.embedder = embedder_map[config.embedding.mode](config.embedding)
+        if config.embedding.mode == "azure_openai" or config.embedding.mode == "openai":
+            self.embedder = embedder_map[config.embedding.mode](config.embedding, config.processing.chunk_size)
+        else:
+            self.embedder = embedder_map[config.embedding.mode](config.embedding)
        self.indexer = MilvusIndexer(config.indexing, self.embedder.embedding_dimensions)


Should this and embedder_map be moved into the CDK?

Not a blocker for this round, but it seems it would be good to have this logic centrally managed. (Also appears for Milvus.)

Yeah, I think that would make sense. I'll create an issue for that

aaronsteers · 2023-09-26T08:48:05Z

airbyte-integrations/connectors/destination-milvus/destination_milvus/indexer.py

+        for key, value in metadata.items():
+            normalized_key = key
+            # the primary key can't be set directly with auto_id, so we prefix it with an underscore
+            if key == self._primary_key:
+                normalized_key = f"_{key}"
+            result[normalized_key] = value


I'm not sure I 100% follow the issue or the mitigation. I understand that we can't modify the primary key when it is of type auto_id - so are we setting a different key using the underscore? Is it in the case of the 'natural' primary key of the stream colliding with a built-in column name?

Does the problem become less if (in the future) we generate our own unique keys instead of using auto_id?

Seems like this is not a big issue either way (as long as we stop the failure scenario) so I'm not asking to change anything - just would be good for me to understand longer-term.

Thanks.

The Milvus collection has to be set to auto_id, but if that's the case Milvus will complain if an entity contains a value for the primary key (as it will pick a value itself). So let's say you create a collection with primary key id set to auto_id, but a record coming from some source contains an id field as well, it will error.

This field changes the logic so the id field from the Airbyte record is stored as _id in the collection. This is necessary because one record might be split into multiple entities, so _id won't be guaranteed to be unique.

…update

airbyte-oss-build-runner · 2023-09-26T09:09:59Z

destination-qdrant test report (commit `15fde9abfa`) - ✅

⏲️ Total pipeline duration: 01mn29s

Step	Result
Connector package install	✅
Build destination-qdrant docker image for platform linux/x86_64	✅
Unit tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/destination-qdrant/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-qdrant test

airbyte-oss-build-runner · 2023-09-26T09:10:00Z

destination-milvus test report (commit `15fde9abfa`) - ❌

⏲️ Total pipeline duration: 01mn41s

Step	Result
Connector package install	✅
Build destination-milvus docker image for platform linux/x86_64	✅
Unit tests	❌
Code format checks	✅
Validate airbyte-integrations/connectors/destination-milvus/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-milvus test

airbyte-oss-build-runner · 2023-09-26T09:10:11Z

destination-pinecone test report (commit `15fde9abfa`) - ✅

⏲️ Total pipeline duration: 02mn25s

Step	Result
Connector package install	✅
Build destination-pinecone docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/destination-pinecone/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-pinecone test

airbyte-oss-build-runner · 2023-09-26T09:37:21Z

destination-qdrant test report (commit `fcdc9a2db4`) - ✅

⏲️ Total pipeline duration: 01mn51s

Step	Result
Connector package install	✅
Build destination-qdrant docker image for platform linux/x86_64	✅
Unit tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/destination-qdrant/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-qdrant test

airbyte-oss-build-runner · 2023-09-26T09:37:25Z

destination-pinecone test report (commit `fcdc9a2db4`) - ✅

⏲️ Total pipeline duration: 01mn59s

Step	Result
Connector package install	✅
Build destination-pinecone docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/destination-pinecone/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-pinecone test

airbyte-oss-build-runner · 2023-09-26T09:38:07Z

destination-milvus test report (commit `fcdc9a2db4`) - ✅

⏲️ Total pipeline duration: 02mn43s

Step	Result
Connector package install	✅
Build destination-milvus docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/destination-milvus/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-milvus test

airbyte-oss-build-runner · 2023-09-26T09:48:16Z

destination-qdrant test report (commit `c7c7ef3ecf`) - ✅

⏲️ Total pipeline duration: 01mn23s

Step	Result
Connector package install	✅
Build destination-qdrant docker image for platform linux/x86_64	✅
Unit tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/destination-qdrant/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-qdrant test

airbyte-oss-build-runner · 2023-09-26T09:48:26Z

destination-pinecone test report (commit `c7c7ef3ecf`) - ✅

⏲️ Total pipeline duration: 02mn16s

Step	Result
Connector package install	✅
Build destination-pinecone docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/destination-pinecone/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-pinecone test

airbyte-oss-build-runner · 2023-09-26T09:48:31Z

destination-milvus test report (commit `c7c7ef3ecf`) - ✅

⏲️ Total pipeline duration: 02mn21s

Step	Result
Connector package install	✅
Build destination-milvus docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/destination-milvus/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-milvus test

Joe Reuter added 2 commits September 22, 2023 13:54

adjust

0913a2c

resource requirements

4b910b1

octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/destination/milvus connectors/destination/qdrant labels Sep 22, 2023

Joe Reuter added 2 commits September 22, 2023 15:07

Update milvus.md

36c5305

Update qdrant.md

feef522

vercel bot deployed to Preview September 22, 2023 13:11 View deployment

Automated Commit - Formatting Changes

956cd4a

octavia-approvington requested review from a team as code owners September 22, 2023 13:31

Joe Reuter added 4 commits September 26, 2023 10:13

Merge remote-tracking branch 'upstream/master' into flash1293/milvus-…

ef0c9db

…update

roll bugfix into update pr

96003e6

Merge remote-tracking branch 'upstream/master' into flash1293/pinecon…

0f1ab49

…e-fix

Merge branch 'flash1293/pinecone-fix' into flash1293/milvus-update

97fdd83

flash1293 changed the base branch from master to flash1293/pinecone-fix September 26, 2023 08:18

flash1293 mentioned this pull request Sep 26, 2023

Milvus connector: Normalize primary key field #30725

Closed

aaronsteers approved these changes Sep 26, 2023

View reviewed changes

aaronsteers reviewed Sep 26, 2023

View reviewed changes

Base automatically changed from flash1293/pinecone-fix to master September 26, 2023 09:05

Merge remote-tracking branch 'upstream/master' into flash1293/milvus-…

15fde9a

…update

fix test

fcdc9a2

set to certified

c7c7ef3

vercel bot deployed to Preview September 26, 2023 09:46 View deployment

flash1293 merged commit e100c27 into master Sep 26, 2023
27 checks passed

flash1293 deleted the flash1293/milvus-update branch September 26, 2023 10:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Milvus / Qdrant / Pinecone: Update CDK #30689

Milvus / Qdrant / Pinecone: Update CDK #30689

flash1293 commented Sep 22, 2023 •

edited

vercel bot commented Sep 22, 2023 •

edited

airbyte-oss-build-runner commented Sep 22, 2023

airbyte-oss-build-runner commented Sep 22, 2023

airbyte-oss-build-runner commented Sep 25, 2023

airbyte-oss-build-runner commented Sep 25, 2023

airbyte-oss-build-runner commented Sep 25, 2023

airbyte-oss-build-runner commented Sep 26, 2023

airbyte-oss-build-runner commented Sep 26, 2023

airbyte-oss-build-runner commented Sep 26, 2023

aaronsteers left a comment

aaronsteers Sep 26, 2023

flash1293 Sep 26, 2023

aaronsteers Sep 26, 2023

flash1293 Sep 26, 2023

aaronsteers Sep 26, 2023

flash1293 Sep 26, 2023

airbyte-oss-build-runner commented Sep 26, 2023

airbyte-oss-build-runner commented Sep 26, 2023

airbyte-oss-build-runner commented Sep 26, 2023

airbyte-oss-build-runner commented Sep 26, 2023

airbyte-oss-build-runner commented Sep 26, 2023

airbyte-oss-build-runner commented Sep 26, 2023

airbyte-oss-build-runner commented Sep 26, 2023

airbyte-oss-build-runner commented Sep 26, 2023

airbyte-oss-build-runner commented Sep 26, 2023

Milvus / Qdrant / Pinecone: Update CDK #30689

Milvus / Qdrant / Pinecone: Update CDK #30689

Conversation

flash1293 commented Sep 22, 2023 • edited

What

vercel bot commented Sep 22, 2023 • edited

airbyte-oss-build-runner commented Sep 22, 2023

destination-qdrant test report (commit feef5222a6) - ✅

airbyte-oss-build-runner commented Sep 22, 2023

destination-milvus test report (commit feef5222a6) - ❌

airbyte-oss-build-runner commented Sep 25, 2023

destination-qdrant test report (commit 25fc12e857) - ✅

airbyte-oss-build-runner commented Sep 25, 2023

destination-pinecone test report (commit 25fc12e857) - ✅

airbyte-oss-build-runner commented Sep 25, 2023

destination-milvus test report (commit 25fc12e857) - ❌

airbyte-oss-build-runner commented Sep 26, 2023

destination-milvus test report (commit 97fdd834b7) - ❌

airbyte-oss-build-runner commented Sep 26, 2023

destination-pinecone test report (commit 97fdd834b7) - ✅

airbyte-oss-build-runner commented Sep 26, 2023

destination-qdrant test report (commit 97fdd834b7) - ✅

aaronsteers left a comment

Choose a reason for hiding this comment

aaronsteers Sep 26, 2023

Choose a reason for hiding this comment

flash1293 Sep 26, 2023

Choose a reason for hiding this comment

aaronsteers Sep 26, 2023

Choose a reason for hiding this comment

flash1293 Sep 26, 2023

Choose a reason for hiding this comment

aaronsteers Sep 26, 2023

Choose a reason for hiding this comment

flash1293 Sep 26, 2023

Choose a reason for hiding this comment

airbyte-oss-build-runner commented Sep 26, 2023

destination-qdrant test report (commit 15fde9abfa) - ✅

airbyte-oss-build-runner commented Sep 26, 2023

destination-milvus test report (commit 15fde9abfa) - ❌

airbyte-oss-build-runner commented Sep 26, 2023

destination-pinecone test report (commit 15fde9abfa) - ✅

airbyte-oss-build-runner commented Sep 26, 2023

destination-qdrant test report (commit fcdc9a2db4) - ✅

airbyte-oss-build-runner commented Sep 26, 2023

destination-pinecone test report (commit fcdc9a2db4) - ✅

airbyte-oss-build-runner commented Sep 26, 2023

destination-milvus test report (commit fcdc9a2db4) - ✅

airbyte-oss-build-runner commented Sep 26, 2023

destination-qdrant test report (commit c7c7ef3ecf) - ✅

airbyte-oss-build-runner commented Sep 26, 2023

destination-pinecone test report (commit c7c7ef3ecf) - ✅

airbyte-oss-build-runner commented Sep 26, 2023

destination-milvus test report (commit c7c7ef3ecf) - ✅

flash1293 commented Sep 22, 2023 •

edited

vercel bot commented Sep 22, 2023 •

edited

destination-qdrant test report (commit `feef5222a6`) - ✅

destination-milvus test report (commit `feef5222a6`) - ❌

destination-qdrant test report (commit `25fc12e857`) - ✅

destination-pinecone test report (commit `25fc12e857`) - ✅

destination-milvus test report (commit `25fc12e857`) - ❌

destination-milvus test report (commit `97fdd834b7`) - ❌

destination-pinecone test report (commit `97fdd834b7`) - ✅

destination-qdrant test report (commit `97fdd834b7`) - ✅

destination-qdrant test report (commit `15fde9abfa`) - ✅

destination-milvus test report (commit `15fde9abfa`) - ❌

destination-pinecone test report (commit `15fde9abfa`) - ✅

destination-qdrant test report (commit `fcdc9a2db4`) - ✅

destination-pinecone test report (commit `fcdc9a2db4`) - ✅

destination-milvus test report (commit `fcdc9a2db4`) - ✅

destination-qdrant test report (commit `c7c7ef3ecf`) - ✅

destination-pinecone test report (commit `c7c7ef3ecf`) - ✅

destination-milvus test report (commit `c7c7ef3ecf`) - ✅