Vector DB Destinations: Fix OpenAI batch size and Weaviate field name collision #31075

flash1293 · 2023-10-04T20:01:13Z

This PR fixes two problems:

Rolling out Vector DB CDK: Fix chunk size for openai embedder #31067
Disambiguating fields properly in Weaviate to not cause field names that are incompatible with GraphQL (as it's commonly used in Weaviate)

vercel · 2023-10-04T20:01:19Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 6, 2023 4:23pm

github-actions · 2023-10-04T20:01:36Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

airbyte-oss-build-runner · 2023-10-04T20:06:33Z

destination-weaviate test report (commit `881c21fcfc`) - ✅

⏲️ Total pipeline duration: 02mn32s

Step	Result
Connector package install	✅
Build destination-weaviate docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for destination-weaviate	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-weaviate test

airbyte-oss-build-runner · 2023-10-06T16:27:43Z

destination-qdrant test report (commit `01c3e28eff`) - ✅

⏲️ Total pipeline duration: 04mn34s

Step	Result
Connector package install	✅
Build destination-qdrant docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Code format checks	✅
Validate metadata for destination-qdrant	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-qdrant test

airbyte-oss-build-runner · 2023-10-06T16:27:44Z

destination-chroma test report (commit `01c3e28eff`) - ✅

⏲️ Total pipeline duration: 04mn43s

Step	Result
Connector package install	✅
Build destination-chroma docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Code format checks	✅
Validate metadata for destination-chroma	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-chroma test

airbyte-oss-build-runner · 2023-10-06T16:27:58Z

destination-weaviate test report (commit `01c3e28eff`) - ✅

⏲️ Total pipeline duration: 05mn29s

Step	Result
Connector package install	✅
Build destination-weaviate docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for destination-weaviate	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-weaviate test

airbyte-oss-build-runner · 2023-10-06T16:27:59Z

destination-milvus test report (commit `01c3e28eff`) - ✅

⏲️ Total pipeline duration: 05mn30s

Step	Result
Connector package install	✅
Build destination-milvus docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for destination-milvus	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-milvus test

airbyte-oss-build-runner · 2023-10-06T16:28:00Z

destination-pinecone test report (commit `01c3e28eff`) - ✅

⏲️ Total pipeline duration: 05mn31s

Step	Result
Connector package install	✅
Build destination-pinecone docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for destination-pinecone	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-pinecone test

pedroslopez · 2023-10-06T17:28:48Z

airbyte-integrations/connectors/destination-weaviate/destination_weaviate/indexer.py

@@ -146,7 +146,7 @@ def _normalize(self, metadata: dict) -> dict:
            normalized_key = key[0].lower() + key[1:]
            # "id" and "additional" are reserved properties in Weaviate, prefix to disambiguate
            if key == "id" or key == "_id" or key == "_additional":
-                normalized_key = f"__{key}"


is this breaking?

… collision (#31075)

fix weaviate problem

881c21f

octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/destination/weaviate labels Oct 4, 2023

vercel bot deployed to Preview October 4, 2023 20:04 View deployment

update cdk

01c3e28

octavia-squidington-iii added connectors/destination/milvus connectors/destination/pinecone connectors/destination/qdrant connectors/destination/chroma labels Oct 6, 2023

vercel bot deployed to Preview October 6, 2023 16:23 View deployment

pedroslopez approved these changes Oct 6, 2023

View reviewed changes

flash1293 merged commit 64964cc into master Oct 6, 2023
31 of 36 checks passed

flash1293 deleted the flash1293/fix-openai-batch-size-connectors branch October 6, 2023 21:18

edgao pushed a commit that referenced this pull request Oct 11, 2023

Vector DB Destinations: Fix OpenAI batch size and Weaviate field name…

882f917

… collision (#31075)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Vector DB Destinations: Fix OpenAI batch size and Weaviate field name collision #31075

Vector DB Destinations: Fix OpenAI batch size and Weaviate field name collision #31075

flash1293 commented Oct 4, 2023 •

edited

vercel bot commented Oct 4, 2023 •

edited

github-actions bot commented Oct 4, 2023 •

edited by flash1293

airbyte-oss-build-runner commented Oct 4, 2023

airbyte-oss-build-runner commented Oct 6, 2023

airbyte-oss-build-runner commented Oct 6, 2023

airbyte-oss-build-runner commented Oct 6, 2023

airbyte-oss-build-runner commented Oct 6, 2023

airbyte-oss-build-runner commented Oct 6, 2023

pedroslopez Oct 6, 2023

Vector DB Destinations: Fix OpenAI batch size and Weaviate field name collision #31075

Vector DB Destinations: Fix OpenAI batch size and Weaviate field name collision #31075

Conversation

flash1293 commented Oct 4, 2023 • edited

vercel bot commented Oct 4, 2023 • edited

github-actions bot commented Oct 4, 2023 • edited by flash1293

Before Merging a Connector Pull Request

airbyte-oss-build-runner commented Oct 4, 2023

destination-weaviate test report (commit 881c21fcfc) - ✅

airbyte-oss-build-runner commented Oct 6, 2023

destination-qdrant test report (commit 01c3e28eff) - ✅

airbyte-oss-build-runner commented Oct 6, 2023

destination-chroma test report (commit 01c3e28eff) - ✅

airbyte-oss-build-runner commented Oct 6, 2023

destination-weaviate test report (commit 01c3e28eff) - ✅

airbyte-oss-build-runner commented Oct 6, 2023

destination-milvus test report (commit 01c3e28eff) - ✅

airbyte-oss-build-runner commented Oct 6, 2023

destination-pinecone test report (commit 01c3e28eff) - ✅

pedroslopez Oct 6, 2023

Choose a reason for hiding this comment

flash1293 commented Oct 4, 2023 •

edited

vercel bot commented Oct 4, 2023 •

edited

github-actions bot commented Oct 4, 2023 •

edited by flash1293

destination-weaviate test report (commit `881c21fcfc`) - ✅

destination-qdrant test report (commit `01c3e28eff`) - ✅

destination-chroma test report (commit `01c3e28eff`) - ✅

destination-weaviate test report (commit `01c3e28eff`) - ✅

destination-milvus test report (commit `01c3e28eff`) - ✅

destination-pinecone test report (commit `01c3e28eff`) - ✅