🚨 Weaviate destination: Add embedding capabilities, overwrite and dedup support, API key auth mode and available on Airbyte Cloud #30151

flash1293 · 2023-09-05T10:14:31Z

What

Fixes #29663 and brings Weaviate destination to Airbyte Cloud.

By aligning the Weaviate destination with other vector db destinations.

How

Restructure the destination based on Pinecone / Milvus:

Use vector db CDK
Implement custom indexer

Reading order:

config.py
destination.py
indexer.py
integration_test.py

Special cases:

Add "no embedding" as embedding option if this part is handled in Weaviate itself
Normalize complex fields by serializing as JSON
Rename field names to always start with a lowercase letter (this is necessary in Weaviate)
Internally retry failed loads within the batch - I kept this from the previous code base, not sure how relevant in practice
Do not allow non-https connections on cloud specifically
Removed the old integration test that were mostly focused around the different schema handling - just rely on properly created schema in the destination or on auto-creating being enabled
The internal fields _ab_stream and _ab_record_id are queried to delete existing data in the class, but this only works if the fields exist already - as they will only be created after data gets loaded the first time, the connector checks whether they exist and skip the deletion step if they don't exist.

🚨 User Impact 🚨

Several breaking changes are made in this PR - check out weaviate_migrations.md for the details: https://github.com/airbytehq/airbyte/pull/30151/files#diff-6ce12b19886f87d02ff8a5b2141ca99fbfd0284ee8a857a833c6d5bb15e85e14

github-actions · 2023-09-05T10:15:06Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

…rewrite

…byte into flash1293/weaviate-rewrite

…rewrite

…byte into flash1293/weaviate-rewrite

…provements

…rewrite

airbyte-oss-build-runner · 2023-09-25T14:10:29Z

destination-weaviate test report (commit `c7497eb06a`) - ✅

⏲️ Total pipeline duration: 02mn48s

Step	Result
Connector package install	✅
Build destination-weaviate docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/destination-weaviate/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-weaviate test

airbyte-oss-build-runner · 2023-09-25T16:56:33Z

destination-weaviate test report (commit `3a6b0e2544`) - ❌

⏲️ Total pipeline duration: 01mn55s

Step	Result
Connector package install	✅
Build destination-weaviate docker image for platform linux/x86_64	✅
Unit tests	❌
Code format checks	✅
Validate airbyte-integrations/connectors/destination-weaviate/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-weaviate test

airbyte-oss-build-runner · 2023-09-25T17:04:05Z

destination-weaviate test report (commit `47c682dffe`) - ✅

⏲️ Total pipeline duration: 01mn54s

Step	Result
Connector package install	✅
Build destination-weaviate docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/destination-weaviate/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-weaviate test

…e-rewrite

airbyte-oss-build-runner · 2023-09-26T10:22:00Z

destination-weaviate test report (commit `828855dd05`) - ✅

⏲️ Total pipeline duration: 02mn42s

Step	Result
Connector package install	✅
Build destination-weaviate docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/destination-weaviate/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-weaviate test

airbyte-oss-build-runner · 2023-09-27T08:41:33Z

destination-weaviate test report (commit `e7de8516e9`) - ✅

⏲️ Total pipeline duration: 01mn39s

Step	Result
Connector package install	✅
Build destination-weaviate docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/destination-weaviate/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-weaviate test

aaronsteers

Nice work on this - it's a significant rewrite.

I did a full readthrough of the code and I don't see any major issues. I added some notes/suggestions inline for your review. 👍

aaronsteers · 2023-09-27T10:38:58Z

airbyte-integrations/connectors/destination-weaviate/Dockerfile

Should this logic be moved to a post-build shell script? I heard Dockerfiles are on their way out.

(Just a question for consideration; I wouldn't block on this.)

To my knowledge this new Dockerfile is closer to the "default logic", so I expect it to not cause any problems.

docs/integrations/destinations/weaviate.md

docs/integrations/destinations/weaviate-migrations.md

Co-authored-by: Aaron ("AJ") Steers <aj@airbyte.io>

airbyte-oss-build-runner · 2023-09-27T15:01:45Z

destination-weaviate test report (commit `f1c33a36ed`) - ✅

⏲️ Total pipeline duration: 03mn43s

Step	Result
Connector package install	✅
Build destination-weaviate docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/destination-weaviate/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-weaviate test

…byte into flash1293/weaviate-rewrite

airbyte-oss-build-runner · 2023-09-27T17:21:35Z

destination-weaviate test report (commit `359cd06f68`) - ✅

⏲️ Total pipeline duration: 03mn42s

Step	Result
Connector package install	✅
Build destination-weaviate docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/destination-weaviate/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-weaviate test

airbyte-oss-build-runner · 2023-09-28T08:37:37Z

destination-weaviate test report (commit `8c6c583397`) - ❌

⏲️ Total pipeline duration: 01mn55s

Step	Result
Connector package install	✅
Build destination-weaviate docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	❌
Validate airbyte-integrations/connectors/destination-weaviate/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-weaviate test

airbyte-oss-build-runner · 2023-09-28T08:57:02Z

destination-weaviate test report (commit `f8d9237765`) - ✅

⏲️ Total pipeline duration: 01mn55s

Step	Result
Connector package install	✅
Build destination-weaviate docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/destination-weaviate/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-weaviate test

…up support, API key auth mode and available on Airbyte Cloud (#30151) Co-authored-by: flash1293 <flash1293@users.noreply.github.com> Co-authored-by: Aaron ("AJ") Steers <aj@airbyte.io>

Joe Reuter added 5 commits September 4, 2023 10:20

better error message for misconfigured text fields

83c97bb

improve message

de0cf30

improve error message

5a88145

refactor vector db cdk helpers a bit

6c74055

make most of the things work

6b7a3b0

flash1293 changed the title ~~Weaviate destination: Add embedding capabilities and API key auth mode~~ Weaviate destination: Add embedding capabilities, overwrite and dedup support and API key auth mode Sep 5, 2023

octavia-squidington-iii added area/connectors Connector related issues CDK Connector Development Kit connectors/destination/weaviate labels Sep 5, 2023

flash1293 and others added 5 commits September 5, 2023 10:20

Automated Commit - Formatting Changes

2ff7617

adjust component interaction

a414d88

Merge branch 'flash1293/ai-cdk-improvements' into flash1293/weaviate-…

3cc529a

…rewrite

Merge branch 'flash1293/weaviate-rewrite' of github.com:airbytehq/air…

b47eadf

…byte into flash1293/weaviate-rewrite

make no_embedding mode work

139c0ab

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Sep 5, 2023

flash1293 and others added 4 commits September 5, 2023 11:06

Automated Commit - Formatting Changes

51ca514

format fixes

347f333

Merge branch 'flash1293/ai-cdk-improvements' into flash1293/weaviate-…

5d76df2

…rewrite

Merge branch 'flash1293/weaviate-rewrite' of github.com:airbytehq/air…

52c2095

…byte into flash1293/weaviate-rewrite

flash1293 changed the title ~~Weaviate destination: Add embedding capabilities, overwrite and dedup support and API key auth mode~~ 🚨 Weaviate destination: Add embedding capabilities, overwrite and dedup support and API key auth mode Sep 5, 2023

Joe Reuter added 9 commits September 5, 2023 17:16

docs, auto-create classes

17a5c30

review comments

a5c5ff8

Merge remote-tracking branch 'origin/master' into flash1293/ai-cdk-im…

716abc8

…provements

revert formatting

bd0f3c2

fix unit tests

3ce6334

Merge branch 'flash1293/ai-cdk-improvements' into flash1293/weaviate-…

db2f04e

…rewrite

work on tests

aa8461a

work on tests

4084ddb

Merge remote-tracking branch 'origin/master' into flash1293/weaviate-…

3c2f06c

…rewrite

disallow no auth on cloud

3a6b0e2

fix test

47c682d

Joe Reuter added 2 commits September 26, 2023 12:13

Merge remote-tracking branch 'upstream/master' into flash1293/weaviat…

ab9dcdd

…e-rewrite

set to certified

828855d

vercel bot deployed to Preview September 26, 2023 10:19 View deployment

Merge branch 'master' into flash1293/weaviate-rewrite

e7de851

vercel bot deployed to Preview September 27, 2023 08:39 View deployment

aaronsteers approved these changes Sep 27, 2023

View reviewed changes

Joe Reuter and others added 3 commits September 27, 2023 16:55

Update docs/integrations/destinations/weaviate.md

c7a0b8f

Co-authored-by: Aaron ("AJ") Steers <aj@airbyte.io>

Update docs/integrations/destinations/weaviate-migrations.md

751025e

Co-authored-by: Aaron ("AJ") Steers <aj@airbyte.io>

Update docs/integrations/destinations/weaviate-migrations.md

f1c33a3

Co-authored-by: Aaron ("AJ") Steers <aj@airbyte.io>

vercel bot deployed to Preview September 27, 2023 14:58 View deployment

Joe Reuter added 2 commits September 27, 2023 19:12

update cdk

68f7a94

Merge branch 'flash1293/weaviate-rewrite' of github.com:airbytehq/air…

359cd06

…byte into flash1293/weaviate-rewrite

vercel bot deployed to Preview September 27, 2023 17:17 View deployment

chunk as configured

8c6c583

fix format

f8d9237

flash1293 merged commit f161e2e into master Sep 28, 2023
25 checks passed

flash1293 deleted the flash1293/weaviate-rewrite branch September 28, 2023 09:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🚨 Weaviate destination: Add embedding capabilities, overwrite and dedup support, API key auth mode and available on Airbyte Cloud #30151

🚨 Weaviate destination: Add embedding capabilities, overwrite and dedup support, API key auth mode and available on Airbyte Cloud #30151

flash1293 commented Sep 5, 2023 •

edited

github-actions bot commented Sep 5, 2023 •

edited by flash1293

airbyte-oss-build-runner commented Sep 25, 2023

airbyte-oss-build-runner commented Sep 25, 2023

airbyte-oss-build-runner commented Sep 25, 2023

airbyte-oss-build-runner commented Sep 26, 2023

airbyte-oss-build-runner commented Sep 27, 2023

aaronsteers left a comment

aaronsteers Sep 27, 2023

flash1293 Sep 27, 2023

airbyte-oss-build-runner commented Sep 27, 2023

airbyte-oss-build-runner commented Sep 27, 2023

airbyte-oss-build-runner commented Sep 28, 2023

airbyte-oss-build-runner commented Sep 28, 2023

🚨 Weaviate destination: Add embedding capabilities, overwrite and dedup support, API key auth mode and available on Airbyte Cloud #30151

🚨 Weaviate destination: Add embedding capabilities, overwrite and dedup support, API key auth mode and available on Airbyte Cloud #30151

Conversation

flash1293 commented Sep 5, 2023 • edited

What

How

🚨 User Impact 🚨

github-actions bot commented Sep 5, 2023 • edited by flash1293

Before Merging a Connector Pull Request

airbyte-oss-build-runner commented Sep 25, 2023

destination-weaviate test report (commit c7497eb06a) - ✅

airbyte-oss-build-runner commented Sep 25, 2023

destination-weaviate test report (commit 3a6b0e2544) - ❌

airbyte-oss-build-runner commented Sep 25, 2023

destination-weaviate test report (commit 47c682dffe) - ✅

airbyte-oss-build-runner commented Sep 26, 2023

destination-weaviate test report (commit 828855dd05) - ✅

airbyte-oss-build-runner commented Sep 27, 2023

destination-weaviate test report (commit e7de8516e9) - ✅

aaronsteers left a comment

Choose a reason for hiding this comment

aaronsteers Sep 27, 2023

Choose a reason for hiding this comment

flash1293 Sep 27, 2023

Choose a reason for hiding this comment

airbyte-oss-build-runner commented Sep 27, 2023

destination-weaviate test report (commit f1c33a36ed) - ✅

airbyte-oss-build-runner commented Sep 27, 2023

destination-weaviate test report (commit 359cd06f68) - ✅

airbyte-oss-build-runner commented Sep 28, 2023

destination-weaviate test report (commit 8c6c583397) - ❌

airbyte-oss-build-runner commented Sep 28, 2023

destination-weaviate test report (commit f8d9237765) - ✅

flash1293 commented Sep 5, 2023 •

edited

github-actions bot commented Sep 5, 2023 •

edited by flash1293

destination-weaviate test report (commit `c7497eb06a`) - ✅

destination-weaviate test report (commit `3a6b0e2544`) - ❌

destination-weaviate test report (commit `47c682dffe`) - ✅

destination-weaviate test report (commit `828855dd05`) - ✅

destination-weaviate test report (commit `e7de8516e9`) - ✅

destination-weaviate test report (commit `f1c33a36ed`) - ✅

destination-weaviate test report (commit `359cd06f68`) - ✅

destination-weaviate test report (commit `8c6c583397`) - ❌

destination-weaviate test report (commit `f8d9237765`) - ✅