🎉 New Destination: Vectara (Vector Database) #31958

topefolorunso · 2023-10-30T13:49:52Z

What

Adding a new vector destination - Vectara resolves #29012

How

Generated the Python destination boilerplate codes and wrote the logic for connecting and writing to the database

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user?

For connector PRs, use this section to explain which type of semantic versioning bump occurs as a result of the changes. Refer to our Semantic Versioning for Connectors guidelines for more information. Breaking changes to connectors must be documented by an Airbyte engineer (PR author, or reviewer for community PRs) by using the Breaking Change Release Playbook.

If there are breaking changes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Actions

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Community member? Grant edit access to maintainers (instructions)
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Connector version is set to 0.0.1
- Dockerfile has version 0.0.1
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- docs/integrations/<source or destination>/<name>.md including changelog with an entry for the initial version. See changelog example
- docs/integrations/README.md

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

…ctara-destination

vercel · 2023-10-30T13:49:58Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Dec 13, 2023 7:10pm

github-actions · 2023-10-30T13:50:15Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

airbyte-integrations/connectors/destination-vectara/destination_vectara/indexer.py

flash1293

@topefolorunso Is this PR ready to review? I took a basic look and the following things seem off:

There is still stuff around from the vector db cdk usage
When building the docker image, the spec command is failing with ModuleNotFoundError: No module named 'langchain' (most likely related to the above)
Basic documentation on how to set up a destination this way
I'm not sure whether OAuth is the right approach here - isn't this more of a use case for API keys? I will reach out to the vectara devs for this question

flash1293 · 2023-11-08T15:18:24Z

@topefolorunso Just chatted with vectara devs and going with oauth only for now makes sense. Please let me know once I can take a look, really excited for this connector :)

topefolorunso · 2023-11-08T22:18:45Z

@topefolorunso Is this PR ready to review?

@flash1293 No it is not, still working on it. I will take note of your comments and update once complete. Please bear with me. Thanks.

…ctara-destination

topefolorunso · 2023-11-10T15:28:22Z

Please review now @flash1293

flash1293

Besides some smaller notes, this PR doesn't implement this part specified in the issue:

To map a record to a document, the user should configure text fields and metadata fields as part of the connector configuration.

For each text field, create a corresponding section of the document, and copy the text of the field into that section.
Set metadata on the section, e.g. source_type with the value being the name of the text field.

Ideally, the corpus needs to be created with source_type specified as a filterable column. This will allow customers to run searches over some columns, and exclude others.

Fields specified as metadata field will be attached to the document as metadata.

It shouldn't just drop everything the record as-is into the first section, there should be a user-configurable text vs. metadata field configuration like in the vector db destinations (just two fields with array of strings).

Then, each text field should become the text field of one seaprate section in the document sent to vectara and all the metadata fields should be added to the metadatajson (not just the stream name).

Also, if a stream has a configured primary key, it should be used as the documentId, concatenating all of these:

Namespace
Stream name
all parts of the primary key

This needs to be done to make sure if the record gets updated and loaded again, it will replace the old version instead of creating a new document.

airbyte-integrations/connectors/destination-vectara/destination_vectara/destination.py

flash1293 · 2023-11-13T10:46:53Z

airbyte-integrations/connectors/destination-vectara/unit_tests/test_destination.py

+from destination_vectara.destination import DestinationVectara
+
+
+class TestDestinationVectara(unittest.TestCase):


These tests need to be updated?

airbyte-integrations/connectors/destination-vectara/destination_vectara/writer.py

flash1293 · 2023-11-13T10:48:14Z

airbyte-integrations/connectors/destination-vectara/icon.jpg

This needs to be an svg

airbyte-integrations/connectors/destination-vectara/destination_vectara/client.py

marcosmarxm · 2023-12-05T19:08:55Z

@topefolorunso are you still planning to work on this?

…ctara-destination

topefolorunso · 2023-12-10T00:59:21Z

@topefolorunso are you still planning to work on this?

Yes I will work on it @marcosmarxm.

flash1293 · 2023-12-10T07:34:56Z

@topefolorunso as it laid bare a while, @ofermend from Vectara took over to bring it I’ve Ruhe finish line.

…ctara-destination

topefolorunso · 2023-12-13T19:13:04Z

@topefolorunso as it laid bare a while, @ofermend from Vectara took over to bring it I’ve Ruhe finish line.

Oh alright. That's fine I guess. Well done @ofermend.
Sorry for the delay on my part. Though I just pushed some changes just in case.

ofermend · 2023-12-14T20:46:05Z

Thanks @topefolorunso - will pull your latest and try to get it to the finale. Appreciate your work so far and I'll reach out if I need any further help here.

marcosmarxm · 2023-12-19T10:44:08Z

Close in favor of #33616

topefolorunso added 5 commits October 12, 2023 13:13

generate boilerplate

18b90ef

Merge branch 'master' of https://github.com/airbytehq/airbyte into ve…

f0abc58

…ctara-destination

add vectara destination

1c23a8b

add unit test, fix code

a0da857

Merge branch 'master' of https://github.com/airbytehq/airbyte into ve…

0f2320e

…ctara-destination

octavia-squidington-iii added area/connectors Connector related issues community connectors/destination/vectara labels Oct 30, 2023

vercel bot deployed to Preview October 30, 2023 13:51 View deployment

marcosmarxm requested a review from flash1293 October 30, 2023 14:00

marcosmarxm assigned flash1293 Oct 30, 2023

flash1293 suggested changes Oct 30, 2023

View reviewed changes

airbyte-integrations/connectors/destination-vectara/destination_vectara/indexer.py Outdated Show resolved Hide resolved

marcosmarxm added hackathon-2023-10 new-destination labels Oct 30, 2023

revert to regular destination cdk

d143ed5

flash1293 reviewed Nov 8, 2023

View reviewed changes

fix vectara writer and client

995670a

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Nov 10, 2023

Merge branch 'master' of https://github.com/airbytehq/airbyte into ve…

dc3c30a

…ctara-destination

topefolorunso requested a review from flash1293 November 10, 2023 15:28

vercel bot deployed to Preview November 10, 2023 15:31 View deployment

make config pretty

47f4482

flash1293 suggested changes Nov 13, 2023

View reviewed changes

Merge branch 'master' of https://github.com/airbytehq/airbyte into ve…

e14463f

…ctara-destination

topefolorunso added 2 commits December 13, 2023 20:01

Merge branch 'master' of https://github.com/airbytehq/airbyte into ve…

2d385a4

…ctara-destination

add text and metadata fields in config

3fad06a

ofermend mentioned this pull request Dec 19, 2023

🎉 New Destination: Vectara #33616

Merged

marcosmarxm closed this Dec 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New Destination: Vectara (Vector Database) #31958

🎉 New Destination: Vectara (Vector Database) #31958

topefolorunso commented Oct 30, 2023

vercel bot commented Oct 30, 2023 •

edited

github-actions bot commented Oct 30, 2023

flash1293 left a comment

flash1293 commented Nov 8, 2023

topefolorunso commented Nov 8, 2023

topefolorunso commented Nov 10, 2023

flash1293 left a comment •

edited

flash1293 Nov 13, 2023

flash1293 Nov 13, 2023

marcosmarxm commented Dec 5, 2023

topefolorunso commented Dec 10, 2023

flash1293 commented Dec 10, 2023

topefolorunso commented Dec 13, 2023

ofermend commented Dec 14, 2023

marcosmarxm commented Dec 19, 2023

		from destination_vectara.destination import DestinationVectara


		class TestDestinationVectara(unittest.TestCase):

🎉 New Destination: Vectara (Vector Database) #31958

🎉 New Destination: Vectara (Vector Database) #31958

Conversation

topefolorunso commented Oct 30, 2023

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Actions

Community member or Airbyter

Airbyter

vercel bot commented Oct 30, 2023 • edited

github-actions bot commented Oct 30, 2023

Before Merging a Connector Pull Request

flash1293 left a comment

Choose a reason for hiding this comment

flash1293 commented Nov 8, 2023

topefolorunso commented Nov 8, 2023

topefolorunso commented Nov 10, 2023

flash1293 left a comment • edited

Choose a reason for hiding this comment

flash1293 Nov 13, 2023

Choose a reason for hiding this comment

flash1293 Nov 13, 2023

Choose a reason for hiding this comment

marcosmarxm commented Dec 5, 2023

topefolorunso commented Dec 10, 2023

flash1293 commented Dec 10, 2023

topefolorunso commented Dec 13, 2023

ofermend commented Dec 14, 2023

marcosmarxm commented Dec 19, 2023

vercel bot commented Oct 30, 2023 •

edited

flash1293 left a comment •

edited