Fix Vector destination tests and move to Poetry

[aaronsteers]

Destinations addressed in this pull request

• Milvus: https://connectors.airbyte.com/files/generated_reports/test_summary/destination-milvus/index.html
• Pinecone: https://connectors.airbyte.com/files/generated_reports/test_summary/destination-pinecone/index.html
• Weaviate: https://connectors.airbyte.com/files/generated_reports/test_summary/destination-weaviate/index.html

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Mar 23, 2024 3:09pm

[aaronsteers]

Milvus account has reached OpenAI quota:

• Job result: RateLimitError: You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors

Update: Weaviate is showing the same issue. (OpenAI quota exceeded.)

[aaronsteers]

Re: Pinecone failures looks like breaking changes in the pinecone library. Probably need to pin to prior version:

• TypeError: <lambda>() got an unexpected keyword argument 'replicas'
• AttributeError: init is no longer a top-level attribute of the pinecone package.�Please create an instance of the Pinecone class instead.�Example:�import os; from pinecone import Pinecone, ServerlessSpec; pc = Pinecone(api_key=os.environ.get("PINECONE_API_KEY"))... pc.create_index(...

Update: I've pinned to <3.0. For future ref, migration guide is here.

[aaronsteers]

I've updated the configs in GSM so that each of these three uses embeddings "mode": "fake" instead of "mode": "openai".

[bindipankhudi]

Besides the OpenAI quota issues, are these all the changes we need to comply with the new connector requirements?

[aaronsteers]

Are these all the changes we need to comply with the new connector requirements? Will these changes make the pre-release tests pass ?

As of now I'm using this PR as a placeholder and vehicle to run/debug CI tests.

It only contains one fix as of now (the pinecone library pin) but the idea is that before merging we'll get all passing tests again.

[bindipankhudi]

I've updated the configs in GSM so that each of these three uses embeddings "mode": "fake" instead of "mode": "openai".

assuming we are good for now then?

[aaronsteers]

I've updated the configs in GSM so that each of these three uses embeddings "mode": "fake" instead of "mode": "openai".

assuming we are good for now then?

Still a bit more work to do. One connector (Vectara, I think) went green after this. Another (pinecone) appears to need a more specific version to be pinned, and the third (weaviate) had some integration tests that strictly required the OpenAI key.

All of these also have the (new) requirement to be on Poetry. If we don't fix that poetry failure, we'll be blind to other failure types because status will stay constantly red.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ aaronsteers
❌ bindipankhudi

---

bindipankhudi seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[bindipankhudi]

Tests for Milvus, Pinecone, and Weaviate destinations are green now.

Changes:

• Poetry configuration for the destinations above
• Open AI key updated for all destinations (no longer using faker)
• Milvus cluster re-enabled - this was causing the Milvus tests to fail

[aaronsteers]

✅ Approved!

This is huge to have these passing tests again!

Two points for follow-up, but non-blocking:

1. The run.run CLI references in the pyproject.toml files won't work yet, but also they shouldn't hurt anything in the meanwhile.
2. It's probably worth looping back sometime soon to bump the CDK versions to latest. The version here is 57ish in most cases, whereas I think the latest version is around 75 now.

I think we can handle both in future cycles.