✨ New: Snowflake Cortex Destination 🚀

[bindipankhudi]

Related to: airbytehq/PyAirbyte#120

Destination is working as expected. All unit tests and integration tests are passing. Majority of the code is generated from the vector-db SDK, or taken from Pinecone destination. The most interesting bit (write logic), specific to Cortex is in SnowflakeCortexIndexer class.

To be done as fast follow

• Add an option for embedding via Snowflake models.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 14, 2024 11:16pm

[aaronsteers]

This looks great! A few changes requested inline, but lmk if you want to push back on any of them, and/or defer.

The one that is probably hardest to implement (and also hardest to change later) is the layout of the password input in the config spec. Happy to help on that if needed.

[aaronsteers]

I compared these with the Java-based snowflake destination to see if these parameters match the similar destination. If we can make the auth/config flow consistent between the two, this could help end users who might end up using both for different use cases.

It looks like all of our specified inputs match except account, which the Java Snowflake destination calls host ("Host").

Also, it looks like we are not accepting schema ("Default Schema") and we should probably add that so users can control which schema is written to.

The last difference I noted was that the way we are configuring password is slightly different, just because the Java destination supports more auth options (OAuth and Key Pair private key). Even if we just have a single auth option, it probably makes sense to try to mirror the same structure, which is to place "password" under a "credentials" object. The challenge here is that we can't necessarily copy code from the Snowflake source - since it is in Java, but maybe another Python connector that supports multiple auth flows could be a model here.

Here is the spec.json for destination-snowflake: https://github.com/airbytehq/airbyte/blob/72b083cbd259b3d922e522bee70e81d60f88f481/airbyte-integrations/connectors/destination-snowflake/src/main/resources/spec.json

[bindipankhudi]

Agreed! The parameters should ideally match the snowflake destination. Made the following changes:

• replaced "account" with "host" for UI.
• reordered params to match the order in snowflake destination
• added "default_schema" field
• added a new credentials section with password field.
  Implementing the Oauth option seems to much at the moment, but above changes should help us do so in future.

[bindipankhudi]

@aaronsteers PR is ready for another look.

• I added all proposed minor updates (type hint, etc)
• Created this new ticket (https://github.com/airbytehq/airbyte-internal-issues/issues/7917) for updating how we do writes curently.
• Updated cortex config fields and order to match Snowflake destination. (more comments on this inline)

[aaronsteers]

Should password be nested under credentials now?

Weird that this didn't throw an error.

[bindipankhudi]

good catch.. this file is for reference purpose, doesn't get used :) i will update it.

[aaronsteers]

Looks great! 🚀

I think you can merge when ready.