Skip to content

Commit

Permalink
update Snowflake destination docs with more info (#10213)
Browse files Browse the repository at this point in the history
* update Snowflake destination docs with more info

* more edits

* address review comments
  • Loading branch information
subodh1810 committed Feb 16, 2022
1 parent c27e2a0 commit 531ed1a
Showing 1 changed file with 20 additions and 13 deletions.
33 changes: 20 additions & 13 deletions docs/integrations/destinations/snowflake.md
Expand Up @@ -27,9 +27,14 @@ Note that Airbyte will create **permanent** tables. If you prefer to create tran

## Getting started

### Requirements

1. Active Snowflake warehouse

We recommend creating an Airbyte-specific warehouse, database, schema, user, and role for writing data into Snowflake so it is possible to track costs specifically related to Airbyte \(including the cost of running this warehouse\) and control permissions at a granular level. Since the Airbyte user creates, drops, and alters tables, `OWNERSHIP` permissions are required in Snowflake. If you are not following the recommended script below, please limit the `OWNERSHIP` permissions to only the necessary database and schema for the Airbyte user.

We provide the following script to create these resources. Before running, you must change the password to something secure. You may change the names of the other resources if you desire.
Login into your Snowflake warehouse, copy and paste the following script in a new [worksheet](https://docs.snowflake.com/en/user-guide/ui-worksheet.html). Select the `All Queries` checkbox and then press the `Run` button.

```text
-- set variables (these need to be uppercase)
Expand Down Expand Up @@ -106,14 +111,14 @@ commit;

You should now have all the requirements needed to configure Snowflake as a destination in the UI. You'll need the following information to configure the Snowflake destination:

* **Host**
* **Role**
* **Warehouse**
* **Database**
* **Schema**
* **Username**
* **Password**
* **JDBC URL Params** (Optional)
* **[Host](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html)** : The host domain of the snowflake instance (must include the account, region, cloud environment, and end with snowflakecomputing.com). Example - `accountname.us-east-2.aws.snowflakecomputing.com`
* **[Role](https://docs.snowflake.com/en/user-guide/security-access-control-overview.html#roles)** : The role you created for Airbyte to access Snowflake. Example - `AIRBYTE_ROLE`
* **[Warehouse](https://docs.snowflake.com/en/user-guide/warehouses-overview.html#overview-of-warehouses)** : The warehouse you created for Airbyte to sync data into. Example - `AIRBYTE_WAREHOUSE`
* **[Database](https://docs.snowflake.com/en/sql-reference/ddl-database.html#database-schema-share-ddl)** : The database you created for Airbyte to sync data into. Example - `AIRBYTE_DATABASE`
* **[Schema](https://docs.snowflake.com/en/sql-reference/ddl-database.html#database-schema-share-ddl)** : The default schema is used as the target schema for all statements issued from the connection that do not explicitly specify a schema name. Schema name would be transformed to allowed by Snowflake if it not follow [Snowflake Naming Conventions](https://docs.airbyte.io/integrations/destinations/snowflake#notes-about-snowflake-naming-conventions).
* **Username** : The username you created to allow Airbyte to access the database. Example - `AIRBYTE_USER`
* **Password** : The password associated with the username.
* **[JDBC URL Params](https://docs.snowflake.com/en/user-guide/jdbc-parameters.html)** (Optional) : Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3).

## Notes about Snowflake Naming Conventions

Expand Down Expand Up @@ -147,17 +152,17 @@ When an identifier is double-quoted, it is stored and resolved exactly as entere

Therefore, Airbyte Snowflake destination will create tables and schemas using the Unquoted identifiers when possible or fallback to Quoted Identifiers if the names are containing special characters.

## Cloud Storage Staging
## Loading Method

By default, Airbyte uses batches of `INSERT` commands to add data to a temporary table before copying it over to the final table in Snowflake. This is too slow for larger/multi-GB replications. For those larger replications we recommend configuring using cloud storage to allow batch writes and loading.
By default, Airbyte uses [INTERNAL STAGING](https://docs.airbyte.com/integrations/destinations/snowflake#internal-staging)

### Internal Staging

Internal named stages are storage location objects within a Snowflake database/schema. Because they are database objects, the same security permissions apply as with any other database objects. No need to provide additional properties for internal staging
Internal named stages are storage location objects within a Snowflake database/schema. Because they are database objects, the same security permissions apply as with any other database objects. No need to provide additional properties for internal staging. This is also the recommended way of using the connector. It doesn't require any external resources and is quick to setup and use.

**Operating on a stage also requires the USAGE privilege on the parent database and schema.**

### AWS S3
### AWS S3 Staging

For AWS S3, you will need to create a bucket and provide credentials to access the bucket. We recommend creating a bucket that is only used for Airbyte to stage data to Snowflake. Airbyte needs read/write access to interact with this bucket.

Expand All @@ -180,7 +185,7 @@ Optional parameters:
* Whether to delete the staging files from S3 after completing the sync. Specifically, the connector will create CSV files named `bucketPath/namespace/streamName/syncDate_epochMillis_randomUuid.csv` containing three columns (`ab_id`, `data`, `emitted_at`). Normally these files are deleted after the `COPY` command completes; if you want to keep them for other purposes, set `purge_staging_data` to `false`.


### Google Cloud Storage \(GCS\)
### Google Cloud Storage \(GCS\) Staging

First you will need to create a GCS bucket.

Expand Down Expand Up @@ -215,6 +220,8 @@ The final query should show a `STORAGE_GCP_SERVICE_ACCOUNT` property with an ema

Finally, you need to add read/write permissions to your bucket with that email.

## Changelog

| Version | Date | Pull Request | Subject |
|:--------|:-----------| :----- | :------ |
| 0.4.12 | 2022-02-15 | [\#10342](https://github.com/airbytehq/airbyte/pull/10342) | Use connection pool, and fix connection leak. |
Expand Down

0 comments on commit 531ed1a

Please sign in to comment.