Skip to content

Commit

Permalink
📚 Destinations V2: update data types for BigQuery + Snowflake (#33471)
Browse files Browse the repository at this point in the history
  • Loading branch information
@cynthiaxyin
cynthiaxyin committed Dec 14, 2023
1 parent 7eba1ed commit 73b7e7b
Show file tree
Hide file tree
Showing 2 changed files with 38 additions and 17 deletions.
22 changes: 12 additions & 10 deletions docs/integrations/destinations/bigquery.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ Airbyte outputs each stream into its own raw table in `airbyte_internal` dataset
### Raw Table schema

| Airbyte field | Description | Column type |
| ---------------------- | ------------------------------------------------------------------ | ----------- |
|------------------------|--------------------------------------------------------------------|-------------|
| \_airbyte_raw_id | A UUID assigned to each processed event | STRING |
| \_airbyte_extracted_at | A timestamp for when the event was pulled from the data source | TIMESTAMP |
| \_airbyte_loaded_at | Timestamp to indicate when the record was loaded into Typed tables | TIMESTAMP |
Expand All @@ -102,19 +102,21 @@ Airbyte converts any invalid characters into `_` characters when writing data. H
## Data type map

| Airbyte type | BigQuery type |
| :---------------------------------- | :------------ |
| DATE | DATE |
| STRING (BASE64) | STRING |
| NUMBER | FLOAT |
| OBJECT | STRING |
|:------------------------------------|:--------------|
| STRING | STRING |
| BOOLEAN | BOOLEAN |
| INTEGER | INTEGER |
| STRING (BASE64) | STRING |
| STRING (BIG_NUMBER) | STRING |
| STRING (BIG_INTEGER) | STRING |
| ARRAY | REPEATED |
| NUMBER | NUMERIC |
| INTEGER | INT64 |
| BOOLEAN | BOOL |
| STRING (TIMESTAMP_WITH_TIMEZONE) | TIMESTAMP |
| STRING (TIMESTAMP_WITHOUT_TIMEZONE) | TIMESTAMP |
| STRING (TIMESTAMP_WITHOUT_TIMEZONE) | DATETIME |
| STRING (TIME_WITH_TIMEZONE) | STRING |
| STRING (TIME_WITHOUT_TIMEZONE) | TIME |
| DATE | DATE |
| OBJECT | JSON |
| ARRAY | JSON |

## Troubleshooting permission issues

Expand Down
33 changes: 26 additions & 7 deletions docs/integrations/destinations/snowflake.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ Navigate to the Airbyte UI to set up Snowflake as a destination. You can authent
### Login and Password

| Field | Description |
| ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|-------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [Host](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html) | The host domain of the snowflake instance (must include the account, region, cloud environment, and end with snowflakecomputing.com). Example: `accountname.us-east-2.aws.snowflakecomputing.com` |
| [Role](https://docs.snowflake.com/en/user-guide/security-access-control-overview.html#roles) | The role you created in Step 1 for Airbyte to access Snowflake. Example: `AIRBYTE_ROLE` |
| [Warehouse](https://docs.snowflake.com/en/user-guide/warehouses-overview.html#overview-of-warehouses) | The warehouse you created in Step 1 for Airbyte to sync data into. Example: `AIRBYTE_WAREHOUSE` |
Expand All @@ -142,7 +142,7 @@ Navigate to the Airbyte UI to set up Snowflake as a destination. You can authent
### OAuth 2.0

| Field | Description |
| :---------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|:------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [Host](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html) | The host domain of the snowflake instance (must include the account, region, cloud environment, and end with snowflakecomputing.com). Example: `accountname.us-east-2.aws.snowflakecomputing.com` |
| [Role](https://docs.snowflake.com/en/user-guide/security-access-control-overview.html#roles) | The role you created in Step 1 for Airbyte to access Snowflake. Example: `AIRBYTE_ROLE` |
| [Warehouse](https://docs.snowflake.com/en/user-guide/warehouses-overview.html#overview-of-warehouses) | The warehouse you created in Step 1 for Airbyte to sync data into. Example: `AIRBYTE_WAREHOUSE` |
Expand Down Expand Up @@ -183,7 +183,7 @@ Airbyte outputs each stream into its own raw table in `airbyte_internal` schema
### Raw Table schema

| Airbyte field | Description | Column type |
| ---------------------- | ------------------------------------------------------------------ | ------------------------ |
|------------------------|--------------------------------------------------------------------|--------------------------|
| \_airbyte_raw_id | A UUID assigned to each processed event | VARCHAR |
| \_airbyte_extracted_at | A timestamp for when the event was pulled from the data source | TIMESTAMP WITH TIME ZONE |
| \_airbyte_loaded_at | Timestamp to indicate when the record was loaded into Typed tables | TIMESTAMP WITH TIME ZONE |
Expand All @@ -193,6 +193,25 @@ Airbyte outputs each stream into its own raw table in `airbyte_internal` schema

**Note:** By default, Airbyte creates permanent tables. If you prefer transient tables, create a dedicated transient database for Airbyte. For more information, refer to[ Working with Temporary and Transient Tables](https://docs.snowflake.com/en/user-guide/tables-temp-transient.html)

## Data type map

| Airbyte type | Snowflake type |
|:------------------------------------|:---------------|
| STRING | TEXT |
| STRING (BASE64) | TEXT |
| STRING (BIG_NUMBER) | TEXT |
| STRING (BIG_INTEGER) | TEXT |
| NUMBER | FLOAT |
| INTEGER | NUMBER |
| BOOLEAN | BOOLEAN |
| STRING (TIMESTAMP_WITH_TIMEZONE) | TIMESTAMP_TZ |
| STRING (TIMESTAMP_WITHOUT_TIMEZONE) | TIMESTAMP_NTZ |
| STRING (TIME_WITH_TIMEZONE) | TEXT |
| STRING (TIME_WITHOUT_TIMEZONE) | TIME |
| DATE | DATE |
| OBJECT | OBJECT |
| ARRAY | ARRAY |

## Supported sync modes

The Snowflake destination supports the following sync modes:
Expand Down Expand Up @@ -227,10 +246,10 @@ Otherwise, make sure to grant the role the required permissions in the desired n

| Version | Date | Pull Request | Subject |
|:----------------|:-----------|:-----------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 3.4.15 | 2023-12-13 | [33263](https://github.com/airbytehq/airbyte/pull/33263) | Only run typing+deduping for a stream if the stream had any records |
| 3.4.14 | 2023-12-08 | [33263](https://github.com/airbytehq/airbyte/pull/33263) | Adopt java CDK version 0.7.0 |
| 3.4.13 | 2023-12-05 | [32326](https://github.com/airbytehq/airbyte/pull/32326) | Use jdbc metadata for table existence check |
| 3.4.12 | 2023-12-04 | [33084](https://github.com/airbytehq/airbyte/pull/33084) | T&D SQL statements moved to debug log level |
| 3.4.15 | 2023-12-13 | [\#33232](https://github.com/airbytehq/airbyte/pull/33232) | Only run typing+deduping for a stream if the stream had any records |
| 3.4.14 | 2023-12-08 | [\#33263](https://github.com/airbytehq/airbyte/pull/33263) | Adopt java CDK version 0.7.0 |
| 3.4.13 | 2023-12-05 | [\#32326](https://github.com/airbytehq/airbyte/pull/32326) | Use jdbc metadata for table existence check |
| 3.4.12 | 2023-12-04 | [\#33084](https://github.com/airbytehq/airbyte/pull/33084) | T&D SQL statements moved to debug log level |
| 3.4.11 | 2023-11-14 | [\#32526](https://github.com/airbytehq/airbyte/pull/32526) | Clean up memory manager logs. |
| 3.4.10 | 2023-11-08 | [\#32125](https://github.com/airbytehq/airbyte/pull/32125) | Fix compilation warnings. |
| 3.4.9 | 2023-11-06 | [\#32026](https://github.com/airbytehq/airbyte/pull/32026) | Add separate TRY_CAST transaction to reduce compute usage |
Expand Down

0 comments on commit 73b7e7b

Please sign in to comment.