Skip to content

Commit

Permalink
Update typing-deduping.md - Loading Data Incrementally to Final Tables (
Browse files Browse the repository at this point in the history
  • Loading branch information
evantahler committed Jan 8, 2024
1 parent 0e4ffb5 commit 1737ab1
Showing 1 changed file with 11 additions and 6 deletions.
17 changes: 11 additions & 6 deletions docs/using-airbyte/core-concepts/typing-deduping.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,6 @@ This page refers to new functionality added by [Destinations V2](/release_notes/
- Internal Airbyte tables in the `airbyte_internal` schema: Airbyte will now generate all raw tables in the `airbyte_internal` schema. We no longer clutter your desired schema with raw data tables.
- Incremental delivery for large syncs: Data will be incrementally delivered to your final tables when possible. No more waiting hours to see the first rows in your destination table.

:::note

Typing and Deduping may cause an increase in your destination's compute cost. This cost will vary depending on the amount of data that is transformed and is not related to Airbyte credit usage.

:::

## `_airbyte_meta` Errors

"Per-row error handling" is a new paradigm for Airbyte which provides greater flexibility for our users. Airbyte now separates `data-moving problems` from `data-content problems`. Prior to Destinations V2, both types of errors were handled the same way: by failing the sync. Now, a failing sync means that Airbyte could not _move_ all of your data. You can query the `_airbyte_meta` column to see which rows failed for _content_ reasons, and why. This is a more flexible approach, as you can now decide how to handle rows with errors on a case-by-case basis.
Expand Down Expand Up @@ -81,3 +75,14 @@ In legacy normalization, columns of [Airbyte type](/understanding-airbyte/suppor
You also now see the following changes in Airbyte-provided columns:

![Airbyte Destinations V2 Column Changes](../../release_notes/assets/updated_table_columns.png)

## Loading Data Incrementally to Final Tables

:::note

Typing and Deduping may cause an increase in your destination's compute cost. This cost will vary depending on the amount of data that is transformed and is not related to Airbyte credit usage. Enabling loading data incrementally to final tables may further increase this cost.

:::

V2 destinations may include the option "Enable Loading Data Incrementally to Final Tables". When enabled your data will load into your final tables incrementally while your data is still being synced. When Disabled (the default), your data loads into your final tables once at the end of a sync. Note that this option only applies if you elect to create Final tables.

0 comments on commit 1737ab1

Please sign in to comment.