Skip to content

Commit

Permalink
Update connector builder docs for auto-import schema change (#27218)
Browse files Browse the repository at this point in the history
* adjust connector builder docs to account for auto-import schema

* undo formatting changes to record-processing

* undo formatting changes to tutorial

* reword
  • Loading branch information
lmossman committed Jun 9, 2023
1 parent 7351b6f commit ca41f23
Show file tree
Hide file tree
Showing 2 changed files with 5 additions and 29 deletions.
Expand Up @@ -302,7 +302,8 @@ This information is used by the Airbyte system for different purposes:
* **Recreating the data structure with right columns** in destination - this allows a warehouse destination to create a SQL table which the columns matching the fields of records
* **Detecting schema changes** - if the schema of a stream changes for an existing connection, this situation can be handled gracefully by Airbyte instead of causing errors in the destination

When doing test reads, the connector builder analyzes the test records and shows the derived schema in the "Detected schema" tab. This schema can be copied over as the declared schema of the stream with a single click.
When doing test reads, the connector builder analyzes the test records and shows the derived schema in the "Detected schema" tab. By default, new streams are configured to automatically import the detected schema into the declared schema on every test read.
This behavior can be toggled off by disabling the `Automatically import declared schema` switch, in which case the declared schema can be manually edited in the UI and it will no longer be automatically updated when triggering test reads.

For example the following test records:
```
Expand Down Expand Up @@ -351,7 +352,7 @@ result in the following schema:

More strict is always better, but the detected schema is a good default to rely on. See the documentation about [supported data types](https://docs.airbyte.com/understanding-airbyte/supported-data-types/) for JSON schema structures that will be picked up by the Airbyte system.

In case the declared schema deviates from the detected schema, the "Detected schema" tab in the testing panel highlights the differences. It's important to note that differences are not necessarily a problem that needs to be fixed - in some cases the currently loaded set of records in the testing panel doesn't feature all possible cases so the detected schema is too strict. However, if the declared schema is incompatible with the detected schema based on the test records, it's very likely there will be errors when running syncs.
If `Automatically import detected schema` is disabled, and the declared schema deviates from the detected schema, the "Detected schema" tab in the testing panel highlights the differences. It's important to note that differences are not necessarily a problem that needs to be fixed - in some cases the currently loaded set of records in the testing panel doesn't feature all possible cases so the detected schema is too strict. However, if the declared schema is incompatible with the detected schema based on the test records, it's very likely there will be errors when running syncs.

<img src={Diff} width="600" alt="Detected schema with highlighted differences" />

Expand Down
29 changes: 2 additions & 27 deletions docs/connector-development/connector-builder-ui/tutorial.mdx
Expand Up @@ -104,17 +104,7 @@ In a real sync, this record will be passed on to a destination like a warehouse.

The request/response tabs are helpful during development to see which requests and responses your connector will send and receive from the API.

### Declaring the record schema

<div style={{ position: "relative", paddingBottom: "59.66850828729282%", height: 0 }}><iframe src="https://www.loom.com/embed/56899a92afe047eba461f32d19c8ca94" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen style={{position: "absolute", top: 0, left: 0, width: "100%", height: "100%"}}></iframe></div>

Each stream of a connector needs to declare how emitted records will look like (which properties do they have, what data types will be used, ...). During a sync, this information will be passed on to the destination to configure it correctly - for example a SQL database destination can use it to properly set up the destination table, assigning the right type to each column.

By default, the stream schema is set to a simple object with unspecified properties. However, the connector builder can infer the schema based on the test read you just issued. To use the infered schema, switch to the "Detected schema" tab and click the "Import schema" button.

The warning icon disappears indicating that the declared schema of your stream matches the test data.

You can find more information about schema declaration on the [record processing concept page](./record-processing).
The detected schema tab indicates the schema that was detected by analyzing the returned records; this detected schema is automatically set as the declared schema for this stream, which you can see by visiting the Declared schema tab in the center stream configuration view.

## Step 3 - Advanced configuration

Expand Down Expand Up @@ -179,21 +169,6 @@ When used in a connection, the connector will make sure exchange rates for the s

You can find more information about incremental syncs on the [incremental sync concept page](./incremental-sync).

### Adding transformations

<div style={{ position: "relative", paddingBottom: "59.66850828729282%", height: 0 }}><iframe src="https://www.loom.com/embed/42ab3872c9284296b452a99e90b26738" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen style={{position: "absolute", top: 0, left: 0, width: "100%", height: "100%"}}></iframe></div>

Note that a warning icon should show next to the "Detected schema" tab - using the per-date endpoint instead of the latest endpoint slightly changed the shape of the records by adding a `historical` property. As we don't need this property in our destination, we can remove it using a transformation.

To do so, follow these steps:
* Enable the "Transformations" section
* Set the "Path" to `historical`
* Trigger a new test read

The `historical` property in the records tab and the schema warning should disappear.

You can find more information about incremental syncs on the [incremental sync concept page](./incremental-sync).

## Step 4 - Publishing and syncing

<div style={{ position: "relative", paddingBottom: "59.66850828729282%", height: 0 }}><iframe src="https://www.loom.com/embed/a6896c6aa8f047f0aeefec08d37a1384" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen style={{position: "absolute", top: 0, left: 0, width: "100%", height: "100%"}}></iframe></div>
Expand Down Expand Up @@ -226,8 +201,8 @@ This tutorial didn't go into depth about all features that can be used in the co
* [Authentication](/connector-development/connector-builder-ui/authentication/)
* [Record processing](/connector-development/connector-builder-ui/record-processing/)
* [Pagination](/connector-development/connector-builder-ui/pagination/)
* [Error handling](/connector-development/connector-builder-ui/error-handling/)
* [Incremental sync](/connector-development/connector-builder-ui/incremental-sync/)
* [Partitioning](/connector-development/connector-builder-ui/partitioning/)
* [Error handling](/connector-development/connector-builder-ui/error-handling/)

Not every possible API can be consumed by connectors configured in the connector builder. The [compatibility guide](/connector-development/connector-builder-ui/connector-builder-compatibility#oauth) can help determining whether another technology should be used to integrate an API with the Airbyte platform.

0 comments on commit ca41f23

Please sign in to comment.