Skip to content

01. Airbyte

Jump to bottom
Janemesi edited this page Oct 22, 2025 · 6 revisions

Airbyte is an open source data integration platform that securely extracts data from the different data sources and reliably loads the data to a data warehouse or database.

Getting started with Airbyte Cloud

To create the connectors on Airbyte, an invitation to the workspace on Airbyte cloud will be sent. Upon receiving an invitation, click on the verification link received through an email to the provided address to be redirected to the relevant workspace.

Within the workspace, there are different sections;

Connections: This section contains the created data pipelines. A connection is an automated data pipeline that replicates data from a source to a destination allowing users to specify the frequency of the replication.

Sources: This section has the defined sources. A source can be an API, a database or a file that you want to ingest your data from.

Destinations: A destination is a database, data warehouse or an analytics tool that you want to load your extracted data pulled from a source.

Billing: This section shows the total credits usage, usage per connection etc.

Settings: Within this section we have different sub sections;
Account: Displays the name and email address associated with the account. One can also update the account password from this section.
Cookie preferences: Allows you to set how you want the retrieved browser data to be used for.
General settings: Displays the workspace name.
Data residency: Allows you to choose the default data processing location for all your connectors.
Sources: Displays the available source connectors.
Destinations: Displays the available destination connectors.
Access management: Displays the users who have access to the workspace.
Notifications: Allows you to set up the actions that you would like to receive notifications for such as monitoring sync failures.

Setting up a source

To set up a source;

  1. Click on Sources.

sources_scre

  1. On the Sources page, click on +New Source. +newsource_doc

  2. Select the source you want to set up from the source options available on the set up the source page. select_source_type

  3. On the Create a source page specify the relevant details.The details provided depends on the type of source you are setting up. For instance if you are setting up an Ona Data source you can provide the;

  4. Once you have submitted the details above, click on Set up source to create the source.

Setting up a destination

There are different destinations available under the destinations page however, on this guide, we will look into how to set up a Postgres destination. To set up a postgres destination;

  1. On the Destinations page, click on + New destination. +newdestination

  2. Select Postgres on the set up destination page. destination_setup

  3. You will need to specify the details under the Create a destination page to be able to successfully set up a destination. Then, click on Set up destination to create the destination. For a Postgres destination specify the;
    Destination name: Name of the destination. For instance, you can use the database name then add Canopy as a prefix.
    Host: Hostname of the database.
    Port: Port of the database.
    DB Name: Name of the database.
    Default Schema: The default schema the tables are written to.
    User: Username to use to access the database.
    SSL modes: SSL connection modes. Highly recommended to select Require or at least Prefer.
    SSH Tunnel Method: Whether to initiate an SSH tunnel and the authentication method.

Setting up a connection

A connection is an automated data pipeline that replicates data from a source to a destination. You can create a connection from an existing source or destination or set up a new source or destination while creating the connection. To set up a connection;

  1. Click on + New connection under the connections page.

  2. On the define source page, you can select an existing source or set up a new source. To set up a new source, take a look at setting up a source section.

  3. Under the define destination page, select a destination. You can select an existing destination or set up a new destination. To set up a postgres destination, take a look at setting up a destination section.

  4. Once you select a destination, you will be redirected to the set up connection page. configureconnection_doc

    • From the Replication frequency dropdown, select how often you want the data to sync from the source to the destination.The default replication frequency is Every 24 hours.
    • From the Destination Namespace, select the location which you want to store the data in the destination. The default configuration is Destination default.
    • In the Destination Stream Prefix (Optional) field, add a prefix to stream names. For instance templates_ renames cases to templates_cases.
    • Under the Activate the streams you want to sync section, configure the sync settings;
    • Toggle the Sync button to enable sync for the stream.
    • Sync mode: select how you want the data to be replicated from the source to the destination.
    • For the source: Select Full refresh to copy the entire dataset each time you sync.
    • For the destination:
    • Select Overwrite to erase the old data and replace it completely.
    • Select Append to capture changes to your table Note: This creates duplicate records.

  5. One done specifying the details, click Set up connection to complete setting up the connection, Airbyte tests the connection. If the sync is successful, the connection page is displayed.

Airbyte Sync Mode changes.

The zip file contains a script written in Python to interact with the Airbyte API. The script performs the following tasks:

  • Fetch Data from Airbyte API: Sends a GET request to the Airbyte API to retrieve all connections. Extracts the data from the response if the request is successful.

  • Update Stream Sync Mode: Iterates through each connection retrieved. For each connection, updates the sync mode of all streams to "incremental_deduped_history". Sends a PATCH request to the Airbyte API to apply the updates to each connection.

With this script you will be able to:

  1. Change the airbyte sync mode for one stream
  2. Change the sync mode for multiple streams
  3. Obtain Ona Data forms from airbyte

Airbyte api sync mode and onadata.zip

Colab file here

Depending on the supported sync Modes for the stream.

The code interacts with Airbyte's API to manage data integration settings. It does the following:

  • Fetch Workspaces: Retrieves a list of workspaces from Airbyte.
  • Fetch Source Definitions: Retrieves details about data sources within each workspace.
  • Update Docker Image Tag: Updates the Docker image tag for specific data sources.
  • Main Logic: Iterates through workspaces, identifies data sources by Docker repository, and updates their Docker image tags.
  1. Version connector change.zip
  2. colab file

Xforms and choices updates

  • Fetch workspaces: Retrieves a list of workspaces from Airbyte.
  • Fetch Connections: Retrieves details about connections within each workspace.
  • Update Connections: Updates connection settings by removing the schedule field and adjusting the selected status for streams starting with xfm or chc to False.
  • Xforms and choices colab

Custom Airbyte Connectors

  • A list of ONA created custom airbyte connectors (sources and destinations) exist in this repository: https://github.com/onainsights/airbyte-connectors
  • The repo also contains a catalog of all connector images for both airbyte stage and airbyte cloud

Security, Access, Deployment, And Data Flows

Clone this wiki locally