Skip to content

Commit

Permalink
Docs: create overview architecture diagram + reorder docs (#33896)
Browse files Browse the repository at this point in the history
  • Loading branch information
marcosmarxm committed Jan 24, 2024
1 parent 3601ac6 commit c8d06f4
Show file tree
Hide file tree
Showing 2 changed files with 38 additions and 19 deletions.
45 changes: 32 additions & 13 deletions docs/understanding-airbyte/high-level-view.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,16 +11,35 @@ The platform provides all the horizontal services required to configure and run
Connectors are independent modules which push/pull data to/from sources and destinations. Connectors are built in accordance with the [Airbyte Specification](./airbyte-protocol.md), which describes the interface with which data can be moved between a source and a destination using Airbyte. Connectors are packaged as Docker images, which allows total flexibility over the technologies used to implement them.

A more concrete diagram can be seen below:

![3.048-Kilometer view](../.gitbook/assets/understanding_airbyte_high_level_architecture.png)

* `UI`: An easy-to-use graphical interface for interacting with the Airbyte API.
* `WebApp Server`: Handles connection between UI and API.
* `Config Store`: Stores all the connections information \(credentials, frequency...\).
* `Scheduler Store`: Stores statuses and job information for the scheduler bookkeeping.
* `Config API`: Airbyte's main control plane. All operations in Airbyte such as creating sources, destinations, connections, managing configurations, etc.. are configured and invoked from the API.
* `Scheduler`: The scheduler takes work requests from the API and sends them to the Temporal service to parallelize. It is responsible for tracking success/failure and for triggering syncs based on the configured frequency.
* `Temporal Service`: Manages the task queue and workflows for the Scheduler.
* `Worker`: The worker connects to a source connector, pulls the data and writes it to a destination.
* `Temporary Storage`: A storage that workers can use whenever they need to spill data on a disk.

```mermaid
---
title: Architecture Overview
config:
theme: neutral
---
flowchart LR
W[fa:fa-display WebApp/UI]
S[fa:fa-server Server/Config API]
D[(fa:fa-table Config & Jobs)]
T(fa:fa-calendar Temporal)
W2[1..n Airbyte Workers]
W -->|sends API requests| S
S -->|store data| D
S -->|create workflow| T
T -->|launch task| W2
W2 -->|return job| T
W2 -->|launches| Source
W2 -->|launches| Destination
```

* **Web App/UI** [`airbyte-webapp`, `airbyte-proxy`]: An easy-to-use graphical interface for interacting with the Airbyte API.
* **Server/Config API** [`airbyte-server`, `airbyte-server-api`]: Handles connection between UI and API. Airbyte's main control plane. All operations in Airbyte such as creating sources, destinations, connections, managing configurations, etc.. are configured and invoked from the API.
* **Database Config & Jobs** [`airbyte-db`]: Stores all the connections information \(credentials, frequency...\).
* **Temporal Service** [`airbyte-temporal`]: Manages the task queue and workflows.
* **Worker** [`airbyte-worker`]: The worker connects to a source connector, pulls the data and writes it to a destination.

The diagram shows the steady-state operation of Airbyte, there are components not described you'll see in your deployment:
* **Cron** [`airbyte-cron`]: Clean the server and sync logs (when using local logs)
* **Bootloader** [`airbyte-bootloader`]: Upgrade and Migrate the Database tables and confirm the enviroment is ready to work.

This is a holistic high-level description of each component. For Airbyte deployed in Kubernetes the structure is very similar with a few changes.
12 changes: 6 additions & 6 deletions docusaurus/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -381,18 +381,18 @@ const understandingAirbyte = {
type: "category",
label: "Understand Airbyte",
items: [
"understanding-airbyte/beginners-guide-to-catalog",
"understanding-airbyte/high-level-view",
"understanding-airbyte/airbyte-protocol",
"understanding-airbyte/airbyte-protocol-docker",
"understanding-airbyte/operations",
"understanding-airbyte/high-level-view",
"understanding-airbyte/jobs",
"understanding-airbyte/tech-stack",
"understanding-airbyte/cdc",
"understanding-airbyte/database-data-catalog",
"understanding-airbyte/beginners-guide-to-catalog",
"understanding-airbyte/supported-data-types",
"understanding-airbyte/operations",
"understanding-airbyte/cdc",
"understanding-airbyte/json-avro-conversion",
"understanding-airbyte/database-data-catalog",
"understanding-airbyte/schemaless-sources-and-destinations",
"understanding-airbyte/tech-stack",
],
};

Expand Down

0 comments on commit c8d06f4

Please sign in to comment.