Skip to content

Commit

Permalink
Remove note and improve Airflow Tutorial (#3548)
Browse files Browse the repository at this point in the history
* remove note and improve docs

* change abhi comments and remove other changes

* remove microsoft team icon change

* remove change from microsoft-icon
  • Loading branch information
marcosmarxm committed May 24, 2021
1 parent 7ccc4fa commit 7d5e2ca
Showing 1 changed file with 15 additions and 12 deletions.
27 changes: 15 additions & 12 deletions docs/tutorials/using-the-airflow-airbyte-operator.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
---
description: Start triggering Airbyte jobs with Airflow in minutes
description: Start triggering Airbyte jobs with Apache Airflow in minutes
---

# Using the Airflow Airbyte Operator
# Using Apache Airflow with the Airbyte Operator

Airbyte allows you to trigger synchronization jobs using Airflow Operators. This tutorial explains how to configure your Airflow DAG to do so.
Airbyte is an official community provider for the Apache Airflow project.
The Airbyte operator allows you to trigger synchronization jobs in Apache Airflow,
and this tutorial will walk through configuring your Airflow DAG to do so.

{% hint style="warning" %}
This tutorial is a preview. The Airbyte Operator is [code complete](https://github.com/apache/airflow/pull/14492) and will be available in a couple of weeks with Airflow's next release.
{% endhint %}
The Airbyte Provider documentation on Airflow project can be found [here](https://airflow.apache.org/docs/apache-airflow-providers-airbyte/stable/index.html).

## 1. Set up the tools

Expand All @@ -20,13 +20,15 @@ If this is your first time using Airbyte, we suggest going through our [Basic Tu

For the purposes of this tutorial, set your Connection's **sync frequency** to **manual**. Airflow will be responsible for manually triggering the Airbyte job.

### **Start Airflow**
### **Start Apache Airflow**

If you don't have an Airflow instance, we recommend following this [Quick Start Airflow Tutorial](https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html) to set one up.
If you don't have an Airflow instance, we recommend following this [guide](https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html) to set one up.
Additionally, you will need to install the `apache-airflow-providers-airbyte` package to use Airbyte Operator on Apache Airflow.
You can read more about it [here](https://airflow.apache.org/docs/apache-airflow-providers-airbyte/stable/index.html)

## 2. Create a DAG in Airflow to trigger your Airbyte job
## 2. Create a DAG in Apache Airflow to trigger your Airbyte job

### Create an Airbyte connection in Airflow
### Create an Airbyte connection in Apache Airflow

Once Airflow starts, navigate to Airflow's `Connections` page as seen below. The Airflow UI can be accessed at [http://localhost:8080/](http://localhost:8080/).

Expand All @@ -53,7 +55,7 @@ Place the following file inside the `/dags` directory. Name this file `dag_airby
```python
from airflow import DAG
from airflow.utils.dates import days_ago
from airflow.providers.airbyte.operator import AirbyteTriggerSyncOperator
from airflow.providers.airbyte.operators.airbyte import AirbyteTriggerSyncOperator

with DAG(dag_id='trigger_airbyte_job_example',
default_args={'owner': 'airflow'},
Expand Down Expand Up @@ -96,7 +98,8 @@ If your Airflow instance has limited resources and/or is under load, setting the
```python
from airflow import DAG
from airflow.utils.dates import days_ago
from airflow.providers.airbyte.operator import AirbyteTriggerSyncOperator
from airflow.providers.airbyte.operators.airbyte import AirbyteTriggerSyncOperator
from airflow.providers.airbyte.sensors.airbyte import AirbyteJobSensor

with DAG(dag_id='airbyte_trigger_job_example_async',
default_args={'owner': 'airflow'},
Expand Down

0 comments on commit 7d5e2ca

Please sign in to comment.