This is a Singer tap that produces JSON-formatted data following the Singer spec.
The tap is developed to get data from Airflow REST API.
Current tap fetches data from the endpoint /dags/{dag_id}/dagRuns.
-
Creating and activating virtual environment:
python3 -m venv ~/.virtualenvs/tap-airflow source ~/.virtualenvs/tap-airflow/bin/activate
-
Installation:
git clone https://github.com/RFAInc/tap-airflow.git cd tap-airflow python install setup.py deactivate
- Client configuration:
{ "host": "http://localhost:8080/api/v1", "username": "username", "password": "password", "selected_all": true }
> Note: here <b>batch</b> is a limit for paging which supports upto <b>10000</b> records per request > Note: if <b>sync_tables</b> is not specified, then the tap will fetch data for all tables
-
Running
-
Run in discovery mode (for generating schemas and catalog)
~/.virtualenvs/tap-airflow/bin/tap-airflow --config client_config.json --discover > catalog.json
-
Run for full sync (for initial run):
~/.virtualenvs/tap-airflow/bin/tap-airflow --config client_config.json --catalog catalog.json
-
Run with state (for further run):
~/.virtualenvs/tap-airflow/bin/tap-airflow --config client_config.json --catalog catalog.json --state state.json
-