Skip to content
This repository has been archived by the owner on Apr 11, 2024. It is now read-only.

Commit

Permalink
Add example dag for SFTP and Local (#1866)
Browse files Browse the repository at this point in the history
# Description
## What is the current behavior?
Missing example dag for SFTP and Local data provides.



## What is the new behavior?
Added example dag for SFTP and Local data provides

## Does this introduce a breaking change?
Nope


### Checklist
- [ ] Created tests which fail without the change (if possible)
- [ ] Extended the README / documentation, if necessary

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
  • Loading branch information
utkarsharma2 and pre-commit-ci[bot] committed Mar 24, 2023
1 parent 5b2f827 commit 251ed96
Showing 1 changed file with 10 additions and 0 deletions.
10 changes: 10 additions & 0 deletions example_dags/example_universal_transfer_operator.py
@@ -1,4 +1,5 @@
import os
import pathlib
from datetime import datetime

from airflow import DAG
Expand All @@ -11,6 +12,9 @@
s3_bucket = os.getenv("S3_BUCKET", "s3://astro-sdk-test")
gcs_bucket = os.getenv("GCS_BUCKET", "gs://uto-test")

CWD = pathlib.Path(__file__).parent
DATA_DIR = str(CWD) + "/../../data/"

# [START dataset_file]
input_file = File(path=f"{gcs_bucket}/uto/", conn_id="google_cloud_default")
# [END dataset_file]
Expand Down Expand Up @@ -134,3 +138,9 @@
destination_dataset=Table(name="uto_bigquery_to_sqlite_table", conn_id="sqlite_default"),
)
# [END transfer_non_native_bigquery_to_sqlite]

transfer_non_native_local_to_sftp = UniversalTransferOperator(
task_id="transfer_non_native_local_to_sftp",
source_dataset=File(path=f"{DATA_DIR}sample.csv", filetype=FileType.CSV),
destination_dataset=File(path="sftp://upload/sample.csv", conn_id="sftp_conn", filetype=FileType.CSV),
)

0 comments on commit 251ed96

Please sign in to comment.