Can you add a slightly more realistic example of a data pipeline in the cloud?

[r39132]

For example, the Luigi example walks through a case that involves importing data into a DB. It would be cool if there were some examples that read from one location (e.g. S3) and wrote to another (e.g. DB).

[artwr]

We have an operator that transfers files from s3 to Hive, as well as a Hive to Mysql operator, which is unfortunately missing from the documentation. The hooks being already in place, a s3 to Postgres operator should not be too difficult. Here is an example of how our s3 to Hive operator would work :

from collections import OrderedDict
from airflow.operators import S3ToHiveTransfer

S3ToHiveTransfer(
    task_id='s3_to_hive',
    s3_key='s3://bucket-name/user_list_{{ ds }}.tsv',
    field_dict=OrderedDict([("user_id", "BIGINT")
                            , ("first_name", "STRING")
                            , ("last_name", "STRING")
                            , ("registered_at", "TIMESTAMP")]),
    hive_table='{{ params.db_name }}.user_list_{{ ds_nodash }}',
    create=True,
    recreate=True,
    delimiter='\t',
    s3_conn_id='s3_connection_name',
    dag=dag)

Usually, the transfer is a separate operation of any transform and has its own operator.
I hope this helps.
Best,
Arthur

[mistercrunch]

Agreed, the current tutorial is really just focused on the mechanics of Airflow with very foobar-y examples. I didn't want to write a pipeline that was too stack specific (MySQL / Hive / ...) and wanted to make sure it would work for anyone, regardless of the stack they might have.

Maybe using a SqliteOperator to do some analytics on some data scraped from the Internet would be a good example. It could be interesting to re-write the Luigi example for comparison :)

But yeah, it's on the TODO list.

[r39132]

Cool. I'll close this for now.