Airflow example running Snowflake queries and publish OpenLineage events. #2

fm100 · 2022-04-13T20:05:55Z

Description

This airflow example uses SnowflakeOperator in order to run queries, and publishes OpenLineage events, which is auto-generated in OPEN_LINEAGE_ACCESS_HISTORY view, to the configured OpenLineage backend.

…nts. Signed-off-by: Minkyu Park <minkyu@datakin.com>

sfc-gh-kbijon

looks good @fm100 . I left some comments. please see if those make sense.
One other thing:
instead of a single directory "dags", should we create two separates directories
a) dag_queries - contains all the files the generate queries for dag
b) dag_exract - contains etl_openlineage.py
WDYT?

sfc-gh-kbijon · 2022-04-14T16:29:37Z

examples/airflow/README.md

+that sends generated OpenLineage events to the configured backend.
+
+## Prerequisite
+* `OPEN_LINEAGE` database and `FOOD_DELIVERY` schema in Snowflake need to be created to run this example.


Should we give a setup.py to create this database and schema in the account and document that in the prerequisite?

Nit: we can probably rename the database OPEN_LINEAGE_EXAMPLE?

I would actually prefer OPENLINEAGE_EXAMPLE, as OpenLineage is one word.

Instead of a script, possibly a code block here with the statements:

CREATE DATABASE ... ...

that way it is easier for the user to follow along. Ross, I think you'll have more feedback when you try it out.

@julienledem Because it's important to keep current_organization as a session variable, I chose to do this as a SnowSQL example. Let me know what you think.

sfc-gh-kbijon · 2022-04-14T16:29:57Z

examples/airflow/README.md

+* `OPEN_LINEAGE` database and `FOOD_DELIVERY` schema in Snowflake need to be created to run this example.
+
+## Environment Variables
+Following environment variables need to be set in order to send the OpenLineage events to the OL backend:


what does OL stand for?

I think we should avoid shortening OpenLineage to OL in these docs, just in case.

agreed. Let's spell out OpenLineage

sfc-gh-kbijon · 2022-04-14T16:32:03Z

examples/airflow/dags/etl_categories.py

@@ -0,0 +1,54 @@
+from airflow import DAG


I wonder if we should briefly explain what this script does. We can probably also document these scripts in the readme. WDYT?

agreed, we could add some doc here

I added a bit of content to the README that explains what these scripts do 👍

sfc-gh-kbijon · 2022-04-14T16:40:11Z

examples/airflow/README.md

+* SNOWFLAKE_USER
+* SNOWFLAKE_PASSWORD
+* SNOWFLAKE_ACCOUNT
+* OPENLINEAGE_URL


I wonder if we should also document the following

what are the things the user needs to install in order to run this example?

The steps to execute
a) First run the all the query generator scripts
b) Wait for 3o min
c) run etl_openlineage.py

Agreed. We could point to documentations about:

installing Marquez

getting a Datakin instance.

I've added some of this to the README, trying to keep it minimal. Let me know what you think.

looks good. ty!

julienledem

The screenshot looks nice!

julienledem · 2022-04-15T00:33:17Z

examples/airflow/README.md

+that sends generated OpenLineage events to the configured backend.
+
+## Prerequisite
+* `OPEN_LINEAGE` database and `FOOD_DELIVERY` schema in Snowflake need to be created to run this example.


Instead of a script, possibly a code block here with the statements:

CREATE DATABASE ... ...

that way it is easier for the user to follow along. Ross, I think you'll have more feedback when you try it out.

julienledem · 2022-04-15T00:34:31Z

examples/airflow/README.md

+* `OPEN_LINEAGE` database and `FOOD_DELIVERY` schema in Snowflake need to be created to run this example.
+
+## Environment Variables
+Following environment variables need to be set in order to send the OpenLineage events to the OL backend:


agreed. Let's spell out OpenLineage

julienledem · 2022-04-15T00:35:33Z

examples/airflow/README.md

+* SNOWFLAKE_USER
+* SNOWFLAKE_PASSWORD
+* SNOWFLAKE_ACCOUNT
+* OPENLINEAGE_URL


Agreed. We could point to documentations about:

installing Marquez

getting a Datakin instance.

julienledem · 2022-04-15T00:36:00Z

examples/airflow/dags/etl_categories.py

@@ -0,0 +1,54 @@
+from airflow import DAG


agreed, we could add some doc here

julienledem · 2022-04-15T00:36:39Z

examples/airflow/dags/etl_categories.py

+        FROM food_delivery.tmp_categories
+        ''',
+        session_parameters={
+            'QUERY_TAG': 'etl_categories'


I would add a comment explaining why we add this

julienledem · 2022-04-15T00:38:13Z

examples/airflow/dags/etl_openlineage.py

+SNOWFLAKE_PASSWORD = os.getenv('SNOWFLAKE_PASSWORD')
+SNOWFLAKE_ACCOUNT = os.getenv('SNOWFLAKE_ACCOUNT')
+
+


We should add some doc explaining that this is the logic forwarding the OpenLineage events from Snowflake to an OpenLineage endpoint

Signed-off-by: Ross Turk <ross@datakin.com>

rossturk · 2022-04-19T03:19:45Z

looks good @fm100 . I left some comments. please see if those make sense.
One other thing:
instead of a single directory "dags", should we create two separates directories
a) dag_queries - contains all the files the generate queries for dag
b) dag_exract - contains etl_openlineage.py
WDYT?

I agree - I chose to name them etl and lineage. My rationale is that they are already in a directory called dags, and the word extract applies to both metadata and data. Let me know if you like these names.

sfc-gh-kbijon

@rossturk The README looks good. I left some comments. Please take a look. thank you!

sfc-gh-kbijon · 2022-04-19T19:24:20Z

examples/airflow/README.md

+
+Once the environment is prepared, initialize Airflow with docker-compose:
+```bash
+% docker-compose up airflow-init


instead of attaching docker-compose.yaml, I wonder if we should document the steps similar to "Installing Marquez", i.e., git clone https://github.com/anilkulkarni87/airflow-docker and document the steps. In that way, this scripts in the repo are purely the examples and not related to any prerequisite files, etc.. WDYT?

That's actually how I started, but it requires a lot of changes to docker-compose.yaml to make this example work smoothly:

custom image build with extra python requirements

pass through environment

skip loading example dags

I opted to keep the procedure simple.

Hmm, that file has a license and I actually didn't talk to our Legal team for that. Is it very complex to document what to do what docker-compose.yaml instead of directly post the file here?

Yeah, it's pretty complex and would be difficult to describe. This file is apache 2 licensed, which is permissive and non-viral.

@sfc-gh-kbijon I have updated the comment block at the top of the file to clarify its intended use, warn that it's not for production, and point folks to the original

left some comments in the file. please address.

examples/airflow/README.md

Signed-off-by: Ross Turk <ross@datakin.com>

sfc-gh-kbijon

thanks @rossturk. It looks good. I left one comment. Please address that. ty!

sfc-gh-kbijon · 2022-04-20T20:31:34Z

examples/airflow/README.md

+* SNOWFLAKE_USER
+* SNOWFLAKE_PASSWORD
+* SNOWFLAKE_ACCOUNT
+* OPENLINEAGE_URL


looks good. ty!

sfc-gh-kbijon · 2022-04-20T20:32:14Z

examples/airflow/README.md

+
+Once the environment is prepared, initialize Airflow with docker-compose:
+```bash
+% docker-compose up airflow-init


left some comments in the file. please address.

examples/airflow/docker-compose.yaml

Signed-off-by: Ross Turk <ross@datakin.com>

sfc-gh-kbijon

@rossturk @fm100 the changes look good. thanks a lot!

sfc-gh-kbijon · 2022-04-20T22:35:10Z

examples/airflow/README.md

+
+This example uses Airflow to run a collection of Snowflake queries for a fictitious food delivery service. Lineage data for these queries is recorded within Snowflake [ACCESS_HISTORY](https://docs.snowflake.com/en/sql-reference/account-usage/access_history.html) and, using the OpenLineage Access History View, emitted to an OpenLineage backend.
+
+This is done using a series of DAGs in `dags/queries` that each use SnowflakeOperator to run queries, along with a DAG in `dags/extract` that uses PythonOperator to send generated OpenLineage events to the configured backend.


@rossturk this needs to be updated to dags/elt and dags/lineage

@sfc-gh-kbijon very odd, I have made this change and committed it to the branch from @fm100 but it does not show up here. I think we should close this PR and open a new one, to see if it includes all of the changes.

Airflow example running Snowflake queries and publish OpenLineage eve…

b9262df

…nts. Signed-off-by: Minkyu Park <minkyu@datakin.com>

sfc-gh-kbijon reviewed Apr 14, 2022

View reviewed changes

sfc-gh-kbijon requested a review from julienledem April 14, 2022 21:45

julienledem reviewed Apr 15, 2022

View reviewed changes

Ross Turk added 5 commits April 18, 2022 23:05

Update OPEN_LINEAGE to OPENLINEAGE

64e63cd

Signed-off-by: Ross Turk <ross@datakin.com>

pass along current_organization in query

03c9c05

Signed-off-by: Ross Turk <ross@datakin.com>

Adding docker-compose for Airflow

cae1e1c

Signed-off-by: Ross Turk <ross@datakin.com>

Updated README with more detailed instructions

097183a

Signed-off-by: Ross Turk <ross@datakin.com>

moving DAGs into separate subdirectories

bdf437d

Signed-off-by: Ross Turk <ross@datakin.com>

rossturk requested review from sfc-gh-kbijon and julienledem April 19, 2022 19:08

sfc-gh-kbijon reviewed Apr 19, 2022

View reviewed changes

Ross Turk added 2 commits April 19, 2022 19:20

restructure README sections

694eb88

Signed-off-by: Ross Turk <ross@datakin.com>

update docker-compose.yaml with notice at top

aeac641

Signed-off-by: Ross Turk <ross@datakin.com>

rossturk requested a review from sfc-gh-kbijon April 20, 2022 17:35

sfc-gh-kbijon requested a review from sfc-gh-tchao April 20, 2022 20:31

sfc-gh-kbijon reviewed Apr 20, 2022

View reviewed changes

add snowflake copyright

933b87a

Signed-off-by: Ross Turk <ross@datakin.com>

sfc-gh-kbijon approved these changes Apr 20, 2022

View reviewed changes

sfc-gh-kbijon reviewed Apr 20, 2022

View reviewed changes

fm100 closed this Apr 21, 2022

fm100 mentioned this pull request Apr 21, 2022

Airflow example running Snowflake queries and publish OpenLineage events. #3

Merged

		SNOWFLAKE_PASSWORD = os.getenv('SNOWFLAKE_PASSWORD')
		SNOWFLAKE_ACCOUNT = os.getenv('SNOWFLAKE_ACCOUNT')


		This example uses Airflow to run a collection of Snowflake queries for a fictitious food delivery service. Lineage data for these queries is recorded within Snowflake [ACCESS_HISTORY](https://docs.snowflake.com/en/sql-reference/account-usage/access_history.html) and, using the OpenLineage Access History View, emitted to an OpenLineage backend.

		This is done using a series of DAGs in `dags/queries` that each use SnowflakeOperator to run queries, along with a DAG in `dags/extract` that uses PythonOperator to send generated OpenLineage events to the configured backend.

Airflow example running Snowflake queries and publish OpenLineage events. #2

Airflow example running Snowflake queries and publish OpenLineage events. #2

Conversation

fm100 commented Apr 13, 2022

Description

sfc-gh-kbijon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

julienledem left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rossturk commented Apr 19, 2022

sfc-gh-kbijon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-kbijon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-kbijon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rossturk Apr 21, 2022 • edited Loading

Choose a reason for hiding this comment

rossturk Apr 21, 2022 •

edited

Loading