Implement `--export-slice-airflow` #320

marov · 2021-10-18T22:54:30Z

Closes #275

Adds a new CLI option --export-slice-airflow (or simply --airflow
Exports to a working DAG sliced_housing_dag.py
Currently simply prints out p from lineapy tests/housing.py --slice "p value" --airflow sliced_housing_dag
Install airflow
Tests and doc

saulshanabrook · 2021-10-18T23:01:24Z

lineapy/instrumentation/tracer.py

@@ -156,6 +161,30 @@ def sliced_func(self, slice_name: str, func_name: str) -> str:
        full_code = format_str(full_code, mode=black_mode)
        return full_code

+    def sliced_aiflow_dag(self, slice_name: str, func_name: str) -> str:


What do you think about moving the airflow specific code to a different folder, like linea/airflow?

I've created plugins/airflow.py, wdyt?

marov · 2021-10-19T05:40:00Z

@saulshanabrook @yifanwu This produces a valid DAG, but I still need to add tests, hence Draft.
Will be adding tests tomorrow evening

yifanwu · 2021-10-19T21:08:03Z

Nit: please add the example to README: lineapy --slice 'p value' --export-slice sliced_housing tests/housing.py

Please also add instructions in the readme about how we can run the airflow jobs?

yifanwu · 2021-10-19T21:08:42Z

Also for future reference, please have, in your PR description, a brief summary of your changes?

…e blocking airflow work. will sync up with Saul to see if this is acceptable or if we have a better solution.

yifanwu · 2021-10-21T00:26:06Z

lineapy/plugins/airflow.py

+        + AIRFLOW_IMPORTS_TEMPLATE
+        + "\n\n"
+        + code_block
+        + f"\n\tprint({artifact_var})"  # TODO What to do with artifact_var in a DAG?


I think we need to discuss and design this in more detail before we can commit this PR. @marov you are better equipped than us to help design what to do with this artifact in a deployed airflow process. Let's look at some reference cases together on our next meeting (Friday).

yifanwu · 2021-10-21T00:29:44Z

CONTRIBUTING.md

@@ -84,6 +84,19 @@ jupyter nbconvert --to notebook --execute tests/test_notebook.ipynb --inplace --
 Or you can open it in a notebook UI (JupyterLab, JupyterNotebook, VS Code, etc.)
 and re-run it manually

+### Airflow


This should really be in the README.md, but since it's not ready for "release", maybe we can create another doc called EXPERIMENTAL.md or something and move this there.

I was under impression that CONTRIBUTING.md serves this function.

yifanwu · 2021-10-21T00:31:08Z

CONTRIBUTING.md

+```
+lineapy tests/housing.py --slice "p value" --airflow sliced_housing_dag
+```
+This creates a `sliced_housing_dag.py` file in the current dir. It can be executed with:


Could we move this into a separate folder (when generating)? Cleaner organization.

Sure, how would you call it? output or export?

Or something like airflow_jobs/airflow_dags?

With this we are saving to the same dir where the original file is, which is tests for the housing.py. This seems to be OK for me.
I can change to any other place of course :)

yifanwu · 2021-10-21T00:35:58Z

lineapy/plugins/airflow.py

+
+dag = DAG(
+    dag_id="DAG_NAME_dag",
+    schedule_interval="*/15 * * * *",  # Every 15 minutes


We need to figure out how to parametrize this for API design. I'm not sure about the workflow where the user goes in and manually modifies things.

Yes, but we need to think about the entire UX. We can't just keep adding options to CLI, right?
I think a config filler a set of configs are in order. I'd vote for YAML, but open to anything

Yes let's talk about it this afternoon.

yifanwu · 2021-10-21T00:40:54Z

lineapy/plugins/airflow.py

+default_dag_args = {"owner": "airflow", "retries": 2, "start_date": days_ago(1)}
+
+dag = DAG(
+    dag_id="DAG_NAME_dag",


What happens if we have two artifacts? Will this logic break?

Currently we can only slice one artifact through --slice, it produces code which we save as this DAG. If there are other artifacts in DB it won't be a problem.

Sorry I'm not following, do the dag_ids not need to be unique?

saulshanabrook · 2021-10-21T17:47:20Z

I was trying to debug the mypy error. I re-run an earlier commit here on CI and found this error in install:

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
The conflict is caused by:
    lineapy[dev] 0.0.1 depends on click>=8.0.0
    black 18.3a0 depends on click
    lineapy 0.0.1 depends on click>=8.0.0
    flask 1.1.4 depends on click<8.0 and >=5.1
    lineapy[dev] 0.0.1 depends on click>=8.0.0
    black 18.3a0 depends on click
    lineapy 0.0.1 depends on click>=8.0.0
    flask 1.1.3 depends on click<8.0 and >=5.1

I wonder why flask is being pulled in?

marov · 2021-10-21T23:43:54Z

I was trying to debug the mypy error. I re-run an earlier commit here on CI and found this error in install:

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
The conflict is caused by:
    lineapy[dev] 0.0.1 depends on click>=8.0.0
    black 18.3a0 depends on click
    lineapy 0.0.1 depends on click>=8.0.0
    flask 1.1.4 depends on click<8.0 and >=5.1
    lineapy[dev] 0.0.1 depends on click>=8.0.0
    black 18.3a0 depends on click
    lineapy 0.0.1 depends on click>=8.0.0
    flask 1.1.3 depends on click<8.0 and >=5.1

I wonder why flask is being pulled in?

I think Airflow uses flask for it's UI, see here https://github.com/apache/airflow/blob/f5ad26dcdd7bcb724992528dce71056965b94d26/setup.py#L422

saulshanabrook · 2021-10-22T00:26:58Z

@marov thanks for checking that out. I am trying to debug the versioning problems with mypy. It looks like currently it's installing an older version of sqlalchemy, because of some transitive dependency of airflow, which doesn't include the mypy plugin, which is breaking the tests. I am looking into if there is a solution that can work for us, to keep a newer version of sqlalchemy.

saulshanabrook · 2021-10-22T01:04:09Z

it seems the issue is that flask-appbuilder depends on an old version of sqlalchemy (dpgaspar/Flask-AppBuilder#1710), which is in turn waiting on this PR to be merged into flask-sqlalchemy (pallets-eco/flask-sqlalchemy#1001)

saulshanabrook · 2021-10-22T01:12:32Z

pyproject.toml

@@ -20,7 +20,7 @@ relative_files = true

 # https://docs.sqlalchemy.org/en/14/orm/extensions/mypy.html
 # https://pydantic-docs.helpmanual.io/mypy_plugin/#enabling-the-plugin
-plugins = ["sqlalchemy.ext.mypy.plugin", "pydantic.mypy"]


Could we try just removing the sqlalchemy plugin but keeping the mypy one? And commenting that its disabled until we can add sqlalchemy >= 1.4, which adds mypy support, but is blocked on flask-appbuilder, which is a dep of airflow?

We could do that. However, as I've mentioned to @yifanwu today, Airflow is but one of the data tools that linea can have "plugins" to.
New tools will come with new dependencies and more issues like this.
Hence I propose:

Merge this PR (removing sqlalchemy plugin and leaving a TODO in comments)

Next create in a different repo (or just a folder?) a place for external plugins with examples and tests

Each of the "plugins" will have its own setup.py decoupling it from the rest

Is Jupyter also falls in this category? One can argue both ways - we can say that Jupyter support is the core feature of lineapy, but I'd argue that it lineapy can work without Jupyter. And Jupyter is not the only notebook server either (Zeppelin etc.)

Next create in a different repo (or just a folder?) a place for external plugins with examples and tests

I am +1 on monorepos when the code is tightly coupled!

Ok, @yifanwu @saulshanabrook how shall we proceed?

saulshanabrook · 2021-10-25T17:31:08Z

We talked about this at standup today and sounds good to merge now.

marov added 3 commits October 17, 2021 23:46

WIP airflow export

8081eee

typo

5623ced

Merge branch 'main' into export-slice-airflow

de9a525

saulshanabrook reviewed Oct 18, 2021

View reviewed changes

marov added 2 commits October 18, 2021 22:05

Refactor

76c036d

Optput a working DAG

c4e119e

marov requested a review from yifanwu October 19, 2021 05:32

marov added 2 commits October 18, 2021 22:34

Unused imports

ff8fb3f

Return type

de89e7a

Add test and docs

06bf582

marov marked this pull request as ready for review October 20, 2021 06:46

marov and others added 7 commits October 19, 2021 23:58

Update docstring

6448004

Airflow without version

fdbfc0f

no airflow install

186af91

apache-airflow==2.1.3

acae984

2.0.1

1dc2cdd

mypy==0.770

0caa77f

skipping mypy plugins (sqlalchemy and pydantic) for now since they ar…

c0b2745

…e blocking airflow work. will sync up with Saul to see if this is acceptable or if we have a better solution.

marov requested a review from lionsardesai October 20, 2021 21:09

yifanwu reviewed Oct 21, 2021

View reviewed changes

yifanwu mentioned this pull request Oct 21, 2021

UX Design: Deploying Workflows #236

Open

saulshanabrook reviewed Oct 22, 2021

View reviewed changes

Merge branch 'main' into export-slice-airflow

20e0899

saulshanabrook approved these changes Oct 25, 2021

View reviewed changes

saulshanabrook merged commit 0ef50e4 into main Oct 25, 2021

saulshanabrook deleted the export-slice-airflow branch October 25, 2021 17:31

saulshanabrook self-assigned this Oct 25, 2021

dorx unassigned saulshanabrook Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement `--export-slice-airflow` #320

Implement `--export-slice-airflow` #320

marov commented Oct 18, 2021 •

edited

saulshanabrook Oct 18, 2021

marov Oct 19, 2021

marov commented Oct 19, 2021

yifanwu commented Oct 19, 2021 •

edited

yifanwu commented Oct 19, 2021

yifanwu Oct 21, 2021 •

edited

yifanwu Oct 21, 2021

marov Oct 21, 2021

yifanwu Oct 21, 2021

marov Oct 21, 2021

yifanwu Oct 21, 2021

marov Oct 24, 2021

yifanwu Oct 21, 2021

marov Oct 21, 2021

yifanwu Oct 21, 2021

yifanwu Oct 21, 2021 •

edited

marov Oct 21, 2021

yifanwu Oct 21, 2021

saulshanabrook commented Oct 21, 2021

marov commented Oct 21, 2021

saulshanabrook commented Oct 22, 2021 •

edited

saulshanabrook commented Oct 22, 2021

saulshanabrook Oct 22, 2021

marov Oct 22, 2021

saulshanabrook Oct 22, 2021

marov Oct 22, 2021

saulshanabrook commented Oct 25, 2021

Implement --export-slice-airflow #320

Implement --export-slice-airflow #320

Conversation

marov commented Oct 18, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marov commented Oct 19, 2021

yifanwu commented Oct 19, 2021 • edited

yifanwu commented Oct 19, 2021

yifanwu Oct 21, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yifanwu Oct 21, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saulshanabrook commented Oct 21, 2021

marov commented Oct 21, 2021

saulshanabrook commented Oct 22, 2021 • edited

saulshanabrook commented Oct 22, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saulshanabrook commented Oct 25, 2021

Implement `--export-slice-airflow` #320

Implement `--export-slice-airflow` #320

marov commented Oct 18, 2021 •

edited

yifanwu commented Oct 19, 2021 •

edited

yifanwu Oct 21, 2021 •

edited

yifanwu Oct 21, 2021 •

edited

saulshanabrook commented Oct 22, 2021 •

edited