Fix is_deferrable in airflow agent

[pingsutw]

Tracking issue

flyteorg/flyte#3936

Why are the changes needed?

dataflow operator should be non-deferrable.

What changes were proposed in this pull request?

fix is_deferrable function.

How was this patch tested?

Run dataflow workflow locally and remotely.

import os

from airflow.providers.apache.beam.operators.beam import BeamRunPythonPipelineOperator, BeamRunJavaPipelineOperator
from airflow.providers.google.cloud.operators.dataflow import DataflowConfiguration
from click.testing import CliRunner

from flytekit import workflow, ImageSpec
from flytekit.clis.sdk_in_container import pyflyte

new_airflow_plugin = "git+https://github.com/flyteorg/flytekit.git@2ef41211b7400a7f4b85d70ea7eb719be5135066#subdirectory=plugins/flytekit-airflow"
image_spec = ImageSpec(
    registry="pingsutw",
    base_image="pingsutw/flytekit:dataflow-v2",
    packages=["apache-airflow[google]", "apache-airflow-providers-apache-beam[google]", new_airflow_plugin],
    apt_packages=["git"],
    env={"AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT": "google-cloud-platform://"}
)


@workflow
def wf():
    BeamRunJavaPipelineOperator(
        task_id="start_java_pipeline_dataflow_runner",
        runner="DataflowRunner",
        job_class="org.apache.beam.examples.WordCount",
        pipeline_options={"output": "gs://opta-gcp-dogfood-gcp/spark",
                          "stagingLocation": "gs://opta-gcp-dogfood-gcp/staging",
                          "tempLocation": "gs://opta-gcp-dogfood-gcp/staging1",
                          "region": "us-east1"},
        jar="gs://opta-gcp-dogfood-gcp/spark/word-count-beam-bundled-0.1.jar",
        dataflow_config=DataflowConfiguration(job_name="test_dataflow",
                                              project_id="dogfood-gcp-dataplane",
                                              location="us-east1",
                                              wait_until_finished=False),
        container_image=image_spec
    )

    BeamRunPythonPipelineOperator(
        task_id="start_python_pipeline_dataflow_runner",
        runner="DataflowRunner",
        py_file="gs://opta-gcp-dogfood-gcp/spark/word_count.py",
        pipeline_options={"output": "gs://opta-gcp-dogfood-gcp/spark",
                          "input": "gs://opta-gcp-dogfood-gcp/spark/-00000-of-00001",
                          "stagingLocation": "gs://opta-gcp-dogfood-gcp/staging1",
                          "region": "us-east1",
                          "tempLocation": "gs://opta-gcp-dogfood-gcp/staging1"},
        py_interpreter="python3",
        dataflow_config=DataflowConfiguration(job_name="test_dataflow", project_id="dogfood-gcp-dataplane",
                                              location="us-east1",
                                              wait_until_finished=False),
        container_image=image_spec
    )


if __name__ == '__main__':
    os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = "/Users/kevin/git/flytekit/somekey.json"
    os.environ['AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT'] = "google-cloud-platform://"
    os.environ['FLYTECTL_CONFIG'] = '/Users/kevin/.flyte/config-dogfood-gcp.yaml'
    runner = CliRunner()
    result = runner.invoke(pyflyte.main,
                           ["run", "--remote", "/Users/kevin/git/flytekit/flyte-example/airflow_example/dataflow_java.py",
                            "wf"])
    print(result.output)

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

[codecov]

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (892b474) 85.77% compared to head (2ef4121) 62.42%.

Files	Patch %	Lines
...s/flytekit-airflow/flytekitplugins/airflow/task.py	85.71%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #2109       +/-   ##
===========================================
- Coverage   85.77%   62.42%   -23.36%     
===========================================
  Files         313      313               
  Lines       23500    23506        +6     
  Branches     3512     3512               
===========================================
- Hits        20158    14673     -5485     
- Misses       2734     8410     +5676     
+ Partials      608      423      -185     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.