New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AIRFLOW-850] Add a PythonSensor #4349

Merged
merged 1 commit into from Dec 22, 2018

Conversation

Projects
None yet
4 participants
@feng-tao
Copy link
Contributor

feng-tao commented Dec 21, 2018

Make sure you have checked all steps below.

Jira

  • My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

Description

  • Here are some details about my PR, including screenshots of any UI changes:
    A general purpose PythonSensor which allows an arbitrary Python callable to delay Task execution until the callable returns True. This is based on a stale pr(#2058)

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:
    yes

Commits

  • My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it.
    • When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
    • All the public functions and the classes in the PR contain docstrings that explain what it does

Code Quality

  • Passes flake8
@feng-tao

This comment has been minimized.

Copy link
Contributor Author

feng-tao commented Dec 21, 2018

PTAL @ashb @Fokko , this is based on the stale pr (#2058) which I think is a good addition to Airflow sensor.


class PythonSensor(BaseSensorOperator):
"""
Waits for a Python callable to return True

This comment has been minimized.

@ashb

ashb Dec 21, 2018

Member

Adding a short example here in the docs specifying what arguments are passed to the callable, how to access templates_ditc etc would be good I think

This comment has been minimized.

@feng-tao

feng-tao Dec 22, 2018

Author Contributor

updated

if self.provide_context:
context.update(self.op_kwargs)
context['templates_dict'] = self.templates_dict
self.op_kwargs = context

This comment has been minimized.

@ashb

ashb Dec 21, 2018

Member

Does it matter that we end up doing this each time we poke?

This comment has been minimized.

@feng-tao

feng-tao Dec 22, 2018

Author Contributor

I think we need to have a way to access the context for the sensor. Not sure if there is a better way to do so. Besides, I think it is ok to do that as it is not an expensive operation.

@feng-tao feng-tao force-pushed the feng-tao:airflow-850 branch from 273b01d to 21e8df4 Dec 22, 2018

@feng-tao

This comment has been minimized.

Copy link
Contributor Author

feng-tao commented Dec 22, 2018

hey @ashb , update the pr with an example. PTAL.

@codecov-io

This comment has been minimized.

Copy link

codecov-io commented Dec 22, 2018

Codecov Report

Merging #4349 into master will increase coverage by 61.98%.
The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff             @@
##           master    #4349       +/-   ##
===========================================
+ Coverage   16.13%   78.12%   +61.98%     
===========================================
  Files         202      202               
  Lines       16483    16483               
===========================================
+ Hits         2660    12877    +10217     
+ Misses      13823     3606    -10217
Impacted Files Coverage Δ
airflow/plugins_manager.py 93.1% <0%> (+1.14%) ⬆️
airflow/executors/dask_executor.py 2% <0%> (+2%) ⬆️
airflow/exceptions.py 100% <0%> (+2.85%) ⬆️
airflow/utils/operator_resources.py 86.95% <0%> (+4.34%) ⬆️
airflow/executors/__init__.py 63.46% <0%> (+5.76%) ⬆️
airflow/utils/module_loading.py 100% <0%> (+10.52%) ⬆️
airflow/utils/decorators.py 91.66% <0%> (+14.58%) ⬆️
airflow/settings.py 80.41% <0%> (+14.68%) ⬆️
airflow/hooks/oracle_hook.py 15.47% <0%> (+15.47%) ⬆️
airflow/__init__.py 80% <0%> (+15.55%) ⬆️
... and 161 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f211390...21e8df4. Read the comment docs.

@kaxil

kaxil approved these changes Dec 22, 2018

@kaxil kaxil merged commit 0e36566 into apache:master Dec 22, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

kaxil added a commit that referenced this pull request Dec 22, 2018

aliceabe pushed a commit to aliceabe/incubator-airflow that referenced this pull request Jan 3, 2019

kaxil added a commit that referenced this pull request Jan 9, 2019

ashb added a commit to ashb/airflow that referenced this pull request Jan 10, 2019

cfei18 pushed a commit to cfei18/incubator-airflow that referenced this pull request Jan 23, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment