Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add max_wait for exponential_backoff in BaseSensor #27597

Merged
merged 7 commits into from
Nov 11, 2022

Conversation

hussein-awala
Copy link
Member

We want to make use of exponential_backoff to increase wait times between pokes, but we need to control the maximum wait time as we do with tasks retries when we activate retry_exponential_backoff ,where we can set max_retry_delay to use a fixed delay after a certain number of retries.

@boring-cyborg boring-cyborg bot added the area:core-operators Operators, Sensors and hooks within Core Airflow label Nov 10, 2022
airflow/sensors/base.py Outdated Show resolved Hide resolved
airflow/sensors/base.py Outdated Show resolved Hide resolved
airflow/sensors/base.py Outdated Show resolved Hide resolved
airflow/sensors/base.py Outdated Show resolved Hide resolved
tests/sensors/test_base.py Outdated Show resolved Hide resolved
airflow/sensors/base.py Outdated Show resolved Hide resolved
hussein-awala and others added 2 commits November 10, 2022 21:46
Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>
airflow/decorators/__init__.pyi Outdated Show resolved Hide resolved
airflow/decorators/__init__.pyi Show resolved Hide resolved
@dstandish dstandish changed the title add an argument to limit the wait interval between pokes when exponential_backoff is on Add max_wait for exponential_backoff in BaseSensor Nov 10, 2022
airflow/sensors/base.py Outdated Show resolved Hide resolved
@jedcunningham jedcunningham merged commit cc4cde9 into apache:main Nov 11, 2022
@boring-cyborg
Copy link

boring-cyborg bot commented Nov 11, 2022

Awesome work, congrats on your first merged pull request!

@jedcunningham
Copy link
Member

Thanks @hussein-awala! Congrats on your first commit 🎉

Adityamalik123 pushed a commit to Adityamalik123/airflow that referenced this pull request Nov 12, 2022
Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>
@ephraimbuddy ephraimbuddy added the type:new-feature Changelog: New Features label Nov 16, 2022
@ephraimbuddy ephraimbuddy added this to the Airflow 2.5.0 milestone Nov 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:core-operators Operators, Sensors and hooks within Core Airflow type:new-feature Changelog: New Features
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants