-
Notifications
You must be signed in to change notification settings - Fork 16.6k
Description
Apache Airflow version:
1.10.10
Environment:
Python 3.6.10
What happened:
The following code block
hook = HttpHook(method='GET', http_conn_id='xxx')
retry_args = dict(
wait=tenacity.wait_exponential(),
stop=tenacity.stop_after_attempt(10),
retry=retry_if_exception_type(
exception_types=(requests.exceptions.ConnectionError, requests.exceptions.HTTPError)
),
)
hook.run_with_advanced_retry(retry_args, ...)
does not retry an HTTP request and raises an AirflowException when check_response is True (by default) and the server responds a 429 status code (too many requests).
What you expected to happen:
I expect the retry parameter is respected. So instead of raising AirflowException, the original exception should be raised.
This is because check_response changes the Exception type from HTTPError to AirflowException check_response. However, only requests.exceptions.ConnectionError is caught.
How to reproduce it:
You can understand the issue by reading the source code, without actually doing something to reproduce it.
Anything else we need to know:
The function document of run_with_advanced_retry is wrong. It should be retry=retry_if_exception_type(exception_types=(requests.exceptions.ConnectionError,)).