Skip to content

HttpHook.run_with_advanced_retry method does not support Exception other than ConnectionError #9569

@lidalei

Description

@lidalei

Apache Airflow version:
1.10.10

Environment:
Python 3.6.10

What happened:
The following code block

hook = HttpHook(method='GET', http_conn_id='xxx')

retry_args = dict(
    wait=tenacity.wait_exponential(),
    stop=tenacity.stop_after_attempt(10),
    retry=retry_if_exception_type(
        exception_types=(requests.exceptions.ConnectionError, requests.exceptions.HTTPError)
    ),
)

hook.run_with_advanced_retry(retry_args, ...)

does not retry an HTTP request and raises an AirflowException when check_response is True (by default) and the server responds a 429 status code (too many requests).

What you expected to happen:
I expect the retry parameter is respected. So instead of raising AirflowException, the original exception should be raised.

This is because check_response changes the Exception type from HTTPError to AirflowException check_response. However, only requests.exceptions.ConnectionError is caught.

How to reproduce it:
You can understand the issue by reading the source code, without actually doing something to reproduce it.

Anything else we need to know:
The function document of run_with_advanced_retry is wrong. It should be retry=retry_if_exception_type(exception_types=(requests.exceptions.ConnectionError,)).

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions