Graceful handle for airflow application pods during K8s API brief interuption

[nayanen]

Description

Description

I have come across this issue where k8s customer using airflow application. The pod get terminated when there is brief API server interruption. API server interruption can happen due to lot of scenarios so ideally the application should handle it gracefully. Another key thing here is this happens only when you set the "is_delete_pod_operator" set to True.

Now if i read the code i understand that the launcher.start_pod or launcher.monitor_pod have some logic which check heartbeat to the AI server and once its find some interruption it goes to the AirflowException part. So if we have some kind of nested exception or some retry logic to handle the brief interruption of API sever unavailability that should help to avoid the pod termination, because it just takes very less time for API server to re-establish the connection.

             'airflow_version': airflow_version.replace('+', '-'),
                'kubernetes_pod_operator': 'True',
            }
        )

        self.log.debug("Starting pod:\n%s", yaml.safe_dump(self.pod.to_dict()))
        final_state = None
        try:
            launcher.start_pod(self.pod, startup_timeout=self.startup_timeout_seconds)
            final_state, remote_pod, result = launcher.monitor_pod(pod=self.pod, get_logs=self.get_logs)
        except AirflowException:

The kubernetesPodOperator is provided as open-source Airflow and more details about the behavior can be found below.

is_delete_operator_pod (bool) – What to do when the pod reaches its final state, or the execution is interrupted. If False (default): do nothing, If True: delete the pod

Use case/motivation

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[raphaelauv]

WIP -> #19572

[uranusjr]

Assigning to avoid acceidental overlap.

[raphaelauv]

I think we can close this issue , now that the new kubernetesPodOperator is retrying.