FIX: do not reattach to finished pod on restart of k8s pod operator

[NotYuki]

For now, if KubernetesPodOperator.reattach_on_restart is enabled, the Operator tries to find a pod regardless of its status. This might lead to an unhealthy deadlock situation if on_finish_action is set to KEEP_POD or DELETE_SUCCEEDED_POD.

For instance, if a task has on_finish_action = OnFinishAction.DELETE_SUCCEEDED_POD and it fails on the first run, any subsequent attempts of the same task will fail immediately with a Kubernetes API error, as Airflow will try to attach to a finished pod.

This PR introduces changes to KPO, allowing it to reattach only to running pods.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[Taragolis]

Could you add a tests. See: https://github.com/apache/airflow/blob/main/contributing-docs/09_testing.rst

[NotYuki]

Thanks for the comment. Actually, after delving deeper into the Kubernetes tests and KPO code, I've noticed similar logic already implemented with patch_already_checked() method. I want to spend some time investigating why it doesn't work as expected on my k8s environment.

[dirrao]

When the next retry attempt comes back, by that time the task might move from running to completed state. Why do you want to rerun the same?

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.