DAGs with external python task are not killed when marked as Failed.

[samkelemen]

It seems that any dag using an external python task are not killed when they are marked as Failed in the UI. Does anyone know what is happening here?

For example, this dag (... is just a placeholder in this example. The actual dag would need the necessary params defined):

def dummy_dag():
    @task.external_python(
         ...
    )
    def external_python_task():
        import os
        import signal
        import time

        while True:
            print("alive", flush=True)
            print(signal.getsignal(signal.SIGTERM))
            time.sleep(1) 

    external_python_task()

dummy_dag()

When I click the "Failed" button in the UI to mark the dag as failed, I get the following log:

[2026-10-05 14:06:01] INFO - Executing cmd: /opt/airflow/venvs/pipeline-test/bin/python /tmp/venv-callhakxr1ad/script.py /tmp/venv-callhakxr1ad/script.in /tmp/venv-callhakxr1ad/script.out /tmp/venv-callhakxr1ad/string_args.txt /tmp/venv-callhakxr1ad/termination.log /tmp/venv-callhakxr1ad/airflow_context.json source=airflow.providers.standard.utils.python_virtualenv pathname=/home/airflow/.local/lib/python3.13/site-packages/airflow/providers/standard/utils/python_virtualenv.py loc=python_virtualenv.py:147
[2026-10-05 14:06:01] INFO - Output: source=airflow.providers.standard.utils.python_virtualenv pathname=/home/airflow/.local/lib/python3.13/site-packages/airflow/providers/standard/utils/python_virtualenv.py loc=python_virtualenv.py:157
[2026-10-05 14:06:02] INFO - alive source=airflow.providers.standard.utils.python_virtualenv pathname=/home/airflow/.local/lib/python3.13/site-packages/airflow/providers/standard/utils/python_virtualenv.py loc=python_virtualenv.py:161
[2026-10-05 14:06:03] INFO - 0 source=airflow.providers.standard.utils.python_virtualenv pathname=/home/airflow/.local/lib/python3.13/site-packages/airflow/providers/standard/utils/python_virtualenv.py loc=python_virtualenv.py:161
[2026-10-05 14:06:03] INFO - alive source=airflow.providers.standard.utils.python_virtualenv pathname=/home/airflow/.local/lib/python3.13/site-packages/airflow/providers/standard/utils/python_virtualenv.py loc=python_virtualenv.py:161
...
[2026-10-05 14:06:11] ERROR - Server indicated the task shouldn't be running anymore. Terminating process detail={"reason":"not_running","message":"TI is no longer in the running state and task should terminate","current_state":"failed"} source=task
[2026-10-05 14:06:12] INFO - 0 source=airflow.providers.standard.utils.python_virtualenv pathname=/home/airflow/.local/lib/python3.13/site-packages/airflow/providers/standard/utils/python_virtualenv.py loc=python_virtualenv.py:161
[2026-10-05 14:06:12] INFO - alive source=airflow.providers.standard.utils.python_virtualenv pathname=/home/airflow/.local/lib/python3.13/site-packages/airflow/providers/standard/utils/python_virtualenv.py loc=python_virtualenv.py:161
[2026-10-05 14:06:13] INFO - 0 source=airflow.providers.standard.utils.python_virtualenv pathname=/home/airflow/.local/lib/python3.13/site-packages/airflow/providers/standard/utils/python_virtualenv.py loc=python_virtualenv.py:161
[2026-10-05 14:06:13] INFO - alive source=airflow.providers.standard.utils.python_virtualenv pathname=/home/airflow/.local/lib/python3.13/site-packages/airflow/providers/standard/utils/python_virtualenv.py loc=python_virtualenv.py:161
[2026-10-05 14:06:14] INFO - 0 source=airflow.providers.standard.utils.python_virtualenv pathname=/home/airflow/.local/lib/python3.13/site-packages/airflow/providers/standard/utils/python_virtualenv.py loc=python_virtualenv.py:161
...

I am using Airflow 3.2.2.

[CaptainAni187]

This is a known bug in the 3.2 line, and it's fixed in Airflow 3.3.1.

When you mark the task as failed, the next heartbeat from the task gets a 409 and the supervisor kills the task process. That's the "Server indicated the task shouldn't be running anymore" line in your log. In 3.2.2 the signal only goes to the task-runner process itself. Your @task.external_python function runs in a separate Python process that the task-runner starts with subprocess.Popen, so it never receives the signal. Once the task-runner is killed, that child gets reparented and keeps running, which is why your log keeps printing alive after the termination message.

It was reported in #65505 and fixed by #65738. The fix puts the task-runner in its own process group and sends the signal to the whole group with os.killpg, so venv and external Python children get it too. I checked the release tags, and that code is in 3.3.1 and 3.3.2 but not in 3.2.2 or 3.3.0, so upgrading to 3.3.1 or later should fix it.

If you can't upgrade yet, one workaround is to have the long-running function notice that its parent is gone and exit on its own:

import os
import sys
import time

parent = os.getppid()
while True:
    if os.getppid() != parent:
        sys.exit(1)
    print("alive", flush=True)
    time.sleep(1)

That only helps for code that loops like this, but it stops the orphaned process from running forever.

[samkelemen]

Thanks so much for the help! I upgraded and the dag was killed.

[mdmmuzzamil]

This is an interesting issue with external Python tasks. The logs clearly show that the task runner detects the failed state and attempts to terminate the process, but the external Python process continues running afterward. It would be useful to understand how the termination signal is propagated to the external environment and whether there is a recommended way to ensure the child process is cleaned up properly.

[samkelemen]

@CaptainAni187's response answers this well