Only update RTIF for terminal task states

[amoghrajesh]

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[ashb]

Needs some thinking about this -- I'm not sure this is the correct behaviour

[amoghrajesh]

Thanks, I anticipated that too. Let me come up with a design for that

[pankajkoti]

At the moment, we're observing that the rtif endpoint does not exist for such intermediate try_numbers of the task run and the post-processing is blocked (e.g. uploading of logs to remote logging store & the logs are not available for those initial try_numbers of the task run) due to the AirflowRuntimeError that gets raised from the flow here.

Logs from the worker

Assigning outlets with DatasetAlias in Airflow 3
[2026-03-16 15:48:23] ERROR - Task failed with exception
CosmosDbtRunError: error here
File "/usr/local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1112 in run
File "/usr/local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1528 in _execute_task
File "/usr/local/astronomer-cosmos/cosmos/operators/base.py", line 342 in execute
File "/usr/local/astronomer-cosmos/cosmos/operators/local.py", line 895 in build_and_run_cmd
File "/usr/local/astronomer-cosmos/cosmos/operators/local.py", line 692 in run_command
File "/usr/local/astronomer-cosmos/cosmos/operators/local.py", line 273 in handle_exception_dbt_runner
File "/usr/local/astronomer-cosmos/cosmos/dbt/runner.py", line 153 in handle_exception_if_needed

[2026-03-16 15:48:23] ERROR - Failed to set rendered fields during finalization task_id=raw_customers_seed dag_id=basic_cosmos_dag
AirflowRuntimeError: API_SERVER_ERROR: {'status_code': 404, 'message': 'Not Found', 'detail': {'detail': 'Not Found'}}
File "/usr/local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1627 in finalize
File "/usr/local/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py", line 206 in send
File "/usr/local/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py", line 270 in _get_response
File "/usr/local/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py", line 257 in _from_frame

404 errors we're seeing in the API server

2026-03-16T10:22:46.992666Z [info     ] Task instance state updated    [airflow.api_fastapi.execution_api.routes.task_instances] loc=task_instances.py:424 new_state=up_for_retry rows_affected=1 ti_id=019cf62b-1924-76c0-b29b-696f4c9867c0
192.168.97.6:34318 - "PATCH /execution/task-instances/019cf62b-1924-76c0-b29b-696f4c9867c0/state HTTP/1.1" 204
2026-03-16T10:22:47.002805Z [info     ] Updating RenderedTaskInstanceFields [airflow.api_fastapi.execution_api.routes.task_instances] field_count=10 loc=task_instances.py:687 ti_id=019cf62b-1924-76c0-b29b-696f4c9867c0
2026-03-16T10:22:47.003740Z [error    ] Task Instance not found        [airflow.api_fastapi.execution_api.routes.task_instances] loc=task_instances.py:691 ti_id=019cf62b-1924-76c0-b29b-696f4c9867c0
192.168.97.6:34318 - "PUT /execution/task-instances/019cf62b-1924-76c0-b29b-696f4c9867c0/rtif HTTP/1.1" 404
2026-03-16T10:22:47.176682Z [info     ] Task instance state updated    [airflow.api_fastapi.execution_api.routes.task_instances] loc=task_instances.py:424 new_state=up_for_retry rows_affected=1 ti_id=019cf62b-1922-7cc4-900b-ac0a636cb0b0
192.168.97.6:34312 - "PATCH /execution/task-instances/019cf62b-1922-7cc4-900b-ac0a636cb0b0/state HTTP/1.1" 204
2026-03-16T10:22:47.187138Z [info     ] Updating RenderedTaskInstanceFields [airflow.api_fastapi.execution_api.routes.task_instances] field_count=10 loc=task_instances.py:687 ti_id=019cf62b-1922-7cc4-900b-ac0a636cb0b0
2026-03-16T10:22:47.188136Z [error    ] Task Instance not found        [airflow.api_fastapi.execution_api.routes.task_instances] loc=task_instances.py:691 ti_id=019cf62b-1922-7cc4-900b-ac0a636cb0b0
192.168.97.6:34312 - "PUT /execution/task-instances/019cf62b-1922-7cc4-900b-ac0a636cb0b0/rtif HTTP/1.1" 404
2026-03-16T10:22:48.079401Z [info     ] Task instance state updated    [airflow.api_fastapi.execution_api.routes.task_instances] loc=task_instances.py:424 new_state=up_for_retry rows_affected=1 ti_id=019cf62b-1923-70cb-a656-1cc31f3bd788
192.168.97.6:34320 - "PATCH /execution/task-instances/019cf62b-1923-70cb-a656-1cc31f3bd788/state HTTP/1.1" 204
2026-03-16T10:22:48.090230Z [info     ] Updating RenderedTaskInstanceFields [airflow.api_fastapi.execution_api.routes.task_instances] field_count=10 loc=task_instances.py:687 ti_id=019cf62b-1923-70cb-a656-1cc31f3bd788
2026-03-16T10:22:48.091175Z [error    ] Task Instance not found        [airflow.api_fastapi.execution_api.routes.task_instances] loc=task_instances.py:691 ti_id=019cf62b-1923-70cb-a656-1cc31f3bd788
192.168.97.6:34320 - "PUT /execution/task-instances/019cf62b-1923-70cb-a656-1cc31f3bd788/rtif HTTP/1.1" 404

[amoghrajesh]

Thanks for sharing that @pankajkoti, I am thinking of a better way to do things