Under which category would you file this issue?
Airflow Core
Apache Airflow version
3.2.2
What happened and how to reproduce it?
What happened
The scheduler enters an unrecoverable crash loop. On startup and on every
[scheduler] orphaned_tasks_check_interval, adopt_or_reset_orphaned_tasks()
builds a log message by calling repr(ti) on each orphaned TaskInstance.
TaskInstance.__repr__ reads self.state, which is a deferred column. When the
TaskInstance is detached from its SQLAlchemy session, that read triggers a
deferred DB load with no bound session and raises DetachedInstanceError. The
exception is unhandled in _run_scheduler_loop, so the scheduler process exits.
Because the scheduler dies before it finishes resetting the orphaned task, the
same orphan is still present on the next start and it crashes again — a
self-perpetuating loop. Nothing outside the scheduler clears the orphan, so it
never recovers without manual DB intervention. While it loops, no DAGs are
scheduled at all.
Traceback (paths are from the official apache/airflow image):
[error ] Exception when executing SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
File ".../airflow/jobs/scheduler_job_runner.py", line 1546, in _execute
ret = execute_callable()
File ".../airflow/jobs/scheduler_job_runner.py", line 1612, in _run_scheduler_loop
self.adopt_or_reset_orphaned_tasks()
File ".../airflow/utils/session.py", line 100, in wrapper
return func(*args, session=session, **kwargs)
File ".../airflow/jobs/scheduler_job_runner.py", line 2814, in adopt_or_reset_orphaned_tasks
for attempt in run_with_db_retries(logger=self.log):
...
File ".../airflow/jobs/scheduler_job_runner.py", line 2864, in adopt_or_reset_orphaned_tasks
reset_tis_message.append(repr(ti))
File ".../airflow/models/taskinstance.py", line 1134, in __repr__
return prefix + f"[{self.state}] ti_id={self.id}>"
File ".../sqlalchemy/orm/attributes.py", line 569, in __get__
return self.impl.get(state, dict_)
...
File ".../sqlalchemy/orm/strategies.py", line 536, in _load_for_state
raise orm_exc.DetachedInstanceError(
sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <TaskInstance at 0x...>
is not bound to a Session; deferred load operation of attribute 'state' cannot
proceed (Background on this error at: https://sqlalche.me/e/20/bhk3)
### How to reproduce
Using KubernetesExecutor:
1. Start a task so a `TaskInstance` is `running`/`queued` under a worker pod.
2. Cause it to become "orphaned" — e.g. the worker pod is lost, or the
scheduler is killed while the task is in flight (SIGKILL / OOM / node drain)
so the owning job is gone.
3. Start the scheduler. `adopt_or_reset_orphaned_tasks()` selects the orphaned
`TaskInstance`(s); when it reaches `reset_tis_message.append(repr(ti))` the
deferred `state` load fails with `DetachedInstanceError` and the scheduler
exits. It then crash-loops on every restart because the orphan persists.
Manual workaround: set the stuck task instances to a terminal state directly in
the metadata DB (or via the UI "Mark Failed"/CLI from a non-scheduler pod), then
restart the scheduler.
### What you think should happen instead?
Adopting/resetting orphaned tasks must not be able to crash the scheduler, and
in particular building a diagnostic log line must never trigger a lazy DB load
on a possibly-detached instance. `adopt_or_reset_orphaned_tasks()` should either:
- construct the message from the task instance identity/key (already-loaded
columns) instead of `repr(ti)`, or
- make `TaskInstance.__repr__` read only already-loaded attributes (e.g. from
`__dict__`) so it degrades gracefully (`state=?`) rather than raising, or
- keep the instances bound/refreshed within the session while the message is
built.
The task instances should still be reset as intended; only the fatal log call
needs to change.
### Operating System
Debian (official `apache/airflow:slim-3.2.2-python3.12` image), Python 3.12.
### Deployment
Other
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes (KubernetesExecutor). Reproducible with
the version pinned by the 3.2.2 constraints file.
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
Happens repeatedly — every scheduler restart while an orphaned task instance
exists. Related (same code path / theme, but not the same `__repr__`/`state`
lazy-load crash): #30709, #39088, #19671, #58570, #13808.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Under which category would you file this issue?
Airflow Core
Apache Airflow version
3.2.2
What happened and how to reproduce it?
What happened
The scheduler enters an unrecoverable crash loop. On startup and on every
[scheduler] orphaned_tasks_check_interval,adopt_or_reset_orphaned_tasks()builds a log message by calling
repr(ti)on each orphanedTaskInstance.TaskInstance.__repr__readsself.state, which is a deferred column. When theTaskInstanceis detached from its SQLAlchemy session, that read triggers adeferred DB load with no bound session and raises
DetachedInstanceError. Theexception is unhandled in
_run_scheduler_loop, so the scheduler process exits.Because the scheduler dies before it finishes resetting the orphaned task, the
same orphan is still present on the next start and it crashes again — a
self-perpetuating loop. Nothing outside the scheduler clears the orphan, so it
never recovers without manual DB intervention. While it loops, no DAGs are
scheduled at all.
Traceback (paths are from the official
apache/airflowimage):