You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When autoscaling with K8s, I have a few minutes of lag between the node appearing as a K8s resources and when the node is ready. For many minutes, this produces logs reporting "Pod not yet started" which likely comes from the K8s event API.
[2024-06-11, 14:12:16 PDT] {pod_manager.py:378} WARNING - Pod not yet started: $SOME_POD_NAME
...
[2024-06-11, 14:18:12 PDT] {pod_manager.py:378} WARNING - Pod not yet started: $SOME_POD_NAME
In the gantt view, I can see the time interval of the above logs coincides with the running state instead of the queued state.
In the docs, the states are defined as
queued: The task has been assigned to an Executor and is awaiting a worker
running: The task is running on a worker (or on a local/synchronous executor)
...and I'm not sure where "task has been assigned but worker is not ready" falls.
Are there any configs that can modify this behavior or am I missing some other mechanic?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
When autoscaling with K8s, I have a few minutes of lag between the node appearing as a K8s resources and when the node is ready. For many minutes, this produces logs reporting "Pod not yet started" which likely comes from the K8s event API.
In the gantt view, I can see the time interval of the above logs coincides with the running state instead of the queued state.
In the docs, the states are defined as
...and I'm not sure where "task has been assigned but worker is not ready" falls.
Are there any configs that can modify this behavior or am I missing some other mechanic?
Beta Was this translation helpful? Give feedback.
All reactions