Repository navigation
Scheduled tasks: tell someone when runs keep failing, and pause after N failures in a row #30444
josephv123
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Problem. A scheduled task bound to a long-running thread fired 18 times in a row over about 5 days, and every run failed with "the prompt exceeds the model's context window". That error can't clear up on its own, so the task would have kept failing at every run. A second task failed 15 hourly runs in a row on DNS errors (
EAI_AGAIN) during a network outage.Nothing told me. Both tasks kept showing
succeeded(that part was #17889, fixed in #34815), and an agent that checkedlist_scheduled_tasksbelieved it. The errors were only inside the bound thread, which nobody was reading. I found them almost 5 days later, by accident.With #34815 the status is right, but you still have to go look at the task to see it.
Proposal. Building on the real run outcome from #34815:
list_scheduled_tasks.The same ask came up in a comment on #17889, where a bound task failed 315 wakes in a row.
Update, Nov 12. #35081 adds a pause after 5 failed runs in a row, but it counts only failed dispatches, so the streaks above would not trigger it (details in my comment there). Notification is still open.
All reactions