Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

signals.TRIGGERFAIL leading to retry attemp reset #1177

Closed
occoder opened this issue Jun 25, 2019 · 3 comments · Fixed by #1556
Closed

signals.TRIGGERFAIL leading to retry attemp reset #1177

occoder opened this issue Jun 25, 2019 · 3 comments · Fixed by #1556

Comments

@occoder
Copy link

occoder commented Jun 25, 2019

Each TriggerFailed state should increment retry attempt number and if retry runs out the task is deemed as a Failed state and the next task starts to run.
However, when I executed following code.

import datetime
from prefect import Flow, Task
from prefect.triggers import manual_only
from prefect.engine import state, signals


class TaskFactory(Task):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def run(self):
        print('Task starts running ------------------------------------------------- Task starts running')
        raise signals.TRIGGERFAIL('Failed due to no external triggering from host')


def on_state_change(task, old_state, new_state):
    print('current work flow running state is '+str(new_state))
    if isinstance(new_state, state.Paused):
        return state.Resume()
    if isinstance(new_state, state.TriggerFailed):
        print('No triggering from host.')


flow = Flow('demo_flow')
tasks_in_sequence = []
for _ in range(3):
    current_task = TaskFactory(name='demo_task_'+str(_), max_retries=5, retry_delay=datetime.timedelta(seconds=1),
                               trigger=manual_only, state_handlers=[on_state_change], timeout=2,
                               skip_on_upstream_skip=False)
    flow.add_task(current_task)
    tasks_in_sequence.append(current_task)

flow.chain(*tasks_in_sequence)

flow.run()

I was always stuck at the first task.

[2019-06-25 02:36:17,690] INFO - prefect.FlowRunner | Beginning Flow run for 'demo_flow'
current work flow running state is <Paused: "Trigger function is "manual_only"">
[2019-06-25 02:36:17,690] INFO - prefect.FlowRunner | Starting flow run.
[2019-06-25 02:36:17,690] INFO - prefect.TaskRunner | Task 'demo_task_0': Starting task run...
[2019-06-25 02:36:17,690] INFO - prefect.TaskRunner | Task 'demo_task_0': finished task run for task with final state: 'Resume'
[2019-06-25 02:36:17,690] INFO - prefect.TaskRunner | Task 'demo_task_1': Starting task run...
[2019-06-25 02:36:17,690] INFO - prefect.TaskRunner | Task 'demo_task_1': finished task run for task with final state: 'Pending'
[2019-06-25 02:36:17,690] INFO - prefect.TaskRunner | Task 'demo_task_2': Starting task run...
[2019-06-25 02:36:17,690] INFO - prefect.TaskRunner | Task 'demo_task_2': finished task run for task with final state: 'Pending'
[2019-06-25 02:36:17,690] INFO - prefect.FlowRunner | Flow run RUNNING: terminal tasks are incomplete.
[2019-06-25 02:36:17,690] INFO - prefect.FlowRunner | Beginning Flow run for 'demo_flow'
current work flow running state is <Running: "Starting task run.">
Task starts running ------------------------------------------------- Task starts running
current work flow running state is <TriggerFailed: "Failed due to no external triggering from host">
No triggering from host.
current work flow running state is <Retrying: "Retrying Task (after attempt 1 of 6)">
[2019-06-25 02:36:17,706] INFO - prefect.TaskRunner | Task 'demo_task_0': Starting task run...
[2019-06-25 02:36:17,706] INFO - prefect.TaskRunner | Task 'demo_task_0': finished task run for task with final state: 'Retrying'
[2019-06-25 02:36:17,706] INFO - prefect.TaskRunner | Task 'demo_task_1': Starting task run...
[2019-06-25 02:36:17,706] INFO - prefect.TaskRunner | Task 'demo_task_1': finished task run for task with final state: 'Pending'
[2019-06-25 02:36:17,706] INFO - prefect.TaskRunner | Task 'demo_task_2': Starting task run...
[2019-06-25 02:36:17,706] INFO - prefect.TaskRunner | Task 'demo_task_2': finished task run for task with final state: 'Pending'
[2019-06-25 02:36:17,706] INFO - prefect.FlowRunner | Flow run RUNNING: terminal tasks are incomplete.
[2019-06-25 02:36:17,706] INFO - prefect.Flow | Waiting for next available Task run at 2019-06-25T02:36:18.706132+00:00
[2019-06-25 02:36:18,721] INFO - prefect.FlowRunner | Beginning Flow run for 'demo_flow'
[2019-06-25 02:36:18,721] INFO - prefect.TaskRunner | Task 'demo_task_0': Starting task run...
current work flow running state is <Paused: "Trigger function is "manual_only"">
current work flow running state is <Running: "Starting task run.">
Task starts running ------------------------------------------------- Task starts running
current work flow running state is <TriggerFailed: "Failed due to no external triggering from host">
No triggering from host.
current work flow running state is <Retrying: "Retrying Task (after attempt 1 of 6)">
[2019-06-25 02:36:18,721] INFO - prefect.TaskRunner | Task 'demo_task_0': finished task run for task with final state: 'Resume'
[2019-06-25 02:36:18,721] INFO - prefect.TaskRunner | Task 'demo_task_1': Starting task run...
[2019-06-25 02:36:18,721] INFO - prefect.TaskRunner | Task 'demo_task_1': finished task run for task with final state: 'Pending'
[2019-06-25 02:36:18,721] INFO - prefect.TaskRunner | Task 'demo_task_2': Starting task run...
[2019-06-25 02:36:18,721] INFO - prefect.TaskRunner | Task 'demo_task_2': finished task run for task with final state: 'Pending'
[2019-06-25 02:36:18,721] INFO - prefect.FlowRunner | Flow run RUNNING: terminal tasks are incomplete.
[2019-06-25 02:36:18,721] INFO - prefect.FlowRunner | Beginning Flow run for 'demo_flow'
[2019-06-25 02:36:18,721] INFO - prefect.TaskRunner | Task 'demo_task_0': Starting task run...
[2019-06-25 02:36:18,721] INFO - prefect.TaskRunner | Task 'demo_task_0': finished task run for task with final state: 'Retrying'
[2019-06-25 02:36:18,721] INFO - prefect.TaskRunner | Task 'demo_task_1': Starting task run...
[2019-06-25 02:36:18,721] INFO - prefect.TaskRunner | Task 'demo_task_1': finished task run for task with final state: 'Pending'
[2019-06-25 02:36:18,721] INFO - prefect.TaskRunner | Task 'demo_task_2': Starting task run...
[2019-06-25 02:36:18,721] INFO - prefect.TaskRunner | Task 'demo_task_2': finished task run for task with final state: 'Pending'
[2019-06-25 02:36:18,721] INFO - prefect.FlowRunner | Flow run RUNNING: terminal tasks are incomplete.
[2019-06-25 02:36:18,721] INFO - prefect.Flow | Waiting for next available Task run at 2019-06-25T02:36:19.721694+00:00
current work flow running state is <Paused: "Trigger function is "manual_only"">
[2019-06-25 02:36:19,737] INFO - prefect.FlowRunner | Beginning Flow run for 'demo_flow'
current work flow running state is <Running: "Starting task run.">
[2019-06-25 02:36:19,737] INFO - prefect.TaskRunner | Task 'demo_task_0': Starting task run...
[2019-06-25 02:36:19,737] INFO - prefect.TaskRunner | Task 'demo_task_0': finished task run for task with final state: 'Resume'
[2019-06-25 02:36:19,737] INFO - prefect.TaskRunner | Task 'demo_task_1': Starting task run...
Task starts running ------------------------------------------------- Task starts running
[2019-06-25 02:36:19,737] INFO - prefect.TaskRunner | Task 'demo_task_1': finished task run for task with final state: 'Pending'
current work flow running state is <TriggerFailed: "Failed due to no external triggering from host">
[2019-06-25 02:36:19,737] INFO - prefect.TaskRunner | Task 'demo_task_2': Starting task run...
No triggering from host.
current work flow running state is <Retrying: "Retrying Task (after attempt 1 of 6)">
[2019-06-25 02:36:19,737] INFO - prefect.TaskRunner | Task 'demo_task_2': finished task run for task with final state: 'Pending'
[2019-06-25 02:36:19,737] INFO - prefect.FlowRunner | Flow run RUNNING: terminal tasks are incomplete.
[2019-06-25 02:36:19,737] INFO - prefect.FlowRunner | Beginning Flow run for 'demo_flow'
[2019-06-25 02:36:19,737] INFO - prefect.TaskRunner | Task 'demo_task_0': Starting task run...
[2019-06-25 02:36:19,737] INFO - prefect.TaskRunner | Task 'demo_task_0': finished task run for task with final state: 'Retrying'
[2019-06-25 02:36:19,737] INFO - prefect.TaskRunner | Task 'demo_task_1': Starting task run...
[2019-06-25 02:36:19,737] INFO - prefect.TaskRunner | Task 'demo_task_1': finished task run for task with final state: 'Pending'
[2019-06-25 02:36:19,737] INFO - prefect.TaskRunner | Task 'demo_task_2': Starting task run...
[2019-06-25 02:36:19,737] INFO - prefect.TaskRunner | Task 'demo_task_2': finished task run for task with final state: 'Pending'
[2019-06-25 02:36:19,737] INFO - prefect.FlowRunner | Flow run RUNNING: terminal tasks are incomplete.
[2019-06-25 02:36:19,737] INFO - prefect.Flow | Waiting for next available Task run at 2019-06-25T02:36:20.737258+00:00
current work flow running state is <Paused: "Trigger function is "manual_only"">
[2019-06-25 02:36:20,752] INFO - prefect.FlowRunner | Beginning Flow run for 'demo_flow'
[2019-06-25 02:36:20,752] INFO - prefect.TaskRunner | Task 'demo_task_0': Starting task run...
current work flow running state is <Running: "Starting task run.">
Task starts running ------------------------------------------------- Task starts running
[2019-06-25 02:36:20,752] INFO - prefect.TaskRunner | Task 'demo_task_0': finished task run for task with final state: 'Resume'
current work flow running state is <TriggerFailed: "Failed due to no external triggering from host">
[2019-06-25 02:36:20,752] INFO - prefect.TaskRunner | Task 'demo_task_1': Starting task run...
No triggering from host.
[2019-06-25 02:36:20,752] INFO - prefect.TaskRunner | Task 'demo_task_1': finished task run for task with final state: 'Pending'
[2019-06-25 02:36:20,752] INFO - prefect.TaskRunner | Task 'demo_task_2': Starting task run...
[2019-06-25 02:36:20,752] INFO - prefect.TaskRunner | Task 'demo_task_2': finished task run for task with final state: 'Pending'
[2019-06-25 02:36:20,752] INFO - prefect.FlowRunner | Flow run RUNNING: terminal tasks are incomplete.
[2019-06-25 02:36:20,752] INFO - prefect.FlowRunner | Beginning Flow run for 'demo_flow'
current work flow running state is <Retrying: "Retrying Task (after attempt 1 of 6)">
[2019-06-25 02:36:20,752] INFO - prefect.TaskRunner | Task 'demo_task_0': Starting task run...
[2019-06-25 02:36:20,752] INFO - prefect.TaskRunner | Task 'demo_task_0': finished task run for task with final state: 'Retrying'
[2019-06-25 02:36:20,752] INFO - prefect.TaskRunner | Task 'demo_task_1': Starting task run...
[2019-06-25 02:36:20,752] INFO - prefect.TaskRunner | Task 'demo_task_1': finished task run for task with final state: 'Pending'
[2019-06-25 02:36:20,752] INFO - prefect.TaskRunner | Task 'demo_task_2': Starting task run...
[2019-06-25 02:36:20,752] INFO - prefect.TaskRunner | Task 'demo_task_2': finished task run for task with final state: 'Pending'
[2019-06-25 02:36:20,752] INFO - prefect.FlowRunner | Flow run RUNNING: terminal tasks are incomplete.
[2019-06-25 02:36:20,752] INFO - prefect.Flow | Waiting for next available Task run at 2019-06-25T02:36:21.752814+00:00
current work flow running state is <Paused: "Trigger function is "manual_only"">
current work flow running state is <Running: "Starting task run.">
[2019-06-25 02:36:21,768] INFO - prefect.FlowRunner | Beginning Flow run for 'demo_flow'
Task starts running ------------------------------------------------- Task starts running
current work flow running state is <TriggerFailed: "Failed due to no external triggering from host">
No triggering from host.
[2019-06-25 02:36:21,768] INFO - prefect.TaskRunner | Task 'demo_task_0': Starting task run...
current work flow running state is <Retrying: "Retrying Task (after attempt 1 of 6)">
[2019-06-25 02:36:21,768] INFO - prefect.TaskRunner | Task 'demo_task_0': finished task run for task with final state: 'Resume'
[2019-06-25 02:36:21,768] INFO - prefect.TaskRunner | Task 'demo_task_1': Starting task run...
[2019-06-25 02:36:21,768] INFO - prefect.TaskRunner | Task 'demo_task_1': finished task run for task with final state: 'Pending'
[2019-06-25 02:36:21,768] INFO - prefect.TaskRunner | Task 'demo_task_2': Starting task run...
[2019-06-25 02:36:21,768] INFO - prefect.TaskRunner | Task 'demo_task_2': finished task run for task with final state: 'Pending'
[2019-06-25 02:36:21,768] INFO - prefect.FlowRunner | Flow run RUNNING: terminal tasks are incomplete.
[2019-06-25 02:36:21,768] INFO - prefect.FlowRunner | Beginning Flow run for 'demo_flow'
[2019-06-25 02:36:21,768] INFO - prefect.TaskRunner | Task 'demo_task_0': Starting task run...
[2019-06-25 02:36:21,768] INFO - prefect.TaskRunner | Task 'demo_task_0': finished task run for task with final state: 'Retrying'
[2019-06-25 02:36:21,768] INFO - prefect.TaskRunner | Task 'demo_task_1': Starting task run...
[2019-06-25 02:36:21,768] INFO - prefect.TaskRunner | Task 'demo_task_1': finished task run for task with final state: 'Pending'
[2019-06-25 02:36:21,768] INFO - prefect.TaskRunner | Task 'demo_task_2': Starting task run...
[2019-06-25 02:36:21,768] INFO - prefect.TaskRunner | Task 'demo_task_2': finished task run for task with final state: 'Pending'
[2019-06-25 02:36:21,768] INFO - prefect.FlowRunner | Flow run RUNNING: terminal tasks are incomplete.
[2019-06-25 02:36:21,768] INFO - prefect.Flow | Waiting for next available Task run at 2019-06-25T02:36:22.768355+00:00
[2019-06-25 02:36:22,783] INFO - prefect.FlowRunner | Beginning Flow run for 'demo_flow'
current work flow running state is <Paused: "Trigger function is "manual_only"">
[2019-06-25 02:36:22,783] INFO - prefect.TaskRunner | Task 'demo_task_0': Starting task run...
current work flow running state is <Running: "Starting task run.">
[2019-06-25 02:36:22,783] INFO - prefect.TaskRunner | Task 'demo_task_0': finished task run for task with final state: 'Resume'
[2019-06-25 02:36:22,783] INFO - prefect.TaskRunner | Task 'demo_task_1': Starting task run...
[2019-06-25 02:36:22,783] INFO - prefect.TaskRunner | Task 'demo_task_1': finished task run for task with final state: 'Pending'
[2019-06-25 02:36:22,783] INFO - prefect.TaskRunner | Task 'demo_task_2': Starting task run...
Task starts running ------------------------------------------------- Task starts running
current work flow running state is <TriggerFailed: "Failed due to no external triggering from host">
No triggering from host.
current work flow running state is <Retrying: "Retrying Task (after attempt 1 of 6)">
[2019-06-25 02:36:22,783] INFO - prefect.TaskRunner | Task 'demo_task_2': finished task run for task with final state: 'Pending'
[2019-06-25 02:36:22,783] INFO - prefect.FlowRunner | Flow run RUNNING: terminal tasks are incomplete.
[2019-06-25 02:36:22,783] INFO - prefect.FlowRunner | Beginning Flow run for 'demo_flow'
[2019-06-25 02:36:22,783] INFO - prefect.TaskRunner | Task 'demo_task_0': Starting task run...
[2019-06-25 02:36:22,783] INFO - prefect.TaskRunner | Task 'demo_task_0': finished task run for task with final state: 'Retrying'
[2019-06-25 02:36:22,783] INFO - prefect.TaskRunner | Task 'demo_task_1': Starting task run...
[2019-06-25 02:36:22,783] INFO - prefect.TaskRunner | Task 'demo_task_1': finished task run for task with final state: 'Pending'
[2019-06-25 02:36:22,783] INFO - prefect.TaskRunner | Task 'demo_task_2': Starting task run...
@occoder
Copy link
Author

occoder commented Jun 25, 2019

After some poking, I found the task's trigger argument set to "manual_only" might be the root cause that leads to resetting the retry attempt counting each time it's invoked.

@cicdw
Copy link
Member

cicdw commented Jun 25, 2019

Hi @occoder - I'm having a hard time understanding what you expect the behavior to be here?

@occoder
Copy link
Author

occoder commented Jun 26, 2019 •
edited

Hi @cicdw The failed state leads that the flow runner auto transit to retrying state which in turn should increment the attemptted retry counter by one. However, the example shown above did not increment the retry counter meaning the task kept retrying and never reached the max retry limit.

@cicdw cicdw mentioned this issue Sep 27, 2019
Merged
3 tasks
zanieb added a commit that referenced this issue Mar 1, 2022
@zanieb
Merge pull request #1177 from PrefectHQ/database-timeouts
d83bd4b 
Add a db connection timeout; lengthen the command timeout
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Persist task_run_count using state context PrefectHQ/prefect
2 participants
@occoder @cicdw