[BUG] Retriability of timeouts appears inconsistent

[fg91]

Describe the bug

Let's consider this task:

@task(timeout=timedelta(seconds=120), retries=10)
def foo():
    from time import sleep
    sleep(3600)

I'm seeing two attempts:

• The first timeout must have been counted as retriable as we see a second attempt.
• The second timeout must have been counted as non-retriable as we don't see further attempts even though we configured retries=10

Expected behavior

I would have expected either no retry or 10 retries.

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[runllm]

Timeouts in Flyte can be inconsistent due to how retries are handled for different types of errors. According to this discussion, node timeouts can sometimes lead to an "unknown" state, making it difficult to debug. Additionally, another discussion highlights that retries might not always behave as expected, especially with map tasks.

Would you like more detail on how to configure retries and handle timeouts in Flyte?

Sources:

• Timeout in node error
• Map task retries
• Spot revocation handling

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!

[hamersaw]

this seems to be caused by differentiation between active and execution deadlines.

[flixr]

See also #361.
We also really want an execution timeout that does not count queuing time or retries:
#2889

[github-actions]

"Hello 👋, this issue has been inactive for over 90 days. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 7 days. Thank you for your contribution and understanding! 🙏"

[fg91]

"Hello 👋, this issue has been inactive for over 90 days. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 7 days. Thank you for your contribution and understanding! 🙏"

Removed the stale label, not addressed yet and would be good to fix :)