[BUG] Unexpected runtimes with resource requests and overridden resource requests

[Mathis-Z]

Describe the bug

I am running the following script on Kubernetes and the runtimes are unexpected:

from flytekit import workflow, Resources, task

from tasks.simple_directory_importer import import_directory_from_url

@task(requests=Resources(cpu="1"))
def fast_task():
    pass

@task
def slow_task():
    pass

@workflow()
def wf() -> None:
    fast_task() # finishes in ~4 seconds
    slow_task().with_overrides(requests=Resources(cpu="1")) # finishes in ~60 seconds

    slow_task().with_overrides(limits=Resources(cpu="1")) # finishes in ~4 seconds

This is related to #6377

Expected behavior

I expect all three tasks to run in the same amount of time.

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[runllm]

It seems likely that the issue you're experiencing with unexpected runtimes when using with_overrides for resource requests and limits is related to how Flyte handles resource allocation. According to Flyte documentation, tasks can have requests and limits which mirror Kubernetes' native equivalents. Requests are hints for scheduling, while limits are hard constraints. If a task's resource requests are not met, it may be scheduled on a node with fewer resources, leading to slower execution.

In your case, the slow_task with requests=Resources(cpu="1") might not be getting the requested resources due to node availability, causing it to run slower. However, when you set limits=Resources(cpu="1"), it enforces the resource allocation, leading to faster execution. This behavior aligns with the default resource management in Flyte, where tasks can be throttled if not properly configured, as discussed in Flyte Issue #6377.

Would you like more detail on how to configure resource requests and limits effectively in Flyte?

Sources:

• Flyte documentation on customizing task resources
• Flyte Issue #6377

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!

[eapolinario]

@Mathis-Z , can you double-check what's present in each pod spec?

[Mathis-Z]

I assume only the assigned resource requests/limits are interesting. So from kubectl describe:
Task 1 (~4s execution time):

    Limits:
      cpu:     1
      memory:  500Mi
    Requests:
      cpu:     1
      memory:  500Mi

Task 2 (~60s execution time)

    Limits:
      cpu:     100m
      memory:  500Mi
    Requests:
      cpu:     100m
      memory:  500Mi

Task 3 (~4s execution time)

    Limits:
      cpu:     1
      memory:  500Mi
    Requests:
      cpu:     100m
      memory:  500Mi

For task 2 the default limits overwrote the request set with with_overrides.

[github-actions]

"Hello 👋, this issue has been inactive for over 90 days. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 7 days. Thank you for your contribution and understanding! 🙏"

[github-actions]

Hello 👋, this issue has been inactive for over 90 days and hasn't received any updates since it was marked as stale. We'll be closing this issue for now, but if you believe this issue is still relevant, please feel free to reopen it. Thank you for your contribution and understanding! 🙏