[Core] Allow us to configure new memory and cpu config upon subsequent retries

[raghumdani]

Description

ray.remote takes in memory, cpu and max_retries and retry_exceptions. We have seen that the most common cause for task failures are OOMs. If we retry them with the same memory config, the task will fail again. Hence, a feature to change the resource config on the subsequent retries of a task would be tremendously useful. We anyway have a workaround to do it on our own but this is a generic improvement that can benefit ray users and can simplify code at our side.

Use case

We will be able to overcome out of memory errors by retrying tasks with increased memory upon subsequent retries.

[rynewang]

Note that memory and CPU are logical resources, that even if you write a small memory task, it can take all memories from the machine.

Would you mind sharing your use case again? Do you have dynamic amount of resources needed for each task? If not, typically you can benchmark your mem usage and set a big enough number for it, and it would schedule well. We don't have plans for "auto-piloting" resource usages for now.

[raghumdani]

Do you have dynamic amount of resources needed for each task?

Yes, memory requirements are calculated based on file sizes each task reads. We also have different types of tasks requiring different amounts of memory. Now we often under-estimate memory and hence, we have to retry the same task with more memory requirement the next time.