Discussion: Alternatives for ASHAscheduler in Ray

[chrstnkgn]

What

• Find alternatives for ASHA scheduler
  • Possible integrations that I can think of now are:
  1. Apply early stop with certain number of patiences
  2. Find another scheduler that ray provides that better fits our need

Why

• While developing and enhancing this project, we have discussed about the scheduler that terminates the training process multiple times and the main issue was that 'ASHA scheduler does not fit our purpose' as it is the algorithm that works well in the multi-processing environment.
• If we have firmly decided to stick with ray tune, then we need to seek algorithms that better fulfill our need and terminate the process at the appropriate timing

How

• There seem to be several options that we can consider according to the ray docs (https://docs.ray.io/en/latest/tune/api_docs/schedulers.html), but if any of them does not seem appropriate, then it might be better to just go with early stopping
• I don't think that it is the part that I can decide alone, so I kindly ask you to freely discuss and provide various opinions here! 🙏

[kdg1993]

Thanks for the clear statements about the important issue that we need to solve @chrstnkgn!
I also take this issue seriously as it relates to the running time

After trying to use several schedulers that are not ASHA and reading the docs, I concluded that it was very difficult to find an optimal grace period and reduction factor

In my personal point of view, understandability & simplicity are sometimes more important than complex SOTA things in some cases

If we cannot provide a persuasive grace period & reduction factor with ASHA or the other complex schedulers automatically, simple early stopping is much better in the case
In other words, since the simple early-stopping method is in line with the complex scheduler in terms of the necessity for the user to find optimal choices, the simpler the better

Additionally, according to the choices for the grace period and reduction factor,
ASHA can not guarantee a significant reduction in running time

[chrstnkgn]

Thank you for your valuable opinion @kdg1993 ! 🌟💯

I agree that sometimes implementing simpler settings could be better in terms of both performance and understanding of the users. Also, it is not a harm to add a simple early-stop code for any of us or the users who want to use the feature as it is a fundamental methodology for the reduction of computational cost. I will try implementing early-stop ASAP as we are trying to conduct our first experiment and any kind of process terminator is necessary. We can start with an early stop first and try applying other algorithms if needed. Again thank you for your opinion! It helped me a lot to think about the issue in different directions.