migrate to ray tune

[aniketmaurya]

Why Ray over Optuna?

Initially, I started with Optuna for HPO because of its Pythonic APIs and powerful search algorithms. Later I realized that Gradsflow will have to incorporate logic for distributed training but Ray already provides this out of the box.

Ray provides a simple, universal API for building distributed applications and it already supports multiple hyperparameter tuning libraries including Optuna.

• With Ray, we get to leverage distributed training and HP search out of the box.
• We get to use search algorithms not only by ray but also optuna and some other cool libraries.
• Easy process and GPU management. Ray even supports gpu_fraction training.

NOTE: User API will remain the same and you won't feel any difference apart from Ray's cool distributed training features. 🔥