Population based training for fine tuning?

[jacoby149]

Hi, I would like to do population based training to train my farm reader.
I know haystack uses ray which has it, I would love if it were a feature of haystack.
is there a way to easily wrap the function farmreader.train so i can do population based training with ray?
I am on a deadline to implement population based training on a farm reader. Thanks! :)

ray population based training : https://docs.ray.io/en/latest/tune/tutorials/tune-advanced-tutorial.html
article saying why population based training is superior : https://medium.com/distributed-computing-with-ray/hyperparameter-optimization-for-transformers-a-guide-c4e32c6c989b

[ZanSara]

Hello @jacoby149, sorry for the late reply. We're glad you're interested in this topic, but unfortunately we didn't have time to investigate that direction yet, so I'd say is not possible with Haystack right now. However, probably @oryx1729 or @tstadel will explore this topic in the future 🙂

[jacoby149]

Thanks for the reply! @ZanSara
In the experiments in the blogs that I linked above, they get a 4% improvement with PBT.

PBT would be very worthwhile to implement :)
Also, it shouldn't be extremely hard since Ray already has PBT capabillities

[julian-risch]

Hi @jacoby149 , I had a brief chat with my colleague @oryx1729 and it seems that population based training is a part of ray tune, whereas we are using ray serve. So far, we didn't have a chance to look into ray tune. If you would like to work on that topic yourself, feel free to create a draft pull request. Unfortunately, we don't have time to further investigate that topic ourselves at the moment.