This work is an exploration into the world of distributed approaches in developing machine learning models.
Mainly in this research we started with large batch algorithms particularly LARS and LAMB which are designed to increase the size of mini batches. However they have some limitations in practice. Afterward, wored on popular algorithms of distributed learning such as: LocalSGD, SlowMo, and LocalAdaScale.
This work is illustrated an evolutionary approach starting from basic centralized model and at each step overcomes the limitations of the previous approaches.