-
Notifications
You must be signed in to change notification settings - Fork 295
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Are the collective APIs provided by gloo thread-safe? #35
Comments
Hi! This is not a concern with Gloo. To do exactly what you describe, Caffe2 creates multiple Gloo contexts and uses a different context for each successive allreduce operation. To limit the number of contexts to be lower than the number of layers (since that might be a lot), it creates a fixed number (e.g. 16) and cycles through the contexts to reuse them. This results in having a maximum of 16 (or whatever you choose) allreduce operations running in parallel. Since every context is completely independent from every other context, you can create as many as you need, and use them from any number of threads. Heads up: for each new context Gloo will create new pairs and the transport layer equivalent (TCP sockets or ibverbs queue pairs), so if you want to use a large number of contexts and have a large number of nodes, you may run into OS level limits for the number of sockets. |
Also see |
Hi @pietern , But can we really benifit from this kind of mutiple threading trick? Have you ever compared single thread allreduce with mutiple threads allreduce, with the same amout of data. For example, 10 jobs, each allreudce 8M float number, running with 10 threads VS. single threads sequencely My experiment on baidu-allreduce says that the total time costs are roughly the same. baidu-allreduce use the same chuncked ring algorithmn as gloo, except that it is built on the top of OpenMPI and gloo is MPI-free It seems we can not benifit from this trick at all, maybe since the real limitation is on hardware such as netword card. Gloo would do better? |
Dear gloo team,
Deep laerning framework like caffe backprops in a layer wise fashion. It is better to do "allreduce" immediately after gradients of each layer figured out. To not only overlap comunication and computing, but also comunication of diffrient layers, these "allreduce" should run in seperate threads, rather than one single thread queued these comunication requests.
Unfortunately, most MPI-like softwares like openmpi and baidu-allreduce do not guarantee multi-thread safety.
Is this also your concern? How can we achieve this using gloo ?
Thanks
The text was updated successfully, but these errors were encountered: