As discussed in #2688, for performance reasons, we should use pipes or unix sockets for local communication.
I believe Amit pointed out that this will not be significantly faster.
Since my name was mentioned, I had to back it up with data.
Tried with 1) tcp client-server socket connection, 2) AF_UNIX socketpair() and 3) Unix pipes
AF_INET domain sockets : 1200 milliseconds
AF_UNIX domain socketpair : 970 milliseconds
Unix pipes : 930 milliseconds
I think we can let this issue be closed.
I dunno, that's actually a pretty substantial gain – 25% faster is not to be sneezed at.
Yeah, but you have to see it in context. The code was doing nothing but transferring 1 GB of data in 1K chunks between 2 processes. In a real world scenario where other computation is taking up the bulk of CPU time, the overall performance gains from this will be limited.
How often do you think we will need to keep transferring huge amounts of data between processes?
I feel that at this stage any additional complexity due to using 2 different IPC mechanisms between processes on the same machine vs processes on remote machines is not commensurate with the gains.
That's fair. If we were going to pick one transport mechanism, mpi which can fallback on tcp would be better. Then we'd get highly optimized tramsport on systems with fancy interconnects.
We should not pick MPI as the default transport for now. I'd rather stay with tcp sockets for now, which allows for robust programming. Most of the high performance libraries are already written in MPI, where most of the cycles will go, and we can just ccall those libraries.
The best performing library for big iron today is most likely GASNet (http://gasnet.cs.berkeley.edu), which is a common interface to a number of platform specific communication interfaces, and also falls back to MPI.
While I was walking around Supercomputing'13 last week, I had this issue in mind while asking questions at booths. I concur that GASNet would be a good communication interface to target.
For a low effort improvement, RSOCKETS should be considered, since it has a sockets-like interface and seems to be available in common Linux distributions as librdmacm.
Does anyone know if there are HPC transports that have ZMQ backends? ZMQ is a nicely designed, flexible library, so if you couple that with high-performance transport, it would be a rather pleasant combination.