Use pipes or unix socket for IPC instead of tcp sockets #2696

Closed
ViralBShah opened this Issue Mar 27, 2013 · 9 comments

5 participants

@ViralBShah
The Julia Language member

As discussed in #2688, for performance reasons, we should use pipes or unix sockets for local communication.

@JeffBezanson
The Julia Language member

I believe Amit pointed out that this will not be significantly faster.

@ViralBShah
The Julia Language member
@amitmurthy
The Julia Language member

Since my name was mentioned, I had to back it up with data.

Tried with 1) tcp client-server socket connection, 2) AF_UNIX socketpair() and 3) Unix pipes

  • Data transmitted 1GB with a final ack from the server.
  • 1 million blocking sends (and recv) of 1024 bytes each in a loop
  • On my 4 year old laptop
  • AF_INET domain sockets were bound to the ip-address on my NIC (specifically NOT 127.0.0.1)
AF_INET domain sockets : 1200 milliseconds
AF_UNIX domain socketpair : 970 milliseconds
Unix pipes : 930 milliseconds

I think we can let this issue be closed.

@StefanKarpinski
The Julia Language member

I dunno, that's actually a pretty substantial gain – 25% faster is not to be sneezed at.

@amitmurthy
The Julia Language member

Yeah, but you have to see it in context. The code was doing nothing but transferring 1 GB of data in 1K chunks between 2 processes. In a real world scenario where other computation is taking up the bulk of CPU time, the overall performance gains from this will be limited.

How often do you think we will need to keep transferring huge amounts of data between processes?

I feel that at this stage any additional complexity due to using 2 different IPC mechanisms between processes on the same machine vs processes on remote machines is not commensurate with the gains.

@StefanKarpinski
The Julia Language member

That's fair. If we were going to pick one transport mechanism, mpi which can fallback on tcp would be better. Then we'd get highly optimized tramsport on systems with fancy interconnects.

@ViralBShah
The Julia Language member

We should not pick MPI as the default transport for now. I'd rather stay with tcp sockets for now, which allows for robust programming. Most of the high performance libraries are already written in MPI, where most of the cycles will go, and we can just ccall those libraries.

The best performing library for big iron today is most likely GASNet (http://gasnet.cs.berkeley.edu), which is a common interface to a number of platform specific communication interfaces, and also falls back to MPI.

@ArchRobison

While I was walking around Supercomputing'13 last week, I had this issue in mind while asking questions at booths. I concur that GASNet would be a good communication interface to target.

For a low effort improvement, RSOCKETS should be considered, since it has a sockets-like interface and seems to be available in common Linux distributions as librdmacm.

@StefanKarpinski
The Julia Language member

Does anyone know if there are HPC transports that have ZMQ backends? ZMQ is a nicely designed, flexible library, so if you couple that with high-performance transport, it would be a rather pleasant combination.

@mgsloan mgsloan referenced this issue in fpco/ide-backend Jun 28, 2015
Closed

Installing on Windows #279

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment