Add a system sync() to start_worker() for aggressive SGE buffering #12439
Conversation
| @@ -980,6 +980,7 @@ function start_worker(out::IO) | |||
| print(out, LPROC.bind_addr) | |||
| print(out, '\n') | |||
| flush(out) | |||
| ccall((:sync, "libc"), Void, ()) # necessary for SGE, which redirects and buffers stdout | |||
There was a problem hiding this comment.
I wonder if the flush function shouldn't do this automatically. In my experience you either don't care about buffering, in which case you wouldn't all flush anyway, or you do and you want to flush everything as thoroughly as possible.
There was a problem hiding this comment.
The whole point is that flush() doesn't do this in this case.
In the case of addprocs_sge(), out == STDOUT. The worker flushes STDOUT, but apparently SGE redirects stdout to a file locally at the worker, and somehow adds its own buffering again in that process. This makes the flush() useless, but the sync() seems to circumvent the problem, because STDOUT has turned into a file on the filesystem.
There was a problem hiding this comment.
My point was that perhaps flush should do this.
|
Looking at the appveyor build, I suppose libc and/or sync does not exist in windows. So the sync should be surrounded by the proper ifdefs. |
|
Is this a problem on master? Would much rather have the PR submitted against master first, and it could be considered for backporting at least several days after being merged. |
|
This would also apply for 0.4-dev. However, I can't compile 0.4-dev on this large cluster, for all kinds of reasons. The latest reason is an "could not allocate pools" error (I do not have virtual address space limits set, afaik). So I won't be able to debug. |
|
That's #10390, there should be a line of code that you can change to hopefully fix that one. |
|
By googling a bit on
|
|
I agree on all points, but the sgemanager needs to be rewritten, probably as you suggested in https://groups.google.com/forum/#!topic/julia-dev/Ms9pTNGoIvA . I am not convinced I'm the one to do that, as I don't really understand the intricacies between julia and clustermanager. The general solution http://docs.julialang.org/en/latest/manual/parallel-computing/#clustermanagers obviously doesn't work for me here, because of the buffering. |
As reported here, on a very large SGE cluster, the i/o buffering for stdout/stderr can be so aggressive that the (port,worker-ip) is never flushed to file---the very file that the master uses to find out what worker machines to connect to. Thus, the workers time-out and the connection cannot be established. Effectively,
addprocs_sge()doesn't work on such a cluster.The only solution I could get working is to include a sync() in the workers, after the flush(), to ensure stdout---which is redirected by SGE to a file---is actually flushed.