-
-
Notifications
You must be signed in to change notification settings - Fork 74
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
stdout not redirected to repl #10
Comments
No, I don't think it is related to the issues mentioned. cc @nlhepler |
more data on this: the remote-process stdout does eventually appear on the local repl, but sometimes not for several tens of seconds. if i simultaneously examine the julia log files (the ones it clutters the home directory with), the stdout there is also delayed. it appears on the repl and in the log files at roughly the same time, both much delayed. so i think the fix is just a matter of flushing the i/o buffer. i tried to add flush(STDOUT) to the remote script, but got an error about serializing a pointer. i tried looking in base for a place to add a flush, but it was not obvious where to put it. thanks for any help. |
You could have a function defined that does the flush and just call it remotely. For example, |
i tried precisely that, and just tried again. here is the error i get:
note that "foo" eventually appears on the repl, and that the call to flush() causes the error |
Strange. It does not seem to be an issue with local workers, i.e. something like Could you try with this definition? |
that doesn't work either. exact same error. and you're right about it working fine on local workers. |
Just to try: Instead of the
|
nope. same error. is the problem that it's referring to the STDOUT in worker 1 and not worker 2? |
That's what I initially thought, but it does not seem to be the case. At least with I just saw |
Also STDOUT seems to be of type TTY and not a regular stream as we have been assuming... |
flush_cstdio has no effect: no error, no stdout on repl, no change to log files in home directory. |
Is it possible that SGE is responsible for the delay? I am not at all familiar with cluster technologies, but a Google search brought up this link - http://scicomp.stackexchange.com/questions/7804/flush-output-in-torque-scheduler . Is there a similar option for SGE? |
the problem is not SGE but rather the buffering done by our high-performance file system. a flush() in julia, as described here JuliaLang/julia#6549, is not sufficient. for STDOUT to actually appear in the julia log files i also had to |
Whew, glad to see that was resolved. |
yesterday i got it to work by adding
note that the remote workers stdout only appears on the repl (and in the log file) after it crashes. this is on an asynchronous NFS3 file system. the sysadmin strongly discourages use of the disk for interprocess communication because of the async buffering. i'm looking into replacing qsub with qrsh... |
qrsh implementation here #11 |
while stdout for a local process appears in the repl, that for a remote sge process does not. see the transcript below. not shown is that it works fine for remote ssh processes. is this related to JuliaLang/julia#6030, JuliaLang/julia#5995, and #6 ?
[arthurb@h06u01 ~]$ juliac
_
_ _ ()_ | A fresh approach to technical computing
() | () () | Documentation: http://docs.julialang.org
_ _ | | __ _ | Type "help()" to list help topics
| | | | | | |/ ` | |
| | || | | | (| | | Version 0.3.0-prerelease+2417 (2014-04-02 18:29 UTC)
/ |_'|||__'| | Commit 193cb11* (0 days old master)
|__/ | x86_64-redhat-linux
julia> using ClusterManagers
julia> ClusterManagers.addprocs_sge(1)
job id is 8083782, waiting for job to start ..................................................
1-element Array{Any,1}:
2
julia> addprocs(1)
1-element Array{Any,1}:
3
julia> remotecall_fetch(3,println,"foo")
From worker 3: foo
julia> remotecall_fetch(2,println,"foo")
julia> remotecall_fetch(1,println,"foo")
foo
julia> versioninfo()
Julia Version 0.3.0-prerelease+2417
Commit 193cb11* (2014-04-02 18:29 UTC)
Platform Info:
System: Linux (x86_64-redhat-linux)
CPU: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY)
LAPACK: libopenblas
LIBM: libopenlibm
julia> Pkg.status()
10 required packages:
11 additional packages:
julia>
The text was updated successfully, but these errors were encountered: