Regression test suite with MPI causes nonblocking IO write errors on macOS #47
Comments
|
Update: it turns out that is a known MPICH bug on macOS pmodels/mpich#1782 It was apparently fixed in the MPICH repository about a month ago: pmodels/mpich#2755 Now, I am unable to reproduce the bug with OpenMPI on macOS. I don't think any modification needs to be made to the Athena++ regression test suite at this time. It is just something for macOS users to be aware of when running it with MPICH version < 3.2.1 which was released on 11/10/2017. Presently, Homebrew users need to install MPICH with the |
25 tasks
felker
added a commit
that referenced
this issue
Nov 2, 2018
- Not compatible with Python 3, see #169. Follow PEP 394 and reserve "python" for scripts that are compatible with both Py2 and Py3. - Remove artificial C++ style error in field.hpp - This is the first change from upstream cpplint.py google/styleguide@1b206ee Calling "python -u" in cpplint_athena.sh has fixed the previously jumbled stdout and stderr from cpplint.py calls to sys.stdout.write() and sys.stderr.write() in the Jenkins log. However, unbuffered writes may have been an issue with regression test crashes in macOS and on Travis CI (see #47), but it was most likely a red herring due to the known O_NONBLOCK bug in MPICH's mpirun(). Monitor this. See if Jenkins log now contains pure output from git ls-tree command.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This is a subtle problem that I have noticed on my MacBook Pro and when using Travis CI with
osxVM builds over the past few months. I have just now diagnosed the issue, but it would be good if others can reproduce this and discuss a workaround (if full macOS support is desired).Bug report
Bug summary
The
make()command intst/regression/scripts/utils/athena.py, invoked by a regression test module'sprepare()step fails in the linker withbeing written in the middle of the stdout stream.
The challenge in tracking down this bug is that it manifests in any regression test only after an MPI-enabled regression test (which itself passes). Currently, the only regression test scripts that use MPI are:
mpi/mpi_linwave.pygrav/jeans_3d.pyCode for reproduction
cd tst/regression python run_tests.py mpi/mpi_linwave hydro/sod_shockOr, another example:
cd tst/regression python run_tests.py grav/jeans_3d outputs/all_outputsThe error only occurs in the
makecommands invoked in the separate test scripts following the MPI regression test. In other words, even though thempi_linwave.pyfirst compiles an MPI binary and then a serial binary in the same module, there is no write error in that test.Furthermore, the bug only occurs either when:
python run_tests.py testnamecommands are executed in the same process/script.So, command line execution of
cd tst/regression python run_tests.py grav/jeans_3d python outputs/all_outputsworks fine, but running those commands in a Bash script will produce the error. Hence, in VM environments like Travis CI, which wraps the user's commands in a command called
travis_run_script, this issue may appear.Actual outcome
I have also observed:
instead of
make: write errorwhen using alternativesubprocesscommands; see below.Version Information
My current MacBook Pro environment is:
clangmpicxx -show:corresponding to one of these Homebrew-built binaries (bottles):
It the process of debugging, I have tried many different versions of compilers, Python environments, build options, and macOS versions. It also occurs with:
gcc7.1 or 4.9 installed by Homebrewvirtualenvfor Python.python -ugccorclang--output-syncoption for GNU Make version > 4.0 to ensure that parallel make output is buffered and well-ordered.Explanation
I have encountered related bug reports in the Travis CI, GNU Make, and macOS communities, but never found a complete explanation until recently. The 12/1/17 reply on travis-ci/travis-ci#4704 explains that this bug is caused by the EAGAIN signal, "try again/ data not ready" from nonblocking socket.
To query if
O_NONBLOCKis set, you can use:Embedding this command in the regression test driver returns 0 (
O_NONBLOCKdisabled) after every test until an MPI-enabled test executes, then it returns a nonzero number. So, the open question is: why do the MPI-enabled regression test scripts turn on nonblocking IO, and why does it occur only after the overall completion of the test?Possible fixes
athena.pyoutput ofmake()command to a file by replacing thesubprocess.check_call()commands withsubprocess.Popen()and pipes. Or, suppress the output by replacing withsubprocess.check_output()(without ever communicating the stream to stdout).The text was updated successfully, but these errors were encountered: