Strange bug on Py3 with 13 or 17 engines

[kwmsmith]

On Python 3:

$ dacluster start -n 13

$ make test
...
======================================================================
FAIL: test_sum_4D_cyclic (distarray.dist.tests.test_distarray.TestReduceMethods)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./distarray/dist/tests/test_distarray.py", line 591, in test_sum_4D_cyclic
    assert_allclose(darr.sum().tondarray(), arr.sum())
  File "/Users/ksmith/Devel/virtualenvs/venv-py3.3.2-64bit/lib/python3.3/site-packages/numpy/testing/utils.py", line 1181, in assert_allclose
    verbose=verbose, header=header)
  File "/Users/ksmith/Devel/virtualenvs/venv-py3.3.2-64bit/lib/python3.3/site-packages/numpy/testing/utils.py", line 644, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rotl=1e-07, atol=0

[bgrant]

I don't see any failures after PR #402.

[kwmsmith]

If that's the case, then 🍪

[cowlicks]

I can't reproduce this on 6db35d5 so I'm closing.

[bgrant]

@cowlicks, @kwmsmith: as discussed in-real-life, my ContextTestCase limited this test to only using the first four engines when running with cluster of size 13. However, when I add an ntargets=13 to the test class (and change the shape parameter of the Distribution in the setUpClass to (1,13)), I do get a mysterious error:

$ python -m unittest distarray.dist.tests.test_distarray.TestReduceMethods.test_sum_4D_cyclic
F
======================================================================
FAIL: test_sum_4D_cyclic (distarray.dist.tests.test_distarray.TestReduceMethods)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./distarray/dist/tests/test_distarray.py", line 535, in test_sum_4D_cyclic
    assert_allclose(darr.sum().tondarray(), arr.sum())
  File "/Users/robertgrant/development/venvs/py33/lib/python3.3/site-packages/numpy/testing/utils.py", line 1183, in assert_allclose
    verbose=verbose, header=header)
  File "/Users/robertgrant/development/venvs/py33/lib/python3.3/site-packages/numpy/testing/utils.py", line 644, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-07, atol=0

(mismatch 100.0%)
 x: array(127205.5)
 y: array(720000.0)

----------------------------------------------------------------------
Ran 1 test in 2.114s

FAILED (failures=1)

That is a large discrepancy.

[bgrant]

Also note that all of the single-axis sums just before the failing line seem to work, it's just the axis-less sum that fails. And, the sum that is produced by our code is kinda nonsense: it should be a multiple of 3 (since it's summing an array filled with 3's).

[bgrant]

See more basic failing test here: https://github.com/enthought/distarray/blob/bugfix/403/distarray/dist/tests/test_distarray.py#L558

[bgrant]

I added an issue to mpi4py here:

https://bitbucket.org/mpi4py/mpi4py/issue/6/incorrect-reduce-result-on-python-3