ERROR: numpy.lib.tests.test_io.TestFromTxt.test_utf8_file_nodtype_unicode

[jwhowarth]

Is anyone else seeing this test suite failure in numpy 1.14.0 on macOS High Sierra 10.13.3 and Xcode 9.2?

Running unit tests for numpy
NumPy version 1.14.0
NumPy relaxed strides checking option: True
NumPy is installed in /sw/src/fink.build/root-numpy-py34-1.14.0-1/sw/lib/python3.4/site-packages/numpy
Python version 3.4.7 (default, Oct  2 2017, 00:50:24) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)]
nose version 1.3.7

======================================================================
ERROR: numpy.lib.tests.test_io.TestFromTxt.test_utf8_file_nodtype_unicode
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sw/lib/python3.4/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/sw/src/fink.build/root-numpy-py34-1.14.0-1/sw/lib/python3.4/site-packages/numpy/lib/tests/test_io.py", line 1990, in test_utf8_file_nodtype_unicode
    f.write(u"norm1," + latin1 + u",norm3\n")
UnicodeEncodeError: 'ascii' codec can't encode characters in position 6-8: ordinal not in range(128)

----------------------------------------------------------------------
Ran 6757 tests in 81.838s

[jwhowarth]

Interestingly, I don't see this test suite failure when numpy 1.14.0 is built on the same system against Python version 2.7.14.

[jwhowarth]

The same test suite error is also seen for numpy 1.14.0 builds against Python version 3.6.4 and Python version 3.5.4 on macOS High Sierra 10.13.3 and Xcode 9.2. So Python version 2.7.14 is the only release immune to the test suite failure in numpy 1.14.0.

[charris]

Hmm, the string contains non-ascii characters on purpose, so the question is why the text file expects ascii. What does sys.getfilesystemencoding() show? Do you have LANG or LC_CTYPE defined in the environment? Where did you get the Python?

[jwhowarth]

These are all python builds from the fink packaging project (http://www.finkproject.org). Their python-2.7.14, python-3.4.7, .python-3.5.4 and python-3.6.4 all produce 'utf-8' for sys.getfilesystemencoding() .

[charris]

I suspect a Python configuration option for the fink builds but I don't where to look.

[eric-wieser]

I think these lines:

# skip test if cannot encode utf8 test string with preferred
        # encoding. The preferred encoding is assumed to be the default
        # encoding of io.open. Will need to change this for PyTest, maybe
        # using pytest.mark.xfail(raises=***).
        try:
            import locale
            encoding = locale.getpreferredencoding()
            utf8.encode(encoding)
        except (UnicodeError, ImportError):
            raise SkipTest('Skipping test_utf8_file_nodtype_unicode, '
                           'unable to encode utf8 in preferred encoding') 

should be changed to

# skip test if cannot encode utf8 test string with preferred
        # encoding. The preferred encoding is assumed to be the default
        # encoding of io.open. Will need to change this for PyTest, maybe
        # using pytest.mark.xfail(raises=***).
        try:
            import locale
            encoding = locale.getpreferredencoding()
            utf8.encode(encoding)
            latin1.encode(encoding)
        except (UnicodeError, ImportError):
            raise SkipTest('Skipping test_utf8_file_nodtype_unicode, '
                           'unable to encode utf8 and latin1 in preferred encoding') 

[charris]

Also, could check the value of locale.getpreferredencoding(False), maybe also with False replaced with True.

[jwhowarth]

Okay. The problem is due to fink building packages as a fink-bld user which doesn't have a shell set so changing the line that executes the test suite from...

SHELL=/bin/sh PYTHONPATH=/sw/src/fink.build/root-numpy-py36-1.14.0-1/sw/lib/python3.6/site-packages /sw/bin/python3.6 -B -c "import numpy, sys; e=numpy.test(); sys.exit(1-e.wasSuccessful())"

to

LANG=en_US.UTF-8 SHELL=/bin/sh PYTHONPATH=/sw/src/fink.build/root-numpy-py36-1.14.0-1/sw/lib/python3.6/site-packages /sw/bin/python3.6 -B -c "import numpy, sys; e=numpy.test(); sys.exit(1-e.wasSuccessful())"

eliminates the failure.

[charris]

Hmm, this all seems a bit fragile if the results depend on the shell. I'm not familiar with how fink works, is the shell something left for the user to specify with no default?

[charris]

@jwhowarth So this was a fink thing? I note that the return type of numpy.test() will change to boolean when NumPy 1.15 is released, that is part of the switch to pytest. Might want to ping the fink folks at some point.

[bsipocz]

A lot has happened to the numpy testing infrastructure in recent years (e.g. switching away from nose), and I'm certain this issue is not relevant anymore. Please do reopen if you think otherwise.