3.0 make test failures on Solaris 10

[smontanaro]

BPO	4506
Nosy	@terryjreedy, @mdickinson, @bitdancer
Files	cmath_debug.patch cmathmodule.S: c_exp without extra printf() calls cmathmodule.S.printf: c_exp with extra printf() calls

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-07-22.19:44:01.247>
created_at = <Date 2008-12-03.21:00:23.440>
labels = []
title = '3.0 make test failures on Solaris 10'
updated_at = <Date 2011-07-22.19:44:01.230>
user = 'https://github.com/smontanaro'

bugs.python.org fields:

activity = <Date 2011-07-22.19:44:01.230>
actor = 'terry.reedy'
assignee = 'none'
closed = True
closed_date = <Date 2011-07-22.19:44:01.247>
closer = 'terry.reedy'
components = []
creation = <Date 2008-12-03.21:00:23.440>
creator = 'skip.montanaro'
dependencies = []
files = ['12211', '12219', '12220']
hgrepos = []
issue_num = 4506
keywords = []
message_count = 18.0
messages = ['76839', '76840', '76841', '76843', '76845', '76890', '76894', '76929', '76933', '77003', '77035', '77140', '77162', '77224', '79065', '90353', '123973', '140902']
nosy_count = 3.0
nosy_names = ['terry.reedy', 'mark.dickinson', 'r.david.murray']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue4506'
versions = []

[smontanaro]

I downloaded the 3.0 tarfile and did a straightforward

configure
make
make test

on Solaris 10 and got several test failures:

290 tests OK.
4 tests failed:
    test_cmath test_math test_posix test_subprocess

Here's the output for just the failing tests:

    test test_cmath failed -- Traceback (most recent call last):
      File "/home/tuba/skipm/src/Python-3.0/Lib/test/test_cmath.py", line 336,
      in test_specific_values
        self.fail('OverflowError not raised in test %s' % test_str)
    AssertionError: OverflowError not raised in test exp0052: exp(complex(710.0,
    1.5))

test test_math failed -- errors occurred; run in verbose mode for details

    test test_posix failed -- Traceback (most recent call last):
      File "/home/tuba/skipm/src/Python-3.0/Lib/test/test_posix.py", line 252,
      in test_getcwd_long_pathnames
        support.rmtree(base_path)
      File "/home/tuba/skipm/src/Python-3.0/Lib/test/support.py", line 98, in
      rmtree    shutil.rmtree(path)
      File "/home/tuba/skipm/src/Python-3.0/Lib/shutil.py", line 225, in rmtree
        onerror(os.rmdir, path, sys.exc_info())
      File "/home/tuba/skipm/src/Python-3.0/Lib/shutil.py", line 223, in rmtree
        os.rmdir(path)
    OSError: [Errno 22] Invalid argument:
    '/home/tuba/skipm/src/Python-3.0/@test.getcwd'

    Could not find platform independent libraries <prefix>
    Could not find platform dependent libraries <exec_prefix>
    Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
    Fatal Python error: Py_Initialize: can't initialize sys standard streams
    ImportError: No module named encodings.utf_8
    .
        this bit of output is from a test of stdout in a different process ...
    test test_subprocess failed -- Traceback (most recent call last):
      File "/home/tuba/skipm/src/Python-3.0/Lib/test/test_subprocess.py", line
      115, in test_executable
        self.assertEqual(p.returncode, 47)
    AssertionError: -6 != 47

Here's the test_math output run through regrtest with the -v option:

\======================================================================
FAIL: testLog (test.test_math.MathTests)
\----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/home/tuba/skipm/src/Python-3.0/Lib/test/test_math.py", line 510, in
      testLog
        self.assertRaises(ValueError, math.log, NINF)
    AssertionError: ValueError not raised by log

\======================================================================
FAIL: testLog10 (test.test_math.MathTests)
\----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/home/tuba/skipm/src/Python-3.0/Lib/test/test_math.py", line 532, in
      testLog10
        self.assertRaises(ValueError, math.log10, NINF)
    AssertionError: ValueError not raised by log10

\----------------------------------------------------------------------
Ran 39 tests in 0.294s

Skip

[mdickinson]

I think you brought up the math and cmath errors before, and I never
managed to get to the bottom of the problem. I'll have another go.

I don't think the (c)math test failures should be regarded as terribly
serious, though; the tests are quite ridiculously strict.

Could you please tell me what

cmath.exp(complex(710.0, 1.5))

actually returns, if it doesn't raise OverflowError?

(Don't know nuffin about posix and subprocess.)

[smontanaro]

Mark> I think you brought up the math and cmath errors before, and I
Mark> never managed to get to the bottom of the problem. I'll have
Mark> another go.

I vaguely remember something about that. If I can be a "test mule" for you,
let me know.

Mark> Could you please tell me what

Mark> cmath.exp(complex(710.0, 1.5))

Mark> actually returns, if it doesn't raise OverflowError?

(1.5802653829857376e+307+inf*j)

Maybe it needs to overflow along both axes???

S

[mdickinson]

(1.5802653829857376e+307+inf*j)

Those values look right; except that there's some code near the end of
the cexp function that's supposed to set errno to ERANGE if either the
real or imaginary component of the result is infinity (and then math_1
knows to raise OverflowError as a result). It looks like that's not
happening for some reason.

If you have time, could you try the attached patch and report what gets
printed when cmath.exp(710+1.5j) is called? On my machine, I get:

Python 3.1a0 (py3k:67510M, Dec  3 2008, 20:56:19) 
[GCC 4.0.1 (Apple Inc. build 5490)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import cmath
>>> cmath.exp(710.0 + 1.5j)
r.real, r.imag: 1.58027e+307, inf
Py_IS_INFINITY(r.real), Py_IS_INFINITY(r.imag): 0, 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: math range error

Mark

[mdickinson]

Tracker issue: I don't seem to be able to remove the 'patch' keyword
without (accidentally) ending up with something else---in this case the
64bit keyword. Is this just me being incompetent, or should I file a
tracker bug?

[smontanaro]

Mark> If you have time, could you try the attached patch and report what
Mark> gets printed when cmath.exp(710+1.5j) is called? On my machine, I
Mark> get:
...

Looks similar here:

    % ./python
    Python 3.0 (r30:67503, Dec  3 2008, 14:46:39)
    [GCC 4.2.2] on sunos5
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import cmath
    >>> cmath.exp(complex(710.0, 1.5))
    r.real, r.imag: 1.58027e+307, Inf
    Py_IS_INFINITY(r.real), Py_IS_INFINITY(r.imag): 0, 1
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    OverflowError: math range error

If I then comment back out your two printf statements and recompile, it once
again fails to raise OverflowError:

    % ./python
    Python 3.0 (r30:67503, Dec  3 2008, 14:46:39)
    [GCC 4.2.2] on sunos5
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import cmath ; cmath.exp(complex(710.0, 1.5))
    (1.5802653829857376e+307+inf*j)

Very weird. If I uncomment either of your printf statements OverflowError
seems to be raised as expected. It's as if some register bit isn't being
set unless you reference r.real or r.imag multiple times.

Skip

[smontanaro]

Mark,

I trimmed down cmathmodule.c to just contain c_exp then
generated assembler files for the non-printf and printf
cases. Perhaps that will help you see what's going on.

Skip

[mdickinson]

Thanks for the assembly code---you're running Solaris on x86! Why
didn't you say so before? :)

I think I have an idea what's going on: it's the old extended-precision
versus double-precision problem. The calculation of c_exp is done in
extended precision in the 80-bit registers of the x87 FPU. The
imaginary part of the result of c_exp(710+1.5j) is representable in
extended precision, but is outside the range of double precision. So
at the point of the Py_IS_INFINITY call, the comparison is done in the
FPU and the value being tested isn't infinity. But sometime after that
this value is forced out of the 80-bit extended precision FPU register
and into memory, where it becomes a 64-bit IEEE 754 double precision
infinity.

I guess the printf calls force the value from register to memory
earlier, so that by the time of the Py_IS_INFINITY call it's already a
64-bit double, and hence already an infinity.

Now to prove my theory by turning this into a fix, somehow.

It seems as though this problem isn't really specific to Solaris; I
guess it's just luck that it hasn't shown up on other x86 platforms. It
should certainly be fixed.

For the test_math failure, there have been problems with log on other
platforms, too, mostly due to strange libm behaviour. The obvious
solution is to adapt the Python implementation to deal with special
values itself, rather than leaving them to the platform math library.
Should be a fairly straightforward change.

[smontanaro]

Mark> Thanks for the assembly code---you're running Solaris on x86! Why
Mark> didn't you say so before? :)

I'm failry sure I can find a SPARC here to run it on as well. They are
rather few and far between though.

Skip

[mdickinson]

I'm failry sure I can find a SPARC here to run it on as well. They
are rather few and far between though.

I don't think that's necessary. Thanks for disabusing me of my 'Solaris
implies SPARC' mindset, though!

There are two more pieces of information that *would* be useful:

1. What happens if you build with the '-ffloat-store' option to gcc?
   If my diagnosis is correct I'd expect the cmath tests to pass with this
   flag. (I'm not 100% sure how to make sure the '-ffloat-store' option
   gets passed through into the Module builds, though
   "BASECFLAGS='-ffloat-store' ./configure" seems to work for me.)

2. It looks as though the configure script isn't finding 'isinf' on
   Solaris 10. Any ideas why? Is there some replacement for isinf that's
   spelt differently? I suspect that if we were using the system lib's
   test for infinity instead of the Py_IS_INFINITY workaround then this
   problem wouldn't have come up.

Mark

[smontanaro]

Mark> 1. What happens if you build with the '-ffloat-store' option to
Mark> gcc?

Doesn't quite work:

    % ./python
    Python 3.0 (r30:67503, Dec  5 2008, 09:48:42)
    [GCC 4.2.2] on sunos5
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import cmath
    >>> cmath.exp(complex(710.0, 1.5))
    Segmentation Fault (core dumped)

:-/

Mark> 2. It looks as though the configure script isn't finding 'isinf'
Mark>    on Solaris 10.  Any ideas why?  Is there some replacement for
Mark>    isinf that's spelt differently?  I suspect that if we were
Mark>    using the system lib's test for infinity instead of the
Mark>    Py_IS_INFINITY workaround then this problem wouldn't have come
Mark>    up.

Thanks for the tip. The configure script doesn't #include <math.h> so
isinf() is not mapped to __builtin_isinf(). Consequently the conftest link
fails:

configure:21401: checking for isinf
configure:21457: gcc -o conftest -g -O2   conftest.c -lresolv -lsocket -lnsl
-lrt -ldl  -lm >&5
conftest.c:252: warning: conflicting types for built-in function 'isinf'
Undefined                        first referenced
 symbol                                                 in file
isinf                               /var/tmp//ccmTAet6.o
ld: fatal: Symbol referencing errors. No output written to conftest
collect2: ld returned 1 exit status

I found this code in /usr/include/iso/math_c99.h, included by <math.h>:

    #undef      isinf
    #define     isinf(x)        __builtin_isinf(x)

Skip

[mdickinson]

The segfault is a little worrying; I don't understand that at all.

Skip, can you come up with a configure patch that would allow isinf to be
detected on Solaris?

I'll also patch Py_IS_INFINITY to make sure that it forces its argument
into memory before testing it; this *should* fix the problem on platforms
that don't have isinf. (Note to self: first find out whether
Py_IS_INFINITY is ever applied to single precision floats, or whether we
can assume the argument is always a double.)

[smontanaro]

can you come up with a configure patch that would allow isinf to be
detected on Solaris?

The plot thickens. I know squat about autoconf sorts of things so I
asked on the autoconf mailing list. Eric Drake responded (in part):

The Python ACHECK_FUNCS test should be rewritten (with proper m4
quoting) as:

  AC_CHECK_FUNCS([acosh asinh atanh expm1 finite log1p])
  AC_CHECK_DECLS([isinf, isnan], [], [], [[#include <math.h>]])

so I gave that a whirl. Whaddya know? isinf *really* isn't available,
at least not until C99 apparently. The AC_CHECK_DECLS macro generates
a conftest.c which looks like this:

  #include <math.h> 
 
  int 
  main () 
  { 
  #ifndef isinf 
    (void) isinf; 
  #endif 
 
    ; 
    return 0; 
  } 

which fails to compile/link on our Sol10 boxes. Turns out libm.so
doesn't export an _isinf symbol. The macro I found (in iso/math_c99.h)
doesn't expand in gcc -E output of the above code, so in my non-c99
environment I don't get to use isinf. Looks like your Py_IS_INFINITY
fix will be necessary. Eric also suggested using the gnu isinf and
isnan modules:

http://git.savannah.gnu.org/gitweb/?
p=gnulib.git;a=blob;f=m4/isinf.m4;h=1b9e45a;hb=67461c3

I'll leave that for you to decide.

In any case, the above changes to AC_CHECK_FUNCS should probably be
made.

Skip

[mdickinson]

Thanks, Skip. It looks like the top priority is fixing Py_IS_INFINITY,
then (still assuming that I'm not barking up entirely the wrong tree).
I've opened bpo-4575 for the Py_IS_INFINITY fix.

I'll look at the changes to AC_CHECK_FUNCS, too.

[mdickinson]

autoconf checks for isinf and isnan fixed in r68299.
I also added a check for isfinite, which should really be used in
preference to finite: isfinite is standard in C99, while finite doesn't
seem to be part of any standard.