Endianness not detected correctly due to AC_RUN_IFELSE macros

[zaytsev]

BPO	20342
Nosy	@doko42, @gpshead, @mdickinson, @vstinner, @moreati
Files	pyconfig.h

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2014-01-22.08:44:31.459>
labels = ['interpreter-core', 'type-bug', 'build']
title = 'Endianness not detected correctly due to AC_RUN_IFELSE macros'
updated_at = <Date 2022-02-07.08:51:47.053>
user = 'https://bugs.python.org/zaytsev'

bugs.python.org fields:

activity = <Date 2022-02-07.08:51:47.053>
actor = 'gregory.p.smith'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core', 'Cross-Build']
creation = <Date 2014-01-22.08:44:31.459>
creator = 'zaytsev'
dependencies = []
files = ['33613']
hgrepos = []
issue_num = 20342
keywords = []
message_count = 8.0
messages = ['208760', '208765', '208772', '208780', '208786', '412719', '412720', '412723']
nosy_count = 6.0
nosy_names = ['doko', 'gregory.p.smith', 'mark.dickinson', 'vstinner', 'zaytsev', 'Alex.Willmer']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue20342'
versions = ['Python 3.4']

[zaytsev]

Hi,

I've managed to get the test suite of Python to run on Python 3.4.0b2 version cross-compiled for a specialized POWER chip. I get a number of failures that seem to have the same nature. I'm under impression, that actually the tests might need to be adjusted, but I'm not sure about it. I'm happy to provide any additional information to diagnose this issue. Please let me know if I've grouped them wrong and have to file a different issue for some tests. Thanks!

== CPython 3.4.0b2 (default, Jan 7 2014, 10:01:53) [GCC 4.4.6]
== Linux-2.6.32-358.11.1.bgq.el6.bgas_20131217.ppc64-ppc64-with-redhat-6.4-Santiago big-endian
== hash algorithm: fnv 64bit
== /tmp/test_python_11312
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=0)

======================================================================
FAIL: fromhex (builtins.float)
Doctest: builtins.float.fromhex
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/XXX/opt/python-3.4.0b2-gcc-cnk/lib/python3.4/doctest.py", line 2187, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for builtins.float.fromhex
  File "builtins", line unknown line number, in fromhex

---

File "builtins", line ?, in builtins.float.fromhex
Failed example:
float.fromhex('-0x1p-1074')
Expected:
-5e-324
Got:
-4.9406564584124654e-324

----------------------------------------------------------------------

======================================================================
FAIL: test_float__format__ (test.test_types.TypesTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/XXX/opt/python-3.4.0b2-gcc-cnk/lib/python3.4/test/test_types.py", line 467, in test_float__format__
    test(1e200, '+', '+1e+200')
  File "/XXX/opt/python-3.4.0b2-gcc-cnk/lib/python3.4/test/test_types.py", line 403, in test
    self.assertEqual(f.__format__(format_spec), result)
AssertionError: '+9.9999999999999997e+199' != '+1e+200'
- +9.9999999999999997e+199
+ +1e+200


----------------------------------------------------------------------

======================================================================
FAIL: SequenceMatcher (difflib)
Doctest: difflib.SequenceMatcher
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/XXX/opt/python-3.4.0b2-gcc-cnk/lib/python3.4/doctest.py", line 2187, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for difflib.SequenceMatcher
  File "/XXX/opt/python-3.4.0b2-gcc-cnk/lib/python3.4/difflib.py", line 47, in SequenceMatcher

---

File "/XXX/opt/python-3.4.0b2-gcc-cnk/lib/python3.4/difflib.py", line 83, in difflib.SequenceMatcher
Failed example:
print(round(s.ratio(), 3))
Expected:
0.866
Got:
0.86599999999999999

----------------------------------------------------------------------

======================================================================
FAIL: test_format_testfile (test.test_float.FormatTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/XXX/opt/python-3.4.0b2-gcc-cnk/lib/python3.4/test/test_float.py", line 592, in test_format_testfile
    self.assertEqual(fmt % float(arg), rhs)
AssertionError: '0.029999999999999999' != '0.03'
- 0.029999999999999999
+ 0.03


----------------------------------------------------------------------

======================================================================
FAIL: test_non_ascii (test.test_format.FormatTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/XXX/opt/python-3.4.0b2-gcc-cnk/lib/python3.4/test/test_format.py", line 272, in test_non_ascii
    self.assertEqual(format(12.3, "\u2007<6"), "12.3\u2007\u2007")
AssertionError: '12.300000000000001' != '12.3\u2007\u2007'
- 12.300000000000001
+ 12.3\u2007\u2007


----------------------------------------------------------------------

======================================================================
FAIL: test_float_default (test.test_optparse.TestExpandDefaults)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/XXX/opt/python-3.4.0b2-gcc-cnk/lib/python3.4/test/test_optparse.py", line 619, in test_float_default
    self.assertHelp(self.parser, expected_help)
  File "/XXX/opt/python-3.4.0b2-gcc-cnk/lib/python3.4/test/test_optparse.py", line 189, in assertHelp
    actual_help + '"\n')
AssertionError: help text failure; expected:
"Usage: test [options]

Options:
-h, --help show this help message and exit
-p PROB, --prob=PROB blow up with probability PROB [default: 0.43]
"; got:
"Usage: test [options]

Options:
-h, --help show this help message and exit
-p PROB, --prob=PROB blow up with probability PROB [default:
0.42999999999999999]
"

----------------------------------------------------------------------

test_doc_tests (test.test_statistics.DocTests) ... **********************************************************************
File "/XXX/opt/python-3.4.0b2-gcc-cnk/lib/python3.4/statistics.py", line 290, in statistics.mean
Failed example:
mean([1, 2, 3, 4, 4])
Expected:
2.8
Got:
2.7999999999999998

---

File "/XXX/opt/python-3.4.0b2-gcc-cnk/lib/python3.4/statistics.py", line 380, in statistics.median_grouped
Failed example:
median_grouped([1, 2, 2, 3, 4, 4, 4, 4, 4, 5])
Expected:
3.7
Got:
3.7000000000000002

---

File "/XXX/opt/python-3.4.0b2-gcc-cnk/lib/python3.4/statistics.py", line 602, in statistics.pstdev
Failed example:
pstdev([1.5, 2.5, 2.5, 2.75, 3.25, 4.75])
Expected:
0.986893273527251
Got:
0.98689327352725098

---

3 items had failures:
1 of 5 in statistics.mean
1 of 4 in statistics.median_grouped
1 of 1 in statistics.pstdev
***Test Failed*** 3 failures.

[zaytsev]

The CPU model is IBM PowerPC A2, the information about the cross-compiler is as follows:

$ /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc64-bgq-linux-gcc -v
Reading specs from /bgsys/drivers/toolchain/V1R2M1_base/gnu-linux/lib/gcc/powerpc64-bgq-linux/4.4.6/specs
Target: powerpc64-bgq-linux
Configured with: /bgsys/drivers/V1R2M1/ppc64/toolchain/gnu/gcc-4.4.6/configure --prefix=/bgsys/drivers/toolchain/V1R2M1_base/gnu-linux --target=powerpc64-bgq-linux --with-pkgversion=BGQ-V1R2M1-130820 --host=powerpc64-linux-gnu --build=powerpc64-linux-gnu   --disable-multilib --enable-shared --enable-secureplt --disable-libmudflap --disable-libspp --with-headers=/bgsys/drivers/V1R2M1/ppc64/toolchain/gnu/build-powerpc64-bgq-linux/tempglibc-4.4.6-install/include --with-libs=/bgsys/drivers/V1R2M1/ppc64/toolchain/gnu/build-powerpc64-bgq-linux/tempglibc-4.4.6-install/lib --with-bin=/bgsys/drivers/V1R2M1/ppc64/toolchain/gnu/build-powerpc64-bgq-linux/tempglibc-4.4.6-install/bin --with-sbin=/bgsys/drivers/V1R2M1/ppc64/toolchain/gnu/build-powerpc64-bgq-linux/tempglibc-4.4.6-install/sbin --with-long-double-128 --enable-threads=posix --enable-languages=c,c++,fortran
Thread model: posix
gcc version 4.4.6 (BGQ-V1R2M1-130820)

[vstinner]

According to what Yury told me on IRC, it's a cross-compilation issue. He wrote me that the following commands returns b'C?\xff\x01\x02\x03\x04\x05' which is the double big endian version, but DOUBLE_IS_BIG_ENDIAN_IEEE754 is not defined from pyconfig.h

./python -c 'import struct; print(ascii(struct.pack("d", 9006104071832581.0)))'

[zaytsev]

I've re-named the issue to reflect the problem.

One possible solution that I can see is to use AC_C_BIGENDIAN macro, either exclusively, or only when cross-compiling.

In the latest autoconf sources, this macro seems to try to detect the endianness from various macros, and then by grepping the produced object files, so it's safe to use during cross-compilation.

Additionally, it allows to set the endianness by hand, if it can't be detected automatically.

Any thoughts?

[vstinner]

doko> One possible solution that I can see is to use AC_C_BIGENDIAN
macro, either exclusively, or only when cross-compiling.

This is wrong. The macro is not used to check the endian but to check
the IEEE 754 implementation. The check is important to decide if David
M. Gay's algorithm for "short float representation" can be used. See
pyport.h:

/* If we can't guarantee 53-bit precision, don't use the code
in Python/dtoa.c, but fall back to standard code. This
means that repr of a float will be long (17 sig digits).

Realistically, there are two things that could go wrong:

(1) doubles aren't IEEE 754 doubles, or
(2) we're on x86 with the rounding precision set to 64-bits
(extended precision), and we don't know how to change
the rounding precision.
*/

#if !defined(DOUBLE_IS_LITTLE_ENDIAN_IEEE754) && \
    !defined(DOUBLE_IS_BIG_ENDIAN_IEEE754) && \
    !defined(DOUBLE_IS_ARM_MIXED_ENDIAN_IEEE754)
#define PY_NO_SHORT_FLOAT_REPR
#endif

/* double rounding is symptomatic of use of extended precision on x86.  If
   we're seeing double rounding, and we don't have any mechanism available for
   changing the FPU rounding precision, then don't use Python/dtoa.c. */
#if defined(X87_DOUBLE_ROUNDING) && !defined(HAVE_PY_SET_53BIT_PRECISION)
#define PY_NO_SHORT_FLOAT_REPR
#endif

--

Python 3.4 already uses AC_C_BIGENDIAN in configure.ac, see how it is
used in pyport.h.

[gpshead]

any idea if this is still relevant?

[zaytsev]

Well, the machine that I had the problem on does no longer exist and I'm not sure whether this is a generic POWER issue.

The problem was that I had to cross-compile Python on one generation of POWER machines for another POWER chip using its own special toolchain. I can't tell whether Python build system was doing something wrong, or else whether it was a problem with the toolchain. In theory, both are valid options.

Also, I can't test whether the current version of Python has this problem or not with this particular chip / toolchain anymore. Maybe someone who has interest in POWER could try to cross-compile from AMD64 to POWER and check whether the result is still subtly broken... this should be much easier, and if there is a genuine problem with Python build system, it will probably expose it just as well.

[gpshead]

I'll leave it open, nothing about this should be ppc64 vs ppc64el specific. Our cross compilation story has historically been not great.

Building for a target of one endianness from a build host of other endianness... sounds like exactly one of many kinds of thing that we might have a mess with until someone can confirm. (I'm not setup with a big endian toolchain to build for a qemu target right now so I can't easily confirm)