''.find() gives wrong result in Python built with ICC

[sandersmuc]

BPO	1084
Nosy	@loewis
Files	findtest.c findtest.c: Version sent to Intel

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2007-09-02.09:58:13.343>
created_at = <Date 2007-09-01.22:57:14.754>
labels = ['interpreter-core', 'build']
title = "''.find() gives wrong result in Python built with ICC"
updated_at = <Date 2007-12-04.15:39:25.939>
user = 'https://bugs.python.org/sandersmuc'

bugs.python.org fields:

activity = <Date 2007-12-04.15:39:25.939>
actor = 'sanders_muc'
assignee = 'none'
closed = True
closed_date = <Date 2007-09-02.09:58:13.343>
closer = 'loewis'
components = ['Build', 'Interpreter Core']
creation = <Date 2007-09-01.22:57:14.754>
creator = 'sanders_muc'
dependencies = []
files = ['8373', '8374']
hgrepos = []
issue_num = 1084
keywords = []
message_count = 6.0
messages = ['55567', '55577', '55579', '55580', '55582', '58188']
nosy_count = 2.0
nosy_names = ['loewis', 'sanders_muc']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue1084'
versions = ['3rd party']

[sandersmuc]

I have just encountered a strange bug affecting Python 2.5.1 on an
x86_64 Linux, but only when compiled with the Intel C Compiler (ICC)
10.0, not a GCC-compiled Python. On my Intel-compiled one, which
otherwise seems to work fine, ''.find() works incorrectly.

I have narrowed down the issue to the simple test case

"foo2/bar/".find ("/bar/")

Observe: On a GCC-compiled Python 2.5.1, the command works as
expected by returning 4:

  [c705213@hc-ma tmp]$ /usr/site/hc-2.6/python/gnu/2.5.1/bin/python2.5
  Python 2.5.1 (r251:54863, Aug 30 2007, 16:21:23)
  [GCC 3.4.6 20060404 (Red Hat 3.4.6-8)] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> print "foo2/**bar**/".find ("/**bar**/")
  4

On my Python 2.5.1 installation which was compiled from source with the
Intel C Compiler (ICC) for Linux, version 10.0, '-1' is returned:

  [c705213@hc-ma tmp]$ /usr/site/hc-2.6/python/intel/2.5.1/bin/python2.5
  Python 2.5.1 (r251:54863, Aug 30 2007, 16:20:06)
  [GCC Intel(R) C++ gcc 3.4 mode] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> print "foo2/**bar**/".find ("/**bar**/")
  -1

What could have possibly gone wrong here? Admittedly, this smacks more
of a bug in icc than in Python, but I report it here, as I feel at loss
of what else to do with it.

Obvious first question: Does anyone else out here have an ICC-compiled
Python handy to check whether the bug reproduces elsewhere?

Any idea what kind of oddity I have stumbled over here? Obviously, it
could simply be that something went wrong when compiling Python from
source with ICC, but it should not happen that the interpreter
nebertheless starts up and fails only silently.

Additional information:

• I have stumbled over the problem when trying to install Numpy 1.0.3.1,
  as the built failed at the point where a script 'conv_template.py',
  which is part of NumPy's installtion system, is started to do some
  pattern replacements in a file called 'scalartypes.inc.src'. My test
  case is reduced from this script.

• The system is the master node of a compute cluster with AMD Opteron
  CPUs. The cluster is not involved, all was done on the master node. The
  box runs RedHat Enterprise Linux 4.0 Advanced Server. It replies to
  'uname -a' with:
  Linux hc-ma.uibk.ac.at 2.6.9-42.0.10.ELsmp Support "bpo-" in Misc/NEWS #1 SMP Fri Feb 16 17:13:42
  EST 2007 x86_64 x86_64 x86_64 GNU/Linux

• The dynamic dependencies of the GCC-compiled and the ICC-compiled
  Python binaries are:
  [c705213@hc-ma tmp]$ ldd /usr/site/hc-2.6/python/gnu/2.5.1/bin/python2.5
  libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000003702900000)
  libdl.so.2 => /lib64/libdl.so.2 (0x0000003701d00000)
  libutil.so.1 => /lib64/libutil.so.1 (0x0000003703900000)
  libm.so.6 => /lib64/tls/libm.so.6 (0x0000003701b00000)
  libc.so.6 => /lib64/tls/libc.so.6 (0x0000003701800000)
  /lib64/ld-linux-x86-64.so.2 (0x0000003701600000)
  [c705213@hc-ma tmp]$ ldd /usr/site/hc-2.6/python/intel/2.5.1/bin/python2.5
  libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000003702900000)
  libdl.so.2 => /lib64/libdl.so.2 (0x0000003701d00000)
  libutil.so.1 => /lib64/libutil.so.1 (0x0000003703900000)
  libimf.so => /usr/site/hc-2.6/intel/10.0/cc/lib/libimf.so
  (0x0000002a95579000)
  libm.so.6 => /lib64/tls/libm.so.6 (0x0000003701b00000)
  libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003705800000)
  libc.so.6 => /lib64/tls/libc.so.6 (0x0000003701800000)
  /lib64/ld-linux-x86-64.so.2 (0x0000003701600000)

• The precise revision of Python is "Python 2.5.1 (r251:54863)".

• The test case ceases to show failure if the string is only slightly
  altered, e.g. if the word 'foo', the word 'bar' or the one of the
  asterisks or one of the slashes is cut out in both search and target string.

[loewis]

It definitely sounds like a compiler bug. Unless you can provide further
details to the specific error in the C code of Python, it's likely that
we can do little about it.

If you want to analyze this further, here is a number of things you can try:

• compile Python at various optimization levels. A compiler bug often
  manifests itself only at a specific set of optimization flags.
• try tracing this invocation of .find() in a debugger. Doing so at a
  lower optimization level is easier, since the compiler may have inlined
  the various functions that form .find() under optimization.
• if the debugger does not allow to pinpoint the erroneous function, add
  printf statements.
  Most of the code to study is in Objects/stringobject.c and
  Objects/stringlib/find.h.

[sandersmuc]

Martin, you are right: is is related to compiler optimization. I have
boiled it down to a call of stringlib_find (defined in
Python-2.5.1/Objects/stringlib/find.h) and this runs fine with 'icc -O2'
but incorrectly for 'icc -O3'. (The test code is attached.)

So, it seems that the lesson is simply once again: Do not use '-O3' with
Intel's C compiler. (At least, for me, it is not the first time that
this caused trouble.)

On the other hand, Python's ./configure script is quite clear in its
preference of GCC, anyway: It more or less ignores with option
'--without-gcc' and uses the content of the CC environment variable only
very occasionally.

[loewis]

If you are curious, we could now try to find out what precisely goes
wrong. The procedure would be this

• after each step, check whether the problem still occurs

a) resolve the includes manually, then strip everything that isn't
needed. This could start with fastsearch.h and find.h; then
remove everything that refers to Python.h (i.e. replace Py_ssize_t
with ssize_t, Py_LOCAL_INLINE with static inline, and so on), then
remove Python.h

b) try simplifying the code, e.g. replace str_len and sub_len with
their (constant) values, drop the sub_len == 0 block, and so on.

c) when further simplification is not possible (while keeping the
actual error), start looking at the assembler code.

Alternatively, sent this or some further-simplified version to Intel
(assuming they have some kind of bug-reporting channel for icc). With my
compiler-vendor's hat on, I'd like to get a test case for bad code
generation that comes as a single file, with no includes; for gcc, the
request is to submit preprocessor output.

Assuming this is too much effort, I'll close this as "won't fix - third
party".

[sandersmuc]

Martin: I've boiled down the test case a bit more and removed all
Python-specific types and macros, so that it can now be compiled
stand-alone. (Updated test case 'findtest.c' attached.) I didn't feel
like diving into the code much deeper, and so I have sent it to Intel
Premier Support as Issue bpo-448807. Let's see if they bother to
investigate it further.

[sandersmuc]

Update to the story: After I submitted the bug report to Intel, they
investigated and quickly confirmed it to be a compiler bug, whcih they
then managed to fix.

I have just got an e-mail from Intel that the newest available version
of ICC, namely version l_cc_c_10.1.008, contains the fix.

In principle the problem should vanish now, but I have not found the
time to verify that.