3.7.0 build error with --enable-optimizations

[jayanthkoushik]

BPO	34112
Nosy	@benjaminp, @jayanthkoushik
Files	setup.py: setup.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-07-15.09:29:00.111>
created_at = <Date 2018-07-14.16:25:41.541>
labels = ['build', '3.7']
title = '3.7.0 build error with --enable-optimizations'
updated_at = <Date 2018-07-15.09:29:00.109>
user = 'https://github.com/jayanthkoushik'

bugs.python.org fields:

activity = <Date 2018-07-15.09:29:00.109>
actor = 'jayanthkoushik'
assignee = 'none'
closed = True
closed_date = <Date 2018-07-15.09:29:00.111>
closer = 'jayanthkoushik'
components = ['Build']
creation = <Date 2018-07-14.16:25:41.541>
creator = 'jayanthkoushik'
dependencies = []
files = ['47691']
hgrepos = []
issue_num = 34112
keywords = []
message_count = 5.0
messages = ['321655', '321657', '321659', '321668', '321684']
nosy_count = 2.0
nosy_names = ['benjamin.peterson', 'jayanthkoushik']
pr_nums = []
priority = 'normal'
resolution = 'works for me'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue34112'
versions = ['Python 3.7']

[jayanthkoushik]

I'm trying to build 3.7.0 locally (not as superuser) on a server. I am able to successfully build without --enable-optimizations enabled, but the build fails with it enabled.

I have dependencies installed in custom locations as well, and this is my environment.

PREF=/scratch

export LC_ALL=C
export CC=gcc
export CXX=g++

export PATH=/opt/gcc/4.9.2/bin:$PREF/binutils/bin:/bin:/usr/bin

export LD_LIBRARY_PATH=/opt/gcc/4.9.2/lib64:/opt/gcc/4.9.2/lib:$PREF/bzip2/lib:$PREF/gdbm/lib:$PREF/libffi/lib64:$PREF/ncurses/lib:$PREF/openmpi/lib:$PREF/openssl/lib:$PREF/readline/lib:$PREF/sqlite/lib:$PREF/util-linux/lib:$PREF/xz/lib:$PREF/zlib/lib:/usr/lib64

export CPATH=/opt/gcc/4.9.2/include:$PREF/bzip2/include:$PREF/gdbm/include:$PREF/libffi/include:$PREF/ncurse
s/include:$PREF/openmpi/include:$PREF/openssl/include:$PREF/readline/include:$PREF/sqlite/include:$PREF/util
-linux/include:$PREF/xz/include:$PREF/zlib/include

I also have to update setup.py to point to the right locations for dependencies. This is the diff, and I've attached the full file.

46c46
< disabled_module_list = ["_tkinter", "_uuid"]
---

disabled_module_list = []
544,567c544,545
< add_dir_to_list(self.compiler.library_dirs,
< '/scratch/readline/lib')
< add_dir_to_list(self.compiler.include_dirs,
< '/scratch/readline/include')
< add_dir_to_list(self.compiler.library_dirs,
< '/scratch/bzip2/lib')
< add_dir_to_list(self.compiler.include_dirs,
< '/scratch/bzip2/include')
< add_dir_to_list(self.compiler.library_dirs,
< '/scratch/ncurses/lib')
< add_dir_to_list(self.compiler.include_dirs,
< '/scratch/ncurses/include')
< add_dir_to_list(self.compiler.library_dirs,
< '/scratch/xz/lib')
< add_dir_to_list(self.compiler.include_dirs,
< '/scratch/xz/include')
< add_dir_to_list(self.compiler.library_dirs,
< '/scratch/gdbm/lib')
< add_dir_to_list(self.compiler.include_dirs,
< '/scratch/gdbm/include')
< add_dir_to_list(self.compiler.library_dirs,
< '/scratch/libffi/lib64')
< add_dir_to_list(self.compiler.include_dirs,
< '/scratch/libffi/include')
---
add_dir_to_list(self.compiler.library_dirs, '/usr/local/lib')
add_dir_to_list(self.compiler.include_dirs, '/usr/local/include')
835,836c813,814
< #elif self.compiler.find_library_file(lib_dirs, 'ncursesw'):
< # curses_library = 'ncursesw'
---
elif self.compiler.find_library_file(lib_dirs, 'ncursesw'):
curses_library = 'ncursesw'
1152c1130,1136
< sqlite_inc_paths = [ '/scratch/sqlite/include' ]
---
sqlite_inc_paths = [ '/usr/include',
'/usr/include/sqlite',
'/usr/include/sqlite3',
'/usr/local/include',
'/usr/local/include/sqlite',
'/usr/local/include/sqlite3',
]
1195a1180
os.path.join(sqlite_incdir, '..', 'lib64'),
1196a1182,1183
os.path.join(sqlite_incdir, '..', '..', 'lib64'),
os.path.join(sqlite_incdir, '..', '..', 'lib'),

This is my build process.

mkdir build && cd build

LDFLAGS="-Wl,--rpath=$PREF/glibc/lib -Wl,--rpath=$PREF/bzip2/lib -Wl,--rpath=$PREF/gdbm/lib -Wl,--rpath=$PRE
F/ncurses/lib -Wl,--rpath=$PREF/openssl/lib -Wl,--rpath=$PREF/readline/lib -Wl,--rpath=$PREF/sqlite/lib -Wl,
--rpath=$PREF/xz/lib -Wl,--rpath=$PREF/zlib/lib -Wl,--rpath=$PREF/libffi/lib64 -Wl,--rpath=$PREF/openmpi/lib
-Wl,--rpath=$PREF/util-linux/lib -Wl,--dynamic-linker=$PREF/glibc/lib/ld-linux-x86-64.so.2" ../configure --
enable-shared --prefix=$PREF/python-latest --with-openssl=$PREF/openssl --enable-optimizations

make -j 8

And this is the error that I get. It's when rebuilding with optimizations.

Rebuilding with profile guided optimizations:
rm -f profile-clean-stamp
make build_all CFLAGS_NODIST=" -fprofile-use -fprofile-correction" LDFLAGS=""
make[1]: Entering directory `/scratch/src/Python-latest/build'
LD_LIBRARY_PATH=/scratch/src/Python-latest/build:/opt/gcc/4.9.2/lib64:/opt/gcc/4.9.2/lib:/scratch/bzip2/lib:/scratch/gdbm/lib:/scratch/libffi/lib64:/scratch/ncurses/lib:/scratch/openmpi/lib:/scratch/openssl/lib:/scratch/readline/lib:/scratch/sqlite/lib:/scratch/util-linux/lib:/scratch/xz/lib:/scratch/zlib/lib:/usr/lib64 ./python -E -S -m sysconfig --generate-posix-vars ;\
        if test $? -ne 0 ; then \
                echo "generate-posix-vars failed" ; \
                rm -f ./pybuilddir.txt ; \
                exit 1 ; \
        fi
Could not import runpy module
Traceback (most recent call last):
  File "/scratch/src/Python-latest/build/../Lib/runpy.py", line 15, in <module>
    import importlib.util
  File "/scratch/src/Python-latest/build/../Lib/importlib/util.py", line 14, in <module>
    from contextlib import contextmanager
  File "/scratch/src/Python-latest/build/../Lib/contextlib.py", line 4, in <module>
    import _collections_abc
SystemError: <built-in function compile> returned NULL without setting an error
generate-posix-vars failed
make[1]: *** [pybuilddir.txt] Error 1
make[1]: Leaving directory `/scratch/src/Python-latest/build'
make: *** [profile-opt] Error 2

I tried searching for the error, and there seem to be similar errors related to improper PYTHONHOME, but I couldn't find anything related to build. I was able to build 3.6 with the same procedure, so I'm not sure if I'm doing something wrong.

[benjaminp]

It's look you're using a fairly old toolchain. I wouldn't reccomend using fancy modern compiler optimizations with GCC 4.

[jayanthkoushik]

Hi. Do you mean turning of --enable-optimizations or reducing GCC optimization level (O3 etc)?

[benjaminp]

--enable-optimizations

On Sat, Jul 14, 2018, at 12:31, Jayanth Koushik wrote:

Jayanth Koushik jnkoushik@gmail.com added the comment:

Hi. Do you mean turning of --enable-optimizations or reducing GCC
optimization level (O3 etc)?

----------

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue34112\>

---

[jayanthkoushik]

Updating gcc to 8.1.0 fixed the problem.