Fatal Python error: _PySys_BeginInit: can't initialize sys module #79894

Tasy · 2019-01-10T22:35:32Z

BPO	35713
Nosy	@vstinner, @tirkarthi
PRs	bpo-35713: Rework Python initialization #11647 bpo-35713: Rework Python initialization #11647 bpo-35713: Rework Python initialization #11647 bpo-35713: Split _Py_InitializeCore into subfunctions #11650 bpo-35713: Split _Py_InitializeCore into subfunctions #11650 bpo-35713: Split _Py_InitializeCore into subfunctions #11650 bpo-35713: Reorganize sys module initialization #11658 bpo-35713: Reorganize sys module initialization #11658 bpo-35713: Reorganize sys module initialization #11658

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-01-18.17:52:46.947>
created_at = <Date 2019-01-10.22:35:31.620>
labels = ['build', '3.7']
title = "Fatal Python error: _PySys_BeginInit: can't initialize sys module"
updated_at = <Date 2019-01-23.14:06:29.967>
user = 'https://bugs.python.org/Tasy'

bugs.python.org fields:

activity = <Date 2019-01-23.14:06:29.967>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-01-18.17:52:46.947>
closer = 'Tasy'
components = ['Build']
creation = <Date 2019-01-10.22:35:31.620>
creator = 'Tasy'
dependencies = []
files = []
hgrepos = []
issue_num = 35713
keywords = []
message_count = 14.0
messages = ['333423', '333425', '333427', '333428', '333429', '333733', '333853', '333984', '333992', '333999', '334219', '334233', '334258', '334259']
nosy_count = 3.0
nosy_names = ['vstinner', 'xtreak', 'Tasy']
pr_nums = ['11647', '11647', '11647', '11650', '11650', '11650', '11658', '11658', '11658']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue35713'
versions = ['Python 3.7']

Tasy · 2019-01-10T22:35:32Z

.
.
.
./python -E -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
echo "generate-posix-vars failed" ; \
rm -f ./pybuilddir.txt ; \
exit 1 ; \
fi
Fatal Python error: _PySys_BeginInit: can't initialize sys module

Current thread 0x00002b4e5f9bf400 (most recent call first):
Aborted (core dumped)
generate-posix-vars failed
make[1]: *** [pybuilddir.txt] Error 1
make[1]: Leaving directory `/usr/local/mysoftware/Python-3.7.2/build'
make: *** [profile-opt] Error 2

vstinner · 2019-01-10T22:48:50Z

.
.
.

Hello. Can you elaborate this part?

Tasy · 2019-01-10T23:38:41Z

Sorry for the confusion. The three dots were for many lines of successful compilation output from make. The rest is the final few lines where the compilation fails.

vstinner · 2019-01-10T23:42:00Z

Because it's late, I will say it shortly: if you don't elaborate how you get this error, I will simply close the issue. You have to describe what you are trying to do, your OS, etc.

(It works for me!)

tirkarthi · 2019-01-10T23:45:02Z

Please attach the full build log too along with the command executed.

Tasy · 2019-01-15T22:03:41Z

Configuration Options:

../configure --prefix=$HOME --enable-shared --enable-optimizations --with-system-expat --with-system-ffi --with-ensurepip=yes

Make throws the following warning:

*** WARNING: renaming "_curses_panel" since importing it failed: No module named '_curses'

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_ssl _uuid
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc atexit pwd
time

Failed to build these modules:
_curses

Following modules built successfully but were removed because they could not be imported:
_curses_panel

Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, libressl/portable#381

There ther is a following error...

0:06:18 load avg: 0.55 [171/416] test_hashlib
*** Error in `./python': corrupted size vs. prev_size: 0x000000000276b7a0 ***
Fatal Python error: Aborted

Current thread 0x00002ba4468c7bc0 (most recent call first):
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/test_hashlib.py", line 904 in _test_pbkdf2_hmac
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/test_hashlib.py", line 935 in test_pbkdf2_hmac_c
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/case.py", line 615 in run
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/case.py", line 663 in call
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 122 in run
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 84 in call
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 122 in run
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 84 in call
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 122 in run
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 84 in call
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/support/testresult.py", line 162 in run
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/support/init.py", line 1895 in _run_suite
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/support/init.py", line 1991 in run_unittest
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/runtest.py", line 178 in test_runner
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/runtest.py", line 182 in runtest_inner
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/runtest.py", line 137 in runtest
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/main.py", line 407 in run_tests_sequential
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/main.py", line 514 in run_tests
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/main.py", line 615 in _main
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/main.py", line 582 in main
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/main.py", line 636 in main
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/regrtest.py", line 46 in _main
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/regrtest.py", line 50 in
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/runpy.py", line 85 in _run_code
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/runpy.py", line 193 in _run_module_as_main
Aborted (core dumped)
make[1]: Leaving directory /usr/local/data/mySoftware/Python-3.7.2/build' make build_all_merge_profile make[1]: Entering directory /usr/local/data/mySoftware/Python-3.7.2/build'
true
make[1]: Leaving directory `/usr/local/data/mySoftware/Python-3.7.2/build'

Remove profile generation binary since we are done with it.

make clean
make[1]: Entering directory /usr/local/data/mySoftware/Python-3.7.2/build' find .. -depth -name '__pycache__' -exec rm -rf {} ';' find .. -name '*.py[co]' -exec rm -f {} ';' find . -name '*.[oa]' -exec rm -f {} ';' find . -name '*.s[ol]' -exec rm -f {} ';' find . -name '*.so.[0-9]*.[0-9]*' -exec rm -f {} ';' find build -name 'fficonfig.h' -exec rm -f {} ';' || true find build -name '*.py' -exec rm -f {} ';' || true find build -name '*.py[co]' -exec rm -f {} ';' || true rm -f pybuilddir.txt rm -f [Lib/lib2to3/](https://github.com/python/cpython/blob/main/Lib/lib2to3)*Grammar*.pickle rm -f [Programs/_testembed](https://github.com/python/cpython/blob/main/Programs/_testembed) [Programs/_freeze_importlib](https://github.com/python/cpython/blob/main/Programs/_freeze_importlib) find build -type f -a ! -name '*.gc??' -exec rm -f {} ';' rm -f [Include/pydtrace_probes.h](https://github.com/python/cpython/blob/main/Include/pydtrace_probes.h) rm -f profile-gen-stamp make[1]: Leaving directory /usr/local/data/mySoftware/Python-3.7.2/build'

This is an expensive target to build and it does not have proper

makefile dependency information. So, we create a "stamp" file

to record its completion and avoid re-running it.

touch profile-run-stamp
Rebuilding with profile guided optimizations:
rm -f profile-clean-stamp
make build_all CFLAGS_NODIST=" -fprofile-use -fprofile-correction" LDFLAGS_NODIST=""
make[1]: Entering directory `/usr/local/data/mySoftware/Python-3.7.2/build'

And then finally :

Modules/_localemodule.o Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o Modules/zipimport.o Modules/faulthandler.o Modules/_tracemalloc.o Modules/hashtable.o Modules/symtablemodule.o Modules/xxsubtype.o Python/frozen.o
gcc -pthread -shared -Wl,--no-as-needed -o libpython3.so -Wl,-hlibpython3.so libpython3.7m.so
gcc -pthread -Xlinker -export-dynamic -o python Programs/python.o -L. -lpython3.7m -lpthread -ldl -lutil -lm
gcc -pthread -Xlinker -export-dynamic -o Programs/_testembed Programs/_testembed.o -L. -lpython3.7m -lpthread -ldl -lutil -lm
LD_LIBRARY_PATH=/usr/local/data/mySoftware/Python-3.7.2/build:$HOME/lib:/usr/local/cuda/lib64:$HOME/lib/cuda/lib64:$HOME/lib/cuda/extras/CUPTI/lib64:/usr/local/data/lib:/usr/local/lib/:/usr/lib/ ./python -E -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
echo "generate-posix-vars failed" ; \
rm -f ./pybuilddir.txt ; \
exit 1 ; \
fi
Fatal Python error: _PySys_BeginInit: can't initialize sys module

Current thread 0x00002adc936069c0 (most recent call first):
Aborted (core dumped)
generate-posix-vars failed
make[1]: *** [pybuilddir.txt] Error 1
make[1]: Leaving directory `/usr/local/data/mySoftware/Python-3.7.2/build'
make: *** [profile-opt] Error 2

vstinner · 2019-01-17T12:12:50Z

What is your OS (name and version)?

What is your compiler (name and version)?

Fatal Python error: _PySys_BeginInit: can't initialize sys module

I have no idea why you get this error. You should try to run this function in a debugger like gdb and run the code step by step to see what happens.

"""

0:06:18 load avg: 0.55 [171/416] test_hashlib
*** Error in `./python': corrupted size vs. prev_size: 0x000000000276b7a0 ***
Fatal Python error: Aborted

Current thread 0x00002ba4468c7bc0 (most recent call first):
File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/test_hashlib.py", line 904 in _test_pbkdf2_hmac
"""

That's maybe unrelated, but _test_pbkdf2_hmac() is not supposed to crash.

Tasy · 2019-01-18T15:51:10Z

Compiler:
$ gcc --version
gcc (Ubuntu 4.8.4-2ubuntu1~14.04.4) 4.8.4

OS:
$ uname -a
Linux machine 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Home directory is of type nfs in case that is relevant.
I'll try to do the gdb thing over the weekend.

vstinner · 2019-01-18T16:55:55Z

Hum. Are you aware that PGO with GCC is broken on such old and unsupported Ubuntu version?

My old note about that:

"PGO is broken on Ubuntu 14.04 LTS with GCC 4.8.4-2ubuntu1~14.04: Modules/socketmodule.c:7743:1: internal compiler error: in edge_badness, at ipa-inline.c:895"

https://pyperformance.readthedocs.io/usage.html#compile

See also https://bugs.python.org/issue31963

I suggest you to not use PGO compilation on old Ubuntu. Maybe upgrade to latest Ubuntu LTS?

Tasy · 2019-01-18T17:52:47Z

compiling without optimizations worked.

vstinner · 2019-01-22T16:39:06Z

New changeset bf4ac2d by Victor Stinner in branch 'master':
bpo-35713: Rework Python initialization (GH-11647)
bf4ac2d

vstinner · 2019-01-22T20:18:21Z

New changeset 6d43f6f by Victor Stinner in branch 'master':
bpo-35713: Split _Py_InitializeCore into subfunctions (GH-11650)
6d43f6f

vstinner · 2019-01-23T14:04:43Z

New changeset ab67281 by Victor Stinner in branch 'master':
bpo-35713: Reorganize sys module initialization (GH-11658)
ab67281

vstinner · 2019-01-23T14:06:30Z

"""

Fatal Python error: _PySys_BeginInit: can't initialize sys module

I have no idea why you get this error. You should try to run this function in a debugger like gdb and run the code step by step to see what happens.
"""

I pushed 3 changes to get working exceptions and working sys.stderr earlier during Python initialization. It should help to display the current exceptions when Py_FatalError() is called during early stage of the Python initialization.

sergey-dryabzhinsky · 2023-06-15T12:26:45Z

Same error while building package for Ubuntu 14.04.

no pgo, no lto
arch: amd64
python: 3.8.17
configure flags: --prefix=/usr --enable-ipv6 --enable-loadable-sqlite-extensions --enable-optimizations --with-dbmliborder=bdb:gdbm --with-computed-gotos --without-ensurepip --with-system-libmpdec --without-system-expat --with-openssl=/opt/openssl111r/usr --with-system-ffi

Previous python version compiled fine.

Wait. It's maybe cos PGO enabled somehow.

vstinner · 2023-06-15T15:59:17Z

You're using old Python on old Ubuntu. Maybe use Ubuntu packages like deadsnakes instead of building Python yourself. Yes, enable-optimizations enables PGO.

This issue is closed and Python 3.8 is no longer supported. Give a try to Python 3.11.

sergey-dryabzhinsky · 2023-06-15T17:41:55Z

Yeah, error gone if enable-optimizations removed from configure args.

yevgen-lasman · 2024-01-22T21:59:22Z

You're using old Python on old Ubuntu. Maybe use Ubuntu packages like deadsnakes instead of building Python yourself. Yes, enable-optimizations enables PGO.

This issue is closed and Python 3.8 is no longer supported. Give a try to Python 3.11.

The problem persists for me for either 3.11 or 3.12 Glad to provide more details about the build environment if it helps to solve this problem.

vstinner · 2024-01-23T13:28:28Z

If you can reproduce the issue on a recent Python version, please open a new issue with complete instructions on how to reproduce it, the output and expected output. This issue is old and closed.

Tasy mannequin added 3.7 (EOL) end of life build The build process and cross-build labels Jan 10, 2019

Tasy mannequin closed this as completed Jan 18, 2019

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fatal Python error: _PySys_BeginInit: can't initialize sys module #79894

Fatal Python error: _PySys_BeginInit: can't initialize sys module #79894

Tasy mannequin commented Jan 10, 2019

Tasy mannequin commented Jan 10, 2019

vstinner commented Jan 10, 2019

Tasy mannequin commented Jan 10, 2019

vstinner commented Jan 10, 2019

tirkarthi commented Jan 10, 2019

Tasy mannequin commented Jan 15, 2019

vstinner commented Jan 17, 2019

Tasy mannequin commented Jan 18, 2019

vstinner commented Jan 18, 2019

Tasy mannequin commented Jan 18, 2019

vstinner commented Jan 22, 2019

vstinner commented Jan 22, 2019

vstinner commented Jan 23, 2019

vstinner commented Jan 23, 2019

sergey-dryabzhinsky commented Jun 15, 2023 •

edited

Loading

vstinner commented Jun 15, 2023

sergey-dryabzhinsky commented Jun 15, 2023

yevgen-lasman commented Jan 22, 2024

vstinner commented Jan 23, 2024

Fatal Python error: _PySys_BeginInit: can't initialize sys module #79894

Fatal Python error: _PySys_BeginInit: can't initialize sys module #79894

Comments

Tasy mannequin commented Jan 10, 2019

Tasy mannequin commented Jan 10, 2019

vstinner commented Jan 10, 2019

Tasy mannequin commented Jan 10, 2019

vstinner commented Jan 10, 2019

tirkarthi commented Jan 10, 2019

Tasy mannequin commented Jan 15, 2019

Remove profile generation binary since we are done with it.

This is an expensive target to build and it does not have proper

makefile dependency information. So, we create a "stamp" file

to record its completion and avoid re-running it.

vstinner commented Jan 17, 2019

Tasy mannequin commented Jan 18, 2019

vstinner commented Jan 18, 2019

Tasy mannequin commented Jan 18, 2019

vstinner commented Jan 22, 2019

vstinner commented Jan 22, 2019

vstinner commented Jan 23, 2019

vstinner commented Jan 23, 2019

sergey-dryabzhinsky commented Jun 15, 2023 • edited Loading

vstinner commented Jun 15, 2023

sergey-dryabzhinsky commented Jun 15, 2023

yevgen-lasman commented Jan 22, 2024

vstinner commented Jan 23, 2024

sergey-dryabzhinsky commented Jun 15, 2023 •

edited

Loading