Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fatal Python error: _PySys_BeginInit: can't initialize sys module #79894

Closed
Tasy mannequin opened this issue Jan 10, 2019 · 17 comments
Closed

Fatal Python error: _PySys_BeginInit: can't initialize sys module #79894

Tasy mannequin opened this issue Jan 10, 2019 · 17 comments
Labels
3.7 (EOL) end of life build The build process and cross-build

Comments

@Tasy
Copy link
Mannequin

Tasy mannequin commented Jan 10, 2019

BPO 35713
Nosy @vstinner, @tirkarthi
PRs
  • bpo-35713: Rework Python initialization #11647
  • bpo-35713: Rework Python initialization #11647
  • bpo-35713: Rework Python initialization #11647
  • bpo-35713: Split _Py_InitializeCore into subfunctions #11650
  • bpo-35713: Split _Py_InitializeCore into subfunctions #11650
  • bpo-35713: Split _Py_InitializeCore into subfunctions #11650
  • bpo-35713: Reorganize sys module initialization #11658
  • bpo-35713: Reorganize sys module initialization #11658
  • bpo-35713: Reorganize sys module initialization #11658
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2019-01-18.17:52:46.947>
    created_at = <Date 2019-01-10.22:35:31.620>
    labels = ['build', '3.7']
    title = "Fatal Python error: _PySys_BeginInit: can't initialize sys module"
    updated_at = <Date 2019-01-23.14:06:29.967>
    user = 'https://bugs.python.org/Tasy'

    bugs.python.org fields:

    activity = <Date 2019-01-23.14:06:29.967>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2019-01-18.17:52:46.947>
    closer = 'Tasy'
    components = ['Build']
    creation = <Date 2019-01-10.22:35:31.620>
    creator = 'Tasy'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 35713
    keywords = []
    message_count = 14.0
    messages = ['333423', '333425', '333427', '333428', '333429', '333733', '333853', '333984', '333992', '333999', '334219', '334233', '334258', '334259']
    nosy_count = 3.0
    nosy_names = ['vstinner', 'xtreak', 'Tasy']
    pr_nums = ['11647', '11647', '11647', '11650', '11650', '11650', '11658', '11658', '11658']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'compile error'
    url = 'https://bugs.python.org/issue35713'
    versions = ['Python 3.7']

    @Tasy
    Copy link
    Mannequin Author

    Tasy mannequin commented Jan 10, 2019

    .
    .
    .
    ./python -E -S -m sysconfig --generate-posix-vars ;\
    if test $? -ne 0 ; then \
    echo "generate-posix-vars failed" ; \
    rm -f ./pybuilddir.txt ; \
    exit 1 ; \
    fi
    Fatal Python error: _PySys_BeginInit: can't initialize sys module

    Current thread 0x00002b4e5f9bf400 (most recent call first):
    Aborted (core dumped)
    generate-posix-vars failed
    make[1]: *** [pybuilddir.txt] Error 1
    make[1]: Leaving directory `/usr/local/mysoftware/Python-3.7.2/build'
    make: *** [profile-opt] Error 2

    @Tasy Tasy mannequin added 3.7 (EOL) end of life build The build process and cross-build labels Jan 10, 2019
    @vstinner
    Copy link
    Member

    .
    .
    .

    Hello. Can you elaborate this part?

    @Tasy
    Copy link
    Mannequin Author

    Tasy mannequin commented Jan 10, 2019

    Sorry for the confusion. The three dots were for many lines of successful compilation output from make. The rest is the final few lines where the compilation fails.

    @vstinner
    Copy link
    Member

    Because it's late, I will say it shortly: if you don't elaborate how you get this error, I will simply close the issue. You have to describe what you are trying to do, your OS, etc.

    (It works for me!)

    @tirkarthi
    Copy link
    Member

    Please attach the full build log too along with the command executed.

    @Tasy
    Copy link
    Mannequin Author

    Tasy mannequin commented Jan 15, 2019

    Configuration Options:

    ../configure --prefix=$HOME --enable-shared --enable-optimizations --with-system-expat --with-system-ffi --with-ensurepip=yes

    Make throws the following warning:

    *** WARNING: renaming "_curses_panel" since importing it failed: No module named '_curses'

    Python build finished successfully!
    The necessary bits to build these optional modules were not found:
    _ssl _uuid
    To find the necessary bits, look in setup.py in detect_modules() for the module's name.

    The following modules found by detect_modules() in setup.py, have been
    built by the Makefile instead, as configured by the Setup files:
    _abc atexit pwd
    time

    Failed to build these modules:
    _curses

    Following modules built successfully but were removed because they could not be imported:
    _curses_panel

    Could not build the ssl module!
    Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
    LibreSSL 2.6.4 and earlier do not provide the necessary APIs, libressl/portable#381

    There ther is a following error...

    0:06:18 load avg: 0.55 [171/416] test_hashlib
    *** Error in `./python': corrupted size vs. prev_size: 0x000000000276b7a0 ***
    Fatal Python error: Aborted

    Current thread 0x00002ba4468c7bc0 (most recent call first):
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/test_hashlib.py", line 904 in _test_pbkdf2_hmac
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/test_hashlib.py", line 935 in test_pbkdf2_hmac_c
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/case.py", line 615 in run
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/case.py", line 663 in call
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 122 in run
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 84 in call
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 122 in run
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 84 in call
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 122 in run
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 84 in call
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/support/testresult.py", line 162 in run
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/support/init.py", line 1895 in _run_suite
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/support/init.py", line 1991 in run_unittest
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/runtest.py", line 178 in test_runner
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/runtest.py", line 182 in runtest_inner
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/runtest.py", line 137 in runtest
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/main.py", line 407 in run_tests_sequential
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/main.py", line 514 in run_tests
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/main.py", line 615 in _main
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/main.py", line 582 in main
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/main.py", line 636 in main
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/regrtest.py", line 46 in _main
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/regrtest.py", line 50 in
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/runpy.py", line 85 in _run_code
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/runpy.py", line 193 in _run_module_as_main
    Aborted (core dumped)
    make[1]: Leaving directory /usr/local/data/mySoftware/Python-3.7.2/build' make build_all_merge_profile make[1]: Entering directory /usr/local/data/mySoftware/Python-3.7.2/build'
    true
    make[1]: Leaving directory `/usr/local/data/mySoftware/Python-3.7.2/build'

    Remove profile generation binary since we are done with it.

    make clean
    make[1]: Entering directory /usr/local/data/mySoftware/Python-3.7.2/build' find .. -depth -name '__pycache__' -exec rm -rf {} ';' find .. -name '*.py[co]' -exec rm -f {} ';' find . -name '*.[oa]' -exec rm -f {} ';' find . -name '*.s[ol]' -exec rm -f {} ';' find . -name '*.so.[0-9]*.[0-9]*' -exec rm -f {} ';' find build -name 'fficonfig.h' -exec rm -f {} ';' || true find build -name '*.py' -exec rm -f {} ';' || true find build -name '*.py[co]' -exec rm -f {} ';' || true rm -f pybuilddir.txt rm -f [Lib/lib2to3/](https://github.com/python/cpython/blob/main/Lib/lib2to3)*Grammar*.pickle rm -f [Programs/_testembed](https://github.com/python/cpython/blob/main/Programs/_testembed) [Programs/_freeze_importlib](https://github.com/python/cpython/blob/main/Programs/_freeze_importlib) find build -type f -a ! -name '*.gc??' -exec rm -f {} ';' rm -f [Include/pydtrace_probes.h](https://github.com/python/cpython/blob/main/Include/pydtrace_probes.h) rm -f profile-gen-stamp make[1]: Leaving directory /usr/local/data/mySoftware/Python-3.7.2/build'

    This is an expensive target to build and it does not have proper

    makefile dependency information. So, we create a "stamp" file

    to record its completion and avoid re-running it.

    touch profile-run-stamp
    Rebuilding with profile guided optimizations:
    rm -f profile-clean-stamp
    make build_all CFLAGS_NODIST=" -fprofile-use -fprofile-correction" LDFLAGS_NODIST=""
    make[1]: Entering directory `/usr/local/data/mySoftware/Python-3.7.2/build'

    And then finally :

    Modules/_localemodule.o Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o Modules/zipimport.o Modules/faulthandler.o Modules/_tracemalloc.o Modules/hashtable.o Modules/symtablemodule.o Modules/xxsubtype.o Python/frozen.o
    gcc -pthread -shared -Wl,--no-as-needed -o libpython3.so -Wl,-hlibpython3.so libpython3.7m.so
    gcc -pthread -Xlinker -export-dynamic -o python Programs/python.o -L. -lpython3.7m -lpthread -ldl -lutil -lm
    gcc -pthread -Xlinker -export-dynamic -o Programs/_testembed Programs/_testembed.o -L. -lpython3.7m -lpthread -ldl -lutil -lm
    LD_LIBRARY_PATH=/usr/local/data/mySoftware/Python-3.7.2/build:$HOME/lib:/usr/local/cuda/lib64:$HOME/lib/cuda/lib64:$HOME/lib/cuda/extras/CUPTI/lib64:/usr/local/data/lib:/usr/local/lib/:/usr/lib/ ./python -E -S -m sysconfig --generate-posix-vars ;\
    if test $? -ne 0 ; then \
    echo "generate-posix-vars failed" ; \
    rm -f ./pybuilddir.txt ; \
    exit 1 ; \
    fi
    Fatal Python error: _PySys_BeginInit: can't initialize sys module

    Current thread 0x00002adc936069c0 (most recent call first):
    Aborted (core dumped)
    generate-posix-vars failed
    make[1]: *** [pybuilddir.txt] Error 1
    make[1]: Leaving directory `/usr/local/data/mySoftware/Python-3.7.2/build'
    make: *** [profile-opt] Error 2

    @vstinner
    Copy link
    Member

    What is your OS (name and version)?

    What is your compiler (name and version)?

    Fatal Python error: _PySys_BeginInit: can't initialize sys module

    I have no idea why you get this error. You should try to run this function in a debugger like gdb and run the code step by step to see what happens.

    """

    0:06:18 load avg: 0.55 [171/416] test_hashlib
    *** Error in `./python': corrupted size vs. prev_size: 0x000000000276b7a0 ***
    Fatal Python error: Aborted

    Current thread 0x00002ba4468c7bc0 (most recent call first):
    File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/test_hashlib.py", line 904 in _test_pbkdf2_hmac
    """

    That's maybe unrelated, but _test_pbkdf2_hmac() is not supposed to crash.

    @Tasy
    Copy link
    Mannequin Author

    Tasy mannequin commented Jan 18, 2019

    Compiler:
    $ gcc --version
    gcc (Ubuntu 4.8.4-2ubuntu1~14.04.4) 4.8.4

    OS:
    $ uname -a
    Linux machine 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

    Home directory is of type nfs in case that is relevant.
    I'll try to do the gdb thing over the weekend.

    @vstinner
    Copy link
    Member

    Hum. Are you aware that PGO with GCC is broken on such old and unsupported Ubuntu version?

    My old note about that:

    "PGO is broken on Ubuntu 14.04 LTS with GCC 4.8.4-2ubuntu1~14.04: Modules/socketmodule.c:7743:1: internal compiler error: in edge_badness, at ipa-inline.c:895"

    https://pyperformance.readthedocs.io/usage.html#compile

    See also https://bugs.python.org/issue31963

    I suggest you to not use PGO compilation on old Ubuntu. Maybe upgrade to latest Ubuntu LTS?

    @Tasy
    Copy link
    Mannequin Author

    Tasy mannequin commented Jan 18, 2019

    compiling without optimizations worked.

    @Tasy Tasy mannequin closed this as completed Jan 18, 2019
    @vstinner
    Copy link
    Member

    New changeset bf4ac2d by Victor Stinner in branch 'master':
    bpo-35713: Rework Python initialization (GH-11647)
    bf4ac2d

    @vstinner
    Copy link
    Member

    New changeset 6d43f6f by Victor Stinner in branch 'master':
    bpo-35713: Split _Py_InitializeCore into subfunctions (GH-11650)
    6d43f6f

    @vstinner
    Copy link
    Member

    New changeset ab67281 by Victor Stinner in branch 'master':
    bpo-35713: Reorganize sys module initialization (GH-11658)
    ab67281

    @vstinner
    Copy link
    Member

    """

    Fatal Python error: _PySys_BeginInit: can't initialize sys module

    I have no idea why you get this error. You should try to run this function in a debugger like gdb and run the code step by step to see what happens.
    """

    I pushed 3 changes to get working exceptions and working sys.stderr earlier during Python initialization. It should help to display the current exceptions when Py_FatalError() is called during early stage of the Python initialization.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @sergey-dryabzhinsky
    Copy link

    sergey-dryabzhinsky commented Jun 15, 2023

    Same error while building package for Ubuntu 14.04.

    • no pgo, no lto
    • arch: amd64
    • python: 3.8.17
    • configure flags: --prefix=/usr --enable-ipv6 --enable-loadable-sqlite-extensions --enable-optimizations --with-dbmliborder=bdb:gdbm --with-computed-gotos --without-ensurepip --with-system-libmpdec --without-system-expat --with-openssl=/opt/openssl111r/usr --with-system-ffi

    Previous python version compiled fine.

    Wait. It's maybe cos PGO enabled somehow.

    @vstinner
    Copy link
    Member

    You're using old Python on old Ubuntu. Maybe use Ubuntu packages like deadsnakes instead of building Python yourself. Yes, enable-optimizations enables PGO.

    This issue is closed and Python 3.8 is no longer supported. Give a try to Python 3.11.

    @sergey-dryabzhinsky
    Copy link

    Yeah, error gone if enable-optimizations removed from configure args.

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life build The build process and cross-build
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants