Installation issue: python is built without curses module #34872

rkevk · 2023-01-10T18:31:30Z

Steps to reproduce the issue

I tested the following combinations, which all resulted in the same behavior:

python@3.10.8 ^ncurses@6.3
python@3.8.10 ^ncurses@6.3
python@3.8.10 ^ncurses@6.2

The concretizations, as reported by spack spec -I python@..., are as follows:

Input spec
--------------------------------
 -   python@3.10.8
 -       ^ncurses@6.3

Concretized
--------------------------------
[+]  python@3.10.8%gcc@9.4.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,7d40923,f2fd060 arch=linux-ubuntu20.04-zen2
[+]      ^bzip2@1.0.8%gcc@9.4.0~debug~pic+shared build_system=generic arch=linux-ubuntu20.04-zen2
[+]          ^diffutils@3.8%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^expat@2.5.0%gcc@9.4.0+libbsd build_system=autotools arch=linux-ubuntu20.04-zen2
[+]          ^libbsd@0.11.5%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-zen2
[+]              ^libmd@1.0.4%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^gdbm@1.23%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^gettext@0.21.1%gcc@9.4.0+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools arch=linux-ubuntu20.04-zen2
[+]          ^libiconv@1.16%gcc@9.4.0 build_system=autotools libs=shared,static arch=linux-ubuntu20.04-zen2
[+]          ^libxml2@2.10.3%gcc@9.4.0~python build_system=autotools arch=linux-ubuntu20.04-zen2
[+]          ^tar@1.34%gcc@9.4.0 build_system=autotools zip=pigz arch=linux-ubuntu20.04-zen2
[+]              ^pigz@2.7%gcc@9.4.0 build_system=makefile arch=linux-ubuntu20.04-zen2
[+]              ^zstd@1.5.2%gcc@9.4.0+programs build_system=makefile compression=none libs=shared,static arch=linux-ubuntu20.04-zen2
[+]      ^libffi@3.4.2%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^libxcrypt@4.4.33%gcc@9.4.0~obsolete_api build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^ncurses@6.3%gcc@9.4.0~symlinks+termlib abi=none build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^openssl@1.1.1s%gcc@9.4.0~docs~shared build_system=generic certs=mozilla arch=linux-ubuntu20.04-zen2
[+]          ^ca-certificates-mozilla@2022-10-11%gcc@9.4.0 build_system=generic arch=linux-ubuntu20.04-zen2
[+]          ^perl@5.36.0%gcc@9.4.0+cpanm+shared+threads build_system=generic arch=linux-ubuntu20.04-zen2
[+]              ^berkeley-db@18.1.40%gcc@9.4.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-ubuntu20.04-zen2
[+]      ^pkgconf@1.8.0%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^readline@8.2%gcc@9.4.0 build_system=autotools patches=bbf97f1 arch=linux-ubuntu20.04-zen2
[+]      ^sqlite@3.40.0%gcc@9.4.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^util-linux-uuid@2.38.1%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^xz@5.2.7%gcc@9.4.0~pic build_system=autotools libs=shared,static arch=linux-ubuntu20.04-zen2
[+]      ^zlib@1.2.13%gcc@9.4.0+optimize+pic+shared build_system=makefile arch=linux-ubuntu20.04-zen2

Input spec
--------------------------------
 -   python@3.8.10
 -       ^ncurses@6.3

Concretized
--------------------------------
 -   python@3.8.10%gcc@9.4.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,4c24573,f2fd060 arch=linux-ubuntu20.04-zen2
[+]      ^bzip2@1.0.8%gcc@9.4.0~debug~pic+shared build_system=generic arch=linux-ubuntu20.04-zen2
[+]          ^diffutils@3.8%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^expat@2.5.0%gcc@9.4.0+libbsd build_system=autotools arch=linux-ubuntu20.04-zen2
[+]          ^libbsd@0.11.5%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-zen2
[+]              ^libmd@1.0.4%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^gdbm@1.23%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^gettext@0.21.1%gcc@9.4.0+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools arch=linux-ubuntu20.04-zen2
[+]          ^libiconv@1.16%gcc@9.4.0 build_system=autotools libs=shared,static arch=linux-ubuntu20.04-zen2
[+]          ^libxml2@2.10.3%gcc@9.4.0~python build_system=autotools arch=linux-ubuntu20.04-zen2
[+]          ^tar@1.34%gcc@9.4.0 build_system=autotools zip=pigz arch=linux-ubuntu20.04-zen2
[+]              ^pigz@2.7%gcc@9.4.0 build_system=makefile arch=linux-ubuntu20.04-zen2
[+]              ^zstd@1.5.2%gcc@9.4.0+programs build_system=makefile compression=none libs=shared,static arch=linux-ubuntu20.04-zen2
[+]      ^libffi@3.4.2%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^libxcrypt@4.4.33%gcc@9.4.0~obsolete_api build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^ncurses@6.3%gcc@9.4.0~symlinks+termlib abi=none build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^openssl@1.1.1s%gcc@9.4.0~docs~shared build_system=generic certs=mozilla arch=linux-ubuntu20.04-zen2
[+]          ^ca-certificates-mozilla@2022-10-11%gcc@9.4.0 build_system=generic arch=linux-ubuntu20.04-zen2
[+]          ^perl@5.36.0%gcc@9.4.0+cpanm+shared+threads build_system=generic arch=linux-ubuntu20.04-zen2
[+]              ^berkeley-db@18.1.40%gcc@9.4.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-ubuntu20.04-zen2
[+]      ^pkgconf@1.8.0%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^readline@8.2%gcc@9.4.0 build_system=autotools patches=bbf97f1 arch=linux-ubuntu20.04-zen2
[+]      ^sqlite@3.40.0%gcc@9.4.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^util-linux-uuid@2.38.1%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^xz@5.2.7%gcc@9.4.0~pic build_system=autotools libs=shared,static arch=linux-ubuntu20.04-zen2
[+]      ^zlib@1.2.13%gcc@9.4.0+optimize+pic+shared build_system=makefile arch=linux-ubuntu20.04-zen2

Input spec
--------------------------------
 -   python@3.8.10
 -       ^ncurses@6.2

Concretized
--------------------------------
[+]  python@3.8.10%gcc@9.4.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,4c24573,f2fd060 arch=linux-ubuntu20.04-zen2
[+]      ^bzip2@1.0.8%gcc@9.4.0~debug~pic+shared build_system=generic arch=linux-ubuntu20.04-zen2
[+]          ^diffutils@3.8%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^expat@2.5.0%gcc@9.4.0+libbsd build_system=autotools arch=linux-ubuntu20.04-zen2
[+]          ^libbsd@0.11.5%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-zen2
[+]              ^libmd@1.0.4%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^gdbm@1.23%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^gettext@0.21.1%gcc@9.4.0+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools arch=linux-ubuntu20.04-zen2
[+]          ^libiconv@1.16%gcc@9.4.0 build_system=autotools libs=shared,static arch=linux-ubuntu20.04-zen2
[+]          ^libxml2@2.10.3%gcc@9.4.0~python build_system=autotools arch=linux-ubuntu20.04-zen2
[+]          ^tar@1.34%gcc@9.4.0 build_system=autotools zip=pigz arch=linux-ubuntu20.04-zen2
[+]              ^pigz@2.7%gcc@9.4.0 build_system=makefile arch=linux-ubuntu20.04-zen2
[+]              ^zstd@1.5.2%gcc@9.4.0+programs build_system=makefile compression=none libs=shared,static arch=linux-ubuntu20.04-zen2
[+]      ^libffi@3.4.2%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^libxcrypt@4.4.33%gcc@9.4.0~obsolete_api build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^ncurses@6.2%gcc@9.4.0~symlinks+termlib abi=none build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^openssl@1.1.1s%gcc@9.4.0~docs~shared build_system=generic certs=mozilla arch=linux-ubuntu20.04-zen2
[+]          ^ca-certificates-mozilla@2022-10-11%gcc@9.4.0 build_system=generic arch=linux-ubuntu20.04-zen2
[+]          ^perl@5.36.0%gcc@9.4.0+cpanm+shared+threads build_system=generic arch=linux-ubuntu20.04-zen2
[+]              ^berkeley-db@18.1.40%gcc@9.4.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-ubuntu20.04-zen2
[+]              ^gdbm@1.23%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-zen2
[+]                  ^readline@8.2%gcc@9.4.0 build_system=autotools patches=bbf97f1 arch=linux-ubuntu20.04-zen2
[+]                      ^ncurses@6.3%gcc@9.4.0~symlinks+termlib abi=none build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^pkgconf@1.8.0%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^readline@8.2%gcc@9.4.0 build_system=autotools patches=bbf97f1 arch=linux-ubuntu20.04-zen2
[+]      ^sqlite@3.40.0%gcc@9.4.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^util-linux-uuid@2.38.1%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-zen2
[+]      ^xz@5.2.7%gcc@9.4.0~pic build_system=autotools libs=shared,static arch=linux-ubuntu20.04-zen2
[+]      ^zlib@1.2.13%gcc@9.4.0+optimize+pic+shared build_system=makefile arch=linux-ubuntu20.04-zen2

Error message

No errors are reported during the build process itself (but see below), but the curses module is quasi-tacitly not built, such that spack load python; python -c "import curses" exits with

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/software/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-9.4.0/python-3.10.8-3a3enkvm74kxnss4l27qrs4qiq4vl2p2/lib/python3.10/curses/__init__.py", line 13, in <module>
    from _curses import *
ModuleNotFoundError: No module named '_curses'

However, inspection of the build log (starting at line 1257) reveals the following:

The necessary bits to build these optional modules were not found:
_curses               _curses_panel         _dbm               
_tkinter                                                       
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

despite lines 707 through 712 stating:

checking curses.h usability... yes
checking curses.h presence... yes
checking for curses.h... yes
checking ncurses.h usability... yes
checking ncurses.h presence... yes
checking for ncurses.h... yes

Information on your system

Spack: 0.20.0.dev0
Python: 3.8.10
Platform: linux-ubuntu20.04-zen2
Concretizer: clingo

Also tested using 0.20.0.dev0 (c3e61664cfa5e9d989f8c9b7854fd4296858a8b0).

Additional information

spack-build-env.txt
spack-build-out.txt

The regular system installation of python has no problem importing and using the curses and readline modules, of course.

As an additional check, I copied the tarball from one of the spack staging directories and successfully built python from that by specifying LDFLAGS and CPPFLAGS to point to the system installations of ncurses (and readline, because I was wondering if the latter led to conflicts, which it apparently did not in the fully manual install).

Package maintainers: @adamjstewart @pradyunsg @scheibelp @skosukhin

General information

I have run spack debug report and reported the version of Spack/Python/Platform
I have run spack maintainers <name-of-the-package> and @mentioned any maintainers
I have uploaded the build log and environment files
I have searched the issues of this repo and believe this is not a duplicate

The text was updated successfully, but these errors were encountered:

adamjstewart · 2023-01-11T17:39:26Z

No idea how to solve this, I don't encounter this same issue on my system so I don't know how to reproduce it. I feel like we've seen bug reports like this in the past, not sure if we resolved any of those.

rkevk · 2023-01-11T17:42:56Z

@adamjstewart So your spack-built python can import curses without issues? What version of spack was it built with and what version of python is it? Maybe I can circumvent it by using a different version. I find it surprising that you can't reproduce the behavior, since we've seen it on three different machines now.

adamjstewart · 2023-01-11T17:49:05Z

[+]  python@3.10.8%apple-clang@12.0.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,7d40923,f2fd060 arch=darwin-catalina-ivybridge
[+]      ^apple-libuuid@1353.100.2%apple-clang@12.0.0 build_system=bundle arch=darwin-catalina-ivybridge
[+]      ^bzip2@1.0.8%apple-clang@12.0.0~debug~pic+shared build_system=generic arch=darwin-catalina-ivybridge
[+]          ^diffutils@3.8%apple-clang@12.0.0 build_system=autotools arch=darwin-catalina-ivybridge
[+]      ^expat@2.5.0%apple-clang@12.0.0~libbsd build_system=autotools arch=darwin-catalina-ivybridge
[+]      ^gdbm@1.23%apple-clang@12.0.0 build_system=autotools arch=darwin-catalina-ivybridge
[+]      ^gettext@0.21.1%apple-clang@12.0.0+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools arch=darwin-catalina-ivybridge
[+]          ^libiconv@1.16%apple-clang@12.0.0 build_system=autotools libs=shared,static arch=darwin-catalina-ivybridge
[+]          ^libxml2@2.10.3%apple-clang@12.0.0~python build_system=autotools arch=darwin-catalina-ivybridge
[+]          ^tar@1.34%apple-clang@12.0.0 build_system=autotools zip=pigz arch=darwin-catalina-ivybridge
[+]              ^pigz@2.7%apple-clang@12.0.0 build_system=makefile arch=darwin-catalina-ivybridge
[+]              ^zstd@1.5.2%apple-clang@12.0.0+programs build_system=makefile compression=none libs=shared,static arch=darwin-catalina-ivybridge
[+]      ^libffi@3.4.2%apple-clang@12.0.0 build_system=autotools arch=darwin-catalina-ivybridge
[+]      ^libxcrypt@4.4.33%apple-clang@12.0.0~obsolete_api build_system=autotools arch=darwin-catalina-ivybridge
[+]      ^ncurses@6.3%apple-clang@12.0.0~symlinks+termlib abi=none build_system=autotools arch=darwin-catalina-ivybridge
[+]      ^openssl@1.1.1s%apple-clang@12.0.0~docs~shared build_system=generic certs=mozilla arch=darwin-catalina-ivybridge
[+]          ^ca-certificates-mozilla@2022-10-11%apple-clang@12.0.0 build_system=generic arch=darwin-catalina-ivybridge
[+]          ^perl@5.36.0%apple-clang@12.0.0+cpanm+shared+threads build_system=generic arch=darwin-catalina-ivybridge
[+]              ^berkeley-db@18.1.40%apple-clang@12.0.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=darwin-catalina-ivybridge
[+]      ^pkgconf@1.8.0%apple-clang@12.0.0 build_system=autotools arch=darwin-catalina-ivybridge
[+]      ^readline@8.2%apple-clang@12.0.0 build_system=autotools patches=bbf97f1 arch=darwin-catalina-ivybridge
[+]      ^sqlite@3.40.0%apple-clang@12.0.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=darwin-catalina-ivybridge
[+]      ^xz@5.2.7%apple-clang@12.0.0~pic build_system=autotools libs=shared,static arch=darwin-catalina-ivybridge
[+]      ^zlib@1.2.13%apple-clang@12.0.0+optimize+pic+shared build_system=makefile arch=darwin-catalina-ivybridge

Spack: 0.20.0.dev0 (6a44a14)
Python: 3.10.8
Platform: darwin-catalina-ivybridge
Concretizer: clingo

rkevk · 2023-01-13T12:24:25Z

Thanks—I'm grasping at straws here, but maybe Mac vs. Linux/Ubuntu makes a difference here? Does one of the devs have access to a Linux machine to test if the behavior is reproducible?

haampie · 2023-01-13T12:30:19Z

I can reproduce

Let me tell you this: Python's build system is an absolute atrocity.

@adamjstewart do you know why we use setup.py and how it ends up in the source tarball in the first place? That file is riddled with assumptions that don't hold for Spack. If I just clone Python and run configure/make/make install I don't have that setup.py.

haampie · 2023-01-13T13:45:04Z

Woah! setup.py is removed in cpython. They've come to senses!

haampie · 2023-01-13T13:52:06Z

Bisected to python/cpython@81dca70

I think we shouldn't fix this issue here.

If you follow the horrible setup.py logic you'll find that the curses library is added if it's not dynamically linked to readline, which Spack does.

rkevk · 2023-01-13T15:29:47Z

@haampie Thanks for looking into this. So the answer is that in python 3.12 the spack install should work as expected? Do you have a suggested workaround for the meantime until 3.12 is released? I'm not sure everyone would be happy running production code on alpha versions of python... Is linking an external, working python installation to spack a sensible workaround?

adamjstewart · 2023-01-13T16:22:48Z

If you follow the horrible setup.py logic you'll find that the curses library is added if it's not dynamically linked to readline, which Spack does.

Could we add a dependency on static readline?

adamjstewart · 2023-01-13T17:39:44Z

I agree that waiting for a 3.12 release isn't a sufficient fix.

pradyunsg · 2023-01-13T20:35:58Z

3.12 removes distutils from the Python standard library (https://peps.python.org/pep-0632/) and setup.py was removed (in part) for that.

Does Spack have the ability to patch CPython's source code before building it? If so, I'd suggest patching to add a if self.missing or self.failed or self.failed_on_import: sys.exit(1) at the end of https://github.com/python/cpython/blob/3.9/setup.py#L473 to avoid missing optional dependencies and such.

https://github.com/python/cpython/blob/3.9/setup.py#L972 is the logic for locating ncurses. The logic ought to be hitting https://github.com/python/cpython/blob/3.9/setup.py#L1107 for this to be missing so it's not as simple as "readline was present". Do we have ncurses available in the build environment? :)

adamjstewart · 2023-01-13T22:50:07Z

Does Spack have the ability to patch CPython's source code before building it?

Yes, we can patch statically (apply a *.patch file) or dynamically (sed-style search and replace in a patch function).

Do we have ncurses available in the build environment? :)

Yes, Spack's Python package depends on both readline and ncurses when the readline module is requested (by default).

I'll try adding this patch.

adamjstewart · 2023-01-13T22:56:21Z

How is this different from setting PYTHONSTRICTEXTENSIONBUILD? Many of these modules are optional, so we only add the needed dependencies when requested. For example, on macOS, I see the following in the build log:

The necessary bits to build these optional modules were not found:
_dbm                  _tkinter              ossaudiodev
spwd
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  pwd                   time

We don't want to crash just because tkinter isn't being built.

pradyunsg · 2023-01-14T12:59:02Z

/me nods.

PYTHONSTRICTEXTENSIONBUILD's checks happen after the build, but I guess it achieves the same goals as the patch I suggested. :)

rkevk · 2023-01-17T13:18:04Z

If I'm following the discussion correctly, then the patch idea is shelved again (because it would just let the build fail without solving the problem?) and the only solution on the table for the time being is to

add a dependency on static readline
?

haampie · 2023-01-17T16:15:15Z

I closed the issue because https://github.com/python/cpython/blob/5e52778b1a2a86fe89578b957b1898d165a350fe/setup.py#L1107 we hit this branch, so it's expected that the _curses module is missing.

rkevk · 2023-01-17T16:24:02Z

I closed the issue because https://github.com/python/cpython/blob/5e52778b1a2a86fe89578b957b1898d165a350fe/setup.py#L1107 we hit this branch, so it's expected that the _curses module is missing.

Which is ultimately due to python/cpython#51633. I haven't had the time yet to comb through the issue and see if there's relevant information for our problem.

haampie · 2023-01-17T16:30:37Z

If I had to guess it's ncurses providing two ABI-incompatible libraries; same set of symbols, but one expects char, the other wchar_t. When using shared linking, if readline links the one, and Python the other, the dynamic linker ultimately uses the symbols of the library that loaded first. So either Python breaks or libreadline, depending on what module is imported first (which ultimately results in a dlopen call). So they guard against it. If you want to fix it, you have to be able to guarantee that the same libcurses is used in both readline an python, or libcurses should be linked statically to python's extension or libreadline or both.

Our ncurses package looks questionable, cause it builds both interfaces.

Note that even if we fix our curses/readline packages, things could still break when using external readline or ncurses.

haampie · 2023-01-18T14:07:15Z

I did a little more digging. So, libtinfo is supposed to be wchar_t / char agnostic, but somehow there still is a libtinfow with w-suffix 😕 maybe that's just because the build system doesn't support building wide / normal in a single build, and always adds suffixes so things can be packaged separately without clashes.

Extra confusion, libtinfo.a and libtinfow.a don't provide the same set of symbols, for example

$ diff <(nm normal/entries.o) <(nm wide/entries.o)
7c7
<                  U _nc_free_termtype
---
>                  U _nc_free_termtype2

which was reported in this thread, where they also talk about Python troubles with curses: https://lists.gnu.org/archive/html/bug-ncurses/2018-02/msg00045.html.

In any case, they promote using one libtinfo to solve the wchar_t / char struggles: https://lists.gnu.org/archive/html/bug-ncurses/2018-02/msg00052.html.

Again, it doesn't really help us if we support both external and spack curses / readline 😩

rkevk · 2023-01-23T13:44:54Z

@haampie I took a look at setup.py and I have a creeping suspicion that the reason for the drama might be a whole lot more stupid than we currently believe:
Starting in https://github.com/python/cpython/blob/v3.10.8/setup.py#L1089 we have (where tmpfile is the result of running ldd /path/to/libreadline.so* > tmpfile)

                with open(tmpfile) as fp:
                    for ln in fp:
                        if 'curses' in ln:
                            readline_termcap_library = re.sub(
                                r'.*lib(n?cursesw?)\.so.*', r'\1', ln
                            ).rstrip()
                            break
                        # termcap interface split out from ncurses
                        if 'tinfo' in ln:
                            readline_termcap_library = 'tinfo'
                            break

Notice the line if 'curses' in ln:. What this is supposed to do, as far as I can tell, is check if libreadline is directly linked to libncurses.so* or one of its relatives. If you check the output of ldd on the spack-installed readline, it is not linked to any variant of lib*curses, but it is linked to libtinfo which is contained in a folder named /path/to/spack/ncurses-version-hash/lib, which does contain the string ncurses, so the setup.py logic gives a false positive.

Have you tried fudging with these setup.py lines to see if that fixes the issue?

rkevk · 2023-01-23T14:56:55Z

Indeed, adding the following brief patch to package.py seems to solve the issue for me:

diff --git a/setup.py b/setup.py
index 85a2b26357..8c83b9f175 100644
--- a/setup.py
+++ b/setup.py
@@ -1088,7 +1088,7 @@ def detect_readline_curses(self):
             if ret == 0:
                 with open(tmpfile) as fp:
                     for ln in fp:
-                        if 'curses' in ln:
+                        if 'libcurses' in ln or 'libncurses' in ln:
                             readline_termcap_library = re.sub(
                                 r'.*lib(n?cursesw?)\.so.*', r'\1', ln
                             ).rstrip()

@haampie Can you double-check?

Edit: See PR in #35092.

adamjstewart · 2023-01-24T22:21:08Z

Can anyone else test #35092?

rkevk added the build-error label Jan 10, 2023

haampie closed this as completed Jan 13, 2023

adamjstewart self-assigned this Jan 13, 2023

adamjstewart reopened this Jan 13, 2023

rkevk linked a pull request Jan 23, 2023 that will close this issue

Fix curses detection when building python #35092

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Installation issue: python is built without curses module #34872

Installation issue: python is built without curses module #34872

rkevk commented Jan 10, 2023

adamjstewart commented Jan 11, 2023

rkevk commented Jan 11, 2023

adamjstewart commented Jan 11, 2023

rkevk commented Jan 13, 2023

haampie commented Jan 13, 2023 •

edited

haampie commented Jan 13, 2023

haampie commented Jan 13, 2023

rkevk commented Jan 13, 2023

adamjstewart commented Jan 13, 2023

adamjstewart commented Jan 13, 2023

pradyunsg commented Jan 13, 2023 •

edited

adamjstewart commented Jan 13, 2023

adamjstewart commented Jan 13, 2023

pradyunsg commented Jan 14, 2023

rkevk commented Jan 17, 2023

haampie commented Jan 17, 2023 •

edited

rkevk commented Jan 17, 2023

haampie commented Jan 17, 2023 •

edited

haampie commented Jan 18, 2023 •

edited

rkevk commented Jan 23, 2023 •

edited

rkevk commented Jan 23, 2023 •

edited

adamjstewart commented Jan 24, 2023

Installation issue: python is built without curses module #34872

Installation issue: python is built without curses module #34872

Comments

rkevk commented Jan 10, 2023

Steps to reproduce the issue

Error message

Information on your system

Additional information

General information

adamjstewart commented Jan 11, 2023

rkevk commented Jan 11, 2023

adamjstewart commented Jan 11, 2023

rkevk commented Jan 13, 2023

haampie commented Jan 13, 2023 • edited

haampie commented Jan 13, 2023

haampie commented Jan 13, 2023

rkevk commented Jan 13, 2023

adamjstewart commented Jan 13, 2023

adamjstewart commented Jan 13, 2023

pradyunsg commented Jan 13, 2023 • edited

adamjstewart commented Jan 13, 2023

adamjstewart commented Jan 13, 2023

pradyunsg commented Jan 14, 2023

rkevk commented Jan 17, 2023

haampie commented Jan 17, 2023 • edited

rkevk commented Jan 17, 2023

haampie commented Jan 17, 2023 • edited

haampie commented Jan 18, 2023 • edited

rkevk commented Jan 23, 2023 • edited

rkevk commented Jan 23, 2023 • edited

adamjstewart commented Jan 24, 2023

haampie commented Jan 13, 2023 •

edited

pradyunsg commented Jan 13, 2023 •

edited

haampie commented Jan 17, 2023 •

edited

haampie commented Jan 17, 2023 •

edited

haampie commented Jan 18, 2023 •

edited

rkevk commented Jan 23, 2023 •

edited

rkevk commented Jan 23, 2023 •

edited