Troubles with Python and ncurses on Cygwin #15317

jpflori · 2013-10-23T06:50:01Z

Python currently fails/is suboptimal on Cygwin because:

it does not properly detect readline which is only installed as a shared library, patch "2.7.3-dylib.patch" from Cygwin package fixes that;
the curses module is not built because of undefined refs, just passing -lcurses when linking is not enough, one should add -ltinfo; in fact the real problem is that setup.py tries to run ldd on the import library (dll.a) which fails to detect what libraries are already linked and anyway would decide to discard -ltinfo as its already linked to readline, but on Cygwin you have to explicitely pass everything...
it segfault at startup when loading the readline module, not sure why. It looks exactly as what is reported at http://trac.macports.org/ticket/29496 . Rebuilding ncurses with debug info (including CFLAGS="-O0 -g" which cannot easily be passed right now) does not give much info. The offending line pointed by GCC is "char *result = exit_attribute_mode;"; maybe some dark magic going on as reported in http://lists.gnu.org/archive/html/bug-ncurses/2006-10/msg00002.html ; please not that on Cygwin the stack is small by default but playing a little bit with that did not really help. Further info: ncurses 5.7 is fine, ncurses 5.8 fails in the same way. Upstream bu report: http://lists.gnu.org/archive/html/bug-ncurses/2013-10/msg00014.html. Should be fixed with http://lists.gnu.org/archive/html/bug-ncurses/2013-10/msg00017.html

This ticket fixes the Python part.
The ncurses part is #15617

Upstream: Workaround found; Bug reported upstream.

CC: @dimpase @vbraun @jdemeyer

Component: porting: Cygwin

Keywords: cygwin spkg ncurses

Author: Jean-Pierre Flori

Branch/Commit: c0a09c5

Reviewer: Travis Scrimshaw

Issue created by migration from https://trac.sagemath.org/ticket/15317

The text was updated successfully, but these errors were encountered:

jpflori · 2013-10-23T06:50:46Z

comment:1

Had no time to investigate the diff between 5.7 and 5.8, especially around the offending lines, function calls.
Any help/clues welcomed.

jpflori · 2013-10-24T07:40:25Z

comment:2

Similar problem on different archs: http://lists.busybox.net/pipermail/buildroot/2011-April/042350.html
No solution though...

jpflori · 2013-10-24T07:45:49Z

comment:3

Maybe have a look at #12282

jpflori · 2013-10-24T11:27:43Z

comment:4

Playing a little bit around, it seems that setting TERM to something non-existing (e.g. "" or "blblbblblbl") makes the segfault disappear.

jpflori · 2013-10-25T06:58:12Z

Upstream: Workaround found; Bug reported upstream.

jpflori · 2013-10-25T06:58:12Z

comment:5

Upstream bug report: http://lists.gnu.org/archive/html/bug-ncurses/2013-10/msg00014.html

The problem is that when the config BROKEN_LINKER is set, then a pointer CurTerm is used but never set to something else than zero.
Setting it as in other cases seems to fix the problem.
Or unsetting BROKEN_LINKER on Cygwin.

jpflori · 2013-10-28T18:02:55Z

comment:6

The BROKEN_LINKER stuff should be fixed in ncurses:

http://lists.gnu.org/archive/html/bug-ncurses/2013-10/msg00017.html

jpflori · 2013-10-28T18:03:11Z

Changed upstream from Workaround found; Bug reported upstream. to Fixed upstream, but not in a stable release.

jpflori · 2013-11-04T10:21:05Z

comment:8

There were further problem with the threading code in the Cygwin dll itself.
Fixed in the > 1.7.25 releases (currently only snpashots). See http://cygwin.com/ml/cygwin/2013-10/msg00371.html.

jpflori · 2013-11-07T10:17:19Z

comment:9

As far as not linking to tinfo is concerned, I'm putting this here for reference: http://bugs.python.org/issue7384.

In particular, python uses ldd for linking detection but applies it on import libraries on cygwin which just fails.
(The situation is more or less the same on OS X but OS X seems less picky about providing explicitely all libraries to be linked in.)

jpflori · 2013-12-27T13:26:19Z

comment:10

Here is how to find the dll corresponding to an import lib:

http://cygwin.com/ml/cygwin/2013-11/msg00167.html$
(dlltool --identify [implib])

dimpase · 2013-12-28T04:57:32Z

comment:11

Replying to @jpflori:

Here is how to find the dll corresponding to an import lib:

http://cygwin.com/ml/cygwin/2013-11/msg00167.html
(dlltool --identify [implib])

Fixed typo in the url.

jpflori · 2014-01-01T20:32:20Z

comment:12

I've opened #15617 for the ncurses update.

jpflori · 2014-01-01T22:06:37Z

comment:13

I've put a dirty but working python spkg at:

http://boxen.math.washington.edu/home/jpflori/cygwin/python-2.7.5.p2.spkg
for those wanting to try a Cygwin(32) build.

tscrim · 2014-01-04T06:09:58Z

comment:14

I installed the new version of ncurses and created a new branch with the python spkg at u/tscrim/cygwin_python-15317 (with #15617 merged in). Now I get somewhere else than I did before (with cygwin32) with it failing at importing the module _socket...

jpflori · 2014-01-04T12:59:42Z

comment:15

Replying to @tscrim:

I installed the new version of ncurses and created a new branch with the python spkg at u/tscrim/cygwin_python-15317 (with #15617 merged in). Now I get somewhere else than I did before (with cygwin32) with it failing at importing the module _socket...

What is in that branch exactly?
From the git log I assume you just updated the tarball with what's within the spkg?
Note that's only for testing purpose and a clean patch should be produced :)
FYI, I think we temporarly could just diff what's in the tarball with the vanilla tarball and ship it as a patch in Sage, but that's definitely not for upstream inclusion.
Right know it basically adds "-ltinfo" all the time which is right for Sage as we build our own ncurses which makes tinfo a separate lib.

The _socket thing reminds me of rebasing issues.
Try to run "find . -name "*.dll" > dlls && rebase -O -T dlls && make" in Sage's root.
And I did not test on Cygwin64 lately, so that might be another problem.

tscrim · 2014-01-05T04:55:51Z

comment:16

Tried that and it didn't work. I also pushed a patch to the branch.

jpflori · 2014-01-05T21:53:34Z

Author: Jean-Pierre Flori

jpflori · 2014-01-05T21:53:34Z

comment:17

The branch I pushed is cleaner and should be good to go.

New commits:

`7916640`	`Make sure tinfo is correctly linked in when needed on Cygwin.`
`9acea9b`	`Actually add the NTL patch.`

jpflori · 2014-01-05T21:53:34Z

Changed upstream from Fixed upstream, but not in a stable release. to Workaround found; Bug reported upstream.

jpflori · 2014-01-05T21:53:34Z

Commit: 7916640

jpflori · 2014-01-05T21:53:34Z

Branch: u/jpflori/ticket/15317

jpflori · 2014-01-08T09:53:41Z

comment:26

Replying to @tscrim:

FTR It doesn't work for me on Cygwin64. Here's the log.

It seems the readline module is not built:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named readline

and indeed readline is not found:

checking how to link readline libs... none
checking for rl_callback_handler_install in -lreadline... no
checking for rl_pre_input_hook in -lreadline... no
checking for rl_completion_display_matches_hook in -lreadline... no
checking for rl_completion_matches in -lreadline... no

Could you post your readline log?

Please let's go one step at a time, at least now installing python on Cygwin32 works, I'll try to go with the pillow and installing python on Cygwin64 issues on follow up tickets quickly.

jpflori · 2014-01-08T09:55:12Z

comment:27

Replying to @jpflori:

Replying to @tscrim:

FTR It doesn't work for me on Cygwin64. Here's the log.

It seems the readline module is not built:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named readline
and indeed readline is not found:
checking how to link readline libs... none
checking for rl_callback_handler_install in -lreadline... no
checking for rl_pre_input_hook in -lreadline... no
checking for rl_completion_display_matches_hook in -lreadline... no
checking for rl_completion_matches in -lreadline... no
Could you post your readline log?Atickets quickly.

Also note this failure surel has nothing to do with the changes here as readline is not detected during the autotools based configuration phase, not the setup.py one which I modified.

jpflori · 2014-01-08T14:42:13Z

comment:28

Strangely on my setup, readline seems to have built fine, is detected, but the python fails because of an ICE in GCC.

jpflori · 2014-01-08T14:43:55Z

comment:29

(I mean my Cygwin64 setup, with Cygwin's GCC 4.8.2.)

jpflori · 2014-01-08T16:56:58Z

comment:30

After fixing my ICE on Cygwin64 (using #10572), I oculd build pytohn.

tscrim · 2014-02-02T05:51:29Z

comment:32

Sorry for the delays, let's get this in.

tscrim · 2014-02-02T05:51:29Z

Reviewer: Travis Scrimshaw

vbraun · 2014-02-02T23:01:03Z

comment:33

Breaks python build on mod:

http://build.sagemath.org/sage/builders/%20%20fast%20UW%20mod%20%28Ubuntu%208.04%20x86_64%29%20incremental/builds/25/steps/compile/logs/python

jpflori · 2014-02-03T09:43:12Z

comment:34

Any chance to get the getbuildinfo.c file?
Doesn't look obvously related, so needs further info.

vbraun · 2014-02-03T13:47:10Z

comment:35

Sorry, the buildbot has already moved on. You'll have to build it yourself.

jpflori · 2014-02-03T15:46:06Z

comment:36

I just did so ... and the build succeeded.

I merged u/jpflori/ticket/15317 with develop, exported MAKE to something sensible and ran make.
Does the buildbot export anything strange before building?

jpflori · 2014-02-03T15:47:28Z

comment:37

Hum, here is waht the pytho log says:

gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall   -I. -IIn
clude -I./Include  -fPIC -DPy_BUILD_CORE -o Python/graminit.o Python/graminit.c
gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall   -I. -IIn
clude -I./Include  -fPIC -DPy_BUILD_CORE \
              -DSVNVERSION="\"`LC_ALL=C svnversion .`\"" \
              -DHGVERSION="\"`LC_ALL=C hg id -i .`\"" \
              -DHGTAG="\"`LC_ALL=C hg id -t .`\"" \
              -DHGBRANCH="\"`LC_ALL=C hg id -b .`\"" \
              -o Modules/getbuildinfo.o ./Modules/getbuildinfo.c
************************************************************************
It seems that you are attempting to run Sage from an unpacked source
tarball, but you have not compiled it yet (or maybe the build has not
finished). You should run `make` in the Sage root directory first.
If you did not intend to build Sage from source, you should download
a binary tarball instead. Read README.txt for more information.
************************************************************************
************************************************************************
It seems that you are attempting to run Sage from an unpacked source
tarball, but you have not compiled it yet (or maybe the build has not
finished). You should run `make` in the Sage root directory first.
If you did not intend to build Sage from source, you should download
a binary tarball instead. Read README.txt for more information.
************************************************************************
************************************************************************
It seems that you are attempting to run Sage from an unpacked source
tarball, but you have not compiled it yet (or maybe the build has not
finished). You should run `make` in the Sage root directory first.
If you did not intend to build Sage from source, you should download
a binary tarball instead. Read README.txt for more information.
************************************************************************

jpflori · 2014-02-03T15:50:10Z

comment:38

Trying to reinstall python then fails:

gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall   -I. -IInclude -I./Include  -fPIC -DPy_BUILD_CORE \
              -DSVNVERSION="\"`LC_ALL=C svnversion .`\"" \
              -DHGVERSION="\"`LC_ALL=C hg id -i .`\"" \
              -DHGTAG="\"`LC_ALL=C hg id -t .`\"" \
              -DHGBRANCH="\"`LC_ALL=C hg id -b .`\"" \
              -o Modules/getbuildinfo.o ./Modules/getbuildinfo.c
<command-line>:0:11: warning: missing terminating " character [enabled by default]
<command-line>:1:7: warning: missing terminating " character [enabled by default]
<command-line>:2:10: warning: missing terminating " character [enabled by default]
./Modules/getbuildinfo.c: In function 'Py_GetBuildInfo':
./Modules/getbuildinfo.c:45:5: error: missing terminating " character
./Modules/getbuildinfo.c:45:48: error: expected expression before ')' token
./Modules/getbuildinfo.c:46:27: error: missing terminating " character
./Modules/getbuildinfo.c:46:27: error: missing terminating " character
./Modules/getbuildinfo.c:47:28: error: missing terminating " character
./Modules/getbuildinfo.c:47:28: error: missing terminating " character
./Modules/getbuildinfo.c:53:19: error: 'buildinfo' undeclared (first use in this function)
./Modules/getbuildinfo.c:53:19: note: each undeclared identifier is reported only once for each function it appears in
./Modules/getbuildinfo.c: In function '_Py_hgversion':
./Modules/getbuildinfo.c:72:5: error: missing terminating " character
./Modules/getbuildinfo.c:72:5: warning: 'return' with no value, in function returning non-void [-Wreturn-type]
./Modules/getbuildinfo.c: In function '_Py_hgidentifier':
./Modules/getbuildinfo.c:79:5: error: missing terminating " character
./Modules/getbuildinfo.c:79:18: error: expected expression before ';' token
./Modules/getbuildinfo.c:83:9: error: missing terminating " character
./Modules/getbuildinfo.c:83:24: error: expected expression before ';' token
./Modules/getbuildinfo.c: In function 'Py_GetBuildInfo':
./Modules/getbuildinfo.c:57:1: warning: control reaches end of non-void function [-Wreturn-type]
make: *** [Modules/getbuildinfo.o] Error 1

jpflori · 2014-02-03T15:54:42Z

comment:39

/usr/local/bin/hg is broken on mod:

#!/bin/sh
sage -hg $*

whence all the above.

In particular, once sage(.git) is built, one get:

(sage-sh) jpflori@mod:src$ which hg
/usr/local/bin/hg
(sage-sh) jpflori@mod:src$ LC_ALL=C hg id -i .
sage-run received unknown option: -hg 
usage: sage [options]
Try 'sage -h' for more information.

jdemeyer · 2014-02-03T15:56:08Z

comment:40

This ticket is not broken, the hg command on mod is broken.

vbraun · 2014-02-03T16:18:07Z

comment:41

How about

./configure HAS_HG=no SVNVERSION=no [...]

jpflori · 2014-02-05T10:20:18Z

comment:42

Replying to @vbraun:

How about

./configure HAS_HG=no SVNVERSION=no [...]

Why not, but let's do this somewhere else.
And maybe only if SAGE_FAT_BINARY is set or something like that.
We cannot deal with every broken systems users may come up with.

vbraun · 2014-02-05T10:50:41Z

comment:43

Fair enough, but this ticket can't be merged until either mod is un-broken or a workaround is implemented. Or both.

jpflori · 2014-02-05T10:52:20Z

comment:44

Maybe William can fix mod (i.e. delete the harmful /usr/local/bin/hg)?

jpflori · 2014-02-05T10:53:35Z

comment:45

(And isn't the failure triggered even without this ticket? I mean: just trying to recompile python should fail.)

vbraun · 2014-02-05T10:53:36Z

comment:46

Can you post to the sage.math users mailing list, this is for admin issues.

jpflori · 2014-02-05T11:00:07Z

comment:47

Done: https://groups.google.com/d/msg/sagemath-users/uMHB5gtlJS8/bKPtZqI-tc0J

vbraun · 2014-02-14T19:44:52Z

Changed branch from u/jpflori/ticket/15317 to c0a09c5

jpflori added this to the sage-6.1 milestone Oct 23, 2013

jpflori added c: porting: cygwin labels Oct 23, 2013

This comment has been minimized.

Sign in to view

sagetrac-vbraun-spam mannequin modified the milestones: sage-6.1, sage-6.2 Jan 30, 2014

tscrim added s: positive review and removed s: needs review labels Feb 2, 2014

vbraun removed the s: positive review label Feb 14, 2014

vbraun closed this as completed in 995ae43 Feb 14, 2014

jpflori mentioned this issue Feb 7, 2014

Update ncurses to dev version 5.9-20131221 #15617

Closed

Troubles with Python and ncurses on Cygwin #15317

Troubles with Python and ncurses on Cygwin #15317

Comments

jpflori commented Oct 23, 2013

jpflori commented Oct 23, 2013

jpflori commented Oct 24, 2013

jpflori commented Oct 24, 2013

jpflori commented Oct 24, 2013

jpflori commented Oct 25, 2013

jpflori commented Oct 25, 2013

This comment has been minimized.

This comment has been minimized.

jpflori commented Oct 28, 2013

jpflori commented Oct 28, 2013

jpflori commented Nov 4, 2013

jpflori commented Nov 7, 2013

jpflori commented Dec 27, 2013

dimpase commented Dec 28, 2013

jpflori commented Jan 1, 2014

jpflori commented Jan 1, 2014

tscrim commented Jan 4, 2014

jpflori commented Jan 4, 2014

tscrim commented Jan 5, 2014

This comment has been minimized.

jpflori commented Jan 5, 2014

jpflori commented Jan 5, 2014

jpflori commented Jan 5, 2014

jpflori commented Jan 5, 2014

jpflori commented Jan 5, 2014

jpflori commented Jan 8, 2014

jpflori commented Jan 8, 2014

jpflori commented Jan 8, 2014

jpflori commented Jan 8, 2014

jpflori commented Jan 8, 2014

tscrim commented Feb 2, 2014

tscrim commented Feb 2, 2014

vbraun commented Feb 2, 2014

jpflori commented Feb 3, 2014

vbraun commented Feb 3, 2014

jpflori commented Feb 3, 2014

jpflori commented Feb 3, 2014

jpflori commented Feb 3, 2014

jpflori commented Feb 3, 2014

jdemeyer commented Feb 3, 2014

vbraun commented Feb 3, 2014

jpflori commented Feb 5, 2014

vbraun commented Feb 5, 2014

jpflori commented Feb 5, 2014

jpflori commented Feb 5, 2014

vbraun commented Feb 5, 2014

jpflori commented Feb 5, 2014

vbraun commented Feb 14, 2014