Fix build failures of pynac on macos with system python3 #29404

mkoeppe · 2020-03-25T02:58:16Z

https://github.com/mkoeppe/sage/runs/531095486

pynac:

checking whether sage-python23 version is >= 2.7... yes
checking for sage-python23 version... 3.7
checking for sage-python23 platform... darwin
checking for sage-python23 script directory... ${prefix}/lib/python3.7/site-packages
checking for sage-python23 extension module directory... ${exec_prefix}/lib/python3.7/site-packages
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... none
checking for python3.7... no
configure: error: Cannot find python3.7 in your system path

Fix in pynac/pynac#351, https://github.com/mkoeppe/pynac/releases/tag/pynac-0.7.26.sage-2020-03-28

The "temporary upstream" tarball URL is in checksums.ini. To test the update on this branch: Use make SAGE_SPKG="sage-spkg -o" pynac-clean pynac; this will download the new upstream package automatically.

Upstream: Completely fixed; Fix reported upstream

CC: @dimpase @kiwifb @videlec @saraedum @isuruf @embray

Component: packages: standard

Author: Matthias Koeppe

Branch: 7344e2d

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/29404

The text was updated successfully, but these errors were encountered:

mkoeppe · 2020-03-25T03:27:46Z

comment:1

pynac's python discovery should be fixed, this looks completely broken

dimpase · 2020-03-25T03:48:50Z

comment:3

This should be probably split into two tickets.

pplpy upstream is https://gitlab.com/videlec/pplpy - also in CC:

kiwifb · 2020-03-25T03:51:03Z

comment:4

Yes for the split. I am surprised that problem shows now in pynac. I would have expected it to happen much earlier. The AX_PYTHON_DEVEL macro is 5 years old - which probably doesn't help.

kiwifb · 2020-03-25T03:54:16Z

comment:5

Note that I don't think the problem is in the macro, but something in there may trigger. I suspect the problem is in AM_PATH_PYTHON and that PYTHON needs to be defined when calling configure in spkg-install.

kiwifb · 2020-03-25T03:58:17Z

comment:6

Which is already done. So I am not sure what happens to override it. Access to config.log would be helpful.

mkoeppe · 2020-03-25T04:02:01Z

comment:7

Yes, please go ahead with splitting the ticket.

dimpase · 2020-03-25T06:22:53Z

comment:8

It looks from the log as here could be some mixing up happening between system python3 and Homebrew's python3 (unless Homebrew's python3 is using system's python3 headers by design).

kiwifb · 2020-03-25T06:44:52Z

comment:9

Replying to @dimpase:

It looks from the log as here could be some mixing up happening between system python3 and Homebrew's python3 (unless Homebrew's python3 is using system's python3 headers by design).

Shouldn't happen. We pass PYTHON=sage-python23 explicitly to configure. That's what should be tested.
https://github.com/sagemath/sage-prod/blob/develop/build/pkgs/pynac/spkg-install.in#L14

My gut feeling is that something funny happened to that line during processing. But I need config.log to establish it.

dimpase · 2020-03-25T07:05:48Z

comment:10

I don't have MacOS machine, but anyone with it may install Homebrew and try.

kiwifb · 2020-03-25T07:23:57Z

comment:11

It is more subtle than my implied gut feeling. The macro AX_PYTHON_DEVEL gets the right value for PYTHON and does its stuff based on that. I think AM_PATH_PYTHON doesn't like python with exotic names because it looks like it goes through a pre-defined list.

dimpase · 2020-03-25T07:27:41Z

comment:12

isn't the location of Sage's Python exotic enough to choke it, if it were the case?

kiwifb · 2020-03-25T07:43:41Z

comment:13

It would be :)

Code inspection says I have it backwards. AM_PATH_PYTHON works and AX_PYTHON_DEVEL is the one failing.

kiwifb · 2020-03-25T08:04:22Z

comment:14

Toxic. This is with an old sage I have around

[pynac-0.7.14] checking whether sage-python23 version is >= 2.7... yes
[pynac-0.7.14] checking for sage-python23 version... 2.7
[pynac-0.7.14] checking for sage-python23 platform... linux2
[pynac-0.7.14] checking for sage-python23 script directory... ${prefix}/lib/python2.7/site-packages
[pynac-0.7.14] checking for sage-python23 extension module directory... ${exec_prefix}/lib/python2.7/site-packages
[pynac-0.7.14] checking for style of include used by make... GNU
[pynac-0.7.14] checking for gcc... gcc
[pynac-0.7.14] checking whether the C compiler works... yes
[pynac-0.7.14] checking for C compiler default output file name... a.out
[pynac-0.7.14] checking for suffix of executables... 
[pynac-0.7.14] checking whether we are cross compiling... no
[pynac-0.7.14] checking for suffix of object files... o
[pynac-0.7.14] checking whether we are using the GNU C compiler... yes
[pynac-0.7.14] checking whether gcc accepts -g... yes
[pynac-0.7.14] checking for gcc option to accept ISO C89... none needed
[pynac-0.7.14] checking whether gcc understands -c and -o together... yes
[pynac-0.7.14] checking dependency style of gcc... gcc3
[pynac-0.7.14] checking for python2.7... /home/fbissey/sagemath/sage/local/bin/python2.7
[pynac-0.7.14] checking for a version of Python >= '2.1.0'... yes
[pynac-0.7.14] checking for the distutils Python package... yes
[pynac-0.7.14] checking for Python include path... -I/home/fbissey/sagemath/sage/local/include/python2.7
[pynac-0.7.14] checking for Python library path... -L/home/fbissey/sagemath/sage/local/lib -lpython2.7
[pynac-0.7.14] checking for Python site-packages path... /home/fbissey/sagemath/sage/local/lib/python2.7/site-packages

As you can see, the first test (AM_PATH_PYTHON) use sage-python23 properly. But not the second that goes directly for pythonX.Y. Worse, that's by design of the macro.

	AC_PATH_PROG([PYTHON],[python[$PYTHON_VERSION]])
	if test -z "$PYTHON"; then
	   AC_MSG_ERROR([Cannot find python$PYTHON_VERSION in your system path])
	   PYTHON_VERSION=""
	fi

And for the cherry on the top of the cake, I believe PYTHON_VERSION is set by AM_PATH_PYTHON. Conclusion AX_PYTHON_DEVEL doesn't do exotic python name, and you better have a regular python executable in the PATH. Now, I'd like to know more about that setup that has sage-python23 in the PATH but not the matching pythonX.Y. Did something happen while I wasn't looking?

kiwifb · 2020-03-25T08:07:44Z

comment:15

Updating AX_PYTHON_DEVEL would still give the same result. There is probably something smarter to do.

kiwifb · 2020-03-25T09:06:27Z

comment:16

All the stuff found by AX_PYTHON_DEVEL could be found by snipping bits of it and using the results from AM_PATH_PYTHON. This is a substantial rewrite of upstream's configure.ac that we are looking at. We only need to lift the bits that figure PYTHON_CPPFLAGS and PYTHON_LDFLAGS (PYTHON_LIBS if lifted from the latest version of AX_PYTHON_DEVEL). The rest is provided by AM_PATH_PYTHON.

mkoeppe · 2020-03-25T11:53:42Z

comment:17

Replying to @kiwifb:

Yes for the split. I am surprised that problem shows now in pynac. I would have expected it to happen much earlier. The AX_PYTHON_DEVEL macro is 5 years old - which probably doesn't help.

It shows now because this is a test with system python3 (#27824) instead of building sage's own python3. On macOS, there is /usr/bin/python3 (which is Python 3.7) but no /usr/bin/python3.7.

dimpase · 2020-03-25T14:58:53Z

comment:18

would it work with Homebrew's python3 ?
https://formulae.brew.sh/formula/python

Python shipped with MacOS is dodgy, IMHO.

mkoeppe · 2020-03-25T17:29:01Z

comment:19

Replying to @kiwifb:

All the stuff found by AX_PYTHON_DEVEL could be found by snipping bits of it and using the results from AM_PATH_PYTHON. This is a substantial rewrite of upstream's configure.ac that we are looking at. We only need to lift the bits that figure PYTHON_CPPFLAGS and PYTHON_LDFLAGS (PYTHON_LIBS if lifted from the latest version of AX_PYTHON_DEVEL). The rest is provided by AM_PATH_PYTHON.

Yes, this sounds like the correct approach.

mkoeppe · 2020-03-26T02:12:21Z

comment:21

Replying to @dimpase:

This should be probably split into two tickets.

pplpy upstream is https://gitlab.com/videlec/pplpy - also in CC:

pplpy is now #29408

mkoeppe · 2020-03-26T02:15:52Z

comment:22

Some links for AX_PYTHON_DEVEL:

current version https://github.com/autoconf-archive/autoconf-archive/blob/master/m4/ax_python_devel.m4
some patch for it https://0xacab.org/schleuder/rpm-gpgme/blob/epel7-schleuder/0001-fix-stupid-ax_python_devel.patch
replacement using python-config https://trac.macports.org/attachment/ticket/39223/python.m4

dimpase · 2020-03-26T02:17:31Z

comment:23

probably we should try replacement using python-config- except that the name should not start with AC_...

mkoeppe · 2020-03-26T02:53:51Z

comment:24

... and except that we do not have sage-python23-config

kiwifb · 2020-03-26T03:10:33Z

comment:25

In the same bag, you could be tempted to use python-X.Y.pc, AM_PATH_PYTHON identifies the python version correctly. I eliminated it from my solutions yesterday but that may be viable if it is shipped consistently. If not, the AX_PYTHON_DEVEL macro just runs python commands to get at sysconfig. It is a fairly valid strategy.

sagetrac-git · 2020-04-03T19:49:11Z

Branch pushed to git repo; I updated commit sha1. New commits:

`cb15a59`	`Use pynac-0.7.26.sage-2020-04-02`
`deb717f`	`use pynac-0.7.26.sage-2020-04-03`

sagetrac-git · 2020-04-03T19:49:11Z

Changed commit from db5074a to deb717f

mkoeppe · 2020-04-03T20:24:32Z

comment:62

This fixes the build on cygwin.

dimpase · 2020-04-04T01:01:47Z

comment:63

please squash the commits

sagetrac-git · 2020-04-04T01:48:00Z

Changed commit from deb717f to 7344e2d

sagetrac-git · 2020-04-04T01:48:00Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`7344e2d`	`Use temporary upstream version pynac-0.7.26.sage-2020-04-03`

mkoeppe · 2020-04-05T17:18:41Z

comment:65

squashed, needs review

mkoeppe · 2020-04-10T04:27:50Z

comment:66

Correct builds can be seen at https://github.com/mkoeppe/sage/actions/runs/74117913

jhpalmieri · 2020-04-10T18:41:48Z

comment:67

It's good on OS X, built with various different Pythons (homebrew, system, Sage).

mkoeppe · 2020-04-12T00:41:17Z

comment:68

Let's get this one into the next beta please

dimpase · 2020-04-12T00:47:13Z

comment:69

looks good.

mkoeppe · 2020-04-12T00:55:08Z

comment:70

Thanks!

DaveWitteMorris · 2020-04-15T06:24:46Z

Attachment: pynac-0.7.26.sage-2020-04-03.log

pynac did not download on Mac OS 10.15.3

DaveWitteMorris · 2020-04-15T06:35:06Z

comment:72

I apologize if this is noise (I may have messed up my installation somehow), but I thought I should report that I had trouble building pynac in 9.1rc0 on Mac OS 10.15.3, even with this ticket:

Attempting to download from https://github.com/mkoeppe/pynac/releases/download/pynac-0.7.26.sage-2020-04-03/pynac-0.7.26.sage-2020-04-03.tar.bz2
[xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
ERROR [transfer|run:135]: [Errno socket error] [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)
************************************************************************

I merged #29408 and this ticket, and ran make build, then realized that the description of this ticket has a different make command, so I tried that, too. But pynac did not download. I am attaching the pynac-0.7.26.sage-2020-04-03.log log file from the more complicated build command, in case it is of use.

The build went smoothly after I manually downloaded pynac and put it in the upstream directory, so I want to say thanks for these patches! I never would have figured out how to fix the build errors on my own.

dimpase · 2020-04-15T06:41:56Z

comment:73

this might be due to (system) python, more precisely its urllib module, being unable to provide correct certs...

does installing python 3.7 from python.org and also making sure it installs certs (there is a 2nd step in installation, iirc) fixes this?

DaveWitteMorris · 2020-04-15T07:53:43Z

comment:74

Yes, that solved the problem. After installing python and certifi, pynac downloaded from mkoeppe's directory (and the build succeeded) when I used make SAGE_SPKG="sage-spkg -o" pynac-clean pynac. Thanks!

vbraun · 2020-04-16T22:33:49Z

Changed branch from u/mkoeppe/fix_build_failures_of_pynac_on_macos_with_system_python3 to 7344e2d

mkoeppe · 2020-06-16T11:41:30Z

Changed commit from 7344e2d to none

mkoeppe · 2020-06-16T11:41:30Z

comment:76

Follow up: #29872

mkoeppe added this to the sage-9.1 milestone Mar 25, 2020

mkoeppe added c: packages: standard labels Mar 25, 2020

mkoeppe changed the title ~~Fix build failures of pplpy on macos with system python3~~ Fix build failures of pplpy and pynac on macos with system python3 Mar 25, 2020

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~Fix build failures of pplpy and pynac on macos with system python3~~ Fix build failures of pynac on macos with system python3 Mar 26, 2020

mkoeppe added the s: needs work label Apr 3, 2020

mkoeppe added s: needs review and removed s: needs work labels Apr 3, 2020

dimpase added s: positive review and removed s: needs review labels Apr 12, 2020

mkoeppe added p: blocker / 1 and removed p: critical / 2 labels Apr 15, 2020

vbraun removed the s: positive review label Apr 16, 2020

vbraun closed this as completed in bf8c781 Apr 16, 2020

This was referenced Apr 15, 2020

Fix build errors with pplpy, scipy, kiwisolver, fpylll on macos with system python3 #29408

Closed

macos: pynac ignores /usr/local #29872

Closed

Fix build failures of pynac on macos with system python3 #29404

Fix build failures of pynac on macos with system python3 #29404

Comments

mkoeppe commented Mar 25, 2020

mkoeppe commented Mar 25, 2020

dimpase commented Mar 25, 2020

kiwifb commented Mar 25, 2020

kiwifb commented Mar 25, 2020

kiwifb commented Mar 25, 2020

mkoeppe commented Mar 25, 2020

dimpase commented Mar 25, 2020

kiwifb commented Mar 25, 2020

dimpase commented Mar 25, 2020

kiwifb commented Mar 25, 2020

dimpase commented Mar 25, 2020

kiwifb commented Mar 25, 2020

kiwifb commented Mar 25, 2020

kiwifb commented Mar 25, 2020

kiwifb commented Mar 25, 2020

mkoeppe commented Mar 25, 2020

dimpase commented Mar 25, 2020

mkoeppe commented Mar 25, 2020

This comment has been minimized.

mkoeppe commented Mar 26, 2020

mkoeppe commented Mar 26, 2020

dimpase commented Mar 26, 2020

mkoeppe commented Mar 26, 2020

kiwifb commented Mar 26, 2020

sagetrac-git mannequin commented Apr 3, 2020

sagetrac-git mannequin commented Apr 3, 2020

mkoeppe commented Apr 3, 2020

dimpase commented Apr 4, 2020

sagetrac-git mannequin commented Apr 4, 2020

sagetrac-git mannequin commented Apr 4, 2020

mkoeppe commented Apr 5, 2020

mkoeppe commented Apr 10, 2020

jhpalmieri commented Apr 10, 2020

mkoeppe commented Apr 12, 2020

dimpase commented Apr 12, 2020

mkoeppe commented Apr 12, 2020

DaveWitteMorris commented Apr 15, 2020

DaveWitteMorris commented Apr 15, 2020

dimpase commented Apr 15, 2020

DaveWitteMorris commented Apr 15, 2020

vbraun commented Apr 16, 2020

mkoeppe commented Jun 16, 2020

mkoeppe commented Jun 16, 2020