python setup.py picks prefix from ~/.pydistutils.cfg

[alexanderdreyer]

Hi,
several packages do use python setup.py. This picks up the customized settings from ~/.pydistutils.cfg. This is bad, because it overrides the prefix setting.

For sage-main, I'll attach a patch: setup.cfg in the corresponding directory overrides the usere settings. Maybe there's a global solution.

Regards,
Alexander Dreyer

......

Update for Sage 7.5. The problem reported on this ancient ticket is still present and affects

• sagelib,
• packages that are installed directly using setup.py
• packages that are installed using pip.

It can be tested by creating the following ~/.pydistutils.cfg:

[install]
root=/does-not-exist

The patch on the ticket uses

• pip --isolated and
• setup.py --no-user-cfg

Note there appear to be some issues regarding these option with early versions of Python 3.x: pypa/pip#3509

Upstream: Reported upstream. Developers deny it's a bug.

CC: @jdemeyer @embray @nexttime @kiwifb @vbraun

Component: build

Author: Matthias Koeppe

Branch/Commit: 375faba

Reviewer: Erik Bray

Issue created by migration from https://trac.sagemath.org/ticket/9536

[alexanderdreyer]

Patch for sage-main

[alexanderdreyer]

comment:1

Attachment: setup_py_issue.patch.gz

[alexanderdreyer]

comment:2

The following packages from the distribution use setup.py and may be affected:

zodb3-3.7.0.p4
scipy_sandbox-20071020.p5
pexpect-2.0.p4
sqlalchemy-0.5.8
glpk-4.44
mpmath-0.15
pil-1.1.6.p2
mercurial-1.3.1.p2
pycrypto-2.0.1.p5
mpir-1.2.2.p1
weave-0.4.9.p0
sympy-0.6.4.p0
scons-1.2.0
numpy-1.3.0.p3
gdmodule-0.56.p7
sagetex-2.2.5
sage-4.5
matplotlib-0.99.3
sage_scripts-4.5
setuptools-0.6c9.p0
python-2.6.4.p9
python_gnutls-1.1.4.p7
sagenb-0.8.1
networkx-1.0.1
jinja-1.2.p0
cython-0.12.1
pygments-0.11.1.p0
scipy-0.7.p5
cvxopt-0.9.p8
jinja2-2.1.1.p0
sphinx-0.6.3.p4
docutils-0.5.p0
twisted-9.0.p2
ipython-0.9.1.p0
moin-1.9.1.p1

The following already have setup.cfg, but it may not have the prefix definition:

zodb3-3.7.0.p4
sqlalchemy-0.5.8
scons-1.2.0
matplotlib-0.99.3
setuptools-0.6c9.p0
sagenb-0.8.1
networkx-1.0.1
jinja-1.2.p0
pygments-0.11.1.p0
jinja2-2.1.1.p0
sphinx-0.6.3.p4
twisted-9.0.p2
moin-1.9.1.p1

[sagetrac-PolyBoRi]

comment:3

Hi!
I am not an export for that for that, but did you read
http://docs.python.org/install/#location-and-names-of-config-files
There seems to be some options:
–no-user-cfg

Cheers,
Michael

[alexanderdreyer]

comment:4

Hi Michael,
indeed, that's a better solution. But both have the problem, that 30+ packages would have to be fixed, and it might be forgotten in the future. (Maybe there some monkey patch for distutils, that all of it can be overwritten by the environment.)

Best regards,
Alexander

[sagetrac-PolyBoRi]

comment:5

maybe, I did not try (just read the docs):
http://docs.python.org/install/#inst-config-files

create system distutils.cfg

[global]
no-user-cfg=1

[alexanderdreyer]

comment:6

Hi,
unfortunately, this is a Python 2.7 feature and it does not support no-user-cfg in distutils.cfg :-(

Regards,
Alexander

[alexanderdreyer]

patch for sage/spkg/base/sage-env sage/local/bin/sage-env (needs python 2.6.4.p10)

[alexanderdreyer]

comment:7

Attachment: sage-env.patch.gz

Hi,
I backported the handling of setup.py --no-user-cfg from Python 2.7 to Python 2.6.4 and also added the handling of the environment variable DISTUTILS_NO_USER_CFG to python's distutils.

The new spkg can be found here:
http://sage.math.washington.edu/home/dreyer/suse101/python-2.6.4.p10.spkg

The last patch adds this variable to sage-env.

Regards,
Alexander

[alexanderdreyer]

comment:8

Reported upstream:
http://bugs.python.org/issue9309

[alexanderdreyer]

Changed upstream from Not yet reported upstream; Will do shortly. to Reported upstream. Little or no feedback.

[alexanderdreyer]

comment:9

Minor update: http://sage.math.washington.edu/home/dreyer/suse101/python-2.6.4.p10.spkg (useful, if the sage-env patch is not used)

[sagetrac-drkirkby]

comment:10

Replying to @alexanderdreyer:

Minor update: http://sage.math.washington.edu/home/dreyer/suse101/python-2.6.4.p10.spkg (useful, if the sage-env patch is not used)

You should also create patches/ACKS.txt.diff as a unified diff file. Although its fairly obvious you can recreate that now, that will not be the case if Python source code is updated.

Would a simpler solution not be for sage-env to set this variable and export it? I don't know much about python, so can't review this myself.

Dave

[alexanderdreyer]

comment:11

Maybe a misunderstanding: the patch of python is necessary to fix that issue anyway. (This first variant didn't work, if the environment variable was not set.)

[alexanderdreyer]

Attachment: python-2.6.4.p10.patch.gz

Adding handling of environment variable DISTUTILS_NO_USER_CFG to python's distutils

[alexanderdreyer]

comment:12

I've added ACKS.txt.patch to the spkg.

[alexanderdreyer]

comment:14

The python community does not like the idea of adding an environment variable, see:
http://bugs.python.org/issue9309

[alexanderdreyer]

Changed upstream from Reported upstream. Little or no feedback. to Reported upstream. Developers deny it's a bug.

[jdemeyer]

comment:16

Please fill in your real name as Author.

[vbraun]

comment:28

  Source in /tmp/pip-mTg5Xg-build has version 16.0.0, which satisfies requirement pyzmq==16.0.0 from file:///mnt/disk/home/release/Sage/local/var/tmp/sage/build/pyzmq-16.0.0/src
Installing collected packages: pyzmq
  Running setup.py install for pyzmq: started
    Running command /mnt/disk/home/release/Sage/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-mTg5Xg-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" configure --zmq=/mnt/disk/home/release/Sage/local --no-user-cfg install --record /tmp/pip-o79sR3-record/install-record.txt --single-version-externally-managed --compile
    running configure
    error: error in command line: command 'Configure' has no such option 'no_user_cfg'
    Running setup.py install for pyzmq: finished with status 'error'
Cleaning up...
  Removing source in /tmp/pip-mTg5Xg-build
Command "/mnt/disk/home/release/Sage/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-mTg5Xg-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" configure --zmq=/mnt/disk/home/release/Sage/local --no-user-cfg install --record /tmp/pip-o79sR3-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-mTg5Xg-build/
Exception information:
Traceback (most recent call last):
  File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/pip/commands/install.py", line 317, in run
    prefix=options.prefix_path,
  File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/pip/req/req_set.py", line 742, in install
    **kwargs
  File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/pip/req/req_install.py", line 880, in install
    spinner=spinner,
  File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/pip/utils/__init__.py", line 718, in call_subprocess
    % (command_desc, proc.returncode, cwd))
InstallationError: Command "/mnt/disk/home/release/Sage/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-mTg5Xg-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" configure --zmq=/mnt/disk/home/release/Sage/local --no-user-cfg install --record /tmp/pip-o79sR3-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-mTg5Xg-build/

[mkoeppe]

comment:29

I've created an upstream issue for this:
zeromq/pyzmq#937

[embray]

comment:30

That's annoying. For now we should probably remove pip install from pyzmq.

[mkoeppe]

Dependencies: #21835

[mkoeppe]

comment:31

Upstream pyzmq reports that it works with pip 9.0.1, setuptools 28.8.0.

Let's try with upgrading these packages. #21835 does the upgrade of pip to 9.0.0...

[mkoeppe]

comment:32

Upgrading setuptools to 28.8.0 does not fix the pyzmq install.

I'll wait for the pip upgrade.

[mkoeppe]

Changed branch from 6dcaebb to u/mkoeppe/6dcaebb5f95abdc78cce729bcced7dc96ee5cb1f

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

bf1b130	Use 'pip --isolated' and 'setup.py --no-user-cfg'
c218299	pyzmq: Fix pip install command line

[sagetrac-git]

Commit: c218299

[mkoeppe]

comment:35

Here's a fix for our pyzmq spkg-install, with help from upstream.

[mkoeppe]

Changed dependencies from #21835 to none

[jdemeyer]

comment:37

Just a comment: it would make sense to use $PIP_INSTALL for a few more packages where no extra options are needed: fpylll, gambit, sagenb, sagetex.

[mkoeppe]

comment:38

I agree, but that should be on a different ticket. I've created #21864 for this.

[mkoeppe]

comment:39

Needs review.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

1ea8221	Use 'pip --isolated' and 'setup.py --no-user-cfg'
375faba	pyzmq: Fix pip install command line

[sagetrac-git]

Changed commit from c218299 to 375faba

[mkoeppe]

comment:41

Rebased, needs review.

[mkoeppe]

comment:42

ping?

[embray]

comment:43

The issue with --global-option and --no-user-cfg still seems odd to me. I think that should be considered a bug in pip. But I guess there's nothing to be done for now.

[vbraun]

Changed branch from u/mkoeppe/6dcaebb5f95abdc78cce729bcced7dc96ee5cb1f to 375faba