MAINT: setup.py cleanups and additions #454

Merged
merged 10 commits into from Mar 24, 2013

Conversation

Projects
None yet
5 participants
Owner

pv commented Feb 27, 2013

Misc. cleanups, fixes and additions regarding setup.py

  • add build_sphinx command
  • add test command
  • remove setup*egg*.py, and always use setuptools if available
  • add missing files to MANIFEST.in (only seems to matter when setuptools is enabled)
  • fix building with pip without Cython

pv added some commits Feb 9, 2013

BLD: fix the setup.py build_sphinx command
This is perhaps the easiest way to build the documentation, so it's good
to have it working.
MAINT: add all source files to MANIFEST.in
sdist with setuptools doesn't play along with source files generated
from Numpy's templates, so they need to be included manually.
BUG: setup.py: pass non-prefixed scipy path to cythonize, but set cwd
This ensures cythonize.dat doesn't depend on cwd. This fixes the build
on pip when Cython is not installed.
Owner

pv commented Feb 27, 2013

UPDATE: aargh, the add_data_dir and add_data_files commands apparently stop working when setuptools is active, so sdist is missing files.

I'd maybe just ditch this part of numpy.distutils and add the stuff to MANIFEST.in. Thoughts?

Member

dlax commented Feb 27, 2013

Pauli Virtanen wrote:

• remove setup_egg_.py, and always use setuptools if available

What's the rationale for "always use setuptools"?

Owner

pv commented Feb 27, 2013

@dlax: the test command is not available otherwise, and it allows getting rid of *egg*.py.

OTOH, I don't know the rationale for the present setup that avoids it.

Owner

rgommers commented Feb 28, 2013

That's not a strong motivation to make changes to numpy.distutils. No telling what it will break. Plus we are still keeping compatibility with numpy 1.5.1; a change in numpy.distutils would help only by the time we drop support for 1.5.1, 1.6.2 and 1.7.x

Owner

pv commented Mar 2, 2013

@rgommers: I meant that we stop relying on this part of numpy.distutils in Scipy, and rely on MANIFEST.in instead so that sdist works correctly both with and without setuptools.

Owner

rgommers commented Mar 2, 2013

Ah that makes sense. +1

Owner

rgommers commented Mar 2, 2013

"always use setuptools" still makes me feel vaguely uncomfortable - it's less stable than distutils.

+prune doc/build
+prune doc/source/generated
+prune */__pycache__
+global-exclude *.pyc *~ *.bak
@rgommers

rgommers Mar 24, 2013

Owner

I propose to add *.swp (Vim tempfiles) and *.pyo here

doc/README.txt
+
+ python setup.py build_sphinx
+
+This will make first build Scipy in-place, and then generate documentation for
@rgommers

rgommers Mar 24, 2013

Owner

typo, remove "make"

Owner

rgommers commented Mar 24, 2013

OK, I can't find a concrete reason right now for why we shouldn't always use setuptools. So let's try this, then we have half a year to notice any issues before the next release.

pv added a commit that referenced this pull request Mar 24, 2013

Merge pull request #454 from pv/setup-cleanups
MAINT: setup.py cleanups and additions

@pv pv merged commit 275e105 into scipy:master Mar 24, 2013

Owner

pv commented Mar 24, 2013

Thanks, merged. Also added the if not ISRELEASED discussed elsewhere before generate_cython to be safe.

Member

WarrenWeckesser commented Mar 25, 2013

@pv: When I build the docs using the make command in the scipy/doc directory, I have the numpy source directory numpy/doc/sphinxext in my PYTHONPATH so the numpydoc extension is available, and this seems to work fine. I just tried python setup.py build_sphinx, and it worked, but the generated HTML says scipy is version 1.7 (i.e. the numpy version). Is this a bug in the build_sphinx command, or do I have to change my setup?

Owner

pv commented Mar 25, 2013

@WarrenWeckesser: the wrong version number is due to __SCIPY_SETUP__ business. I think scipy/__init__.py should be changed so that the check for __SCIPY_SETUP__ is done only if importing the version number fails.

Or even, we could just get rid of it altogether --- I know some of the build scripts import scipy._build_utils, but we could move that under tools/. I'm not sure if there's something else that needs scipy modules at build time.

Member

WarrenWeckesser commented Mar 26, 2013

Since this PR was merged (commit 275e105), when I install with python setup.py install --prefix=/home/warren/local_scipy, the tests don't run. E.g.:

$ python -c "import scipy.signal as sig; sig.test('full')"
Running unit tests for scipy.signal
NumPy version 1.7.0
NumPy is installed in /home/warren/anaconda/lib/python2.7/site-packages/numpy
SciPy version 0.13.0.dev-275e105
SciPy is installed in /home/warren/local_scipy/lib/python2.7/site-packages/scipy-0.13.0.dev_275e105-py2.7-linux-x86_64.egg/scipy
Python version 2.7.3 |Anaconda 1.4.0 (64-bit)| (default, Feb 25 2013, 18:46:31) [GCC 4.1.2 20080704 (Red Hat 4.1.2-52)]
nose version 1.2.1

----------------------------------------------------------------------
Ran 0 tests in 0.001s

OK

My PYTHONPATH:

$ echo $PYTHONPATH 
/home/warren/local_scipy/lib/python2.7/site-packages
Owner

pv commented Mar 26, 2013

That seems to be because setuptools sets the executable bit on the scripts, and we don't supply nose the --exe flag

Owner

rgommers commented Mar 26, 2013

Is there a way to tell setuptools not to do that?

Owner

pv commented Mar 26, 2013

@rgommers: AFAIK, there's no way to stop setuptools doing that. The only fix is AFAIK either to add extra_argv=["--exe"] to test() functions for all scipy subpackages, or ditch setuptools, OR add __init__.py to all test directories.

If revert, do we have any use for eggs (i.e., are the *egg.py scripts useful)? The problem with eggs is that all egg-users will then run into this same problem...

I'd maybe prefer the __init__.py route, it seems Ipython does that.

Member

dlax commented Mar 26, 2013

Would it be possible to add a mechanism to disable setuptools even if it is installed? (e.g. by detecting an environment variable)

Owner

rgommers commented Mar 26, 2013

Egg generation is probably useful to some people, I think I've seen @fonnesbeck use that functionality for his scipy-superpack.

What do we have to gain here besides cleaning up a few files that are 3 LoC each? If that's it, reverting probably makes sense. As for cleanups, removing numscons support altogether and making Bento work again would be a more effective way of removing a bunch of stuff.

Member

WarrenWeckesser commented Mar 28, 2013

What's the argument against putting __init__.py in the test directories (if there is one)?

Member

josef-pkt commented Mar 28, 2013

What's the argument against putting init.py in the test directories
(if there is one)?

I haven't seen any disadvantage in statsmodels,
but Skipper put extra_argv=[--exe] in the our test runner. I don't know if
that's related (since I don't know enough about the Linux details)

It polutes maybe the python path, but we could import some test helper
functions from outside of scipy.

Owner

rgommers commented Mar 28, 2013

@WarrenWeckesser what's the argument for it? I just tried it for optimize and got three test failures. Even if we can easily solve those, it looks to me like the wrong solution for this problem.

Owner

rgommers commented Mar 28, 2013

OK, I read back in the thread, reverting removes the python setup.py test functionality. Kind of handy, but not all that important.

Unless the issues are fixable asap, we should first revert part of this PR.

Owner

pv commented Mar 28, 2013

Reverted in a4e93fb

Owner

rgommers commented Mar 28, 2013

thanks

Owner

rgommers commented Mar 28, 2013

It's not a generic issue for setuptools by the way. pip uses setuptools, but with pip you can run the tests just fine. So guess it's just eggs.

Owner

rgommers commented Mar 28, 2013

Also, the test command didn't work with in-place builds & PYTHONPATH (which is how I prefer to work). As usual.

Member

WarrenWeckesser commented Mar 28, 2013

@rgommers: Earlier, @pv suggested adding __init__.py to all test directories as a possible fix. But I didn't try it myself, so if that wouldn't have worked, you can ignore my comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment