transition to using install program

[sagetrac-felixs]

To keep track of installed files, it's probably most convenient to use a unified install program within all spkg-install scripts, like the one included within #14796.

For transitional purposes, the non-#14796 build-system should provide an install program within $PATH. I propose to place it as sage-dist-install into $SAGE_SRC/bin. It should just ignore -F for now.

For python-distutils packages, a setup.py wrapper is included.

CC: @nexttime @jondo

Component: distribution

Keywords: spkg-install filelist destdir

Author: Felix Salfelder

Branch/Commit: u/felixs/sagedist @ 2584344

Issue created by migration from https://trac.sagemath.org/ticket/14804

[sagetrac-felixs]

Attachment: sage-dist-install.gz

[vbraun]

comment:2

Python distutils at least encapsulates the file operations in distutils.file_utils, so it would be a finite task to hook into that as well.

The alternative is to serialize the install phase and diff the SAGE_ROOT tree before and after.

Yet another possibility is to use separate install trees for each spkg in "sage-the-distribution", as in Nix or Stow.

[sagetrac-felixs]

comment:3

Replying to @vbraun:

Python distutils at least encapsulates the file operations in distutils.file_utils, so it would be a finite task to hook into that as well.

For example. but i have no idea how to "hook into that" yet without patching distutils. is there any python magic you have in mind?

The alternative is to serialize the install phase and diff the SAGE_ROOT tree before and after.

Won't give me DESTDIR. serialization is one of the uglier hacks (no way). diff takes quadratic time (even less way).

Yet another possibility is to use separate install trees for each spkg in "sage-the-distribution", as in Nix or Stow.

Just complicates the issue. better is supporting no "install" at all (like sage-doc currently does).

[nexttime]

comment:4

Replying to @sagetrac-felixs:

Replying to @vbraun:

Python distutils at least encapsulates the file operations in distutils.file_utils, so it would be a finite task to hook into that as well.

For example. but i have no idea how to "hook into that" yet without patching distutils. is there any python magic you have in mind?

You can patch it (Python / setuptools / whatever), but you don't necessarily have to, as most Python objects are mutable, including functions / methods, i.e., you can overwrite them... :-)

In [1]: import distutils.file_util

In [2]: def foo(s,t): print "Hello"
   ...: 

In [3]: distutils.file_util.copy_file=foo

In [4]: distutils.file_util.copy_file("bar","baz")
Hello

And you could e.g. do so in an spkg's setup.py, to use some alternate implementation (provided elsewhere).

[sagetrac-felixs]

comment:5

Replying to @nexttime:

You can patch it (Python / setuptools / whatever), but you don't necessarily have to,

I dont want to. I'd just patch upstream if it's completely hosed.

as most Python objects are mutable, including functions / methods, i.e., you can overwrite them... :-)

...

And you could e.g. do so in an spkg's setup.py, to use some alternate implementation (provided elsewhere).

I don't quite understand. Are you proposing to patch the upstream package? (this would be worse, as it involves more than just distutils patching).

I need something of the kind

setup.py install USEMYOWNFILEUTIL

or

IMPORTPATH=myhackedfileutil_path setup.py install

within spkg-install. is this even possible?

[nexttime]

comment:6

Replying to @sagetrac-felixs:

Replying to @nexttime:

as most Python objects are mutable, including functions / methods, i.e., you can overwrite them... :-)
And you could e.g. do so in an spkg's setup.py, to use some alternate implementation (provided elsewhere).

I don't quite understand. Are you proposing to patch the upstream package? (this would be worse, as it involves more than just distutils patching).

I need something of the kind

setup.py install USEMYOWNFILEUTIL

or

IMPORTPATH=myhackedfileutil_path setup.py install

within spkg-install. is this even possible?

Sure, I was just writing a post scriptum:

And in the latter case, you wouldn't even have to patch / modify the spkg's / upstream's setup.py, but just spkg-install (or whatever that will become), as you can also do things like

python <<EOF
# Do some stuff, e.g. modifying distutils.file_util.*
# (preferably importing an alternate implementation located elsewhere)
import sys
sys.argv = ["setup.py", "install"]
execfile("setup.py")
EOF

Or just create a (shell) script that does similar, and call that instead of doing python setup.py install.

[nexttime]

comment:7

P.P.S.:

Of course, as is that relies on upstream only using distutils (at least to copy / install files), just like your custom [sage-dist-]install mechanism relies on upstream respecting autotools' conventions (and exclusively using install[-sh]) for such purposes.

[nexttime]

comment:8

... but you may have to wrap python for some packages that use both (e.g.) autotools and Python's distutils (some support setting PYTHON).

Still, uninstalling an spkg by (just) removing the files it added (probably restoring those it deleted or modified, too) can easily break a Sage installation, but I guess you're aware of the pitfalls (that aren't specific to Sage)...

[sagetrac-felixs]

comment:9

Attachment: sage-setup.py.gz

Thanks Leif, I wasn't aware of that overriding possibility.

Replying to @nexttime:

Of course, as is that relies on upstream only using distutils (at least to copy / install files), just like your custom [sage-dist-]install mechanism relies on upstream respecting autotools' conventions (and exclusively using install[-sh]) for such purposes.

.. and lets see what else copy_file is abused for. But I think, this approach will finally cut it.

autotools uses $(INSTALL), which can be overriden with anything. also others, otherwise broken make based build systems use $(INSTALL) and $(INSTALLDIRS).

[nexttime]

comment:10

Replying to @sagetrac-felixs:

.. and lets see what else copy_file is abused for. But I think, this approach will finally cut it.

Note that your replacement (or wrapper) function will of course finally have to mimic the full behaviour of the original one, i.e., has to take the same optional arguments (with the same default values), and has to return a tuple.

[sagetrac-felixs]

comment:11

Replying to @nexttime:

Note that your replacement (or wrapper) function will of course finally have to mimic the full behaviour of the original one, i.e., has to take the same optional arguments (with the same default values), and has to return a tuple.

It is supposed to call the original function after it's done with file listing.
I'd expect that

def B(*args, **kwds): return B(args, kwds)

works, except for the default values...

[nexttime]

comment:12

Replying to @sagetrac-felixs:

It is supposed to call the original function after it's done with file listing.
I'd expect that

def B(*args, **kwds): return B(args, kwds)

works, except for the default values...

... and infinite recursion. ;-)

[sagetrac-felixs]

comment:13

Replying to @nexttime:

... and infinite recursion. ;-)

that was typed to fast, also some asterisks are missing. as it seems it even does the right thing wrt default values...

[sagetrac-felixs]

comment:14

And indeed, setup.py install takes --record <filename>. I should have tried to patch distutils right away.... :)

[nexttime]

comment:15

It may make sense to wrap the whole distutils.core.setup() function [too]...

(depending on what else you intend to do)

[sagetrac-felixs]

comment:16

There is now sage-dist-install, sage-dist-make, spkg.mk and sage-setup.py within my WIP branch on github.

With these, most spkg-install scripts can be augmented to support both the #14796 build system (together with #14792) as well as the current one.

Please have a look whether this could break anything.

[sagetrac-felixs]

comment:17

This is now in git. It works for all packages I have tried with.

[sagetrac-felixs]

Branch: u/felixs/sagedist

[sagetrac-felixs]

Commit: fdeefe5

[sagetrac-git]

Changed commit from fdeefe5 to 2584344

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:2584344]

provide SAGE_PREFIX

[jdemeyer]

comment:21

Needs to be rebased.

[embray]

comment:25

This was in a way trying to do what is now accomplished (in spirit--not so much in the details) by some combination of #23160 an #22509. I don't think this particular approach really fits in with what we're doing now. But thank you Felix for your early pioneering work on trying to fix Sage's build system...