If $SAGE_SUDO is set, use it whenever we do "make install" of a package.

[mkoeppe]

That would be a solution for those who want to install Sage in a prefix that's only writable by root.

In this way, building can take place in unprivileged user account.

It could perhaps also be enabled by ./configure --enable-sudo-install. (not on this ticket)

This ticket makes routine changes to all spkg-install scripts. Whenever installing has more steps than just make install, it moves the install part to a separate script spkg-do-install.

*** See #21726 for a slightly different approach ***

CC: @jdemeyer @kiwifb @embray @nexttime @vbraun @mezzarobba

Component: build

Reviewer: Matthias Koeppe, Samuel Lelièvre

Issue created by migration from https://trac.sagemath.org/ticket/21537

[mkoeppe]

Branch: u/mkoeppe/if__sage_sudo_is_set__use_it_whenever_we_do__make_install__of_a_package_

[mkoeppe]

New commits:

444fd6e

Add SAGE_SUDO for packages A-L

[mkoeppe]

Commit: 444fd6e

[kiwifb]

comment:4

Practical matters:

• documentation
• If you need a password and that the time between sudo-ing is longer than the time out, you will have to insert the password quite a number of times. In the case of a parallel install, will it block output to screen until you have entered the password?

I don't know enough of the mechanics of sage standard output on screen to know how requesting a password will affect the flow.

[kiwifb]

comment:5

I guess I should say in the case of parallel make. In which the parallelism of make is translated into installing packages concurrently.

[mkoeppe]

comment:6

Yes, these issues need addressing in documentation and/or implementation.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

666fb57

Add SAGE_SUDO for packages L-P

[sagetrac-git]

Changed commit from 444fd6e to 666fb57

[mkoeppe]

comment:8

Replying to @kiwifb:

In the case of a parallel install, will it block output to screen until you have entered the password?

I don't know enough of the mechanics of sage standard output on screen to know how requesting a password will affect the flow.

The problem of parallel output and prompts has been discussed before in #20884, #21082.
Also #21539 could help.

[sagetrac-git]

Changed commit from 666fb57 to ec6ce24

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

ec6ce24

spkg-do-install: Make executable

[sagetrac-git]

Changed commit from ec6ce24 to 23e3ddc

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

23e3ddc

Packages P-S

[sagetrac-git]

Changed commit from 23e3ddc to 74fbeff

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

341bdb0	Packages M-Z
74fbeff	Call pip with SAGE_SUDO

[mkoeppe]

Author: Matthias Koeppe

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

d8919cc	Add SAGE_SUDO for packages A-L
7f46f6c	Add SAGE_SUDO for packages L-P
ccb3a13	spkg-do-install: Make executable
3f957cc	Packages P-S
c806e9f	Packages M-Z
abd3e96	Call pip with SAGE_SUDO

[sagetrac-git]

Changed commit from 74fbeff to abd3e96

[mkoeppe]

comment:14

Rebased on 7.4.beta6.
Needs review.

[embray]

comment:15

Somehow this makes me feel a bit icky, but I'm not -1 on it either. It's a solution for now. I feel icky because if the Sage distribution were like a "normal" autoconf package I should be able to simply run make && sudo make install and not have tell configure that I intend to use sudo when I install. Nor should sudo be invoked in each sub-make. But the Sage distribution is definitely not a normal package since really it's just a big wrapper around a bunch of other packages, combined with some semblance of dependency management.

That said, it would be nice if the Sage distribution did work like make && sudo make install. To me, this points toward another area where sage packages could and probably should be refactored anyways. If packages had separate spkg-build and spkg-install scripts that would go a long way, I think. Then make/make build would run the spkg-build scripts, and only make install would run the actual spkg-install scripts. It would probably also go a long ways towards further eliminating duplication of effort between the many spkg-install scripts.

On another note, for Python packages installed with pip it still might be best to have separate "build" and "install" phases--in particular for Python packages with extension modules, where one might not want to be running a compiler as root. This could be achieved by running pip install without sudo with a temporary target directory, and then copying into the appropriate site-packages (perhaps after an uninstall of previously installed versions of the package). For that to work it might be best if this waited on #21441 which adds a new sage-pip-install script, which might be further modified to support this usage.

[mkoeppe]

comment:16

Fine with me to wait for #21441 and its sage-pip-install. Is that ticket moving forward, by the way?

[mkoeppe]

comment:17

Replying to @embray:

[...] if the Sage distribution were like a "normal" autoconf package I should be able to simply run make && sudo make install and not have tell configure that I intend to use sudo when I install.

Just for reference for other readers (I know that you are aware of that ticket, Erik): Any discussion of separating make and make install for sage-the-distribution should take place on ticket #21495.

[mkoeppe]

comment:18

Replying to @embray:

To me, this points toward another area where sage packages could and probably should be refactored anyways. If packages had separate spkg-build and spkg-install scripts that would go a long way, I think. Then make/make build would run the spkg-build scripts, and only make install would run the actual spkg-install scripts. It would probably also go a long ways towards further eliminating duplication of effort between the many spkg-install scripts.

Right now, in this patch I am calling spkg-do-install scripts with $SAGE_SUDO from various package's spkg-install scripts.
Of course, each package's spkg-install script needs to be changed for $SAGE_SUDO to work, and installing packages that have not been prepared for it will cause an error when trying to install them to a prefix that is only root-writable.

An "easier" solution would be to just call spkg-build if it exists, unprivileged, and then call spkg-install using $SAGE_SUDO. This would "just work" without making changes to the -- but it would run a lot of stuff as root until a package moves its build part from spkg-install to spkg-build.

Thoughts?

On another note, for Python packages installed with pip it still might be best to have separate "build" and "install" phases--in particular for Python packages with extension modules, where one might not want to be running a compiler as root. This could be achieved by running pip install without sudo with a temporary target directory, and then copying into the appropriate site-packages (perhaps after an uninstall of previously installed versions of the package).

Are you sure that Python packages with extension modules are relocatable in this way? I have no experience with the specifics of the Python distutils world, but I would imagine all kinds of trouble (rpath/sonames, possible hardcoded paths, ...)

For that to work it might be best if this waited on #21441 which adds a new sage-pip-install script, which might be further modified to support this usage.

Sure, I've made it a dependency.

[mkoeppe]

Dependencies: #21441

[embray]

comment:19

Right, regardless of how make is invoked, if there are separate spkg-build and spkg-install scripts the former can be run unprivileged. Then, at least in the current system, you would call $SAGE_SUDO spkg-install.

I think most packages would be easy enough to split up their spkg-install into two scripts. Though as I think I wrote in another ticket one might actually want a third spkg-env script that sets environment variables needed for both the build and install steps (fortunately for most autotools packages most relevant environment variables are only needed for the build step, I think).

[embray]

comment:20

Replying to @mkoeppe:

Replying to @embray:

On another note, for Python packages installed with pip it still might be best to have separate "build" and "install" phases--in particular for Python packages with extension modules, where one might not want to be running a compiler as root. This could be achieved by running pip install without sudo with a temporary target directory, and then copying into the appropriate site-packages (perhaps after an uninstall of previously installed versions of the package).

Are you sure that Python packages with extension modules are relocatable in this way? I have no experience with the specifics of the Python distutils world, but I would imagine all kinds of trouble (rpath/sonames, possible hardcoded paths, ...)

In general, yes, it should not be a problem. This does not affect rpath or soname for extension modules--I mean in principle it could but that would be a pathological case the likes of which I've never seen, and that is definitely not supported.

In principle one could also imagine a module that when installed generates some files containing hard-coded paths, but can't think of anything like that. Most well made Python modules are good about keeping such things relative and dynamic. So I'd be surprised if that's a problem anywhere, though if it is obviously exceptions can be made.

(Update: In fact, I should add, this is how distutils builds packages normally anyways--first it builds everything to build/, and then simply copies from there directly to the install location--what I described simply adds one additional copy. All the "install" commands really do in distutils is copy files--all the real work should usually be done in the "build" command (or sub-commands thereof).)

[mkoeppe]

Changed dependencies from #21441 to #21726

[mkoeppe]

Work Issues: redo patch using the method of 21726

[mkoeppe]

comment:24

This was implemented in #21726. The present ticket can be closed

[mkoeppe]

Changed author from Matthias Koeppe to none

[slel]

comment:25

Ok.

[slel]

Changed branch from u/mkoeppe/if__sage_sudo_is_set__use_it_whenever_we_do__make_install__of_a_package_ to none

[slel]

Changed commit from abd3e96 to none

[slel]

Changed work issues from redo patch using the method of 21726 to none

[slel]

Changed dependencies from #21726 to none

[slel]

Reviewer: Matthias Koeppe, Samuel Lelièvre