Remove remaining dependencies of bootstrap on gettext (AC_LIB_RPATH etc.)

[mkoeppe]

Background: ./bootstrap needs about 112 KB of m4 and shell code from gnulib: https://www.gnu.org/software/gnulib
Before #29549, these files were copied from a system-wide installation of gettext (which uses gnulib): https://www.gnu.org/software/gettext

As opposed to gettext, gnulib does not make releases, and many systems don't package gnulib at all. Thus one cannot uniformly rely on gnulib being already installed. This essentially leaves two options for using gnulib in Sage - copy said 112K into the Sage git tree, or provide gnulib as a Sage pseudo-package (pseudo, due to the chicken vs egg issue: one cannot just make it an ordinary spkg - it needs to be used by bootstrap).

Who needs to run bootstrap and where:

• Any developer who switches to a branch that makes an SPKG upgrade or other changes to the build system, on their development machine.
• Our portability testsuite, run locally using tox -e docker-... and on GH Actions, in the Docker container for the tested platform.

Who does not need to run bootstrap:

• Users who only build released versions from tarballs - because they come pre-bootstrapped
• Users who only build released versions from a git clone - because Sage falls back to downloading a tarball of the configure SPKG.

---

History: #29549 intended to remove the dependency of bootstrap on gettext. #29549 made it possible to run bootstrap on the manylinux platforms needed by cibuildwheel (#33800).

The branch of #29549 followed "copy files" approach. However, some files that gnulib-tool added were left out from the commit, so it breaks on machines on which gettext is missing completely, as reported in https://groups.google.com/g/sage-release/c/FezzF5Q7Wt4/m/xyxVTaYgBAAJ

---

This ticket:
Here we complete the task and remove gettext from various distros/ files etc.

Two approaches have been proposed:

mkoeppe ("copy files") : Branch u/mkoeppe/fix_broken_bootstrap__ac_lib_rpath_etc__ @ 4011cc4608

• Follows approach 3 of the Gnulib manual section 'Integration with Version Control Systems'.
• 5 files, in config/ and m4/, total 112 KB, have been imported by using gnulib-tool import iconv and committed to the branch.
• Matches our practice with imported files m4/ax_*.m4 from autoconf-archive. Different from our practice with SPKGs because the files are needed to generate the configure script.
• Introduces no new dependency.
• Burden on a developer who authors an update ticket for these files: Obtain gnulib by cloning the repo, then re-run gnulib-tool import, remove unneeded files (that's where bootstrap: Clean up use of gettextize #29549 went wrong), commit the changes. (Updates are expected to be necessary only very rarely.)
• Burden on users and other developers: None.
• dimpase's criticism of this approach: "gnulib's code is not Sage's code, it has no place in Sage git tree; this way, updating the imported gnulib files is tricky and error-prone by design".

dimpase ("Sage pseudo-package"): sagemath/sagetrac-mirror@develop...u/dimpase/config/gnulib

• Follows approach 2(C) of the Gnulib manual section 'Integration with Version Control Systems', but instead of using Gnulib's Programs for developing in Git checkouts, implements its own script (making checkout much faster, as full gnulib checkout is 250 MB).
• Introduces a new (although one can't really do Sage development without git installed) dependency on git and on availability of internet connection at the first run of ./bootstrap in a tree, to pull 83 MB (fair to say that internet is needed for Sage development a lot, anyway).
• Burden on a developer who authors an update ticket for these files: Edit the bootstrap file to change the commit hash, verify that the list of files removed by make bootstrap-clean is still correct, commit this change
• Burden on other users: a pseudo-package needs to be installed once (and updated, automatically)
• mkoeppe's criticism of this approach: "Having bootstrap operate a multi-megabyte git clone (on the 1st install and when there's a change of the gnulib commit hash) in the upstream/ directory increases complexity for no benefit."

hybrid ("copy files, spkg-src script for maintenance"): Branch u/mkoeppe/config/gnulib

• Resolves the criticism on the "copy files" approach: "updating the imported gnulib files" is now assisted by the spkg-src script (adapted from dimpase's branch)
• Resolves the criticism on the "Sage pseudo-package" approach: "operate a multi-megabyte git clone (on the 1st install and when there's a change of the gnulib commit hash)" is only done by authors of tickets that update gnulib, not by general users of bootstrap.

---

Voting took place at https://groups.google.com/g/sage-devel/c/E2qstbWXC7g

---

To test: For example tox -r -e local-homebrew-macos-minimal -- config.status

CC: @culler @dimpase @kiwifb @isuruf @nbruin

Component: build: configure

Author: Matthias Koeppe, Dima Pasechnik

Branch: 8bf9d97

Reviewer: François Bissey, Dima Pasechnik, Matthias Koeppe, Kwankyu Lee

Issue created by migration from https://trac.sagemath.org/ticket/34152

[mkoeppe]

Branch: u/mkoeppe/fix_broken_bootstrap__ac_lib_rpath_etc__

[mkoeppe]

New commits:

65638b4

m4/: Add missing dependencies of iconv.m4

[mkoeppe]

Commit: 65638b4

[mkoeppe]

Author: Matthias Koeppe

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

4011cc4

.homebrew-build-env, build/pkgs/{_prereq,_bootstrap}/distros: Remove gettext

[sagetrac-git]

Changed commit from 65638b4 to 4011cc4

[dimpase]

comment:6

vendoring more and more of gnulib...

should we instead install,on demand, gnulib itself?

[mkoeppe]

comment:7

This is how Gnulib is designed to be used.
"Its components are intended to be shared at the source level, rather than being a library that gets built, installed, and linked against. Thus, there is no distribution tarball; the idea is to copy files from Gnulib into your own source tree."
https://www.gnu.org/software/gnulib/

[dimpase]

comment:8

it seems that "copying" is meant to be done from an installed gnulib version.
That's how I read https://www.mail-archive.com/bug-gnulib@gnu.org/msg36476.html
where a way to do this is discussed.
gnulib has a script to facilitate such an installation: https://github.com/coreutils/gnulib/blob/master/top/gitsub.sh

[mkoeppe]

comment:9

There's no such thing as an "installed" gnulib.

[mkoeppe]

comment:10

The workflow with gnulib is documented in https://www.gnu.org/software/gnulib/manual/html_node/Invoking-gnulib_002dtool.html

[mkoeppe]

comment:11

Replying to @dimpase:

gnulib has a script to facilitate such an installation: https://github.com/coreutils/gnulib/blob/master/top/gitsub.sh

from https://github.com/coreutils/gnulib/blob/master/top/gitsub.sh#L36:
"The package maintainer may choose to use or not use git submodules." - I chose to not use git submodules.

[dimpase]

comment:12

Replying to @mkoeppe:

There's no such thing as an "installed" gnulib.

there certainly is an executable/script called gnulib-tool - which is certainly a part of installed gnulib

[mkoeppe]

comment:13

"gnulib-tool is not installed in a standard directory that is contained in the PATH variable. It needs to be run directly in the directory that contains the Gnulib source code." https://www.gnu.org/software/gnulib/manual/html_node/Invoking-gnulib_002dtool.html

[dimpase]

comment:14

gnulib-tool is not installed in a standard directory that is contained in the PATH variable

OK, fine, but its directory has to appear somehow

[mkoeppe]

comment:15

Yes, a developer who wants to update the portions of Gnulib that have been copied into the source tree will use a git clone of gnulib and invoke gnulib-tool from there.

[dimpase]

comment:16

Replying to @mkoeppe:

Replying to @dimpase:

gnulib has a script to facilitate such an installation: https://github.com/coreutils/gnulib/blob/master/top/gitsub.sh

from https://www.gnu.org/software/gnulib/manual/html_node/Invoking-gnulib_002dtool.html:
"The package maintainer may choose to use or not use git submodules." - I chose to not use git submodules.

The quote you attribute to Invoking-gnulib_002dtool.html is actually from gitsub.sh.

gitsub.sh can do subcheckouts rather than submodules.

The docs don't even discuss manual vendoring of files, I think.

[mkoeppe]

comment:17

Replying to @dimpase:

Replying to @mkoeppe:

Replying to @dimpase:

gnulib has a script to facilitate such an installation: https://github.com/coreutils/gnulib/blob/master/top/gitsub.sh

from https://www.gnu.org/software/gnulib/manual/html_node/Invoking-gnulib_002dtool.html:
"The package maintainer may choose to use or not use git submodules." - I chose to not use git submodules.

The quote you attribute to Invoking-gnulib_002dtool.html is actually from gitsub.sh.

Yes, sorry, wrong link.
Fixed now: I meant to paste https://github.com/coreutils/gnulib/blob/master/top/gitsub.sh#L36

[mkoeppe]

comment:18

Replying to @dimpase:

The docs don't even discuss manual vendoring of files, I think.

Please, Dima, the whole manual talks about nothing else.

[dimpase]

comment:19

Replying to @mkoeppe:

Yes, a developer who wants to update the portions of Gnulib that have been copied into the source tree will use a git clone of gnulib and invoke gnulib-tool from there.

but not manually - rather, as a part of pre-configuring a project. Checking in our repo huge chunks of gnulib is suboptimal.

[mkoeppe]

comment:20

Replying to @dimpase:

Replying to @mkoeppe:

Yes, a developer who wants to update the portions of Gnulib that have been copied into the source tree will use a git clone of gnulib and invoke gnulib-tool from there.

but not manually

It's called the "initial import". https://www.gnu.org/software/gnulib/manual/html_node/Initial-import.html
This is what I did in #29549 (except that I forgot to git add some files).

[dimpase]

comment:21

Replying to @mkoeppe:

Replying to @dimpase:

The docs don't even discuss manual vendoring of files, I think.

Please, Dima, the whole manual talks about nothing else.

This manual is older than git, you know, no wonder it won't mention it.

gitsub.sh is for the modern way of doing things, and that's what gnulib maintainer uses.

[mkoeppe]

comment:22

Replying to @dimpase:

gitsub.sh is for the modern way of doing things, and that's what gnulib maintainer uses.

It's a new option of how to work with gnulib, but it offers no benefits for us.

[dimpase]

comment:291

You now seem to acknowledge that it's common practice, which sounds like a way to a peaceful resolution of what we discussed here in this ticket.

No, you are twisting my words, please stop. I never conducted a study of this (and I trust Bruno did, one way or another, arriving to the conclusion opposite).

I also wrote there, and got a confirmation from a Gentoo developer, that "code
injected by
the gnulib-tool got committed into the tree, and blissfully forgotten" is a bad approach.

[mkoeppe]

comment:292

Replying to @dimpase:

Here what Bruno Haible (on of primary authors of gnulib/gettext) has to say about our problem:
https://lists.gnu.org/archive/html/bug-gnulib/2022-08/msg00103.html
Note that he does not suggest to check bits of gnulib in our git tree. [...]

In the previous message, https://lists.gnu.org/archive/html/bug-gnulib/2022-08/msg00102.html, Bruno writes this:

In [1] we document three approaches, dependending on the package's usual
handling of imported/generated files.

Dima Pasechnik wrote:
> [...]
> Why you at gnulib headquarters want large chunks of gnulib
> scattered accross many, many thousands of downstream git trees, I don't know.

This is not what we "want". The documentation [1] gives three options. You
choose the one that fits best for your package.

[1] https://www.gnu.org/software/gnulib/manual/html_node/VCS-Issues.html

He is confirming what the manual is saying, disputing your comment:72

[jhpalmieri]

comment:293

This is fun.

I suggest that we stop the discussion and just agree to abide by the vote on sage-devel, which is still ongoing.

[mkoeppe]

comment:294

+1, with the recommendation for everyone interested to read the whole thread in https://lists.gnu.org/archive/html/bug-gnulib/2022-08/msg00089.html

[sagetrac-tmonteil]

comment:295

Replying to @kwankyu:

Nils seems to be proposing yet another solution for the present ticket, instead of arguing merits or demerits of the candidate solutions of the ongoing vote.

The vote is to finish endless debates about solutions for this ticket. Proposing another solution at this moment, regardless of its own merits, would just disrupt the vote.

By the result of the vote, this ticket will be closed with adopting one solution from the candidates.

Nils may open another ticket for his idea, but that should build upon the present ticket.

This only shows that asking people to vote on a proposal that was constructed by few other people without letting them to debate and improve the proposal first is a bad habit. We already experienced similar "vote don't discuss" procedure by the past, including the "code of conduct" episode. If a better proposal emerges, i do not see in the name of what it could not be chosen over the previous ones.

[kwankyu]

comment:296

Replying to @sagetrac-tmonteil:

This only shows that asking people to vote on a proposal that was constructed by few other people without letting them to debate and improve the proposal first is a bad habit.

We would be choked if we start a discussion on the procedure of voting that is likely to lead to a voting :) If you want the discussion anyway, please open it on sage-devel.

[nbruin]

comment:297

Replying to @sagetrac-tmonteil:

This only shows that asking people to vote on a proposal that was constructed by few other people without letting them to debate and improve the proposal first is a bad habit. We already experienced similar "vote don't discuss" procedure by the past, including the "code of conduct" episode. If a better proposal emerges, i do not see in the name of what it could not be chosen over the previous ones.

This problem must urgently be resolved: presently, you can have a valid setup according to the documentation and yet be unable to build/configure the last couple of sagemath beta releases.

As far as I could see, a valid solution could have been "gettext-devel is a prereq for sagemath" (a really ugly one, though) but the solutions on the ticket at the moment would work too. I think the priority is to get something merged that resolves the acute problem.

Once we have a somewhat workable solution in place that at least allows people to build sage, we can look if the solution, like any piece of sage, can be improved. Who knows, perhaps a better solution even gets merged before a release is actually made! But we cannot really wait for that now.

[kiwifb]

comment:298

Replying to @sagetrac-tmonteil:

Replying to @kwankyu:

Nils seems to be proposing yet another solution for the present ticket, instead of arguing merits or demerits of the candidate solutions of the ongoing vote.

The vote is to finish endless debates about solutions for this ticket. Proposing another solution at this moment, regardless of its own merits, would just disrupt the vote.

By the result of the vote, this ticket will be closed with adopting one solution from the candidates.

Nils may open another ticket for his idea, but that should build upon the present ticket.

This only shows that asking people to vote on a proposal that was constructed by few other people without letting them to debate and improve the proposal first is a bad habit. We already experienced similar "vote don't discuss" procedure by the past, including the "code of conduct" episode. If a better proposal emerges, i do not see in the name of what it could not be chosen over the previous ones.

Your suggestion has some merits but there is no one size fits all for all cases. Problem: you have people arguing over two possible solution. You invite more sage developers in and now you have people arguing over 8 different solutions (Ok, I am exaggerating but I really see that kind of things as a real possibility). I'll take responsibility for calling the vote on this particular occasion. I was invited on the ticket but it became clear that I wouldn't sway one side over the other and I had nothing much more to offer to the ticket. May be calling someone else would have been better in this case.

[kiwifb]

comment:299

Not a large participation but of those who voted, the hybrid approach was clearly prefered.

• hybrid: 8
• pseudo-package: 2
• copy: 0

[mkoeppe]

comment:300

I reviewed Dima's script while transforming it to the spkg-src script on the current branch.

[mkoeppe]

Reviewer: ..., Matthias Koeppe

[kwankyu]

Changed reviewer from ..., Matthias Koeppe to ..., Matthias Koeppe, Kwankyu Lee

[kwankyu]

comment:302

It looks good to me, except those m4 files of external origin. But we decided to accept them.

[nbruin]

comment:303

Replying to @kwankyu:

It looks good to me, except those m4 files of external origin. But we decided to accept them.

If you put it like that, it's almost an argument for checking them in: by keeping them out of the tree, they'd be invisible for review and yet the files are used to build sagemath. When we put them into the repo at least it's clear for all to see those particular pieces of code that we depend on. We have of course plenty of dependencies that we don't put through our usual review process ...

Anyway, thanks for getting the review process for this on track! I'm happy to see this problem is being resolved now. If people care enough to implement a nicer solution they can do so.

[mkoeppe]

comment:304

Replying to @nbruin:

by keeping them out of the tree, they'd be invisible for review and yet the files are used to build sagemath. When we put them into the repo at least it's clear for all to see those particular pieces of code that we depend on. We have of course plenty of dependencies that we don't put through our usual review process ...

Yes, this is an important point, and it matches an earlier discussion that we had on sage-devel regarding monorepo vs. multirepo strategies (in the context of the Sage library, not the Sage distribution): https://groups.google.com/g/sage-devel/c/AaKxoNQAWMg/m/uY1rW5n0BQAJ

Related: https://groups.google.com/g/sage-devel/c/JBFbtUNhqNU/m/KuySNvTuBAAJ

[mkoeppe]

comment:305

Let's finally get this ticket in please.

[kwankyu]

Changed reviewer from ..., Matthias Koeppe, Kwankyu Lee to François Bissey, ​Dima Pasechnik, Matthias Koeppe, Kwankyu Lee

[mkoeppe]

comment:307

Thanks!

[vbraun]

Changed branch from u/mkoeppe/config/gnulib to 8bf9d97

[mkoeppe]

Changed reviewer from François Bissey, ​Dima Pasechnik, Matthias Koeppe, Kwankyu Lee to François Bissey, Dima Pasechnik, Matthias Koeppe, Kwankyu Lee

[mkoeppe]

Changed commit from 8bf9d97 to none