archlinux: Add missing system packages (arch.txt), add arch section to installation manual, fix detection of system packages

[mkoeppe]

https://groups.google.com/d/msg/sage-devel/JgArPVx7dwg/gMxeB4b7AgAJ

To test:

  tox -e docker-archlinux-latest-standard

CC: @videlec @antonio-rojas @dimpase

Component: porting

Author: Matthias Koeppe, Vincent Delecroix

Branch/Commit: 338468f

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/29366

[dimpase]

comment:3

Attachment: config.log

not sure there is much on Sage's side that can be done - apart from keeping correct lists of packages to install.

[videlec]

comment:4

Indeed, I would like to be sure that it is not on Sage side and have an up to date developer manual for the relevant packages.

[videlec]

Attachment: configure-output.log

[mkoeppe]

comment:5

In case you haven't seen it yet: the system package lists in the developer's manual are now autogenerated. But adding some Arch specific instructions would be a useful addition

[mkoeppe]

comment:6

Installation manual, that is

[dimpase]

comment:7

Well, config.log is clear enough on e.g. what's missing w.r.t. Pari:
namely, install elldata, galpol, seadata, as seen in

configure:14982: checking is pari_elldata installed?
1735	  ***   at top-level: r=ellinit("11a1");r[11]
1736	  ***                   ^---------------------
1737	  *** ellinit: error opening elldata file: `/usr/share/pari/elldata/ell0'.
1738	configure:14989: result: no; cannot use system pari/GP without elldata package
1739	configure:14991: Install elldata package and reconfigure.
1740	configure:14993: Otherwise Sage will build its own pari/GP.
1741	configure:14997: checking is pari_galdata installed?
1742	configure:15001: result: yes
1743	configure:15012: checking is pari_galpol installed?
1744	  ***   at top-level: galoisgetname(12,1)=="C3 : C4"
1745	  ***                 ^------------------------------
1746	  *** galoisgetname: error opening galpol file: `/usr/share/pari/galpol/12/nb'.
1747	configure:15019: result: no; cannot use system pari/GP without galpol package
1748	configure:15021: Install galpol package and reconfigure.
1749	configure:15023: Otherwise Sage will build its own pari/GP.
1750	configure:15027: checking is pari_seadata installed?
1751	  ***   at top-level: poldegree(ellmodulareqn(211)[1])
1752	  ***                           ^----------------------
1753	  *** ellmodulareqn: error opening seadata file: `/usr/share/pari/seadata/sea211'.
1754	configure:15034: result: no; cannot use system pari/GP without seadata package
1755	configure:15036: Install seadata package and reconfigure.
1756	configure:15038: Otherwise Sage will build its own pari/GP.
1757	configure:15266: no suitable system package found for SPKG pari
1758	configure:15284: result: using Sage's pari SPKG

[antonio-rojas]

comment:8

Nothing to fix on Arch side either - givaro is not used because you have an older version on /usr/local which is detected first, and pari is not used because of missing optional packages

[mkoeppe]

comment:9

Could you also take a look at the build logs at https://github.com/mkoeppe/sage/runs/518106289 please?

[antonio-rojas]

comment:10

Replying to @mkoeppe:

Could you also take a look at the build logs at https://github.com/mkoeppe/sage/runs/518106289 please?

Looks like Arch's ntl is not accepted because it is built with threads, which in turn causes all its reverse dependencies to be compiled. Other packages (fplll, ecm) are simply not installed on the build machine.

[mkoeppe]

comment:11

Replying to @antonio-rojas:

Replying to @mkoeppe:

Could you also take a look at the build logs at https://github.com/mkoeppe/sage/runs/518106289 please?

Looks like Arch's ntl is not accepted because it is built with threads, which in turn causes all its reverse dependencies to be compiled.

Right, this build is using #29339 - Fix NTL spkg-configure.m4 so it rejects NTLs built with NTL_THREADS

Other packages (fplll, ecm) are simply not installed on the build machine.

OK, this means we need to add arch.txt files for these packages

[mkoeppe]

comment:12

and for more of the following listed here:

configure: notice: the following SPKGs did not find equivalent system packages: arb cbc cliquer cmake eclib ecm flint fplll git givaro gp2c isl lcalc libatomic_ops libsemigroups mpfi ninja_build ntl pari pari_elldata pari_galdata pari_galpol pari_nftables pari_seadata pari_seadata_small perl_term_readline_gnu yasm zeromq

[dimpase]

comment:13

Replying to @mkoeppe:

Replying to @antonio-rojas:

Replying to @mkoeppe:

Could you also take a look at the build logs at https://github.com/mkoeppe/sage/runs/518106289 please?

Looks like Arch's ntl is not accepted because it is built with threads, which in turn causes all its reverse dependencies to be compiled.

Right, this build is using #29339 - Fix NTL spkg-configure.m4 so it rejects NTLs built with NTL_THREADS

I believe that I was able to use Arch's NTL in builds few betas ago.
This again points at #29339 as being too drastic in its approach.

[antonio-rojas]

comment:14

Replying to @dimpase:

Replying to @mkoeppe:

Replying to @antonio-rojas:

Replying to @mkoeppe:

Could you also take a look at the build logs at https://github.com/mkoeppe/sage/runs/518106289 please?

Looks like Arch's ntl is not accepted because it is built with threads, which in turn causes all its reverse dependencies to be compiled.

Right, this build is using #29339 - Fix NTL spkg-configure.m4 so it rejects NTLs built with NTL_THREADS

I believe that I was able to use Arch's NTL in builds few betas ago.
This again points at #29339 as being too drastic in its approach.

And the distro sagemath package has been using multithreaded NTL for years with no issues whatsoever

[mkoeppe]

comment:15

Replying to @antonio-rojas:

And the distro sagemath package has been using multithreaded NTL for years with no issues whatsoever

Well, it is currently disabled as of last NTL upgrade #20590.

[antonio-rojas]

comment:16

Replying to @mkoeppe:

Replying to @antonio-rojas:

And the distro sagemath package has been using multithreaded NTL for years with no issues whatsoever

Well, it is currently disabled as of last NTL upgrade #20590.

I mean the Arch sagemath package, not sage-the-distro

[mkoeppe]

comment:17

Ah, thanks for the clarification

[mkoeppe]

Branch: u/mkoeppe/archlinux__add_missing_system_packages__arch_txt___add_arch_section_to_installation_manual__fix_detection_of_system_packages

[videlec]

comment:21

Replying to @mkoeppe:

and for more of the following listed here:

configure: notice: the following SPKGs did not find equivalent system packages: arb cbc cliquer cmake eclib ecm flint fplll git givaro gp2c isl lcalc libatomic_ops libsemigroups mpfi ninja_build ntl pari pari_elldata pari_galdata pari_galpol pari_nftables pari_seadata pari_seadata_small perl_term_readline_gnu yasm zeromq

Note that the packages pari-seadata and pari-seadata-small conflict.

After removing the pkgconfig file of givaro in /usr/local (my bad) and installing the three pari packages (pari-seadata, pari-galpol, pari-elldata), everything is now taken from the system! Thanks.

---

New commits:

9a21aa7

build/pkgs/readline/distros/arch.txt: New

[videlec]

Commit: 9a21aa7

[mkoeppe]

comment:22

So what are we missing in our arch.txt files?

[videlec]

comment:23

I don't understand what these arch.txt are supposed to say. It looks like a kind of translation sage package -> arch package. However, for pari to be recognized by the sagemath configure, it needs to be installed with the optional packages pari-seadata, pari-galpol, pari-elldata. Where is this specified?

[dimpase]

comment:24

Here is an example from build/pkgs/pari/distros/debian.txt:

pari-gp2c libpari-dev
# #29319: cypari2 needs gphelp at installation time
pari-doc
# We add these data packages because they are checked by spkg-configure.m4
pari-elldata pari-galdata pari-galpol pari-seadata

Put the equivalent of these Debian package names
into build/pkgs/pari/distros/arch.txt

[videlec]

comment:25

Where is the format documented?

[videlec]

Changed commit from 2acd21d to none

[videlec]

Changed branch from u/mkoeppe/archlinux__add_missing_system_packages__arch_txt___add_arch_section_to_installation_manual__fix_detection_of_system_packages to public/29366

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

9a21aa7	build/pkgs/readline/distros/arch.txt: New
2acd21d	src/doc/bootstrap: Also process arch.txt

[sagetrac-git]

Commit: 2acd21d

[sagetrac-git]

Changed commit from 2acd21d to 187761e

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

187761e

packages for pari

[videlec]

comment:32

Also, both pari-seadata and pari-elldata are from AUR.

[mkoeppe]

comment:33

$ tox -e docker-archlinux-latest-standard
docker-archlinux-latest-standard run-test-pre: PYTHONHASHSEED='2835960636'
docker-archlinux-latest-standard run-test: commands[0] | bash -c 'build/bin/write-dockerfile.sh arch "standard" yes no > /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/docker-archlinux-latest-standard/Dockerfile'
docker-archlinux-latest-standard run-test: commands[1] | docker build . -f /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/docker-archlinux-latest-standard/Dockerfile --build-arg EXTRA_CONFIGURE_ARGS= --build-arg BASE_IMAGE=archlinux:latest --build-arg http_proxy=http://host.docker.internal:3128 --build-arg https_proxy=http://host.docker.internal:3128
Sending build context to Docker daemon  179.1MB
Step 1/23 : ARG BASE_IMAGE=archlinux:latest
Step 2/23 : FROM ${BASE_IMAGE}
 ---> 0152bf6f0800
Step 3/23 : ENV PACKAGES="binutils make m4 perl python tar bc gcc which gettext autoconf automake libtool pkg-config arb boost eclib gcc gf2x gfan gcc-fortran glpk gsl lcalc gd lrcalc m4ri m4rie nauty openblas lapack cblas pari pari-galdata pari-seadata pari-elldata pari-galdata pari-seadata-small patch planarity r readline rankwidth sqlite3 symmetrica tachyon"
 ---> Using cache
 ---> 61ceb4b4c200
Step 4/23 : RUN  pacman -Syu --noconfirm binutils make m4 perl python tar bc gcc which gettext autoconf automake libtool pkg-config arb boost eclib gcc gf2x gfan gcc-fortran glpk gsl lcalc gd lrcalc m4ri m4rie nauty openblas lapack cblas pari pari-galdata pari-seadata pari-elldata pari-galdata pari-seadata-small patch planarity r readline rankwidth sqlite3 symmetrica tachyon
 ---> Running in a4547b23d1cc
:: Synchronizing package databases...
downloading core.db...
downloading extra.db...
downloading community.db...
warning: perl-5.30.1-1 is up to date -- reinstalling
warning: tar-1.32-3 is up to date -- reinstalling
error: target not found: pari-seadata
error: target not found: pari-elldata
The command '/bin/sh -c pacman -Syu --noconfirm binutils make m4 perl python tar bc gcc which gettext autoconf automake libtool pkg-config arb boost eclib gcc gf2x gfan gcc-fortran glpk gsl lcalc gd lrcalc m4ri m4rie nauty openblas lapack cblas pari pari-galdata pari-seadata pari-elldata pari-galdata pari-seadata-small patch planarity r readline rankwidth sqlite3 symmetrica tachyon' returned a non-zero code: 1
ERROR: InvocationError for command /usr/local/bin/docker build . -f .tox/docker-archlinux-latest-standard/Dockerfile --build-arg EXTRA_CONFIGURE_ARGS= --build-arg BASE_IMAGE=archlinux:latest --build-arg http_proxy=http://host.docker.internal:3128 --build-arg https_proxy=http://host.docker.internal:3128 (exited with code 1)

[dimpase]

comment:34

pacman cannot install packages from AUR, they need to be built from source:
https://wiki.archlinux.org/index.php/Arch_User_Repository#Installing_packages

(in case of these packages "building" is instant though, it just needs different commands to be run)

Yes, it'd be possible to script this, so these packages need to at least be tagged somehow.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

8fdaa9e	build/pkgs/pari/distros/arch.txt: Comment out these packages (uninstallable)
4cd1b55	src/doc/en/installation/source.rst: Include arch*.txt

[sagetrac-git]

Changed commit from 187761e to 4cd1b55

[mkoeppe]

Author: Matthias Koeppe, Vincent Delecroix

[mkoeppe]

comment:36

This is not complete but let's get it into 9.1

[dimpase]

comment:37

there are replacements for pacman which allow automatic install of packages from AUR and the standard repo, such as yay.

so the script can first install yay, and then use it for the rest.

[antonio-rojas]

comment:38

Replying to @dimpase:

there are replacements for pacman which allow automatic install of packages from AUR and the standard repo, such as yay.

so the script can first install yay, and then use it for the rest.

Please don't. AUR is unsupported, user generated content. There's no guarantee that it couldn't be replaced with broken or even malicious content at any time.

If necessary, I could move these packages to the binary repos. But I don't really see the need to check for those at build time, at least on Arch, given that they can be installed by the user at any time after Sage install.

[dimpase]

comment:39

well, if we recommend AUR packages, it's better if we test with them too, no?
But, indeed, if they can be moved to the regular package repo, why not?

[antonio-rojas]

comment:40

Alright, packages are now in the official repos

[sagetrac-git]

Changed commit from 4cd1b55 to 338468f

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

5a62319	build/pkgs/readline/distros/arch.txt: New
1236a4e	src/doc/bootstrap: Also process arch.txt
5e203e9	packages for pari
ba6f6f9	build/pkgs/pari/distros/arch.txt: Comment out these packages (uninstallable)
cefaf09	src/doc/en/installation/source.rst: Include arch*.txt
338468f	build/pkgs/pari*/distros/arch.txt: Update/add

[mkoeppe]

comment:43

Replying to @antonio-rojas:

Alright, packages are now in the official repos

Thanks very much, that's great.

configure now finds the packages!

tox -e docker-archlinux-latest-standard -- base-toolchain
...
Checking whether SageMath should install SPKG pari...
checking installing gmp/mpir or readline? ... no
checking installing PARI/GP packages? ... checking for gp... /usr/sbin/gp
checking for gphelp... /usr/sbin/gphelp
checking whether gphelp has access to the documentation... yes
checking is pari_elldata installed? ... yes
checking is pari_galdata installed? ... yes
checking is pari_galpol installed? ... yes
checking is pari_seadata installed? ... yes
configure: will use system package and not install SPKG pari
using pari/gp from the system

Needs review.

[dimpase]

Reviewer: Dima Pasechnik

[dimpase]

comment:44

lgtm

[mkoeppe]

comment:45

Thank you!

[vbraun]

Changed branch from public/29366 to 338468f