tox.ini: Add centos-6, docker-nobootstrap, manylinux-*

[mkoeppe]

The docker-nobootstrap option uses bootstrap -D; thus, autotools are not required on the host for the build.

This enables building with centos-6 and the manylinux images based on centos-6 (https://github.com/pypa/manylinux).

Examples:

tox -e docker-manylinux-2014-standard
tox -e docker-manylinux-2014-standard-python3.9
tox -e docker-manylinux-2010-standard-i686-python3.8

Depends on #29929

CC: @kliem @slel

Component: porting

Author: Matthias Koeppe

Branch/Commit: 14da64d

Reviewer: Jonathan Kliem

Issue created by migration from https://trac.sagemath.org/ticket/30195

[mkoeppe]

Dependencies: #29929

[mkoeppe]

Author: Matthias Koeppe

[mkoeppe]

Branch: u/mkoeppe/tox_ini__add_manylinux

[mkoeppe]

Commit: 585bdac

[mkoeppe]

comment:3

Branch is 1 commit on top of #29929, which also needs review

---

Last 10 new commits:

25393b0	Handle SAGE_CHECK_PACKAGES in build/make/Makefile.in, not sage-spkg
5aca238	Merge branch 't/30118/handle_sage_check_packages_in_build_make_makefile_in__not_sage_spkg' into t/29929/tox_ini__add_a_macos_environment_without_homebrew__conda
a17c655	m4/sage_spkg_collect.m4: Do not include script packages in SAGE_SDIST_PACKAGES
827923a	bootstrap: Accept 2020s in config.guess version check
c30ac54	tox.ini: Add local-nobootstrap
de6464d	.github/workflows/tox.yml: Add jobs "dist", "local-macos-nohomebrew
775b7f7	Trac #30088: add a few explanatory comments to m4/sage_spkg_collect.m4.
160862f	m4/sage_spkg_collect.m4: Mention sagelib in the comment
d1a2cbf	Merge branch 't/30088/fix__make_sdist___add_test_run_to_github_actions' into t/29929/tox_ini__add_a_macos_environment_without_homebrew__conda
585bdac	tox.ini: Add centos-6, docker-nobootstrap, manylinux-*

[kliem]

comment:4

Can you provide an example of this at work?

According to the readme, only CPython up to 3.8 are provided with manylinux. Why have you configured python3.9?

[kliem]

Reviewer: Jonathan Kliem

[mkoeppe]

comment:5

The readme does not seem to be up to date. On an actual docker image:

[root@4185c8d9310b /]# /opt/python/cp3 TAB
cp35-cp35m/ cp36-cp36m/ cp37-cp37m/ cp38-cp38/  cp39-cp39/  

[kliem]

comment:6

That was one of my guesses.

Do the new environments work with github actions? If you haven't tested it yet, I have free capacities and can just test this ticket.

[mkoeppe]

comment:7

They should work, just add them to tox.yml for testing. I haven't tested them widely yet, only one run of docker-manylinux-2014-standard on my computer. Best to test together with #30173

[kliem]

comment:8

 File "/usr/lib/python3/dist-packages/tox/session.py", line 38, in main
    config = prepare(args)
  File "/usr/lib/python3/dist-packages/tox/session.py", line 26, in prepare
    config = parseconfig(args)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 239, in parseconfig
    parseini(config, inipath)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 760, in __init__
    self.make_envconfig(name, section, reader._subs, config)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 791, in make_envconfig
    res = meth(env_attr.name, env_attr.default)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 964, in getargvlist
    return _ArgvlistReader.getargvlist(self, s)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1134, in getargvlist
    commands.append(cls.processcommand(reader, current_command))
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1164, in processcommand
    new_word = reader._replace(word)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1008, in _replace
    return Replacer(self, crossonly=crossonly).do_replace(value)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1028, in do_replace
    return self.RE_ITEM_REF.sub(self._replace_match, x)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1057, in _replace_match
    return self._replace_env(match)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1077, in _replace_env
    (envkey, envkey))
tox.ConfigError: ConfigError: substitution env:'BOOTSTRAP': unknown environment variable 'BOOTSTRAP'  or recursive definition.
##[error]Process completed with exit code 1.

https://github.com/kliem/sage/runs/897566509?check_suite_focus=true

[sagetrac-git]

Changed commit from 585bdac to 756f310

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

756f310

tox.ini: Initialize the BOOTSTRAP variable

[mkoeppe]

comment:10

Thanks for catching this

[kliem]

comment:11

Centos 6 seems to work in the way that it fails to buil mpir.

There are many errors with manylinux like

   File "/usr/lib/python3/dist-packages/tox/config.py", line 1008, in _replace
    return Replacer(self, crossonly=crossonly).do_replace(value)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1028, in do_replace
    return self.RE_ITEM_REF.sub(self._replace_match, x)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1057, in _replace_match
    return self._replace_env(match)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1071, in _replace_env
    envvalue = self.reader.get_environ_value(envkey)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 903, in get_environ_value
    return self._setenv.get(name)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 286, in get
    self.resolved[name] = res = self.reader._replace(val)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1008, in _replace
    return Replacer(self, crossonly=crossonly).do_replace(value)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1028, in do_replace
    return self.RE_ITEM_REF.sub(self._replace_match, x)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1057, in _replace_match
    return self._replace_env(match)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1077, in _replace_env
    (envkey, envkey))
tox.ConfigError: ConfigError: substitution env:'BASE_IMAGE': unknown environment variable 'BASE_IMAGE'  or recursive definition.

https://github.com/kliem/sage/runs/899346008?check_suite_focus=true

and

 Step 14/18 : RUN ${BOOTSTRAP}
 ---> Running in d5c9ccb6ceb3
rm -rf config configure build/make/Makefile-auto.in
rm -f src/doc/en/installation/*.txt
rm -rf src/doc/en/reference/spkg/*.rst
rm -f src/doc/en/reference/repl/*.txt
Traceback (most recent call last):
  File "/sage/build/bin/sage-download-file", line 27, in ?
    from sage_bootstrap.download.cmdline import run_safe
  File "/sage/build/bin/../sage_bootstrap/download/__init__.py", line 7, in ?
    from sage_bootstrap.download.transfer import Download
  File "/sage/build/bin/../sage_bootstrap/download/transfer.py", line 43
    n = 0 if chunks_so_far == 0 else self.length // 2
           ^
SyntaxError: invalid syntax
Error: downloading configure-339b4599590f3dc3acec5eace4b7a82f210d3219.tar.gz failed

https://github.com/kliem/sage/runs/899346035?check_suite_focus=true

Should they be fixed here?

[mkoeppe]

comment:12

I'll look into the tox config error first.

[mkoeppe]

comment:13

Replying to @kliem:

Centos 6 seems to work in the way that it fails to buil mpir.

Right... errors from an ancient assembler version.

tmp-mul_1.s: Assembler messages:
tmp-mul_1.s:94: Error: no such instruction: `mulx (%rsi),%r9,%r8'
tmp-mul_1.s:95: Error: no such instruction: `mulx 8(%rsi),%r11,%r10'
tmp-mul_1.s:96: Error: no such instruction: `mulx 16(%rsi),%rcx,%r12'

We'll definitely not try to support centos-6 and shouldn't be adding it to tox.yml.

[mkoeppe]

comment:14

Replying to @kliem:

Centos 6 seems to work in the way that it fails to buil mpir.

There are many errors with manylinux like

tox.ConfigError: ConfigError: substitution env:'BASE_IMAGE': unknown environment variable 'BASE_IMAGE'  or recursive definition.

This is just caused by me not adding documentation and leaving you to guess what is supported. manylinux is not a configuration - only manylinux-1, manylinux-2010, manylinux-2014 are.

I'll make manylinux an alias for manylinux-2014 and add some documentation.

[sagetrac-git]

Changed commit from 756f310 to 8c13ac7

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

8c13ac7

tox.ini: Make manylinux default to manylinux-2014, add documentation

[mkoeppe]

comment:16

Replying to @kliem:

  File "/sage/build/bin/../sage_bootstrap/download/transfer.py", line 43
    n = 0 if chunks_so_far == 0 else self.length // 2
           ^
SyntaxError: invalid syntax
Error: downloading configure-339b4599590f3dc3acec5eace4b7a82f210d3219.tar.gz failed

https://github.com/kliem/sage/runs/899346035?check_suite_focus=true

The added comments also explain this one -- manylinux-1 is too old, after all, and sage cannot build on it

[mkoeppe]

comment:17

On manylinux-2010-minimal (https://github.com/kliem/sage/runs/899346056), building python3 fails with ./python: error while loading shared libraries: libcrypt.so.2: cannot open shared object file: No such file or directory (related to #29012?)
... but actually it was intended to use the installed python3 in /opt. Looks like I need to make adjustments for -2010.

[sagetrac-git]

Changed commit from 8c13ac7 to 2747c37

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

4c670d3	build/pkgs/xz/spkg-configure.m4: Slightly lower the xz version bound set in #26286, for manylinux-2010
2f43164	tox.ini [manylinux]: Use --with-system-python3=force
2747c37	tox.ini [manylinux]: Patch the downloaded configure file so that system xz is accepted

[mkoeppe]

comment:20

When you have a chance, please test this version again. See above on my comments on the supported tox environments. Also note the images for non-x86 archs can't work on Linux Docker (and thus on GH Actions) because they are not multi-arch images. They work on macOS Docker Desktop because they are running inside of an emulator.

[kliem]

comment:21

https://github.com/kliem/sage/actions/runs/180807074

That is what I'm testing for now:

 tox_system_factor: [ubuntu-focal, debian-buster, centos-8, manylinux-1, manylinux-2010, manylinux-2014, manylinux-2014-python3.7]

manylinux-1 will of course fail.

[kliem]

comment:22

I'm getting configure failures with minimal. Will this be a subject of this ticket or of another one?

Otherwise it's looking fine.

[sagetrac-git]

Changed commit from 2747c37 to 14da64d

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

14da64d

tox.ini [manylinux]: Do not use --with-system-python3=force for -minimal

[kliem]

comment:24

manylinux-2014-python3.7, minimal still fails at configuration.

https://github.com/kliem/sage/runs/908012032?check_suite_focus=true

Maybe I just got the name scheme wrong.

[mkoeppe]

comment:25

minimal with python3.7 cannot work because python3 dependencies such as sqlite3 are not available from the system and therefore it cannot use system python3.

[kliem]

comment:26

https://github.com/kliem/sage/actions/runs/181590963

This seems to work well. At least according to the timings we are well in the build process. Any failures are probably subject to different tickets now.

If the run turns out to your satisfaction, you can put it on positive review on my behalf.

[mkoeppe]

comment:27

Thank you!

[vbraun]

Changed branch from u/mkoeppe/tox_ini__add_manylinux to 14da64d