GNU ftp servers are down

[healther]

Our CI is currently choked up, because we can't fetch from ftp.gnu.org anymore.

$ spack fetch bash
==> Fetching https://ftp.gnu.org/gnu/bash/bash-4.4.12.tar.gz
 
curl: (7) Failed to connect to ftp.gnu.org port 443: Network is unreachable
==> Fetching from https://ftp.gnu.org/gnu/bash/bash-4.4.12.tar.gz failed.
==> Error: All fetchers failed for bash-4.4.12-vhdxkdfbkdiqrkoqi3nz34qzq5qzjhrc

Anyone else seeing this kind of problem? It should be pretty wild spread as

$ grep --exclude *pyc -r ftp.gnu.org var/spack/repos/builtin 
var/spack/repos/builtin/packages/coreutils/package.py:    url      = "http://ftp.gnu.org/gnu/coreutils/coreutils-8.26.tar.xz"
var/spack/repos/builtin/packages/gmake/package.py:    url      = "https://ftp.gnu.org/gnu/make/make-4.2.1.tar.gz"
var/spack/repos/builtin/packages/gmp/package.py:    url      = "https://ftp.gnu.org/gnu/gmp/gmp-6.1.2.tar.bz2"
var/spack/repos/builtin/packages/groff/package.py:    url      = "http://ftp.gnu.org/gnu/groff/groff-1.22.3.tar.gz"
var/spack/repos/builtin/packages/libiconv/package.py:    url      = "http://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.15.tar.gz"
var/spack/repos/builtin/packages/wget/package.py:    url      = "http://ftp.gnu.org/gnu/wget/wget-1.19.1.tar.gz"
var/spack/repos/builtin/packages/parallel/package.py:    url      = "http://ftp.gnu.org/gnu/parallel/parallel-20170122.tar.bz2"
var/spack/repos/builtin/packages/gdb/package.py:    url = "http://ftp.gnu.org/gnu/gdb/gdb-7.10.tar.gz"
var/spack/repos/builtin/packages/aspell6-en/package.py:    url      = "ftp://ftp.gnu.org/gnu/aspell/dict/en/aspell6-en-2017.01.22-0.tar.bz2"
var/spack/repos/builtin/packages/ncurses/package.py:    url      = "http://ftp.gnu.org/pub/gnu/ncurses/ncurses-6.0.tar.gz"
var/spack/repos/builtin/packages/nettle/package.py:    url      = "http://ftp.gnu.org/gnu/nettle/nettle-3.3.tar.gz"
var/spack/repos/builtin/packages/gdbm/package.py:    url      = "http://ftp.gnu.org/gnu/gdbm/gdbm-1.13.tar.gz"
var/spack/repos/builtin/packages/mpfr/package.py:    url      = "https://ftp.gnu.org/gnu/mpfr/mpfr-3.1.5.tar.bz2"
var/spack/repos/builtin/packages/aspell6-es/package.py:    url      = "ftp://ftp.gnu.org/gnu/aspell/dict/es/aspell6-es-1.11-2.tar.bz2"
var/spack/repos/builtin/packages/libunistring/package.py:    url      = "http://ftp.gnu.org/gnu/libunistring/libunistring-0.9.7.tar.xz"
var/spack/repos/builtin/packages/aspell6-de/package.py:    url      = "ftp://ftp.gnu.org/gnu/aspell/dict/de/aspell6-de-20030222-1.tar.bz2"
var/spack/repos/builtin/packages/readline/package.py:    url      = "https://ftp.gnu.org/gnu/readline/readline-7.0.tar.gz"
var/spack/repos/builtin/packages/libsigsegv/package.py:    url      = "https://ftp.gnu.org/gnu/libsigsegv/libsigsegv-2.11.tar.gz"
var/spack/repos/builtin/packages/binutils/package.py:    url      = "https://ftp.gnu.org/gnu/binutils/binutils-2.28.tar.bz2"
var/spack/repos/builtin/packages/gcc/package.py:    url      = 'http://ftp.gnu.org/gnu/gcc/gcc-7.1.0/gcc-7.1.0.tar.bz2'
var/spack/repos/builtin/packages/gcc/package.py:    list_url = 'http://ftp.gnu.org/gnu/gcc/'
var/spack/repos/builtin/packages/gcc/package.py:        url = 'http://ftp.gnu.org/gnu/gcc/gcc-{0}/gcc-{0}.tar.{1}'
var/spack/repos/builtin/packages/libmatheval/package.py:    url      = "https://ftp.gnu.org/gnu/libmatheval/libmatheval-1.1.11.tar.gz"
var/spack/repos/builtin/packages/gperf/package.py:    url      = "http://ftp.gnu.org/pub/gnu/gperf/gperf-3.0.4.tar.gz"
var/spack/repos/builtin/packages/emacs/package.py:    url      = "http://ftp.gnu.org/gnu/emacs/emacs-24.5.tar.gz"
var/spack/repos/builtin/packages/bash/package.py:    url      = "https://ftp.gnu.org/gnu/bash/bash-4.4.tar.gz"
var/spack/repos/builtin/packages/units/package.py:    url      = "https://ftp.gnu.org/gnu/units/units-2.13.tar.gz"
var/spack/repos/builtin/packages/patch/package.py:    url      = "http://ftp.gnu.org/gnu/patch/patch-2.7.5.tar.xz"
var/spack/repos/builtin/packages/glpk/package.py:    url = "http://ftp.gnu.org/gnu/glpk/glpk-4.57.tar.gz"
var/spack/repos/builtin/packages/octave/package.py:    url      = "https://ftp.gnu.org/gnu/octave/octave-4.0.0.tar.gz"
var/spack/repos/builtin/packages/gawk/package.py:    url      = "http://ftp.gnu.org/gnu/gawk/gawk-4.1.4.tar.xz"
var/spack/repos/builtin/packages/mpc/package.py:    url      = "https://ftp.gnu.org/gnu/mpc/mpc-1.0.2.tar.gz"
var/spack/repos/builtin/packages/mpc/package.py:            return "https://ftp.gnu.org/gnu/mpc/mpc-%s.tar.gz" % version
var/spack/repos/builtin/packages/screen/package.py:    url      = "http://ftp.gnu.org/gnu/screen/screen-4.3.1.tar.gz"
var/spack/repos/builtin/packages/autoconf/package.py:    url = 'http://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz'
var/spack/repos/builtin/packages/m4/package.py:    url      = "https://ftp.gnu.org/gnu/m4/m4-1.4.18.tar.gz"
var/spack/repos/builtin/packages/cvs/package.py:    url      = "https://ftp.gnu.org/non-gnu/cvs/source/feature/1.12.13/cvs-1.12.13.tar.bz2"
var/spack/repos/builtin/packages/texinfo/package.py:    url      = "http://ftp.gnu.org/gnu/texinfo/texinfo-6.0.tar.gz"
var/spack/repos/builtin/packages/tar/package.py:    url = "https://ftp.gnu.org/gnu/tar/tar-1.29.tar.gz"
var/spack/repos/builtin/packages/datamash/package.py:    url      = "http://ftp.gnu.org/gnu/datamash/datamash-1.0.5.tar.gz"
var/spack/repos/builtin/packages/autogen/package.py:    url      = "https://ftp.gnu.org/gnu/autogen/rel5.18.12/autogen-5.18.12.tar.gz"
var/spack/repos/builtin/packages/autogen/package.py:    list_url = "https://ftp.gnu.org/gnu/autogen"
var/spack/repos/builtin/packages/guile/package.py:    url      = "https://ftp.gnu.org/gnu/guile/guile-2.2.0.tar.gz"
var/spack/repos/builtin/packages/bison/package.py:    url      = "http://ftp.gnu.org/gnu/bison/bison-3.0.4.tar.gz"
var/spack/repos/builtin/packages/automake/package.py:    url      = 'http://ftp.gnu.org/gnu/automake/automake-1.15.tar.gz'

we have quite a number of packages that use this source

[healther]

Okay so I worked around this particular problem by adding all sources to our local mirror (obtained from mirrors of ftp.gnu.org). I know that there is the url_list attribute, can we use this to provide fallback download locations?

[bassenj]

I had the same problem. The list_url approach worked for me. What do you think about allowing list_url as property in packages.yaml? Then it would easier to change to a local mirror without modifying the builtin packages.

[healther]

You can add a local mirror via spack mirror, in that case spack will always try to fetch it from your mirror first. I was more thinking about adding a bunch of the official mirrors to each ftp.gnu.org package...
And generally I don't think it's a good idea to create multiple locations where information about packages are stored.

[adamjstewart]

The list_url thing is kind of a hack. list_url is intended to be a secondary page for a software package that lists links to downloads. It also happens to function as a backup mirror if the primary URL no longer works.

What we should really do is add a new attribute that lists a backup mirror to download from. This was discussed to some extent in #1090, but no progress was ever made, and it may never happen. Once #7121 goes online, I think it will mostly be irrelevant anyway.

Feel free to submit a PR that changes the primary download URL to a mirror for now. Hopefully #7121 will solve this problem in the long run.

[tz-rrze]

For ftp.gnu.org it might be a good idea to generally use ftpmirror.gnu.org instead, cf. https://www.gnu.org/server/mirror.en.html

[adamjstewart]

That definitely seems more reliable if it automatically ignores servers that are offline and finds a working link.

[adamjstewart]

By the way, it looks like ftp.gnu.org is back online now. But I would still welcome a PR that switches everything to use the official GNU mirrors.