INSTALL: Script fails at PETSc while downloading HWLOC

[jonathonmckay]

Describe the error
The installation script fails at the "Installing petsc" phase, while the PETSc script is trying to download HWLOC (Specifically: https://web.cels.anl.gov/projects/petsc/download/externalpackages/hwloc-2.10.0.tar.gz)

Steps to Reproduce
Following steps at https://www.firedrakeproject.org/download.html
Specifically:

curl -O https://raw.githubusercontent.com/firedrakeproject/firedrake/master/scripts/firedrake-install
python3 firedrake-install --verbose --no-package-manager --disable-ssh

Expected behavior
The install script should run and download all required files

Error message
Terminal Messages

Installing petsc/
Depending on your platform, PETSc may take an hour or more to build!
Traceback (most recent call last):
  File "/minerva/source/firedrake/firedrake-install", line 1816, in <module>
    install("petsc/")
  File "/minerva/source/firedrake/firedrake-install", line 1043, in install
    build_and_install_petsc()
  File "/minerva/source/firedrake/firedrake-install", line 1155, in build_and_install_petsc
    check_call([python, "./configure", "PETSC_DIR={}".format(petsc_dir), "PETSC_ARCH={}".format(petsc_arch)] + petsc_options)
  File "/minerva/source/firedrake/firedrake-install", line 672, in check_call
    log.debug(subprocess.check_output(arguments, stderr=subprocess.STDOUT, env=os.environ).decode())
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/minerva/opt/python-3.11.6/lib/python3.11/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/minerva/opt/python-3.11.6/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/minerva/source/firedrake/firedrake/bin/python', './configure', 'PETSC_DIR=/minerva/sourc'PETSC_ARCH=default', '--with-c2html=0', '--with-fortran-bindings=0', '--download-bison', '--with-zlib', '--download-netcdf', '--dotix', '--download-superlu_dist', '--download-mpich', '--download-suitesparse', '--download-pnetcdf', '--download-ptscotch', '--down-download-hypre', '--with-debugging=0', '--download-hwloc', '--with-shared-libraries=1', '--download-metis', '--download-cmake']' r

firedrake-install.log (Tail end)

=============================================================================================
                 Running make install on BISON; this may take several minutes
=============================================================================================
=============================================================================================
  Trying to download https://download.open-mpi.org/release/hwloc/v2.10/hwloc-2.10.0.tar.gz
  for HWLOC
=============================================================================================
=============================================================================================
  Trying to download
  https://web.cels.anl.gov/projects/petsc/download/externalpackages/hwloc-2.10.0.tar.gz for
  HWLOC
=============================================================================================

*********************************************************************************************
           UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details):
---------------------------------------------------------------------------------------------
  Error during download/extract/detection of HWLOC:
  Unable to download package HWLOC from:
  https://download.open-mpi.org/release/hwloc/v2.10/hwloc-2.10.0.tar.gz
  * If URL specified manually - perhaps there is a typo?
  * If your network is disconnected - please reconnect and rerun ./configure
  * Or perhaps you have a firewall blocking the download
  * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct
  you what packages to download manually
  * or you can download the above URL manually, to
  /yourselectedlocation/hwloc-2.10.0.tar.gz
  and use the configure option:
  --download-hwloc=/yourselectedlocation/hwloc-2.10.0.tar.gz
  Unable to download package HWLOC from:
  https://web.cels.anl.gov/projects/petsc/download/externalpackages/hwloc-2.10.0.tar.gz
  * If URL specified manually - perhaps there is a typo?
  * If your network is disconnected - please reconnect and rerun ./configure
  * Or perhaps you have a firewall blocking the download
  * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct
  you what packages to download manually
  * or you can download the above URL manually, to
  /yourselectedlocation/hwloc-2.10.0.tar.gz
  and use the configure option:
  --download-hwloc=/yourselectedlocation/hwloc-2.10.0.tar.gz
*********************************************************************************************

Environment:

• OS: CentOS 7.5
• Python version: 3.11.6

Additional context
The link that fails is https://web.cels.anl.gov/projects/petsc/download/externalpackages/hwloc-2.10.0.tar.gz
I have tried this URL on several computers. All return a "404".

[connorjward]

There appear to be two links that they search to find the URL. The former (https://download.open-mpi.org/release/hwloc/v2.10/hwloc-2.10.0.tar.gz) appears to work. It's possible that that server or your network experienced a temporary issue. Can you please try again?

[jonathonmckay]

Unfortunately, I have been trying this off an on for the last month or so, hoping that I just caught someone with a missing file. It's never worked.

I have also noticed that it is trying two places. The download.open-mpi.org URL works when I try it manually,but the script seems to be hung up on the web.cels.anl.gov URL not working.

Also the help message from the log says Unable to download package HWLOC from: https://download.open-mpi.org/release/hwloc/v2.10/hwloc-2.10.0.tar.gz, yet this is the URL that works, and was the one tried first (according to the order of the log). While the web.cels.anl.gov URL is the one the doesn't work, its not mentioned in the help message (even though the help message happens immediately after trying this URL (again, according to the order of the log)). The order of the messages is kinda odd.

[connorjward]

Strange. Do you have a non-standard network setup? Or some sort of firewall?

This is really an issue with PETSc instead of Firedrake so I would recommend asking for help there. If you want to reproduce the issue without using Firedrake you should be able to run ./configure --download-hwloc from inside the firedrake/src/petsc directory.

Lastly, we updated our fork of PETSc last Friday. I think that is unlikely to have fixed this issue, but you could try a fresh install and see if things work.

[jonathonmckay]

We do in-fact have non-standard network with a very annoying firewall. But this is a know problem with this sort of automatic install script, so I tried downloading the failing hwloc file with a computer on the guest network, and my personal phone. All got the same problem.

Yeah, I know this problem is ultimately in the script provided by PETSc, but I didn't know how much control your script was exerting over it. I will check their forums. I'll post here if I get any meaningful information there for completeness.

While I've been trying this for over a month, I downloaded a new "firedrake-install" script on 2024-06-21.

[connorjward]

We do in-fact have non-standard network with a very annoying firewall. But this is a know problem with this sort of automatic install script, so I tried downloading the failing hwloc file with a computer on the guest network, and my personal phone. All got the same problem.

Yeah, I know this problem is ultimately in the script provided by PETSc, but I didn't know how much control your script was exerting over it. I will check their forums. I'll post here if I get any meaningful information there for completeness.

While I've been trying this for over a month, I downloaded a new "firedrake-install" script on 2024-06-21.

It is also possible to install PETSc separately from the rest of the installation script. If you run firedrake-install --show-petsc-configure-options you can see what flags we pass to PETSc. It is quite annoying and manual but you could then clone your own PETSc, download all of the packages manually and then run something like

$ ./configure --hwloc-dir=/path/to/downloaded/tar/file ...etc

I don't really recommend this though. Figuring out how to satisfy the firewall is definitely a better option.

Alternatively we do have Docker and Singularity containers that might be suitable.

[balay]

Actually the option here is:

$ ./configure --download-hwloc=/path/to/downloaded/tar/file

Alternative:

  --with-packages-download-dir=<dir>
       Skip network download of package tarballs and locate them in specified dir. If not found in dir, print package URL - so it can be obtained manually.

[jonathonmckay]

With the extensive help of @balay (99% sure it's the same person) over on the PETSc Discord, I was able work out the problem was our annoying corporate firewall.
The solution was to set export SSL_CERT_FILE=/path/to/file.crt to a known certificate file I have for our annoying corporate firewall.
This solved the HWLOC problem.

(It's now having the same problem with the PETSc script downloading HDF5, but again its likely a problem with our annoying corporate firewall, which I will have to muddle through now that I know that's what the problem is.)

For reference, here's the link to the Discord thread:
https://discord.com/channels/1119324534303109172/1254799088495362080

Thank you all for your help.

[connorjward]

Glad to hear you're making progress! Thanks for sharing these tips.

[balay]

BTW: something firedrake installer could do is accept additional PETSc configure options - similar to the pip installer - via an env variable - PETSC_CONFIGURE_OPTIONS

$ git grep PETSC_CONFIGURE_OPTIONS
setup.py:  Provide any ``PETSc`` ./configure options using the environmental variable ``PETSC_CONFIGURE_OPTIONS``.
setup.py:    options = os.environ.get('PETSC_CONFIGURE_OPTIONS', '')

If they are append to the end of firedrake generated options - they can over-ride the defaults set here.

[connorjward]

BTW: something firedrake installer could do is accept additional PETSc configure options - similar to the pip installer - via an env variable - PETSC_CONFIGURE_OPTIONS

$ git grep PETSC_CONFIGURE_OPTIONS
setup.py:  Provide any ``PETSc`` ./configure options using the environmental variable ``PETSC_CONFIGURE_OPTIONS``.
setup.py:    options = os.environ.get('PETSC_CONFIGURE_OPTIONS', '')

If they are append to the end of firedrake generated options - they can over-ride the defaults set here.

Firedrake already supports PETSC_CONFIGURE_OPTIONS like this.

[balay]

Ah - then @jonathonmckay can use this feature to add in required petsc configure options to overcome the firewall issues.

[jonathonmckay]

Oh yeah. python3 firedrake-install --help shows this at the end:

The installer will ensure that the required configuration options are
passed to PETSc. In addition, any configure options which you provide
in the PETSC_CONFIGURE_OPTIONS environment variable will be
honoured.

Guess I didn't read the whole thing, or didn't understand what it would mean to my specific case when I did.
Thank you. This will likely come in handy.

[jonathonmckay]

Yes, setting PETSC_CONFIGURE_OPTIONS clears up the HDF5 problem.
And I've learned you can stack the settings (because our annoying corporate firewall is giving me grief for other automatic downloads). You can do this:

export PETSC_CONFIGURE_OPTIONS="--download-hdf5=/path/to/downloads/hdf5-1.14.3-p1.tar.bz2 --download-netcdf=/path/to/downloads/netcdf-c-4.9.2-p1.tar.gz"

(Manually download the required files first, of course)
Then run python3 firedrake-install after.