flux-pmix doesn't install without fortran (that it doesn't need)

[vsoch]

As of tonight's build, flux-pmix is broken in develop (we test the build nightly):

==> Warning: Module file /opt/spack/share/spack/modules/linux-ubuntu22.04-x86_64_v4/py-setuptools-65.5.0-gcc-12.1.0-lkm7fbw exists and will not be overwritten
==> Warning: Module file /opt/spack/share/spack/modules/linux-ubuntu22.04-x86_64_v4/py-wheel-0.37.1-gcc-12.1.0-4fpohtv exists and will not be overwritten
==> Error: InstallError: OpenMPI requires both C and Fortran compilers!

/opt/spack/var/spack/repos/builtin/packages/openmpi/package.py:898, in die_without_fortran:
        895        # dependencies depends_on('mpi'), require Fortran compiler to
        896        # avoid delayed build errors in dependents.
        897        if (self.compiler.f77 is None) or (self.compiler.fc is None):
  >>    898            raise InstallError("OpenMPI requires both C and Fortran compilers!")

See build log for details:
  /tmp/runner/spack-stage/spack-stage-openmpi-4.1.4-klxdtniuzyplfgfzxs22pa3xrs2y22bs/spack-build-out.txt

==> Warning: Skipping build of flux-pmix-0.2.0-fxptrbvk7fly5r4oroa3yut2esvxurpq since openmpi-4.1.4-klxdtniuzyplfgfzxs22pa3xrs2y22bs failed

And see https://github.com/flux-framework/spack/actions/runs/3598832757/jobs/6061943835

Possibly related: 7423f52

Normally for this error I have to do a custom spack.yaml that defines a fortran compiler (because spack never finds them) but this is worrysome become a spack install flux-pmix should work - indeed before today it did. Perhaps whatever change was done to require this could make this new configuration optional? Thanks!

Update with suggestion: I do think it's the linked merge. My suggested fix is that the pmix python bindings should be optional. Adding more complexity to a build means a potential for more errors, and unless it's absolutely needed, should be optional to take the most conservative approach.

[alalazo]

Update with suggestion: I do think it's the linked merge. My suggested fix is that the pmix python bindings should be optional. Adding more complexity to a build means a potential for more errors, and unless it's absolutely needed, should be optional to take the most conservative approach.

The Python bindings are optional in #34107 since they are gated by:

variant("python", default=False, when="@4.1.2:", description="Enable python bindigs")

[tgamblin]

The only difference I see in the result of spack spec flux-pmix before and after 7423f52 is that on 7423f52, the pmix python variant exists but is off by default. The pmix configuration is:

flux-pmix@0.2.0%gcc@11.3.0 build_system=autotools arch=linux-ubuntu22.04-aarch64
...
    ^pmix@4.1.2%gcc@11.3.0~docs+pmi_backwards_compatibility~python~restful build_system=autotools arch=linux-ubuntu22.04-aarch64
        ^libevent@2.1.12%gcc@11.3.0+openssl build_system=autotools arch=linux-ubuntu22.04-aarch64

python is off (~python).

The error is happening in openmpi, which is a hard dependency of flux-pmix in both 7423f52^ and 7423f52. I'm not seeing how 7423f52 would cause this. If it was building with openmpi before I think it would've had the same error.

[vsoch]

If it was building with openmpi before I think it would've had the same error.

It did not - it's been building nightly for months now and this is the first time. I'm not sure why it is just showing up now - we do use a cache but it's definitely not years old (e.g., when this was added https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/openmpi/package.py#L893)

pinging @trws for comment. I'm fixing it by adding a parameter to install fortran to our build action. Maybe someday these compilers could be packages... looks longingly into the distance 🌆

[trws]

The “destroy build because someone might possibly use Fortran downstream” function is only called if the auto record phase is run, and that phase only runs on a specific branch or when spack determines auto record has to be run for some reason. My current theory is something changed that made spack decide to rerun the autotools on openmpi in this specific case, so we start getting the error.

TBH I can’t wait for language dependencies so we can give people a “no you can’t build something that needs Fortran without a Fortran compiler” error at concretization time. Then we can delete functions like this that try to protect other packages from errors downstream by checking for things they may not need. Kinda in favor of deleting the function anyway really, it doesn’t belong there.

[vsoch]

The plot thickens! To be clear, this won't fix the bug here so spack install flux-pmix works, but will help our nightly automated builds to run. So when we add gfortran before we install spack, it resulted in that error, appearing that the compiler wasn't detected. but actually it was -

compilers:
- compiler:
    spec: clang@12.0.1
    paths:
      cc: /usr/bin/clang-12
      cxx: /usr/bin/clang++-12
      f77: null
      fc: null
    flags: {}
    operating_system: ubuntu22.04
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []
- compiler:
    spec: clang@13.0.1
    paths:
      cc: /usr/bin/clang-13
      cxx: /usr/bin/clang++-13
      f77: null
      fc: null
    flags: {}
    operating_system: ubuntu22.04
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []
- compiler:
    spec: clang@14.0.0
    paths:
      cc: /usr/bin/clang
      cxx: /usr/bin/clang++
      f77: null
      fc: null
    flags: {}
    operating_system: ubuntu22.04
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []
- compiler:
    spec: gcc@10.4.0
    paths:
      cc: /usr/local/bin/gcc
      cxx: /usr/local/bin/g++
      f77: /usr/local/bin/gfortran
      fc: /usr/local/bin/gfortran
    flags: {}
    operating_system: ubuntu22.04
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []
- compiler:
    spec: gcc@11.3.0
    paths:
      cc: /usr/bin/gcc
      cxx: /usr/bin/g++
      f77: /usr/bin/gfortran
      fc: /usr/bin/gfortran
    flags: {}
    operating_system: ubuntu22.04
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []
- compiler:
    spec: gcc@12.1.0
    paths:
      cc: /usr/bin/gcc-12
      cxx: /usr/bin/g++-12
      f77: null
      fc: null
    flags: {}
    operating_system: ubuntu22.04
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []
- compiler:
    spec: gcc@9.5.0
    paths:
      cc: /usr/bin/gcc-9
      cxx: /usr/bin/g++-9
      f77: /usr/bin/gfortran-9
      fc: /usr/bin/gfortran-9
    flags: {}
    operating_system: ubuntu22.04
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []

but just not for the latest compiler being used, gcc 12 - and that's what everything is using 😭

Warning: Module file /opt/spack/share/spack/modules/linux-ubuntu22.04-x86_64_v4/py-wheel-0.37.1-gcc-12.1.0-4fpohtv exists and will not be overwritten

So I don't want to force using an older compiler - I think my questions are why gfortran isn't added / supported for gcc 12? And/or is there a way (outside of providing on the command line) to tell spack to choose a compiler that has everything it needs (because clearly it's there - if we have a compiler that works why should the build bail out?) 🤔

[vsoch]

Okay the workaround for this was to:

1. install fortran compilers
2. force spack to use an older version since newer was not detected, e.g.

spack config add "packages:all:compiler:[gcc@11.3.0]"

That will get the build working. I've updated the PR to reflect the remaining issue with respect to building flux-pmix - no rush in fixing this, if there is a label for "TODO when compilers are packages" this would be a good contender!

[alalazo]

I think my questions are why gfortran isn't added / supported for gcc 12?

Do you have a gfortran-12 installed? If so, can you post the output of:

$ gfortran --version

? (We use regexes to extract the version from that output)

[tgamblin]

My current theory is something changed that made spack decide to rerun the autotools on openmpi in this specific case, so we start getting the error.

I’m interested in this, since it sounds like the selected version of openmpi changed from some release version to master/develop/etc. can someone verify that? Do we have the full flux-pmix specs from CI before and after the change?

Would probably be sufficient to run spack spec flux-pmix in the working and broken CI environments.

[vsoch]

@alalazo it was very logical actually, we just didn't have the gfortran-12 binary. gfortran (given the installer) was linked to gfortran-10 - here is the action we are using to install: https://github.com/modflowpy/install-gfortran-action.

[vsoch]

? (We use regexes to extract the version from that output)

Ah! And that explains why my hackadoo attempt to link gfortran-10 to 12 didn't get picked up in spack. TIL!