make compilers a build dependency without using depends_on()

[davydden]

this idea materialised in numerous discussions [1] : #882 (comment) #876 (comment)

Essentially, each package should know which compilers it needs. Same applies to mpi. We may need just C/C++ part of the mpi library, or we may need both c/c++ and fortran compiler wrappers.
For opempi one could have

variant ('cc', default=True, 'build with C compilers')
variant ('cxx', default=True, 'build with CXX compilers')
depends_on('cc', when='+cc');
depends_on('cc', when='+cxx');

and then in some package

depends_on('mpi+cc+cxx+f77+fc')

or alike. (update: see #896 (comment)).

That's how it is done in Homebrew (see for example Hypre https://github.com/Homebrew/homebrew-science/blob/master/hypre.rb#L17). I think it is a good idea to adopt a similar approach.

I see several reasons why this is important, here is one. I am currently helping our sysadmin to use spack. On some Linuxes gfortran was not available when he first run spack isntall dealii. dealii has a lot of dependencies and quite big DAG was created. openmpi together with some other packages were installed successfully. Luckily for us, openblas failed due to missing fortran compilers. I helped him to debug the issue, we removed compilers.yaml and everything went fine with openblas. However hypre failed even when compilers.yaml had proper links to fortran compilers. That is because openmpi happened to be installed before we fixed gfortran issue and thus did not compile Fortran part of the library. Obviously a fix is to reinstall openmpi.

You can imagine that for newcomers those errors would be very difficult to track down. What is worse is that DAG fails to build somewhere in the middle. Such issues could be made more transparent if packages would declare which compilers they need. In this case, the whole DAG would probably have failed to conretize as no fortran compiler would be found. This would be a much easier to fix even for a newcomer to spack.

Additionally once compilers are made dependencies of packages, their mixing c/c++ and fortran from different families (clang/gcc) would probably be easier as one would search for a working fortran compiler among all compilers known to spack. Of course, the whole conretization logic would be more complicated, probably with restrictions to use at maximum two different compiler suites for c/c++ and f77/fc each.

[1] i will update the links if I find more

[alalazo]

Just my two cents : compilers are not dependencies. If we proceed with this we will lose the possibility to say something like:

spack install A%intel ^ B%gcc

because we will end up with 2 providers for the same virtual dependency in one DAG. In fact the line above will look to spack the same as:

spack install A ^mvapich2 ^ B^openmpi

as long as virtual dependency resolution is concerned. I think the ability to handle situations like this (mixing compilers in one DAG) is one of the key differences between spack and other package managers, and I am not at all sure that the thing is worth a major reworking just to :

• remove compilers from the spec language
• permit a different virtual dependency resolution just for compilers, in order to allow for more than one provider

[davydden]

@alalazo : i agree with you that treating compilers exactly the same as packages using depends_on is not a good idea. But let me elaborate on the first post...

I think we can all agree that if a package has Fortran code it depends on a fortran compiler being available. This, however, does not mean that we need to use depends_on() to treat compilers. The main issue I outlined above is that packages do not state which compilers they need. We could fix that by something like:

class Openmpi(Package):
    variant ('fc', default=True, "compile with fortran support");

    compiles_with('fc', when='+fc');

Namely, by introducing new command compiles_with() where the first argument is one from the list fc|cc|cxx|f77.
When a user does spack install openmpi+fc%gcc-5.3.0, spack should check that fc is actually defined for gcc-5.3.0 compiler family. Otherwise DAG should fail to conretize with an error along the lines:

openmpi needs FC compiler, which appears to be empty in gcc-5.3.0

.
By doing so we avoid putting compilers as nodes in DAG and thereby retain the possibility to compile different packages in DAG using different compilers. But we also fix the issue when a user tries to build a package without the needed compiler.

[alalazo]

If you need just a check that a particular compiler is present, then an optional language directive (to mimic the familiar language of cmake) would just suffice. But it won't be a dependency, it will just be an assertion on a compiler being present. In that case we may elaborate further your example:

class Openmpi(Package):

    variant ('fc', default=True, "compile with fortran support");

    language('c')  # Checking for a C compiler is unconditional 
    language('fortran', standard='f90', when='+fc')  # ...while checking for a f90 compiler depends on the variant fc being active
    ....

I think something like this may be done quickly and neatly, but it's something different (at least to my understanding) from what the issue and related link initially proposed.

[adamjstewart]

I believe I was one of the first people to bring this up (#568). A big reason why this would be useful would be to allow you to mix compilers. Right now, you can hack compilers.yaml to contain something like:

compilers:
  linux-x86_64:
    nag@6.0:
      cc: /blues/gpfs/home/software/spack/opt/spack/linux-x86_64/gcc-4.4.7/gcc-5.3.0-fygfl7rvyuiteto27dlhmilp5cstw2o2/bin/gcc
      cxx: /blues/gpfs/home/software/spack/opt/spack/linux-x86_64/gcc-4.4.7/gcc-5.3.0-fygfl7rvyuiteto27dlhmilp5cstw2o2/bin/g++
      f77: /blues/gpfs/home/software/nag/6.0/bin/nagfor
      fc: /blues/gpfs/home/software/nag/6.0/bin/nagfor

But the resulting package is not NAG. It's GCC C/C++ and NAG Fortran. Without including the version number, there is no difference between NAG 6.0/GCC 5.3.0 and NAG 6.0/GCC 4.4.7. There is also no difference between NAG/GCC and NAG/Intel, or Clang/GCC and Clang/Intel. As @tgamblin suggested, adding version suffixes like %nag@6.0-gcc-5.3.0 can help alleviate this problem, but it is still a hack.

Being able to explicitly run something on the command line like:

spack install openmpi %gcc.c %nag.fortran

would solve this in my opinion. As for the actual implementation, I have no allegiances. Like @davydden, I want a way to specify that a particular package needs a certain compiler (CC, CXX, F77, and/or FC) and a way to explicitly choose which one to satisfy that virtual dependency/need. It doesn't need to be the same mechanic as depends_on, but that happens to be a convenient terminology that is already built into Spack.

As for specifying the compiler to build a subpackage for, I think this should still work:

spack install openmpi %gcc.c %nag.fortran ^libpciaccess%intel

[eschnett]

There are build-time and run-time dependency. A compiler is a build-time dependency. Thus the expression

spack install A%intel ^ B%gcc

will be just fine -- they don't conflict. It's like building different packages with different versions of cmake. Once this is possible, compilers will be the same.

[davydden]

@alalazo :

but it's something different (at least to my understanding) from what the issue

i think it still solves the issue i described above.

and related link initially proposed.

agree. It is different in that these however-we-call-these-requreiemts-or-needs are not part of the DAG.

@adamjstewart :

Being able to explicitly run something on the command line like: spack install openmpi %gcc.c %nag.fortran would solve this in my opinion .

agree.

I would say the trick should be to make, in @alalazo 's terminology, language('c') to influence conretization and thereby be more than just a check/assert. Say a user runs spack install openmpi without specification of c/fortran. Then spack could follow the order of compilers in compilers.yaml to satisfy those needs. So if we have

   clang@7.3.0-apple:
     cc: /usr/bin/clang
      cxx: /usr/bin/clang++
      f77: null
      fc: null
    gcc@5.3.0:
      cc: /usr/bin/gcc
      cxx: /usr/bin/g++
      f77: /usr/bin/gfortran
      fc: /usr/bin/gfortran

together with

packages:
  all:
    compiler: [clang, gcc]

then spack_cc and spack_cxx would point to clang/clang++ whereas spack_f77 and spack_fc would point to gfortran.

This whole conretization check only affects a single node in DAG in the same way compilers are conretized. So i hope it should be doable, perhaps even quickly.

It doesn't need to be the same mechanic as depends_on

👍

@eschnett :

A compiler is a build-time dependency.

if I understand you correct, you suggest to keep build time dependencies outside of DAG whereas run-time dependencies within the DAG. In this sense this is indeed a related issue/solution strategy.

[mathstuf]

if I understand you correct, you suggest to keep build time dependencies outside of DAG whereas run-time dependencies within the DAG. In this sense this is indeed a related issue/solution strategy.

See #839 for how this could work.

[becker33]

Closing in favor of #31357