Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error: set() does not equal {'-lnvcpumath', '-lnvhpcatm'} #914

Closed
jonasjucker opened this issue Feb 22, 2024 · 5 comments
Closed

Error: set() does not equal {'-lnvcpumath', '-lnvhpcatm'} #914

jonasjucker opened this issue Feb 22, 2024 · 5 comments
Labels

Comments

@jonasjucker
Copy link
Contributor

jonasjucker commented Feb 22, 2024

spack v0.21.1 has a problem with the setup of nvhpc 21.3 in compilers.yaml:

- compiler:
    environment: {}
    extra_rpaths: []
    flags:
      fflags: -lnvcpumath -lnvhpcatm
    modules:
    - PrgEnv-nvidia
    - nvidia/21.3
    - daint-gpu
    - CrayNvidia/.21.09
    operating_system: cnl7

Due to a wrong installation of nvhpc 21.3, -lnvcpumath -lnvhpcatm have to be passed as fflags.

It seems that spec with mixed compilers sometimes don't like this:

spack spec pytorch-fortran@0.4%nvhpc ^pytorch-fortran-proxy@0.4%gcc ^python@3.10

==> Error: set() does not equal {'-lnvcpumath', '-lnvhpcatm'}

This problem appears for

  • COSMO
  • pytorch-fortran
  • ICON

This is a known problem to spack and was adresses in several issues/PR

The relevant code in spack is spack/lib/spack/spack/solver/asp.py

    def reorder_flags(self):
        """Order compiler flags on specs in predefined order.

        We order flags so that any node's flags will take priority over
        those of its dependents.  That is, the deepest node in the DAG's
        flags will appear last on the compile line, in the order they
        were specified.

        The solver determines which flags are on nodes; this routine
        imposes order afterwards.
        """
        # reverse compilers so we get highest priority compilers that share a spec
        compilers = dict((c.spec, c) for c in reversed(all_compilers_in_config()))
        cmd_specs = dict((s.name, s) for spec in self._command_line_specs for s in spec.traverse())

        for spec in self._specs.values():
            # if bootstrapping, compiler is not in config and has no flags
            flagmap_from_compiler = {}
            if spec.compiler in compilers:
                flagmap_from_compiler = compilers[spec.compiler].flags

            for flag_type in spec.compiler_flags.valid_compiler_flags():
                from_compiler = flagmap_from_compiler.get(flag_type, [])
                from_sources = []

                # order is determined by the  DAG. A spec's flags come after any of its ancestors
                # on the compile line
                node = SpecBuilder.make_node(pkg=spec.name)
                source_key = (node, flag_type)
                if source_key in self._flag_sources:
                    order = [
                        SpecBuilder.make_node(pkg=s.name)
                        for s in spec.traverse(order="post", direction="parents")
                    ]
                    sorted_sources = sorted(
                        self._flag_sources[source_key], key=lambda s: order.index(s)
                    )

                    # add flags from each source, lowest to highest precedence
                    for node in sorted_sources:
                        all_src_flags = list()
                        per_pkg_sources = [self._specs[node]]
                        if node.pkg in cmd_specs:
                            per_pkg_sources.append(cmd_specs[node.pkg])
                        for source in per_pkg_sources:
                            all_src_flags.extend(source.compiler_flags.get(flag_type, []))
                        extend_flag_list(from_sources, all_src_flags)

                # compiler flags from compilers config are lowest precedence
                ordered_compiler_flags = list(llnl.util.lang.dedupe(from_compiler + from_sources))
                compiler_flags = spec.compiler_flags.get(flag_type, [])

                msg = "%s does not equal %s" % (set(compiler_flags), set(ordered_compiler_flags))
                assert set(compiler_flags) == set(ordered_compiler_flags), msg

                spec.compiler_flags.update({flag_type: ordered_compiler_flags})
@jonasjucker jonasjucker added bug Something isn't working high-prio v0.21.1 labels Feb 22, 2024
@jonasjucker
Copy link
Contributor Author

ping @leclairm @mjaehn because this is a major blocker atm.

@leclairm
Copy link
Contributor

As I see it, it's only an issue on Daint, right? If yes, that would speak for freezing the Daint config and only moving to 21.x on Alps.

@leclairm
Copy link
Contributor

So after a lot of printouts, I have a start of an answer to what is happening and what looks like a workaround

My 2 cents on what happens

in the reorder_flags method, this statement node = SpecBuilder.make_node(pkg=spec.name) generates a node that seem to only consider the package name and not the rest of the spec, e.g. the compiler. In our case, it leads to including nvhpc flags provided by compilers.yaml when trying to reorder flags for cmake%gcc and gmake%gcc.

Workarounds

2 things have worked so far:

  • explicitly include ^gmake%gcc ^cmake%gcc in the spec, like spack spec pytorch-fortran@0.4%nvhpc ^pytorch-fortran-proxy@0.4%gcc ^python@3.10 ^gmake%gcc ^cmake%gcc
  • make the package depend on gmake%gcc and cmake%gcc. So directly include depends_on('gmake %gcc') and depends_on('cmake %gcc') in /repos/c2sm/packages/pytorch-fortran-proxy/package.py

@leclairm
Copy link
Contributor

So after a lot of printouts, I have a start of an answer to what is happening and what looks like a workaround

My 2 cents on what happens

in the reorder_flags method, this statement node = SpecBuilder.make_node(pkg=spec.name) generates a node that seem to only consider the package name and not the rest of the spec, e.g. the compiler. In our case, it leads to including nvhpc flags provided by compilers.yaml when trying to reorder flags for cmake%gcc and gmake%gcc.

Workarounds

2 things have worked so far:

* explicitly include `^gmake%gcc ^cmake%gcc` in the spec, like `spack spec pytorch-fortran@0.4%nvhpc ^pytorch-fortran-proxy@0.4%gcc ^python@3.10 ^gmake%gcc ^cmake%gcc`

* make the package depend on `gmake%gcc` and `cmake%gcc`. So directly include `depends_on('gmake %gcc')` and `depends_on('cmake %gcc')` in `/repos/c2sm/packages/pytorch-fortran-proxy/package.py`

Well I was wrong for what might be happening there. I think it's more in these lines:

sorted_sources = sorted(
    self._flag_sources[source_key], key=lambda s: order.index(s)
)

@jonasjucker
Copy link
Contributor Author

Daint will be decomissioned soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants