Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Umpire incompatible with rocm 5.3.0 #800

Closed
jaelynlitz opened this issue Jan 6, 2023 · 14 comments
Closed

Umpire incompatible with rocm 5.3.0 #800

jaelynlitz opened this issue Jan 6, 2023 · 14 comments

Comments

@jaelynlitz
Copy link

Describe the bug

Hi, I'm a developer on ExaGO within ExaSGD. I'm trying to upgrade our dependency on Umpire from version 6.0.0 to 2022.3.1. I'm building Umpire via Spack within our software stack and am running into some build errors. This is an AMD system and I'm trying to use ROCm 5.3.0.

@cameronrutherford

To Reproduce

Concretize with gcc 10.2.0, rocm 5.3.0 with the options below. Try to spack install umpire.

Expected behavior

Code to compile.

Compilers & Libraries (please complete the following information):

  • Compiler & version: gcc10.2.0 rocm 5.3.0

Additional context

Below the cut are the errors from the build log:

Logs
Spack spec: umpire@2022.03.1%gcc@10.2.0+c~cuda+device_alloc~deviceconst~examples~fortran~ipo~numa~openmp+rocm+shared amdgpu_target=gfx90a build_system=cmake build_type=RelWithDebInfo patches=78daca9 tests=none arch=linux-centos7-zen2
==> Installing umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4
==> umpire exists in binary cache but with different hash
==> No binary for umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4 found: installing from source
==> Warning: download_tarball() was unable to download umpire@2022.03.1%gcc@10.2.0+c~cuda+device_alloc~deviceconst~examples~fortran~ipo~numa~openmp+rocm+shared amdgpu_target=gfx90a build_system=cmake build_type=RelWithDebInfo patches=78daca9 tests=none arch=linux-centos7-zen2 ^blt@0.5.2%gcc@10.2.0 build_system=generic arch=linux-centos7-zen2 ^ca-certificates-mozilla@2022-10-11%gcc@10.2.0 build_system=generic arch=linux-centos7-zen2 ^camp@2022.03.2%gcc@10.2.0~cuda~ipo~openmp+rocm~tests amdgpu_target=gfx90a build_system=cmake build_type=RelWithDebInfo arch=linux-centos7-zen2 ^cmake@3.20.6%gcc@10.2.0~doc+ncurses+ownlibs~qt build_system=generic build_type=Release arch=linux-centos7-zen2 ^hip@5.3.0%gcc@10.2.0~ipo build_system=cmake build_type=Release patches=ca523f1 arch=linux-centos7-zen2 ^hsa-rocr-dev@5.3.0%gcc@10.2.0+image~ipo+shared build_system=cmake build_type=Release patches=71e6851 arch=linux-centos7-zen2 ^llvm-amdgpu@5.3.0%gcc@10.2.0~ipo~link_llvm_dylib~llvm_dylib~openmp+rocm-device-libs build_system=cmake build_type=Release patches=a08bbe1 arch=linux-centos7-zen2 ^ncurses@6.3%gcc@10.2.0~symlinks+termlib abi=none build_system=autotools arch=linux-centos7-zen2 ^openssl@1.1.1s%gcc@10.2.0~docs~shared build_system=generic certs=mozilla arch=linux-centos7-zen2 ^perl@5.26.0%gcc@10.2.0+cpanm+shared+threads build_system=generic patches=0eac10e,8cf4302 arch=linux-centos7-zen2 ^pkgconf@1.8.0%gcc@10.2.0 build_system=autotools arch=linux-centos7-zen2 ^zlib@1.2.13%gcc@10.2.0+optimize+pic+shared build_system=makefile arch=linux-centos7-zen2 from any configured mirrors
==> Using cached archive: /qfs/projects/exasgd/src/svcexasgd/cache/source-cache/_source-cache/git//LLNL/Umpire.git/v2022.03.1.tar.gz
==> Applied patch /qfs/people/svcexasgd/gitlab/18959/spack_incline/tpl/spack/var/spack/repos/builtin/packages/umpire/std-filesystem-pr784.patch
==> Warning: Fetching from mirror without a checksum!
  This package is normally checked out from a version control system, but it has been archived on a spack mirror.  This means we cannot know a checksum for the tarball in advance. Be sure that your connection to this mirror is secure!
==> umpire: Executing phase: 'initconfig'
==> umpire: Executing phase: 'cmake'
==> umpire: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j24'

12 errors found in build log:
     185    /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-stage-um
            pire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-src/src/umpir
            e/strategy/FixedSizePool.hpp:133:9: warning: variable 'i' set but n
            ot used [-Wunused-but-set-variable]
     186        int i = 0;
     187            ^
     188    In file included from /qfs/projects/exasgd/src/svcexasgd/cache/buil
            d-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnn
            v4/spack-src/src/umpire/interface/c_fortran/wrapResourceManager.cpp
            :17:
     189    In file included from /qfs/projects/exasgd/src/svcexasgd/cache/buil
            d-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnn
            v4/spack-src/src/umpire/strategy/QuickPool.hpp:11:
     190    In file included from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../
            ../../include/c++/4.8.5/map:62:
  >> 191    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.
            5/bits/stl_multimap.h:130:41: error: no member named 'rebind' in 'u
            mpire::strategy::QuickPool::pool_allocator<std::pair<const unsigned
             long, umpire::strategy::QuickPool::Chunk *>>'
     192          typedef typename _Alloc::template rebind<value_type>::other
     193                           ~~~~~~~~         ^
     194    /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-stage-um
            pire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-src/src/umpir
            e/strategy/QuickPool.hpp:150:5: note: in instantiation of template 
            class 'std::multimap<unsigned long, umpire::strategy::QuickPool::Ch
            unk *, std::less<unsigned long>, umpire::strategy::QuickPool::pool_
            allocator<std::pair<const unsigned long, umpire::strategy::QuickPoo
            l::Chunk *>>>' requested here
     195        SizeMap::iterator size_map_it;
     196        ^
     197    In file included from /qfs/projects/exasgd/src/svcexasgd/cache/buil
            d-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnn
            v4/spack-src/src/umpire/interface/c_fortran/wrapResourceManager.cpp
            :17:
     198    In file included from /qfs/projects/exasgd/src/svcexasgd/cache/buil
            d-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnn
            v4/spack-src/src/umpire/strategy/QuickPool.hpp:11:
     199    In file included from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../
            ../../include/c++/4.8.5/map:60:
  >> 200    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.
            5/bits/stl_tree.h:335:24: error: type 'int' cannot be used prior to
             '::' because it has no members
     201          typedef typename _Alloc::template rebind<_Rb_tree_node<_Val> 
            >::other
     202                           ^
     203    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.
            5/bits/stl_multimap.h:136:17: note: in instantiation of template cl
            ass 'std::_Rb_tree<unsigned long, std::pair<const unsigned long, um
            pire::strategy::QuickPool::Chunk *>, std::_Select1st<std::pair<cons
            t unsigned long, umpire::strategy::QuickPool::Chunk *>>, std::less<
            unsigned long>, int>' requested here
     204          _Rep_type _M_t;
     205                    ^
     206    /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-stage-um
            pire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-src/src/umpir
            e/strategy/QuickPool.hpp:150:5: note: in instantiation of template 
            class 'std::multimap<unsigned long, umpire::strategy::QuickPool::Ch
            unk *, std::less<unsigned long>, umpire::strategy::QuickPool::pool_
            allocator<std::pair<const unsigned long, umpire::strategy::QuickPoo
            l::Chunk *>>>' requested here
     207        SizeMap::iterator size_map_it;
     208        ^
     209    In file included from /qfs/projects/exasgd/src/svcexasgd/cache/buil
            d-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnn
            v4/spack-src/src/umpire/interface/c_fortran/wrapResourceManager.cpp
            :17:
     210    In file included from /qfs/projects/exasgd/src/svcexasgd/cache/buil
            d-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnn
            v4/spack-src/src/umpire/strategy/QuickPool.hpp:11:
     211    In file included from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../
            ../../include/c++/4.8.5/map:60:
  >> 212    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.
            5/bits/stl_tree.h:439:39: error: base specifier must name a class
     213            struct _Rb_tree_impl : public _Node_allocator
     214                                   ~~~~~~~^~~~~~~~~~~~~~~
     215    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.
            5/bits/stl_tree.h:473:31: note: in instantiation of template class 
            'std::_Rb_tree<unsigned long, std::pair<const unsigned long, umpire
            ::strategy::QuickPool::Chunk *>, std::_Select1st<std::pair<const un
            signed long, umpire::strategy::QuickPool::Chunk *>>, std::less<unsi
            gned long>, int>::_Rb_tree_impl<std::less<unsigned long>, true>' re
            quested here
     216          _Rb_tree_impl<_Compare> _M_impl;
     217                                  ^
     218    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.
            5/bits/stl_multimap.h:136:17: note: in instantiation of template cl
            ass 'std::_Rb_tree<unsigned long, std::pair<const unsigned long, um
            pire::strategy::QuickPool::Chunk *>, std::_Select1st<std::pair<cons
            t unsigned long, umpire::strategy::QuickPool::Chunk *>>, std::less<
            unsigned long>, int>' requested here

     ...

     220                    ^
     221    /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-stage-um
            pire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-src/src/umpir
            e/strategy/QuickPool.hpp:150:5: note: in instantiation of template 
            class 'std::multimap<unsigned long, umpire::strategy::QuickPool::Ch
            unk *, std::less<unsigned long>, umpire::strategy::QuickPool::pool_
            allocator<std::pair<const unsigned long, umpire::strategy::QuickPoo
            l::Chunk *>>>' requested here
     222        SizeMap::iterator size_map_it;
     223        ^
     224    In file included from /qfs/projects/exasgd/src/svcexasgd/cache/buil
            d-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnn
            v4/spack-src/src/umpire/interface/c_fortran/wrapResourceManager.cpp
            :10:
     225    In file included from /qfs/projects/exasgd/src/svcexasgd/cache/buil
            d-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnn
            v4/spack-src/src/umpire/ResourceManager.hpp:354:
  >> 226    /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-stage-um
            pire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-src/src/umpir
            e/ResourceManager.inl:41:13: error: no viable overloaded '='
     227      allocator = util::make_unique<Strategy>(name, getNextId(), std::f
            orward<Args>(args)...);
     228      ~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            ~~~~~~~~~~~~~~~~~~~~~~
     229    /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-stage-um
            pire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-src/src/umpir
            e/ResourceManager.inl:61:10: note: in instantiation of function tem
            plate specialization 'umpire::ResourceManager::makeAllocator<umpire
            ::strategy::QuickPool, umpire::Allocator &, unsigned long &, unsign
            ed long &>' requested here
     230      return makeAllocator<Strategy>(name, tracked, std::forward<Args>(
            args)...);
     231             ^
     232    /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-stage-um
            pire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-src/src/umpir
            e/interface/c_fortran/wrapResourceManager.cpp:141:26: note: in inst
            antiation of function template specialization 'umpire::ResourceMana
            ger::makeAllocator<umpire::strategy::QuickPool, true, umpire::Alloc
            ator &, unsigned long &, unsigned long &>' requested here

     ...

     242          operator=(nullptr_t) noexcept
     243          ^
     244    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.
            5/bits/unique_ptr.h:274:19: note: candidate function not viable: no
             known conversion from 'unique_ptr<umpire::strategy::QuickPool, def
            ault_delete<umpire::strategy::QuickPool>>' to 'const unique_ptr<ump
            ire::strategy::AllocationStrategy, default_delete<umpire::strategy:
            :AllocationStrategy>>' for 1st argument
     245          unique_ptr& operator=(const unique_ptr&) = delete;
     246                      ^
     247    1 warning and 4 errors generated when compiling for gfx90a.
  >> 248    make[2]: *** [src/umpire/interface/c_fortran/CMakeFiles/umpire_inte
            rface.dir/wrapResourceManager.cpp.o] Error 1
     249    make[2]: *** Waiting for unfinished jobs....
     250    [ 46%] Building CXX object src/umpire/strategy/CMakeFiles/umpire_st
            rategy.dir/AllocationPrefetcher.cpp.o
     251    cd /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-stage
            -umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-build-xiw6
            mnk/src/umpire/strategy && /opt/rocm-5.3.0/hip/bin/hipcc -DCAMP_HAV
            E_HIP -D__HIP_PLATFORM_AMD__=1 -D__HIP_PLATFORM_HCC__=1 -I/qfs/proj
            ects/exasgd/src/ci-incline/linux-centos7-zen2/gcc-10.2.0/camp-2022.
            03.2-63pjiqilapgeolgflds5ykmf5prar3ns/include -I/qfs/projects/exasg
            d/src/svcexasgd/cache/build-stage/spack-stage-umpire-2022.03.1-xiw6
            mnko466g2s3fylruimgprf4fnnv4/spack-src/src -I/qfs/projects/exasgd/s
            rc/svcexasgd/cache/build-stage/spack-stage-umpire-2022.03.1-xiw6mnk
            o466g2s3fylruimgprf4fnnv4/spack-build-xiw6mnk/include -Wall -Wextra
                  -O2 -g -DNDEBUG -fPIC -std=c++14 -MD -MT src/umpire/strategy/
            CMakeFiles/umpire_strategy.dir/AllocationPrefetcher.cpp.o -MF CMake
            Files/umpire_strategy.dir/AllocationPrefetcher.cpp.o.d -o CMakeFile
            s/umpire_strategy.dir/AllocationPrefetcher.cpp.o -c /qfs/projects/e
            xasgd/src/svcexasgd/cache/build-stage/spack-stage-umpire-2022.03.1-
            xiw6mnko466g2s3fylruimgprf4fnnv4/spack-src/src/umpire/strategy/Allo
            cationPrefetcher.cpp
     252    [ 50%] Building CXX object src/umpire/strategy/CMakeFiles/umpire_st
            rategy.dir/AllocationStrategy.cpp.o
     253    [ 50%] Building CXX object src/umpire/op/CMakeFiles/umpire_op.dir/M
            emoryOperation.cpp.o
     254    cd /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-stage
            -umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-build-xiw6
            mnk/src/umpire/op && /opt/rocm-5.3.0/hip/bin/hipcc -DCAMP_HAVE_HIP 
            -D__HIP_PLATFORM_AMD__=1 -D__HIP_PLATFORM_HCC__=1 -I/qfs/projects/e
            xasgd/src/ci-incline/linux-centos7-zen2/gcc-10.2.0/camp-2022.03.2-6
            3pjiqilapgeolgflds5ykmf5prar3ns/include -I/qfs/projects/exasgd/src/
            svcexasgd/cache/build-stage/spack-stage-umpire-2022.03.1-xiw6mnko46
            6g2s3fylruimgprf4fnnv4/spack-src/src -I/qfs/projects/exasgd/src/svc
            exasgd/cache/build-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2
            s3fylruimgprf4fnnv4/spack-build-xiw6mnk/include -Wall -Wextra      
            -O2 -g -DNDEBUG -fPIC -std=c++14 -MD -MT src/umpire/op/CMakeFiles/u
            mpire_op.dir/MemoryOperation.cpp.o -MF CMakeFiles/umpire_op.dir/Mem
            oryOperation.cpp.o.d -o CMakeFiles/umpire_op.dir/MemoryOperation.cp
            p.o -c /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-s
            tage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-src/sr
            c/umpire/op/MemoryOperation.cpp

     ...

     287    /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-stage-um
            pire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-src/src/umpir
            e/strategy/FixedSizePool.hpp:133:9: warning: variable 'i' set but n
            ot used [-Wunused-but-set-variable]
     288        int i = 0;
     289            ^
     290    make[2]: Leaving directory `/qfs/projects/exasgd/src/svcexasgd/cach
            e/build-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgp
            rf4fnnv4/spack-build-xiw6mnk'
     291    [ 75%] Built target umpire_util
     292    make[2]: Leaving directory `/qfs/projects/exasgd/src/svcexasgd/cach
            e/build-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgp
            rf4fnnv4/spack-build-xiw6mnk'
  >> 293    make[1]: *** [src/umpire/interface/c_fortran/CMakeFiles/umpire_inte
            rface.dir/all] Error 2
     294    make[1]: *** Waiting for unfinished jobs....
     295    [ 76%] Building CXX object src/umpire/op/CMakeFiles/umpire_op.dir/H
            ipMemsetOperation.cpp.o
     296    cd /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-stage
            -umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-build-xiw6
            mnk/src/umpire/op && /opt/rocm-5.3.0/hip/bin/hipcc -DCAMP_HAVE_HIP 
            -D__HIP_PLATFORM_AMD__=1 -D__HIP_PLATFORM_HCC__=1 -I/qfs/projects/e
            xasgd/src/ci-incline/linux-centos7-zen2/gcc-10.2.0/camp-2022.03.2-6
            3pjiqilapgeolgflds5ykmf5prar3ns/include -I/qfs/projects/exasgd/src/
            svcexasgd/cache/build-stage/spack-stage-umpire-2022.03.1-xiw6mnko46
            6g2s3fylruimgprf4fnnv4/spack-src/src -I/qfs/projects/exasgd/src/svc
            exasgd/cache/build-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2
            s3fylruimgprf4fnnv4/spack-build-xiw6mnk/include -Wall -Wextra      
            -O2 -g -DNDEBUG -fPIC -std=c++14 -MD -MT src/umpire/op/CMakeFiles/u
            mpire_op.dir/HipMemsetOperation.cpp.o -MF CMakeFiles/umpire_op.dir/
            HipMemsetOperation.cpp.o.d -o CMakeFiles/umpire_op.dir/HipMemsetOpe
            ration.cpp.o -c /qfs/projects/exasgd/src/svcexasgd/cache/build-stag
            e/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spa
            ck-src/src/umpire/op/HipMemsetOperation.cpp
     297    [ 78%] Building CXX object src/umpire/strategy/CMakeFiles/umpire_st
            rategy.dir/mixins/AllocateNull.cpp.o
     298    cd /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-stage
            -umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-build-xiw6
            mnk/src/umpire/strategy && /opt/rocm-5.3.0/hip/bin/hipcc -DCAMP_HAV
            E_HIP -D__HIP_PLATFORM_AMD__=1 -D__HIP_PLATFORM_HCC__=1 -I/qfs/proj
            ects/exasgd/src/ci-incline/linux-centos7-zen2/gcc-10.2.0/camp-2022.
            03.2-63pjiqilapgeolgflds5ykmf5prar3ns/include -I/qfs/projects/exasg
            d/src/svcexasgd/cache/build-stage/spack-stage-umpire-2022.03.1-xiw6
            mnko466g2s3fylruimgprf4fnnv4/spack-src/src -I/qfs/projects/exasgd/s
            rc/svcexasgd/cache/build-stage/spack-stage-umpire-2022.03.1-xiw6mnk
            o466g2s3fylruimgprf4fnnv4/spack-build-xiw6mnk/include -Wall -Wextra
                  -O2 -g -DNDEBUG -fPIC -std=c++14 -MD -MT src/umpire/strategy/
            CMakeFiles/umpire_strategy.dir/mixins/AllocateNull.cpp.o -MF CMakeF
            iles/umpire_strategy.dir/mixins/AllocateNull.cpp.o.d -o CMakeFiles/
            umpire_strategy.dir/mixins/AllocateNull.cpp.o -c /qfs/projects/exas
            gd/src/svcexasgd/cache/build-stage/spack-stage-umpire-2022.03.1-xiw
            6mnko466g2s3fylruimgprf4fnnv4/spack-src/src/umpire/strategy/mixins/
            AllocateNull.cpp
     299    /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-stage-um
            pire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-src/src/umpir
            e/resource/FileMemoryResource.cpp:72:9: warning: unused variable 'e
            rrno_save' [-Wunused-variable]

     ...

     302    /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-stage-um
            pire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-src/src/umpir
            e/resource/FileMemoryResource.cpp:80:9: warning: unused variable 'e
            rrno_save' [-Wunused-variable]
     303        int errno_save = errno;
     304            ^
     305    In file included from /qfs/projects/exasgd/src/svcexasgd/cache/buil
            d-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnn
            v4/spack-src/src/umpire/strategy/MixedPool.cpp:8:
     306    In file included from /qfs/projects/exasgd/src/svcexasgd/cache/buil
            d-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnn
            v4/spack-src/src/umpire/strategy/MixedPool.hpp:10:
     307    In file included from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../
            ../../include/c++/4.8.5/map:62:
  >> 308    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.
            5/bits/stl_multimap.h:130:41: error: no member named 'rebind' in 'u
            mpire::strategy::QuickPool::pool_allocator<std::pair<const unsigned
             long, umpire::strategy::QuickPool::Chunk *>>'
     309          typedef typename _Alloc::template rebind<value_type>::other
     310                           ~~~~~~~~         ^
     311    /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-stage-um
            pire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-src/src/umpir
            e/strategy/QuickPool.hpp:150:5: note: in instantiation of template 
            class 'std::multimap<unsigned long, umpire::strategy::QuickPool::Ch
            unk *, std::less<unsigned long>, umpire::strategy::QuickPool::pool_
            allocator<std::pair<const unsigned long, umpire::strategy::QuickPoo
            l::Chunk *>>>' requested here
     312        SizeMap::iterator size_map_it;
     313        ^
     314    In file included from /qfs/projects/exasgd/src/svcexasgd/cache/buil
            d-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnn
            v4/spack-src/src/umpire/strategy/MixedPool.cpp:8:
     315    In file included from /qfs/projects/exasgd/src/svcexasgd/cache/buil
            d-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnn
            v4/spack-src/src/umpire/strategy/MixedPool.hpp:10:
     316    In file included from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../
            ../../include/c++/4.8.5/map:60:
  >> 317    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.
            5/bits/stl_tree.h:335:24: error: type 'int' cannot be used prior to
             '::' because it has no members
     318          typedef typename _Alloc::template rebind<_Rb_tree_node<_Val> 
            >::other
     319                           ^
     320    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.
            5/bits/stl_multimap.h:136:17: note: in instantiation of template cl
            ass 'std::_Rb_tree<unsigned long, std::pair<const unsigned long, um
            pire::strategy::QuickPool::Chunk *>, std::_Select1st<std::pair<cons
            t unsigned long, umpire::strategy::QuickPool::Chunk *>>, std::less<
            unsigned long>, int>' requested here
     321          _Rep_type _M_t;
     322                    ^
     323    /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-stage-um
            pire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-src/src/umpir
            e/strategy/QuickPool.hpp:150:5: note: in instantiation of template 
            class 'std::multimap<unsigned long, umpire::strategy::QuickPool::Ch
            unk *, std::less<unsigned long>, umpire::strategy::QuickPool::pool_
            allocator<std::pair<const unsigned long, umpire::strategy::QuickPoo
            l::Chunk *>>>' requested here
     324        SizeMap::iterator size_map_it;
     325        ^
     326    In file included from /qfs/projects/exasgd/src/svcexasgd/cache/buil
            d-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnn
            v4/spack-src/src/umpire/strategy/MixedPool.cpp:8:
     327    In file included from /qfs/projects/exasgd/src/svcexasgd/cache/buil
            d-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnn
            v4/spack-src/src/umpire/strategy/MixedPool.hpp:10:
     328    In file included from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../
            ../../include/c++/4.8.5/map:60:
  >> 329    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.
            5/bits/stl_tree.h:439:39: error: base specifier must name a class
     330            struct _Rb_tree_impl : public _Node_allocator
     331                                   ~~~~~~~^~~~~~~~~~~~~~~
     332    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.
            5/bits/stl_tree.h:473:31: note: in instantiation of template class 
            'std::_Rb_tree<unsigned long, std::pair<const unsigned long, umpire
            ::strategy::QuickPool::Chunk *>, std::_Select1st<std::pair<const un
            signed long, umpire::strategy::QuickPool::Chunk *>>, std::less<unsi
            gned long>, int>::_Rb_tree_impl<std::less<unsigned long>, true>' re
            quested here
     333          _Rb_tree_impl<_Compare> _M_impl;
     334                                  ^
     335    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.
            5/bits/stl_multimap.h:136:17: note: in instantiation of template cl
            ass 'std::_Rb_tree<unsigned long, std::pair<const unsigned long, um
            pire::strategy::QuickPool::Chunk *>, std::_Select1st<std::pair<cons
            t unsigned long, umpire::strategy::QuickPool::Chunk *>>, std::less<
            unsigned long>, int>' requested here

     ...

     343    [ 82%] Building CXX object src/umpire/strategy/CMakeFiles/umpire_st
            rategy.dir/mixins/Inspector.cpp.o
     344    cd /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-stage
            -umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-build-xiw6
            mnk/src/umpire/strategy && /opt/rocm-5.3.0/hip/bin/hipcc -DCAMP_HAV
            E_HIP -D__HIP_PLATFORM_AMD__=1 -D__HIP_PLATFORM_HCC__=1 -I/qfs/proj
            ects/exasgd/src/ci-incline/linux-centos7-zen2/gcc-10.2.0/camp-2022.
            03.2-63pjiqilapgeolgflds5ykmf5prar3ns/include -I/qfs/projects/exasg
            d/src/svcexasgd/cache/build-stage/spack-stage-umpire-2022.03.1-xiw6
            mnko466g2s3fylruimgprf4fnnv4/spack-src/src -I/qfs/projects/exasgd/s
            rc/svcexasgd/cache/build-stage/spack-stage-umpire-2022.03.1-xiw6mnk
            o466g2s3fylruimgprf4fnnv4/spack-build-xiw6mnk/include -Wall -Wextra
                  -O2 -g -DNDEBUG -fPIC -std=c++14 -MD -MT src/umpire/strategy/
            CMakeFiles/umpire_strategy.dir/MonotonicAllocationStrategy.cpp.o -M
            F CMakeFiles/umpire_strategy.dir/MonotonicAllocationStrategy.cpp.o.
            d -o CMakeFiles/umpire_strategy.dir/MonotonicAllocationStrategy.cpp
            .o -c /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-st
            age-umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-src/src
            /umpire/strategy/MonotonicAllocationStrategy.cpp
     345    cd /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-stage
            -umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-build-xiw6
            mnk/src/umpire/strategy && /opt/rocm-5.3.0/hip/bin/hipcc -DCAMP_HAV
            E_HIP -D__HIP_PLATFORM_AMD__=1 -D__HIP_PLATFORM_HCC__=1 -I/qfs/proj
            ects/exasgd/src/ci-incline/linux-centos7-zen2/gcc-10.2.0/camp-2022.
            03.2-63pjiqilapgeolgflds5ykmf5prar3ns/include -I/qfs/projects/exasg
            d/src/svcexasgd/cache/build-stage/spack-stage-umpire-2022.03.1-xiw6
            mnko466g2s3fylruimgprf4fnnv4/spack-src/src -I/qfs/projects/exasgd/s
            rc/svcexasgd/cache/build-stage/spack-stage-umpire-2022.03.1-xiw6mnk
            o466g2s3fylruimgprf4fnnv4/spack-build-xiw6mnk/include -Wall -Wextra
                  -O2 -g -DNDEBUG -fPIC -std=c++14 -MD -MT src/umpire/strategy/
            CMakeFiles/umpire_strategy.dir/mixins/Inspector.cpp.o -MF CMakeFile
            s/umpire_strategy.dir/mixins/Inspector.cpp.o.d -o CMakeFiles/umpire
            _strategy.dir/mixins/Inspector.cpp.o -c /qfs/projects/exasgd/src/sv
            cexasgd/cache/build-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g
            2s3fylruimgprf4fnnv4/spack-src/src/umpire/strategy/mixins/Inspector
            .cpp
     346    [ 83%] Building CXX object src/umpire/strategy/CMakeFiles/umpire_st
            rategy.dir/NamedAllocationStrategy.cpp.o
     347    cd /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-stage
            -umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-build-xiw6
            mnk/src/umpire/strategy && /opt/rocm-5.3.0/hip/bin/hipcc -DCAMP_HAV
            E_HIP -D__HIP_PLATFORM_AMD__=1 -D__HIP_PLATFORM_HCC__=1 -I/qfs/proj
            ects/exasgd/src/ci-incline/linux-centos7-zen2/gcc-10.2.0/camp-2022.
            03.2-63pjiqilapgeolgflds5ykmf5prar3ns/include -I/qfs/projects/exasg
            d/src/svcexasgd/cache/build-stage/spack-stage-umpire-2022.03.1-xiw6
            mnko466g2s3fylruimgprf4fnnv4/spack-src/src -I/qfs/projects/exasgd/s
            rc/svcexasgd/cache/build-stage/spack-stage-umpire-2022.03.1-xiw6mnk
            o466g2s3fylruimgprf4fnnv4/spack-build-xiw6mnk/include -Wall -Wextra
                  -O2 -g -DNDEBUG -fPIC -std=c++14 -MD -MT src/umpire/strategy/
            CMakeFiles/umpire_strategy.dir/NamedAllocationStrategy.cpp.o -MF CM
            akeFiles/umpire_strategy.dir/NamedAllocationStrategy.cpp.o.d -o CMa
            keFiles/umpire_strategy.dir/NamedAllocationStrategy.cpp.o -c /qfs/p
            rojects/exasgd/src/svcexasgd/cache/build-stage/spack-stage-umpire-2
            022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-src/src/umpire/stra
            tegy/NamedAllocationStrategy.cpp
     348    3 errors generated when compiling for gfx90a.
  >> 349    make[2]: *** [src/umpire/strategy/CMakeFiles/umpire_strategy.dir/Mi
            xedPool.cpp.o] Error 1
     350    make[2]: *** Waiting for unfinished jobs....
     351    1 warning generated when compiling for gfx90a.
     352    In file included from /qfs/projects/exasgd/src/svcexasgd/cache/buil
            d-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnn
            v4/spack-src/src/umpire/strategy/DynamicPoolList.cpp:7:
     353    In file included from /qfs/projects/exasgd/src/svcexasgd/cache/buil
            d-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnn
            v4/spack-src/src/umpire/strategy/DynamicPoolList.hpp:15:
     354    In file included from /qfs/projects/exasgd/src/svcexasgd/cache/buil
            d-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnn
            v4/spack-src/src/umpire/strategy/DynamicSizePool.hpp:18:
     355    /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-stage-um
            pire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-src/src/umpir
            e/strategy/FixedSizePool.hpp:133:9: warning: variable 'i' set but n
            ot used [-Wunused-but-set-variable]

     ...

     358    1 warning generated when compiling for host.
     359    make[2]: Leaving directory `/qfs/projects/exasgd/src/svcexasgd/cach
            e/build-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgp
            rf4fnnv4/spack-build-xiw6mnk'
     360    make[2]: Leaving directory `/qfs/projects/exasgd/src/svcexasgd/cach
            e/build-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgp
            rf4fnnv4/spack-build-xiw6mnk'
     361    [ 83%] Built target umpire_resource
     362    [ 83%] Built target umpire_op
     363    make[2]: Leaving directory `/qfs/projects/exasgd/src/svcexasgd/cach
            e/build-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgp
            rf4fnnv4/spack-build-xiw6mnk'
  >> 364    make[1]: *** [src/umpire/strategy/CMakeFiles/umpire_strategy.dir/al
            l] Error 2
     365    make[1]: Leaving directory `/qfs/projects/exasgd/src/svcexasgd/cach
            e/build-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgp
            rf4fnnv4/spack-build-xiw6mnk'
  >> 366    make: *** [all] Error 2

See build log for details:
  /qfs/projects/exasgd/src/svcexasgd/cache/build-stage/spack-stage-umpire-2022.03.1-xiw6mnko466g2s3fylruimgprf4fnnv4/spack-build-out.txt
@cameronrutherford
Copy link

cc @pelesh

@jaelynlitz
Copy link
Author

Tried reverting to umpire 6.0.0, our last working version, and am getting the same error as above. Tried toggling ~shared with no change in the output.

v6.0.0 logs below the cut

spec
umpire@6.0.0%gcc@10.2.0+c~cuda~device_alloc~deviceconst~examples~fortran~ipo~numa~openmp+rocm~shared amdgpu_target=gfx90a build_system=cmake build_type=RelWithDebInfo tests=none arch=linux-centos7-zen2

==> Installing umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff
==> umpire exists in binary cache but with different hash
==> No binary for umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff found: installing from source
==> Using cached archive: /qfs/projects/exasgd/src/litz372/cache/source-cache/_source-cache/git//LLNL/Umpire.git/v6.0.0.tar.gz
==> Warning: Fetching from mirror without a checksum!
  This package is normally checked out from a version control system, but it has been archived on a spack mirror.  This means we cannot know a checksum for the tarball in advance. Be sure that your connection to this mirror is secure!
==> No patches needed for umpire
==> umpire: Executing phase: 'initconfig'
==> umpire: Executing phase: 'cmake'
==> umpire: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j24'

20 errors found in build log:
     176    /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/strategy
            /FixedSizePool.hpp:133:9: warning: variable 'i' set but not used [-Wunused-but-set-variable]
     177        int i = 0;
     178            ^
     179    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/interface/c_fortran/wrapResourceManager.cpp:15:
     180    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/strategy/DynamicPoolMap.hpp:11:
     181    In file included from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/map:62:
  >> 182    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/bits/stl_multimap.h:130:41: error: no member named 'rebind' in 'umpir
            e::strategy::QuickPool::pool_allocator<std::pair<const unsigned long, umpire::strategy::QuickPool::Chunk *>>'
     183          typedef typename _Alloc::template rebind<value_type>::other
     184                           ~~~~~~~~         ^
     185    /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/strategy
            /QuickPool.hpp:150:5: note: in instantiation of template class 'std::multimap<unsigned long, umpire::strategy::QuickPool::Chunk *, std::le
            ss<unsigned long>, umpire::strategy::QuickPool::pool_allocator<std::pair<const unsigned long, umpire::strategy::QuickPool::Chunk *>>>' req
            uested here
     186        SizeMap::iterator size_map_it;
     187        ^
     188    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/interface/c_fortran/wrapResourceManager.cpp:15:
     189    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/strategy/DynamicPoolMap.hpp:11:
     190    In file included from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/map:60:
  >> 191    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/bits/stl_tree.h:335:24: error: type 'int' cannot be used prior to '::
            ' because it has no members
     192          typedef typename _Alloc::template rebind<_Rb_tree_node<_Val> >::other
     193                           ^
     194    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/bits/stl_multimap.h:136:17: note: in instantiation of template class
            'std::_Rb_tree<unsigned long, std::pair<const unsigned long, umpire::strategy::QuickPool::Chunk *>, std::_Select1st<std::pair<const unsign
            ed long, umpire::strategy::QuickPool::Chunk *>>, std::less<unsigned long>, int>' requested here
     195          _Rep_type _M_t;
     196                    ^
     197    /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/strategy
            /QuickPool.hpp:150:5: note: in instantiation of template class 'std::multimap<unsigned long, umpire::strategy::QuickPool::Chunk *, std::le
            ss<unsigned long>, umpire::strategy::QuickPool::pool_allocator<std::pair<const unsigned long, umpire::strategy::QuickPool::Chunk *>>>' req
            uested here
     198        SizeMap::iterator size_map_it;
     199        ^
     200    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/interface/c_fortran/wrapResourceManager.cpp:15:
     201    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/strategy/DynamicPoolMap.hpp:11:
     202    In file included from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/map:60:
  >> 203    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/bits/stl_tree.h:439:39: error: base specifier must name a class
     204            struct _Rb_tree_impl : public _Node_allocator
     205                                   ~~~~~~~^~~~~~~~~~~~~~~
     206    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/bits/stl_tree.h:473:31: note: in instantiation of template class 'std
            ::_Rb_tree<unsigned long, std::pair<const unsigned long, umpire::strategy::QuickPool::Chunk *>, std::_Select1st<std::pair<const unsigned l
            ong, umpire::strategy::QuickPool::Chunk *>>, std::less<unsigned long>, int>::_Rb_tree_impl<std::less<unsigned long>, true>' requested here
     207          _Rb_tree_impl<_Compare> _M_impl;
     208                                  ^
     209    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/bits/stl_multimap.h:136:17: note: in instantiation of template class
            'std::_Rb_tree<unsigned long, std::pair<const unsigned long, umpire::strategy::QuickPool::Chunk *>, std::_Select1st<std::pair<const unsign
            ed long, umpire::strategy::QuickPool::Chunk *>>, std::less<unsigned long>, int>' requested here

     ...

     211                    ^
     212    /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/strategy
            /QuickPool.hpp:150:5: note: in instantiation of template class 'std::multimap<unsigned long, umpire::strategy::QuickPool::Chunk *, std::le
            ss<unsigned long>, umpire::strategy::QuickPool::pool_allocator<std::pair<const unsigned long, umpire::strategy::QuickPool::Chunk *>>>' req
            uested here
     213        SizeMap::iterator size_map_it;
     214        ^
     215    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/interface/c_fortran/wrapResourceManager.cpp:10:
     216    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/ResourceManager.hpp:363:
  >> 217    /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/Resource
            Manager.inl:41:13: error: no viable overloaded '='
     218      allocator = util::make_unique<Strategy>(name, getNextId(), std::forward<Args>(args)...);
     219      ~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     220    /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/Resource
            Manager.inl:61:10: note: in instantiation of function template specialization 'umpire::ResourceManager::makeAllocator<umpire::strategy::Qu
            ickPool, umpire::Allocator &, unsigned long &, unsigned long &>' requested here
     221      return makeAllocator<Strategy>(name, tracked, std::forward<Args>(args)...);
     222             ^
     223    /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/interfac
            e/c_fortran/wrapResourceManager.cpp:184:26: note: in instantiation of function template specialization 'umpire::ResourceManager::makeAlloc
            ator<umpire::strategy::QuickPool, true, umpire::Allocator &, unsigned long &, unsigned long &>' requested here

     ...

     233          operator=(nullptr_t) noexcept
     234          ^
     235    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/bits/unique_ptr.h:274:19: note: candidate function not viable: no kno
            wn conversion from 'unique_ptr<umpire::strategy::QuickPool, default_delete<umpire::strategy::QuickPool>>' to 'const unique_ptr<umpire::str
            ategy::AllocationStrategy, default_delete<umpire::strategy::AllocationStrategy>>' for 1st argument
     236          unique_ptr& operator=(const unique_ptr&) = delete;
     237                      ^
     238    1 warning and 4 errors generated when compiling for gfx90a.
  >> 239    make[2]: *** [src/umpire/CMakeFiles/umpire_interface.dir/interface/c_fortran/wrapResourceManager.cpp.o] Error 1
     240    make[2]: *** Waiting for unfinished jobs....
     241    [ 45%] Building CXX object src/umpire/strategy/CMakeFiles/umpire_strategy.dir/AllocationAdvisor.cpp.o
     242    cd /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-build-pnun2wp/src/um
            pire/strategy && /opt/rocm-5.3.0/hip/bin/hipcc -DCAMP_HAVE_HIP -D__HIP_PLATFORM_AMD__ -D__HIP_PLATFORM_HCC__ -D__HIP_ROCclr__ -I/qfs/proje
            cts/exasgd/src/ci-incline/linux-centos7-zen2/gcc-10.2.0/camp-0.2.3-x5lxh775b2xhjrndoku54hd2l4x4rywh/include -I/qfs/projects/exasgd/src/lit
            z372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src -I/qfs/projects/exasgd/src/litz372/cache/bu
            ild-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-build-pnun2wp/include -Wpedantic       -Wall -Wextra  -O2 -g -DN
            DEBUG -fPIC -Wno-unused-parameter -std=c++11 -MD -MT src/umpire/strategy/CMakeFiles/umpire_strategy.dir/AllocationAdvisor.cpp.o -MF CMakeF
            iles/umpire_strategy.dir/AllocationAdvisor.cpp.o.d -o CMakeFiles/umpire_strategy.dir/AllocationAdvisor.cpp.o -c /qfs/projects/exasgd/src/l
            itz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/strategy/AllocationAdvisor.cpp
     243    [ 47%] Building CXX object src/umpire/strategy/CMakeFiles/umpire_strategy.dir/AllocationPrefetcher.cpp.o
     244    cd /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-build-pnun2wp/src/um
            pire/strategy && /opt/rocm-5.3.0/hip/bin/hipcc -DCAMP_HAVE_HIP -D__HIP_PLATFORM_AMD__ -D__HIP_PLATFORM_HCC__ -D__HIP_ROCclr__ -I/qfs/proje
            cts/exasgd/src/ci-incline/linux-centos7-zen2/gcc-10.2.0/camp-0.2.3-x5lxh775b2xhjrndoku54hd2l4x4rywh/include -I/qfs/projects/exasgd/src/lit
            z372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src -I/qfs/projects/exasgd/src/litz372/cache/bu
            ild-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-build-pnun2wp/include -Wpedantic       -Wall -Wextra  -O2 -g -DN
            DEBUG -fPIC -Wno-unused-parameter -std=c++11 -MD -MT src/umpire/strategy/CMakeFiles/umpire_strategy.dir/AllocationPrefetcher.cpp.o -MF CMa
            keFiles/umpire_strategy.dir/AllocationPrefetcher.cpp.o.d -o CMakeFiles/umpire_strategy.dir/AllocationPrefetcher.cpp.o -c /qfs/projects/exa
            sgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/strategy/AllocationPrefet
            cher.cpp
     245    [ 49%] Building CXX object src/umpire/strategy/CMakeFiles/umpire_strategy.dir/AllocationStrategy.cpp.o

     ...

     273    [ 73%] Building CXX object src/umpire/op/CMakeFiles/umpire_op.dir/HipCopyFromOperation.cpp.o
     274    cd /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-build-pnun2wp/src/um
            pire/op && /opt/rocm-5.3.0/hip/bin/hipcc -DCAMP_HAVE_HIP -D__HIP_PLATFORM_AMD__ -D__HIP_PLATFORM_HCC__ -D__HIP_ROCclr__ -I/qfs/projects/ex
            asgd/src/ci-incline/linux-centos7-zen2/gcc-10.2.0/camp-0.2.3-x5lxh775b2xhjrndoku54hd2l4x4rywh/include -I/qfs/projects/exasgd/src/litz372/c
            ache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src -I/qfs/projects/exasgd/src/litz372/cache/build-st
            age/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-build-pnun2wp/include -Wpedantic       -Wall -Wextra  -O2 -g -DNDEBUG
            -fPIC -Wno-unused-parameter -std=c++11 -MD -MT src/umpire/op/CMakeFiles/umpire_op.dir/HipCopyFromOperation.cpp.o -MF CMakeFiles/umpire_op.
            dir/HipCopyFromOperation.cpp.o.d -o CMakeFiles/umpire_op.dir/HipCopyFromOperation.cpp.o -c /qfs/projects/exasgd/src/litz372/cache/build-st
            age/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/op/HipCopyFromOperation.cpp
     275    [ 75%] Building CXX object src/umpire/op/CMakeFiles/umpire_op.dir/HipCopyToOperation.cpp.o
     276    cd /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-build-pnun2wp/src/um
            pire/op && /opt/rocm-5.3.0/hip/bin/hipcc -DCAMP_HAVE_HIP -D__HIP_PLATFORM_AMD__ -D__HIP_PLATFORM_HCC__ -D__HIP_ROCclr__ -I/qfs/projects/ex
            asgd/src/ci-incline/linux-centos7-zen2/gcc-10.2.0/camp-0.2.3-x5lxh775b2xhjrndoku54hd2l4x4rywh/include -I/qfs/projects/exasgd/src/litz372/c
            ache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src -I/qfs/projects/exasgd/src/litz372/cache/build-st
            age/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-build-pnun2wp/include -Wpedantic       -Wall -Wextra  -O2 -g -DNDEBUG
            -fPIC -Wno-unused-parameter -std=c++11 -MD -MT src/umpire/op/CMakeFiles/umpire_op.dir/HipCopyToOperation.cpp.o -MF CMakeFiles/umpire_op.di
            r/HipCopyToOperation.cpp.o.d -o CMakeFiles/umpire_op.dir/HipCopyToOperation.cpp.o -c /qfs/projects/exasgd/src/litz372/cache/build-stage/sp
            ack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/op/HipCopyToOperation.cpp
     277    make[2]: Leaving directory /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/s
            pack-build-pnun2wp'
     278    make[2]: Leaving directory /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/s
            pack-build-pnun2wp'
  >> 279    make[1]: *** [src/umpire/CMakeFiles/umpire_interface.dir/all] Error 2
     280    make[1]: *** Waiting for unfinished jobs....
     281    [ 77%] Building CXX object src/umpire/op/CMakeFiles/umpire_op.dir/HipMemsetOperation.cpp.o
     282    cd /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-build-pnun2wp/src/um
            pire/op && /opt/rocm-5.3.0/hip/bin/hipcc -DCAMP_HAVE_HIP -D__HIP_PLATFORM_AMD__ -D__HIP_PLATFORM_HCC__ -D__HIP_ROCclr__ -I/qfs/projects/ex
            asgd/src/ci-incline/linux-centos7-zen2/gcc-10.2.0/camp-0.2.3-x5lxh775b2xhjrndoku54hd2l4x4rywh/include -I/qfs/projects/exasgd/src/litz372/c
            ache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src -I/qfs/projects/exasgd/src/litz372/cache/build-st
            age/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-build-pnun2wp/include -Wpedantic       -Wall -Wextra  -O2 -g -DNDEBUG
            -fPIC -Wno-unused-parameter -std=c++11 -MD -MT src/umpire/op/CMakeFiles/umpire_op.dir/HipMemsetOperation.cpp.o -MF CMakeFiles/umpire_op.di
            r/HipMemsetOperation.cpp.o.d -o CMakeFiles/umpire_op.dir/HipMemsetOperation.cpp.o -c /qfs/projects/exasgd/src/litz372/cache/build-stage/sp
            ack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/op/HipMemsetOperation.cpp
     283    [ 77%] Built target umpire_util
     284    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/strategy/DynamicPoolList.cpp:7:
     285    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/strategy/DynamicPoolList.hpp:15:

     ...

     304    cd /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-build-pnun2wp/src/um
            pire/strategy && /opt/rocm-5.3.0/hip/bin/hipcc -DCAMP_HAVE_HIP -D__HIP_PLATFORM_AMD__ -D__HIP_PLATFORM_HCC__ -D__HIP_ROCclr__ -I/qfs/proje
            cts/exasgd/src/ci-incline/linux-centos7-zen2/gcc-10.2.0/camp-0.2.3-x5lxh775b2xhjrndoku54hd2l4x4rywh/include -I/qfs/projects/exasgd/src/lit
            z372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src -I/qfs/projects/exasgd/src/litz372/cache/bu
            ild-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-build-pnun2wp/include -Wpedantic       -Wall -Wextra  -O2 -g -DN
            DEBUG -fPIC -Wno-unused-parameter -std=c++11 -MD -MT src/umpire/strategy/CMakeFiles/umpire_strategy.dir/NamedAllocationStrategy.cpp.o -MF
            CMakeFiles/umpire_strategy.dir/NamedAllocationStrategy.cpp.o.d -o CMakeFiles/umpire_strategy.dir/NamedAllocationStrategy.cpp.o -c /qfs/pro
            jects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/strategy/NamedAl
            locationStrategy.cpp
     305    [ 85%] Building CXX object src/umpire/strategy/CMakeFiles/umpire_strategy.dir/QuickPool.cpp.o
     306    cd /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-build-pnun2wp/src/um
            pire/strategy && /opt/rocm-5.3.0/hip/bin/hipcc -DCAMP_HAVE_HIP -D__HIP_PLATFORM_AMD__ -D__HIP_PLATFORM_HCC__ -D__HIP_ROCclr__ -I/qfs/proje
            cts/exasgd/src/ci-incline/linux-centos7-zen2/gcc-10.2.0/camp-0.2.3-x5lxh775b2xhjrndoku54hd2l4x4rywh/include -I/qfs/projects/exasgd/src/lit
            z372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src -I/qfs/projects/exasgd/src/litz372/cache/bu
            ild-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-build-pnun2wp/include -Wpedantic       -Wall -Wextra  -O2 -g -DN
            DEBUG -fPIC -Wno-unused-parameter -std=c++11 -MD -MT src/umpire/strategy/CMakeFiles/umpire_strategy.dir/QuickPool.cpp.o -MF CMakeFiles/ump
            ire_strategy.dir/QuickPool.cpp.o.d -o CMakeFiles/umpire_strategy.dir/QuickPool.cpp.o -c /qfs/projects/exasgd/src/litz372/cache/build-stage
            /spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/strategy/QuickPool.cpp
     307    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/strategy/MixedPool.cpp:8:
     308    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/strategy/MixedPool.hpp:10:
     309    In file included from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/map:62:
  >> 310    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/bits/stl_multimap.h:130:41: error: no member named 'rebind' in 'umpir
            e::strategy::QuickPool::pool_allocator<std::pair<const unsigned long, umpire::strategy::QuickPool::Chunk *>>'
     311          typedef typename _Alloc::template rebind<value_type>::other
     312                           ~~~~~~~~         ^
     313    /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/strategy
            /QuickPool.hpp:150:5: note: in instantiation of template class 'std::multimap<unsigned long, umpire::strategy::QuickPool::Chunk *, std::le
            ss<unsigned long>, umpire::strategy::QuickPool::pool_allocator<std::pair<const unsigned long, umpire::strategy::QuickPool::Chunk *>>>' req
            uested here
     314        SizeMap::iterator size_map_it;
     315        ^
     316    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/strategy/MixedPool.cpp:8:
     317    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/strategy/MixedPool.hpp:10:
     318    In file included from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/map:60:
  >> 319    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/bits/stl_tree.h:335:24: error: type 'int' cannot be used prior to '::
            ' because it has no members
     320          typedef typename _Alloc::template rebind<_Rb_tree_node<_Val> >::other
     321                           ^
     322    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/bits/stl_multimap.h:136:17: note: in instantiation of template class
            'std::_Rb_tree<unsigned long, std::pair<const unsigned long, umpire::strategy::QuickPool::Chunk *>, std::_Select1st<std::pair<const unsign
            ed long, umpire::strategy::QuickPool::Chunk *>>, std::less<unsigned long>, int>' requested here
     323          _Rep_type _M_t;
     324                    ^
     325    /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/strategy
            /QuickPool.hpp:150:5: note: in instantiation of template class 'std::multimap<unsigned long, umpire::strategy::QuickPool::Chunk *, std::le
            ss<unsigned long>, umpire::strategy::QuickPool::pool_allocator<std::pair<const unsigned long, umpire::strategy::QuickPool::Chunk *>>>' req
            uested here
     326        SizeMap::iterator size_map_it;
     327        ^
     328    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/strategy/MixedPool.cpp:8:
     329    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/strategy/MixedPool.hpp:10:
     330    In file included from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/map:60:
  >> 331    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/bits/stl_tree.h:439:39: error: base specifier must name a class
     332            struct _Rb_tree_impl : public _Node_allocator
     333                                   ~~~~~~~^~~~~~~~~~~~~~~
     334    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/bits/stl_tree.h:473:31: note: in instantiation of template class 'std
            ::_Rb_tree<unsigned long, std::pair<const unsigned long, umpire::strategy::QuickPool::Chunk *>, std::_Select1st<std::pair<const unsigned l
            ong, umpire::strategy::QuickPool::Chunk *>>, std::less<unsigned long>, int>::_Rb_tree_impl<std::less<unsigned long>, true>' requested here
     335          _Rb_tree_impl<_Compare> _M_impl;
     336                                  ^
     337    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/bits/stl_multimap.h:136:17: note: in instantiation of template class
            'std::_Rb_tree<unsigned long, std::pair<const unsigned long, umpire::strategy::QuickPool::Chunk *>, std::_Select1st<std::pair<const unsign
            ed long, umpire::strategy::QuickPool::Chunk *>>, std::less<unsigned long>, int>' requested here

     ...

     339                    ^
     340    /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/strategy
            /QuickPool.hpp:150:5: note: in instantiation of template class 'std::multimap<unsigned long, umpire::strategy::QuickPool::Chunk *, std::le
            ss<unsigned long>, umpire::strategy::QuickPool::pool_allocator<std::pair<const unsigned long, umpire::strategy::QuickPool::Chunk *>>>' req
            uested here
     341        SizeMap::iterator size_map_it;
     342        ^
     343    1 warning generated when compiling for gfx90a.
     344    3 errors generated when compiling for gfx90a.
  >> 345    make[2]: *** [src/umpire/strategy/CMakeFiles/umpire_strategy.dir/MixedPool.cpp.o] Error 1
     346    make[2]: *** Waiting for unfinished jobs....
     347    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/strategy/QuickPool.cpp:7:
     348    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/strategy/QuickPool.hpp:11:
     349    In file included from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/map:62:
  >> 350    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/bits/stl_multimap.h:130:41: error: no member named 'rebind' in 'umpir
            e::strategy::QuickPool::pool_allocator<std::pair<const unsigned long, umpire::strategy::QuickPool::Chunk *>>'
     351          typedef typename _Alloc::template rebind<value_type>::other
     352                           ~~~~~~~~         ^
     353    /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/strategy
            /QuickPool.hpp:150:5: note: in instantiation of template class 'std::multimap<unsigned long, umpire::strategy::QuickPool::Chunk *, std::le
            ss<unsigned long>, umpire::strategy::QuickPool::pool_allocator<std::pair<const unsigned long, umpire::strategy::QuickPool::Chunk *>>>' req
            uested here
     354        SizeMap::iterator size_map_it;
     355        ^
     356    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/strategy/QuickPool.cpp:7:
     357    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/strategy/QuickPool.hpp:11:
     358    In file included from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/map:60:
  >> 359    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/bits/stl_tree.h:335:24: error: type 'int' cannot be used prior to '::
            ' because it has no members
     360          typedef typename _Alloc::template rebind<_Rb_tree_node<_Val> >::other
     361                           ^
     362    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/bits/stl_multimap.h:136:17: note: in instantiation of template class
            'std::_Rb_tree<unsigned long, std::pair<const unsigned long, umpire::strategy::QuickPool::Chunk *>, std::_Select1st<std::pair<const unsign
            ed long, umpire::strategy::QuickPool::Chunk *>>, std::less<unsigned long>, int>' requested here
     363          _Rep_type _M_t;
     364                    ^
     365    /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/strategy
            /QuickPool.hpp:150:5: note: in instantiation of template class 'std::multimap<unsigned long, umpire::strategy::QuickPool::Chunk *, std::le
            ss<unsigned long>, umpire::strategy::QuickPool::pool_allocator<std::pair<const unsigned long, umpire::strategy::QuickPool::Chunk *>>>' req
            uested here
     366        SizeMap::iterator size_map_it;
     367        ^
     368    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/strategy/QuickPool.cpp:7:
     369    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/strategy/QuickPool.hpp:11:
     370    In file included from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/map:60:
  >> 371    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/bits/stl_tree.h:439:39: error: base specifier must name a class
     372            struct _Rb_tree_impl : public _Node_allocator
     373                                   ~~~~~~~^~~~~~~~~~~~~~~
     374    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/bits/stl_tree.h:473:31: note: in instantiation of template class 'std
            ::_Rb_tree<unsigned long, std::pair<const unsigned long, umpire::strategy::QuickPool::Chunk *>, std::_Select1st<std::pair<const unsigned l
            ong, umpire::strategy::QuickPool::Chunk *>>, std::less<unsigned long>, int>::_Rb_tree_impl<std::less<unsigned long>, true>' requested here
     375          _Rb_tree_impl<_Compare> _M_impl;
     376                                  ^
     377    /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/bits/stl_multimap.h:136:17: note: in instantiation of template class
            'std::_Rb_tree<unsigned long, std::pair<const unsigned long, umpire::strategy::QuickPool::Chunk *>, std::_Select1st<std::pair<const unsign
            ed long, umpire::strategy::QuickPool::Chunk *>>, std::less<unsigned long>, int>' requested here

     ...

     383    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/strategy/DynamicPoolList.cpp:7:
     384    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/strategy/DynamicPoolList.hpp:15:
     385    In file included from /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-s
            rc/src/umpire/strategy/DynamicSizePool.hpp:18:
     386    /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/strategy
            /FixedSizePool.hpp:133:9: warning: variable 'i' set but not used [-Wunused-but-set-variable]
     387        int i = 0;
     388            ^
  >> 389    /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/strategy
            /QuickPool.cpp:47:35: error: cannot initialize object parameter of type 'umpire::strategy::mixins::AlignedAllocation' with an expression o
            f type 'umpire::strategy::QuickPool'
     390      const std::size_t rounded_bytes{aligned_round_up(bytes)};
     391                                      ^~~~~~~~~~~~~~~~
  >> 392    /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/strategy
            /QuickPool.cpp:70:13: error: cannot initialize object parameter of type 'umpire::strategy::mixins::AlignedAllocation' with an expression o
            f type 'umpire::strategy::QuickPool'
     393          ret = aligned_allocate(size); // Will Poison
     394                ^~~~~~~~~~~~~~~~
  >> 395    /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/strategy
            /QuickPool.cpp:77:15: error: cannot initialize object parameter of type 'umpire::strategy::mixins::AlignedAllocation' with an expression o
            f type 'umpire::strategy::QuickPool'
     396            ret = aligned_allocate(size); // Will Poison
     397                  ^~~~~~~~~~~~~~~~
  >> 398    /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/strategy
            /QuickPool.cpp:315:85: error: cannot initialize object parameter of type 'umpire::strategy::AllocationStrategy' with an expression of type
             'umpire::strategy::QuickPool'
     399      UMPIRE_REPLAY("\"event\": \"coalesce\", \"payload\": { \"allocator_name\": \"" << getName() << "\" }");
     400                                                                                        ^~~~~~~
     401    /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-src/src/umpire/Replay.h
            pp:76:83: note: expanded from macro 'UMPIRE_REPLAY'
     402                    << "\"timestamp\":" << static_cast<long>(time.count()) << ", " << msg << " }" << std::endl;            \
     403                                                                                      ^~~
     404    7 errors generated when compiling for gfx90a.
  >> 405    make[2]: *** [src/umpire/strategy/CMakeFiles/umpire_strategy.dir/QuickPool.cpp.o] Error 1
     406    make[2]: Leaving directory /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/s
            pack-build-pnun2wp'
     407    [ 85%] Built target umpire_resource
     408    1 warning generated when compiling for host.
     409    make[2]: Leaving directory /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/s
            pack-build-pnun2wp'
  >> 410    make[1]: *** [src/umpire/strategy/CMakeFiles/umpire_strategy.dir/all] Error 2
     411    make[2]: Leaving directory /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/s
            pack-build-pnun2wp'
     412    [ 85%] Built target umpire_op
     413    make[1]: Leaving directory /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/s
            pack-build-pnun2wp'
  >> 414    make: *** [all] Error 2

See build log for details:
  /qfs/projects/exasgd/src/litz372/cache/build-stage/spack-stage-umpire-6.0.0-pnun2wpnynk7c6psswijvn4rt5gufhff/spack-build-out.txt

@jaelynlitz jaelynlitz changed the title Spack cannot build v2022.3.1 Incompatible with rocm 5.3.0 Feb 2, 2023
@jaelynlitz jaelynlitz changed the title Incompatible with rocm 5.3.0 Umpire incompatible with rocm 5.3.0 Feb 2, 2023
@davidbeckingsale
Copy link
Member

Sorry I missed this. It looks like a system configuration problem. Your build is pulling in standard library headers from GCC 4.8.5: /usr/lib/gcc/x86_64-redhat-linux/4.8.5.

Umpire now requires C++14, and previously C++11, and needs a newer compiler.

You may need to provide a --gcc-toolchain flag to get amdclang++ to use a newer GCC.

@jaelynlitz
Copy link
Author

Thanks for the suggestion @davidbeckingsale. I tried adding the --gcc-toolchain flag to our spack.yaml (attached), but it seems as though the flag was not picked up. Maybe my syntax is off.. The spack build output is also attached - it's picking up gcc10.2.0 and hipcc as the compilers, but still finding the 4.8.5 include dir.

spack-build-out.txt
spack.txt

@jaelynlitz
Copy link
Author

jaelynlitz commented Feb 9, 2023

I am trying to compile from source to pin down if it is a problem with Umpire or our compiler and I have attached the full error log from trying to compile with v6.0.0 and clang15.0.0. The --gcc-toolchain flag seems correctly populated, but it is still trying to link against the 4.8.5 library -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5. I get the same error output with v2022.03.1.

Here is the most notable snippet from the output:

[ 35%] Linking CXX executable ../../bin/judy_test
cd /qfs/projects/exasgd/src/jaelyn-spack/Umpire/build/tests/tpl && /qfs/projects/ops/rh7_gpu/rocmapps/views/cmake/3.21.4/bin/cmake -E cmake_link_script CMakeFiles/judy_test.dir/link.txt --verbose=1
/opt/rocm-5.3.0/llvm/bin/clang++  -Wpedantic  --gcc-toolchain=/share/apps/gcc/8.4.0     -Wall -Wextra  CMakeFiles/judy_test.dir/judy_test.cpp.o ../../src/umpire/tpl/judy/CMakeFiles/umpire_tpl_judy.dir/judy.c.o -o ../../bin/judy_test   -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5  ../../lib/libgtest_main.a ../../lib/libgtest.a -lpthread -lpthread
ld.lld: error: undefined symbol: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::c_str() const
>>> referenced by judy_test.cpp
>>>               CMakeFiles/judy_test.dir/judy_test.cpp.o:(JudyTest_Construction_Test::TestBody())
>>> referenced by judy_test.cpp
>>>               CMakeFiles/judy_test.dir/judy_test.cpp.o:(JudyTest_WriteValues_Test::TestBody())
>>> referenced by judy_test.cpp
>>>               CMakeFiles/judy_test.dir/judy_test.cpp.o:(JudyTest_WriteValues_Test::TestBody())
>>> referenced 108 more times
...

compile umpire from source rocm5.3.0.txt

@davidbeckingsale
Copy link
Member

Modifying some of the logic from here:

https://github.com/LLNL/radiuss-spack-configs/blob/main/packages/camp/package.py#L86-L112

may help. You can add the /usr/lib/gcc/x86_64-redhat-linux/4.8.5 directory to BLT_CMAKE_IMPLICIT_LINK_DIRECTORIES_EXCLUDE, and it should fix it.

@pelesh
Copy link

pelesh commented Feb 9, 2023

A couple of additional suggestions:

  • You can check with a CMake editing tool such as ccmake what compiler/libs the cmake command has found. Just hit t to toggle to advanced mode.
  • It often helps to specify your compiler at invocation of the cmake command. In your case it would be something like CC=clang CXX=clang++ cmake ... I think CMake will by default use whatever is linked to /usr/bin/cc.

@davidbeckingsale
Copy link
Member

@jaelynlitz I missed the bit where you mentioned building from source, rather than Spack. You can try passing BLT_CMAKE_IMPLICIT_LINK_DIRECTORIES_EXCLUDE directly to CMake, e.g.:

cmake <blah blah blah> -DBLT_CMAKE_IMPLICIT_LINK_DIRECTORIES_EXCLUDE=/usr/lib/gcc/x86_64-redhat-linux/4.8.5 ..

@jaelynlitz
Copy link
Author

@davidbeckingsale that worked for compiling from source! I'll see what I can do to transfer into spack

@jaelynlitz
Copy link
Author

After success compiling from source with gcc10.2.0, I have been trying to build umpire via spack with gcc10.2.0. However, Umpire is still trying to use the clang15.0.0 compiler and still linking against gcc4.8.5. The --gcc-toolchain flags are still given in the spack.yaml for clang and hipcc, and I have added the BLT exclude dirs line to the Umpire spack package:
Under line 149
entries.append(cmake_cache_string("BLT_CMAKE_IMPLICIT_LINK_DIRECTORIES_EXCLUDE", "/usr/lib/gcc/x86_64-redhat-linux/4.8.2;/usr/lib/gcc/x86_64-redhat-linux/4.8.5;/usr/tce/packages/gcc/gcc-4.9.3/lib64;/usr/tce/packages/gcc/gcc-4.9.3/gnu/lib64/gcc/powerpc64le-unknown-linux-gnu/4.9.3;/usr/tce/packages/gcc/gcc-4.9.3/gnu/lib64;/usr/tce/packages/gcc/gcc-4.9.3/lib64/gcc/x86_64-unknown-linux-gnu/4.9.3"))

Attached spack.yaml and concretization and build output
spack.yaml
umpire gcc vs clang .txt

@cameronrutherford
Copy link

cameronrutherford commented Apr 18, 2023

I was able to repro the same issues as @jaelynlitz using latest spack develop and the same box. I verified that the BLT exclude configuration was being passed to umpire, and nothing on the system path should be linked in. I also verified with cpp while in the spack environment that /usr/lib/;/lib were not on the include search path.

This leads me to believe something funky is going on with the spack compiler wrapper and how it has gcc/clang configured. I can posted updated logs/spack configuration but they are largely the same as ones already posted in the issue.

Maybe getting more information from spack would be helpful? I am a little at a loss why building by hand and through spack are giving such different results.

@kab163
Copy link
Contributor

kab163 commented Jan 19, 2024

Seeing as we are up to rocm/6.0.0 now and we have the CI working for rocm/5.6.0, can this be closed?

@kab163
Copy link
Contributor

kab163 commented Jan 24, 2024

Waiting a few more days.. if no response, I will close this issue.

@cameronrutherford
Copy link

This isn't critical path for any project work since ECP came to and end, and so I don't think we are going to get to testing this anytime soon. I assume that this issue can be closed given your reports, and we can re-open any issues that we might have found with ROCm 5.6.0/6.0.0 as necessary

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants