Fix HIP, hwloc and test_install #712

tcojean · 2021-02-27T14:13:40Z

HIP overhaul, Hwloc and test_install fixes.

On hwloc

There were a few issues with the CMake hwloc setup.

hwloc was always built even when a valid version already existed on
the system.
The FindHWLOC module had a few issues in terms of organization, search
paths as well as variable management. This has been fixed.
hwloc is a public dependency of Ginkgo but was not installed when we
use the one from our build directory, that would create errors with
missing libraries in some cases.
The HWLOC_XMLFILE should not be managed by Ginkgo, hwloc already
takes care of it, if the environment variable HWLOC_XMLFILE is set, it
uses that as base, otherwise it uses the machine topology it can
detect.
The core/devices which are always compiled were moved into their own
root directory. This better reflects the new library that it creates as
well as the fact all libraries depend on it.

On test install

Test install was missing a HIP version which I added. In addition, the
commands were not properly triggered thus only the base (non CUDA
version) was tested, that is now fixed.

On HIP

This part saw a lot of changes.

ROCm < 3.5 support is dropped as it is now old and all HPC clusters
always update the ROCm version since they know the ecosystem is somewhat
unstable.
Instead of linking against hip::device in ginkgo_hip, I link
against libamdhip64.so publicly.
In general, the purpose of this rework is to depend more on CMake
transitive properties such that we remove special cases. Thanks to this
and the previous point as well, it is possible to remove entirely the
hip_helpers.cmake file and remove all of the bans we were doing.
Thanks to relying on transitive properties, there is no need for the
special_linkage test case anymore.
Instead of building the RPATH_FOR_HIP chain, in Ginkgo everything
uses the CXX language for linking, which properly takes into account
RPATH information unlike when liking with the HIP language. For users,
a simple string GINKGO_INSTALL_RPATH_FOR_HIP is provided which is a
linking flag to be given to the HIP language linker after installing
Ginkgo.
In general, there is also better management or relying more on the
defaults for some environment variables and HIP variables.

Open issues

An annoying issue (still), although it's mitigated thanks to linking
publicly against libamdhip64.so which doesn't create inconvenience to
the user, is that we have to include hip/hip_runtime.hpp in our public
headers (to find __forceinline__ as well as HIP_DEVICE_COMPILE, see
https://github.com/ginkgo-project/ginkgo/blob/develop/include/ginkgo/core/base/types.hpp#L46-L85.
In addition, the GKO_ASSERT cannot be reworked as well otherwise it
creates mysterious issues in the check_headers test, see:
https://gitlab.com/ginkgo-project/ginkgo-public-ci/-/jobs/1073846520

yhmtsai · 2021-03-10T07:24:57Z

Could you also update the ReadMe for hip and maybe INSTALL for hwloc?

tcojean · 2021-03-10T08:24:13Z

All requested changes should have been done for now.

pratikvn

LGTM!

Did you figure out why @nbeams was having an issue with the get_pci_device function in MFEM ?

yhmtsai

LGTM in general.
I would like check whether hip test is still compiled with correct arch flags or not.
For RPATH thing, I would like to depends on LD_LIBRARY_PATH by default.
also, I think if we upload compiled ginkgo library via github release, the package should use LD_LIBRARY_PATH to find hip stuff.
the one benefit of embedding all info into ginkgo is maybe for application with many library which use different backend libs.
(I am not sure whether it really works or not)

yhmtsai · 2021-03-10T12:21:56Z

cmake/create_test.cmake

+        # With how HIP works, to prevent redifinition we need to comment these.
+        # They might need to be used later though.
+        #HIPCC_OPTIONS ${GINKGO_HIPCC_OPTIONS}
+        #NVCC_OPTIONS  ${GINKGO_HIP_NVCC_OPTIONS}
+        #CLANG_OPTIONS ${GINKGO_HIP_CLANG_OPTIONS}


what do you mean redefinition here?
If it does not contain these option, can it get the arch flag and optimization?

I have trouble explaining how it works but indeed with the current setting everything is passed correctly. I can only explain it as a bug in FindHIP.cmake, that bug would explain why we have so many duplications of all the flags as well.

For example:

/opt/rocm-4.0.0/hip/bin/hipcc -M /home/terry/ginkgo/hip/test/base/hip_executor_topology.hip.cpp -o /home/terry/ginkgo/build/hip/test/base/CMakeFiles/hip_test_base_hip_executor_topology.dir//hip_test_base_hip_executor_topology_generated_hip_executor_topology.hip.cpp.o.depend.pre -std=c++14 -fPIC --amdgpu-target=gfx906 --amdgpu-target=gfx908 -fPIC -O2 -DNDEBUG -I/home/terry/ginkgo/build -I/opt/rocm-4.0.0/include -I/opt/rocm-4.0.0/hipblas/include -I/opt/rocm-4.0.0/hiprand/include -I/opt/rocm-4.0.0/hipsparse/include -I/home/terry/ginkgo/build/include -I/home/terry/ginkgo/include -I/home/terry/ginkgo -I/usr/local/include -I/home/terry/ginkgo/build/include -I/home/terry/ginkgo/include -I/home/terry/ginkgo -I/usr/local/include -I/usr/local/include -I/home/terry/ginkgo/build/include -I/home/terry/ginkgo/include -I/home/terry/ginkgo -I/usr/local/include -I/home/terry/ginkgo/build/include -I/home/terry/ginkgo/include -I/home/terry/ginkgo -I/usr/local/include -I/home/terry/ginkgo/build/include -I/home/terry/ginkgo/include -I/home/terry/ginkgo -I/usr/local/include -I/opt/rocm-4.0.0/hip/include -I/home/terry/ginkgo/build/include -I/home/terry/ginkgo/include -I/home/terry/ginkgo -I/usr/local/include -I/home/terry/ginkgo/build/include -I/home/terry/ginkgo/include -I/home/terry/ginkgo -I/usr/local/include -I/home/terry/ginkgo/build/third_party/gtest/src/googletest/include -I/home/terry/ginkgo/build/third_party/gtest/src/googletest/include

Note that If I pass the flags again (at least HIPCC_OPTIONS), there is two std=c++14 and somehow that makes the command invalid. I can maybe reactivate the other flags at least for safety?

I didn't try it to make 100% sure but I think the problem is due to these:
https://github.com/ROCm-Developer-Tools/HIP/blob/main/cmake/FindHIP.cmake#L463-L466
https://github.com/ROCm-Developer-Tools/HIP/blob/main/cmake/FindHIP.cmake#L381-L397

By looking only at these two, since no reset seem to be done, they always append the flags that they find to the global variables HIP_HIPCC_FLAGS and whatnot. The macro doing this is called whenever we call a hip_add_library or hip_add_executable. Thus, when we do hip_add_executable for the test, we already had the definitions from libginkgo_hip.so which get reused and thus we do not need to pass the options again (and it creates an error in the case of -std=c++14).
If they used a function that would make things much simpler and they could use a scoped variable to manage the flags that they find. Or they could just add them to the property of the object they are defining. I do not understand what they are doing.

In any case, I will try to put them back and see if it works, although that will duplicate everything at least twice.

For completeness, here is the failing build, I think it's only nvcc which creates issues so I will make a special case out of it:
https://gitlab.com/ginkgo-project/ginkgo-public-ci/-/jobs/1060533273

nvcc fatal : redefinition of argument 'std'

yhmtsai · 2021-03-10T12:34:54Z

cmake/create_test.cmake

@@ -101,63 +101,31 @@ function(ginkgo_create_cuda_test test_name)
    add_test(NAME ${REL_BINARY_DIR}/${test_name} COMMAND ${TEST_TARGET_NAME})
 endfunction(ginkgo_create_cuda_test)

-function(ginkgo_create_hip_test_special_linkage test_name)


I see. I also put the previous comments link here for reference.
#393 (comment)

yhmtsai · 2021-03-10T12:40:28Z

test_install/CMakeLists.txt

-    # --cuda-gpu-arch=gfx900;--cuda-gpu-arch=gfx906
-    set_target_properties(hip::device PROPERTIES INTERFACE_LINK_LIBRARIES "hip::host")
+if(GINKGO_BUILD_HIP)
+    find_package(HIP REQUIRED)


do we need to add version check here?

The HIP version isn't the same as ROCm and in some cases defaults to 0 so this is hard to do.

nbeams · 2021-03-10T15:24:54Z

LGTM!

Did you figure out why @nbeams was having an issue with the get_pci_device function in MFEM ?

The get_pci_device issue was independent of MFEM -- it was an error I would get when trying to build Ginkgo itself.

pratikvn · 2021-03-10T17:39:49Z

@nbeams , but it was when you were building this branch and not develop right ? Do you still get it with the current status of the fix_hip branch ?

nbeams · 2021-03-10T19:46:00Z

I just tried with the latest version of this branch and also the latest version of develop; they both behave the same way. If I compile with gcc (9.2.0), I see the errors, but if I compile with ROCm clang, I don't. It only happens when building the examples, e.g.

[ 84%] Linking CXX executable ginkgo-ranges
../../hip/libginkgo_hip.so.1.3.0: undefined reference to `gko::MachineTopology::get_pci_device(std::string const&) const'
collect2: error: ld returned 1 exit status
make[2]: *** [examples/ginkgo-ranges/ginkgo-ranges] Error 1
make[1]: *** [examples/ginkgo-ranges/CMakeFiles/ginkgo-ranges.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 84%] Linking CXX executable cb-gmres
../../hip/libginkgo_hip.so.1.3.0: undefined reference to `gko::MachineTopology::get_pci_device(std::string const&) const'
collect2: error: ld returned 1 exit status
make[2]: *** [examples/cb-gmres/cb-gmres] Error 1
make[1]: *** [examples/cb-gmres/CMakeFiles/cb-gmres.dir/all] Error 2

So whatever is happening, whether it's my configuration on this machine or the hwloc I'm using or something else, it's not related to any changes in this branch.

pratikvn · 2021-03-11T09:01:53Z

I see. thanks. I will look into that.

upsj

LGTM! Only a few clarifying questions

upsj · 2021-03-24T08:40:08Z

cmake/install_helpers.cmake

+            if (APPLE)
+                set(ORIGIN_OR_LOADER_PATH "@loader_path")
+            else()
+                set(ORIGIN_OR_LOADER_PATH "$ORIGIN")


I guess this would be a no-op on Windows?

Yes there is anyway no RPATH support on Windows (except some experimental one somewhere).

side note, Windows can use the registry to register the library path.
I do not want to change any system settings, so I do not dig into this direction.

upsj · 2021-03-24T08:42:08Z

cmake/install_helpers.cmake

-    # libraries are present in the directory, these will be used first and
-    # foremost and the environment variable `LD_LIBRARY_PATH` will have no
-    # effect.
+function(ginkgo_add_install_rpath name subdir)


I can't see the subdir parameter being used anywhere. Is it still relevant here?

Right, it's a relica of ginkgo_install_library where it's also useless now. Since this has been annoying me for a while, I'm removing this parameter everywhere.

upsj · 2021-03-24T08:42:57Z

core/CMakeLists.txt

 if (GINKGO_BUILD_HIP AND GINKGO_HIP_PLATFORM MATCHES "hcc")
-    set(RPATH_ADD_HIP_PATH "${HIP_PATH}/lib")
+    list(APPEND GKO_RPATH_ADDITIONS "${HIP_PATH}/lib")


Do we always add this to the RPATH, or just in case GINKGO_INSTALL_RPATH_DEPENDENCIES is ON?

Yes, this is passed to the ginkgo_install_library which gets then passed to the ginkgo_add_install_rpath, and nothing gets done when we are not either in BUILD_SHARED_LIBS or GINKGO_INSTALL_RPATH or GINKGO_INSTALL_RPATH_DEPENDENCIES (where ARGN is used).
https://github.com/ginkgo-project/ginkgo/blob/fix_hip/cmake/install_helpers.cmake#L20-L21
https://github.com/ginkgo-project/ginkgo/blob/fix_hip/cmake/install_helpers.cmake#L27-L30

I think it already did. it passed by ARGN.

I will add more description on that topic.

yhmtsai

LGTM. Do you think it is possible to add some related tests for rpath check? (In CI not cmake, I guess)

yhmtsai · 2021-03-24T08:53:51Z

cmake/create_test.cmake

+            # NVCC_OPTIONS  ${GINKGO_HIP_NVCC_OPTIONS}
+            # CLANG_OPTIONS ${GINKGO_HIP_CLANG_OPTIONS}


Could these two also uncomment if the problem is only in HIPCC_OPTIONS?
I think we use GINKGO_HIP_NVCC_OPTIONS to pass the cuda architecture flag.

No it's in everything. HIPCC is in the name of the macro but it's used for all options, including NVCC and CLANG. The main issue actually comes from nvcc (also clang sometimes if I recall) which really dislikes having two times some important flags like -std=c++14.

I see. I also see what you wrote in the previous conversation.
We still have these flag added by hip on Nvidia (including optimization/arch).
The benchmark and test should have the same settings if they do not fix or change hip.
If we need to compare hip and nvcc on NVIDIA GPU again, we should remember to check these flags are still in library/benchmark/test.
Thanks

yhmtsai · 2021-03-24T08:59:19Z

cmake/install_helpers.cmake

-            set(ORIGIN_OR_LOADER_PATH "@loader_path")
-        else()
-            set(ORIGIN_OR_LOADER_PATH "$ORIGIN")
+        if (GINKGO_INSTALL_RPATH)


If GINKGO_INTSALL_RPATH is OFF, ginkgo does not include rpath information no matter if the other two config set or not, right?
If it is, adding it to install description might be great.

Yes that's the case. I thought it was already clear, I will add more details then.

yhmtsai · 2021-03-24T09:44:11Z

core/CMakeLists.txt

 if (GINKGO_BUILD_HIP AND GINKGO_HIP_PLATFORM MATCHES "hcc")
-    set(RPATH_ADD_HIP_PATH "${HIP_PATH}/lib")
+    list(APPEND GKO_RPATH_ADDITIONS "${HIP_PATH}/lib")


I think it already did. it passed by ARGN.

tcojean

On the topic of testing RPATH in CI, I don't think it can be easily done since it depends on platform, where libraries are located, etc...

tcojean · 2021-03-24T09:54:02Z

cmake/install_helpers.cmake

-            set(ORIGIN_OR_LOADER_PATH "@loader_path")
-        else()
-            set(ORIGIN_OR_LOADER_PATH "$ORIGIN")
+        if (GINKGO_INSTALL_RPATH)


Yes that's the case. I thought it was already clear, I will add more details then.

tcojean · 2021-03-24T09:54:43Z

core/CMakeLists.txt

 if (GINKGO_BUILD_HIP AND GINKGO_HIP_PLATFORM MATCHES "hcc")
-    set(RPATH_ADD_HIP_PATH "${HIP_PATH}/lib")
+    list(APPEND GKO_RPATH_ADDITIONS "${HIP_PATH}/lib")


I will add more description on that topic.

tcojean · 2021-03-24T09:55:16Z

cmake/install_helpers.cmake

-    # libraries are present in the directory, these will be used first and
-    # foremost and the environment variable `LD_LIBRARY_PATH` will have no
-    # effect.
+function(ginkgo_add_install_rpath name subdir)


Right, it's a relica of ginkgo_install_library where it's also useless now. Since this has been annoying me for a while, I'm removing this parameter everywhere.

tcojean · 2021-03-24T09:57:40Z

cmake/create_test.cmake

+            # NVCC_OPTIONS  ${GINKGO_HIP_NVCC_OPTIONS}
+            # CLANG_OPTIONS ${GINKGO_HIP_CLANG_OPTIONS}


No it's in everything. HIPCC is in the name of the macro but it's used for all options, including NVCC and CLANG. The main issue actually comes from nvcc (also clang sometimes if I recall) which really dislikes having two times some important flags like -std=c++14.

On hwloc ________ There were a few issues with the CMake hwloc setup. + hwloc was always built even when a valid version already existed on the system. + The FindHWLOC module had a few issues in terms of organization, search paths as well as variable management. This has been fixed. + hwloc is a public dependency of Ginkgo but was not installed when we use the one from our build directory, that would create errors with missing libraries in some cases. + The `HWLOC_XMLFILE` should not be managed by Ginkgo, hwloc already takes care of it, if the environment variable `HWLOC_XMLFILE` is set, it uses that as base, otherwise it uses the machine topology it can detect. + The core/devices which are always compiled were moved into their own root directory. This better reflects the new library that it creates as well as the fact all libraries depend on it. On test install ________ Test install was missing a HIP version which I added. In addition, the commands were not properly triggered thus only the base (non CUDA version) was tested, that is now fixed. On RPATH management _________ For HIP and HWLOC, the INSTALL_RPATH property is now properly managed. Since all libraries require HWLOC, this setting is added directly into `ginkgo_install_libraries`. For HIP, the setting is added only when actually building HIP. On HIP ________ This part saw a lot of changes. + ROCm < 3.5 support is dropped as it is now old and all HPC clusters always update the ROCm version since they know the ecosystem is somewhat unstable. + Instead of linking against `hip::device` in `ginkgo_hip`, I link against `libamdhip64.so` publicly. + In general, the purpose of this rework is to depend more on CMake transitive properties such that we remove special cases. Thanks to this and the previous point as well, it is possible to remove entirely the `hip_helpers.cmake` file and remove all of the bans we were doing. + Thanks to relying on transitive properties, there is no need for the `special_linkage` test case anymore. + Instead of building the `RPATH_FOR_HIP` chain, in Ginkgo everything uses the `CXX` language for linking, which properly takes into account RPATH information unlike when liking with the `HIP` language. For users, a simple string `GINKGO_INSTALL_RPATH_FOR_HIP` is provided which is a linking flag to be given to the `HIP` language linker after installing Ginkgo. + In general, there is also better management or relying more on the defaults for some environment variables and HIP variables. Open issues ________ An annoying issue (still), although it's mitigated thanks to linking publicly against `libamdhip64.so` which doesn't create inconvenience to the user, is that we have to include `hip/hip_runtime.hpp` in our public headers (to find `__forceinline__` as well as `HIP_DEVICE_COMPILE`, see https://github.com/ginkgo-project/ginkgo/blob/develop/include/ginkgo/core/base/types.hpp#L46-L85. In addition, the `GKO_ASSERT` cannot be reworked as well otherwise it creates mysterious issues in the `check_headers` test, see: https://gitlab.com/ginkgo-project/ginkgo-public-ci/-/jobs/1073846520

+ Change behavior and name of the ginkgo_core_device library (to ginkgo_device). + Update the documentation in INSTALL.md and README.md and in cmake/install_helpers.cmake. + Fix some code style issues. Co-authored-by: Yuhsiang Tsai <yhmtsai@gmail.com> Co-authored-by: Pratik Nayak <pratikvn@protonmail.com> Co-authored-by: Tobias Ribizel <ribizel@kit.edu>

+ Add documentation to the new RPATH macro in install_helpers. + Remove the usage of `subdir` in `ginkgo_install_library` and the new `ginkgo_add_install_rpath` since this is unused. + Some minor improvements to other documentations. Co-authored-by: Yuhsiang Tsai <yhmtsai@gmail.com> Co-authored-by: Tobias Ribizel <ribizel@kit.edu>

sonarcloud · 2021-03-24T21:54:48Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
24 Code Smells

83.5% Coverage
13.2% Duplication

Ginkgo release 1.4.0 The Ginkgo team is proud to announce the new Ginkgo minor release 1.4.0. This release brings most of the Ginkgo functionality to the Intel DPC++ ecosystem which enables Intel-GPU and CPU execution. The only Ginkgo features which have not been ported yet are some preconditioners. Ginkgo's mixed-precision support is greatly enhanced thanks to: 1. The new Accessor concept, which allows writing kernels featuring on-the-fly memory compression, among other features. The accessor can be used as header-only, see the [accessor BLAS benchmarks repository](https://github.com/ginkgo-project/accessor-BLAS/tree/develop) as a usage example. 2. All LinOps now transparently support mixed-precision execution. By default, this is done through a temporary copy which may have a performance impact but already allows mixed-precision research. Native mixed-precision ELL kernels are implemented which do not see this cost. The accessor is also leveraged in a new CB-GMRES solver which allows for performance improvements by compressing the Krylov basis vectors. Many other features have been added to Ginkgo, such as reordering support, a new IDR solver, Incomplete Cholesky preconditioner, matrix assembly support (only CPU for now), machine topology information, and more! Supported systems and requirements: + For all platforms, cmake 3.13+ + C++14 compliant compiler + Linux and MacOS + gcc: 5.3+, 6.3+, 7.3+, all versions after 8.1+ + clang: 3.9+ + Intel compiler: 2018+ + Apple LLVM: 8.0+ + CUDA module: CUDA 9.0+ + HIP module: ROCm 3.5+ + DPC++ module: Intel OneAPI 2021.3. Set the CXX compiler to `dpcpp`. + Windows + MinGW and Cygwin: gcc 5.3+, 6.3+, 7.3+, all versions after 8.1+ + Microsoft Visual Studio: VS 2019 + CUDA module: CUDA 9.0+, Microsoft Visual Studio + OpenMP module: MinGW or Cygwin. Algorithm and important feature additions: + Add a new DPC++ Executor for SYCL execution and other base utilities [#648](#648), [#661](#661), [#757](#757), [#832](#832) + Port matrix formats, solvers and related kernels to DPC++. For some kernels, also make use of a shared kernel implementation for all executors (except Reference). [#710](#710), [#799](#799), [#779](#779), [#733](#733), [#844](#844), [#843](#843), [#789](#789), [#845](#845), [#849](#849), [#855](#855), [#856](#856) + Add accessors which allow multi-precision kernels, among other things. [#643](#643), [#708](#708) + Add support for mixed precision operations through apply in all LinOps. [#677](#677) + Add incomplete Cholesky factorizations and preconditioners as well as some improvements to ILU. [#672](#672), [#837](#837), [#846](#846) + Add an AMGX implementation and kernels on all devices but DPC++. [#528](#528), [#695](#695), [#860](#860) + Add a new mixed-precision capability solver, Compressed Basis GMRES (CB-GMRES). [#693](#693), [#763](#763) + Add the IDR(s) solver. [#620](#620) + Add a new fixed-size block CSR matrix format (for the Reference executor). [#671](#671), [#730](#730) + Add native mixed-precision support to the ELL format. [#717](#717), [#780](#780) + Add Reverse Cuthill-McKee reordering [#500](#500), [#649](#649) + Add matrix assembly support on CPUs. [#644](#644) + Extends ISAI from triangular to general and spd matrices. [#690](#690) Other additions: + Add the possibility to apply real matrices to complex vectors. [#655](#655), [#658](#658) + Add functions to compute the absolute of a matrix format. [#636](#636) + Add symmetric permutation and improve existing permutations. [#684](#684), [#657](#657), [#663](#663) + Add a MachineTopology class with HWLOC support [#554](#554), [#697](#697) + Add an implicit residual norm criterion. [#702](#702), [#818](#818), [#850](#850) + Row-major accessor is generalized to more than 2 dimensions and a new "block column-major" accessor has been added. [#707](#707) + Add an heat equation example. [#698](#698), [#706](#706) + Add ccache support in CMake and CI. [#725](#725), [#739](#739) + Allow tuning and benchmarking variables non intrusively. [#692](#692) + Add triangular solver benchmark [#664](#664) + Add benchmarks for BLAS operations [#772](#772), [#829](#829) + Add support for different precisions and consistent index types in benchmarks. [#675](#675), [#828](#828) + Add a Github bot system to facilitate development and PR management. [#667](#667), [#674](#674), [#689](#689), [#853](#853) + Add Intel (DPC++) CI support and enable CI on HPC systems. [#736](#736), [#751](#751), [#781](#781) + Add ssh debugging for Github Actions CI. [#749](#749) + Add pipeline segmentation for better CI speed. [#737](#737) Changes: + Add a Scalar Jacobi specialization and kernels. [#808](#808), [#834](#834), [#854](#854) + Add implicit residual log for solvers and benchmarks. [#714](#714) + Change handling of the conjugate in the dense dot product. [#755](#755) + Improved Dense stride handling. [#774](#774) + Multiple improvements to the OpenMP kernels performance, including COO, an exclusive prefix sum, and more. [#703](#703), [#765](#765), [#740](#740) + Allow specialization of submatrix and other dense creation functions in solvers. [#718](#718) + Improved Identity constructor and treatment of rectangular matrices. [#646](#646) + Allow CUDA/HIP executors to select allocation mode. [#758](#758) + Check if executors share the same memory. [#670](#670) + Improve test install and smoke testing support. [#721](#721) + Update the JOSS paper citation and add publications in the documentation. [#629](#629), [#724](#724) + Improve the version output. [#806](#806) + Add some utilities for dim and span. [#821](#821) + Improved solver and preconditioner benchmarks. [#660](#660) + Improve benchmark timing and output. [#669](#669), [#791](#791), [#801](#801), [#812](#812) Fixes: + Sorting fix for the Jacobi preconditioner. [#659](#659) + Also log the first residual norm in CGS [#735](#735) + Fix BiCG and HIP CSR to work with complex matrices. [#651](#651) + Fix Coo SpMV on strided vectors. [#807](#807) + Fix segfault of extract_diagonal, add short-and-fat test. [#769](#769) + Fix device_reset issue by moving counter/mutex to device. [#810](#810) + Fix `EnableLogging` superclass. [#841](#841) + Support ROCm 4.1.x and breaking HIP_PLATFORM changes. [#726](#726) + Decreased test size for a few device tests. [#742](#742) + Fix multiple issues with our CMake HIP and RPATH setup. [#712](#712), [#745](#745), [#709](#709) + Cleanup our CMake installation step. [#713](#713) + Various simplification and fixes to the Windows CMake setup. [#720](#720), [#785](#785) + Simplify third-party integration. [#786](#786) + Improve Ginkgo device arch flags management. [#696](#696) + Other fixes and improvements to the CMake setup. [#685](#685), [#792](#792), [#705](#705), [#836](#836) + Clarification of dense norm documentation [#784](#784) + Various development tools fixes and improvements [#738](#738), [#830](#830), [#840](#840) + Make multiple operators/constructors explicit. [#650](#650), [#761](#761) + Fix some issues, memory leaks and warnings found by MSVC. [#666](#666), [#731](#731) + Improved solver memory estimates and consistent iteration counts [#691](#691) + Various logger improvements and fixes [#728](#728), [#743](#743), [#754](#754) + Fix for ForwardIterator requirements in iterator_factory. [#665](#665) + Various benchmark fixes. [#647](#647), [#673](#673), [#722](#722) + Various CI fixes and improvements. [#642](#642), [#641](#641), [#795](#795), [#783](#783), [#793](#793), [#852](#852) Related PR: #857

Release 1.4.0 to master The Ginkgo team is proud to announce the new Ginkgo minor release 1.4.0. This release brings most of the Ginkgo functionality to the Intel DPC++ ecosystem which enables Intel-GPU and CPU execution. The only Ginkgo features which have not been ported yet are some preconditioners. Ginkgo's mixed-precision support is greatly enhanced thanks to: 1. The new Accessor concept, which allows writing kernels featuring on-the-fly memory compression, among other features. The accessor can be used as header-only, see the [accessor BLAS benchmarks repository](https://github.com/ginkgo-project/accessor-BLAS/tree/develop) as a usage example. 2. All LinOps now transparently support mixed-precision execution. By default, this is done through a temporary copy which may have a performance impact but already allows mixed-precision research. Native mixed-precision ELL kernels are implemented which do not see this cost. The accessor is also leveraged in a new CB-GMRES solver which allows for performance improvements by compressing the Krylov basis vectors. Many other features have been added to Ginkgo, such as reordering support, a new IDR solver, Incomplete Cholesky preconditioner, matrix assembly support (only CPU for now), machine topology information, and more! Supported systems and requirements: + For all platforms, cmake 3.13+ + C++14 compliant compiler + Linux and MacOS + gcc: 5.3+, 6.3+, 7.3+, all versions after 8.1+ + clang: 3.9+ + Intel compiler: 2018+ + Apple LLVM: 8.0+ + CUDA module: CUDA 9.0+ + HIP module: ROCm 3.5+ + DPC++ module: Intel OneAPI 2021.3. Set the CXX compiler to `dpcpp`. + Windows + MinGW and Cygwin: gcc 5.3+, 6.3+, 7.3+, all versions after 8.1+ + Microsoft Visual Studio: VS 2019 + CUDA module: CUDA 9.0+, Microsoft Visual Studio + OpenMP module: MinGW or Cygwin. Algorithm and important feature additions: + Add a new DPC++ Executor for SYCL execution and other base utilities [#648](#648), [#661](#661), [#757](#757), [#832](#832) + Port matrix formats, solvers and related kernels to DPC++. For some kernels, also make use of a shared kernel implementation for all executors (except Reference). [#710](#710), [#799](#799), [#779](#779), [#733](#733), [#844](#844), [#843](#843), [#789](#789), [#845](#845), [#849](#849), [#855](#855), [#856](#856) + Add accessors which allow multi-precision kernels, among other things. [#643](#643), [#708](#708) + Add support for mixed precision operations through apply in all LinOps. [#677](#677) + Add incomplete Cholesky factorizations and preconditioners as well as some improvements to ILU. [#672](#672), [#837](#837), [#846](#846) + Add an AMGX implementation and kernels on all devices but DPC++. [#528](#528), [#695](#695), [#860](#860) + Add a new mixed-precision capability solver, Compressed Basis GMRES (CB-GMRES). [#693](#693), [#763](#763) + Add the IDR(s) solver. [#620](#620) + Add a new fixed-size block CSR matrix format (for the Reference executor). [#671](#671), [#730](#730) + Add native mixed-precision support to the ELL format. [#717](#717), [#780](#780) + Add Reverse Cuthill-McKee reordering [#500](#500), [#649](#649) + Add matrix assembly support on CPUs. [#644](#644) + Extends ISAI from triangular to general and spd matrices. [#690](#690) Other additions: + Add the possibility to apply real matrices to complex vectors. [#655](#655), [#658](#658) + Add functions to compute the absolute of a matrix format. [#636](#636) + Add symmetric permutation and improve existing permutations. [#684](#684), [#657](#657), [#663](#663) + Add a MachineTopology class with HWLOC support [#554](#554), [#697](#697) + Add an implicit residual norm criterion. [#702](#702), [#818](#818), [#850](#850) + Row-major accessor is generalized to more than 2 dimensions and a new "block column-major" accessor has been added. [#707](#707) + Add an heat equation example. [#698](#698), [#706](#706) + Add ccache support in CMake and CI. [#725](#725), [#739](#739) + Allow tuning and benchmarking variables non intrusively. [#692](#692) + Add triangular solver benchmark [#664](#664) + Add benchmarks for BLAS operations [#772](#772), [#829](#829) + Add support for different precisions and consistent index types in benchmarks. [#675](#675), [#828](#828) + Add a Github bot system to facilitate development and PR management. [#667](#667), [#674](#674), [#689](#689), [#853](#853) + Add Intel (DPC++) CI support and enable CI on HPC systems. [#736](#736), [#751](#751), [#781](#781) + Add ssh debugging for Github Actions CI. [#749](#749) + Add pipeline segmentation for better CI speed. [#737](#737) Changes: + Add a Scalar Jacobi specialization and kernels. [#808](#808), [#834](#834), [#854](#854) + Add implicit residual log for solvers and benchmarks. [#714](#714) + Change handling of the conjugate in the dense dot product. [#755](#755) + Improved Dense stride handling. [#774](#774) + Multiple improvements to the OpenMP kernels performance, including COO, an exclusive prefix sum, and more. [#703](#703), [#765](#765), [#740](#740) + Allow specialization of submatrix and other dense creation functions in solvers. [#718](#718) + Improved Identity constructor and treatment of rectangular matrices. [#646](#646) + Allow CUDA/HIP executors to select allocation mode. [#758](#758) + Check if executors share the same memory. [#670](#670) + Improve test install and smoke testing support. [#721](#721) + Update the JOSS paper citation and add publications in the documentation. [#629](#629), [#724](#724) + Improve the version output. [#806](#806) + Add some utilities for dim and span. [#821](#821) + Improved solver and preconditioner benchmarks. [#660](#660) + Improve benchmark timing and output. [#669](#669), [#791](#791), [#801](#801), [#812](#812) Fixes: + Sorting fix for the Jacobi preconditioner. [#659](#659) + Also log the first residual norm in CGS [#735](#735) + Fix BiCG and HIP CSR to work with complex matrices. [#651](#651) + Fix Coo SpMV on strided vectors. [#807](#807) + Fix segfault of extract_diagonal, add short-and-fat test. [#769](#769) + Fix device_reset issue by moving counter/mutex to device. [#810](#810) + Fix `EnableLogging` superclass. [#841](#841) + Support ROCm 4.1.x and breaking HIP_PLATFORM changes. [#726](#726) + Decreased test size for a few device tests. [#742](#742) + Fix multiple issues with our CMake HIP and RPATH setup. [#712](#712), [#745](#745), [#709](#709) + Cleanup our CMake installation step. [#713](#713) + Various simplification and fixes to the Windows CMake setup. [#720](#720), [#785](#785) + Simplify third-party integration. [#786](#786) + Improve Ginkgo device arch flags management. [#696](#696) + Other fixes and improvements to the CMake setup. [#685](#685), [#792](#792), [#705](#705), [#836](#836) + Clarification of dense norm documentation [#784](#784) + Various development tools fixes and improvements [#738](#738), [#830](#830), [#840](#840) + Make multiple operators/constructors explicit. [#650](#650), [#761](#761) + Fix some issues, memory leaks and warnings found by MSVC. [#666](#666), [#731](#731) + Improved solver memory estimates and consistent iteration counts [#691](#691) + Various logger improvements and fixes [#728](#728), [#743](#743), [#754](#754) + Fix for ForwardIterator requirements in iterator_factory. [#665](#665) + Various benchmark fixes. [#647](#647), [#673](#673), [#722](#722) + Various CI fixes and improvements. [#642](#642), [#641](#641), [#795](#795), [#783](#783), [#793](#793), [#852](#852) Related PR: #866

tcojean added the 1:ST:WIP This PR is a work in progress. Not ready for review. label Feb 27, 2021

tcojean self-assigned this Feb 27, 2021

tcojean force-pushed the fix_hip branch 14 times, most recently from ed91ad7 to e447828 Compare March 5, 2021 08:39

tcojean requested review from upsj, pratikvn, yhmtsai, fritzgoebel and Slaedr March 5, 2021 09:18

pratikvn approved these changes Mar 10, 2021

View reviewed changes

yhmtsai approved these changes Mar 10, 2021

View reviewed changes

tcojean force-pushed the fix_hip branch 2 times, most recently from 0bcba17 to 456b16f Compare March 12, 2021 07:39

tcojean mentioned this pull request Mar 12, 2021

Improve test install #721

Merged

2 tasks

tcojean force-pushed the fix_hip branch 3 times, most recently from 9c45bee to 0afbd93 Compare March 23, 2021 10:43

upsj approved these changes Mar 24, 2021

View reviewed changes

yhmtsai approved these changes Mar 24, 2021

View reviewed changes

tcojean commented Mar 24, 2021

View reviewed changes

tcojean and others added 5 commits March 24, 2021 11:14

Pass the HIP options to tests without the NVCC bug

9c7fbd8

Add three new options to control RPATH behavior.

386c65b

tcojean force-pushed the fix_hip branch from 8985d33 to f43a5e6 Compare March 24, 2021 10:15

tcojean added 1:ST:ready-to-merge This PR is ready to merge. and removed 1:ST:ready-for-review This PR is ready for review labels Mar 24, 2021

tcojean merged commit 20bf7cf into develop Mar 25, 2021

tcojean deleted the fix_hip branch March 25, 2021 10:21

		# NVCC_OPTIONS ${GINKGO_HIP_NVCC_OPTIONS}
		# CLANG_OPTIONS ${GINKGO_HIP_CLANG_OPTIONS}

Fix HIP, hwloc and test_install #712

Fix HIP, hwloc and test_install #712

Conversation

tcojean commented Feb 27, 2021 • edited Loading

On hwloc

On test install

On HIP

Open issues

yhmtsai commented Mar 10, 2021

tcojean commented Mar 10, 2021

pratikvn left a comment

Choose a reason for hiding this comment

yhmtsai left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tcojean Mar 11, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nbeams commented Mar 10, 2021

pratikvn commented Mar 10, 2021

nbeams commented Mar 10, 2021

pratikvn commented Mar 11, 2021

upsj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tcojean Mar 24, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yhmtsai left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tcojean Mar 24, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tcojean left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tcojean Mar 24, 2021 • edited Loading

Choose a reason for hiding this comment

sonarcloud bot commented Mar 24, 2021

tcojean commented Feb 27, 2021 •

edited

Loading

tcojean Mar 11, 2021 •

edited

Loading

tcojean Mar 24, 2021 •

edited

Loading

tcojean Mar 24, 2021 •

edited

Loading

tcojean Mar 24, 2021 •

edited

Loading