draft #5 Added google benchmark to kokkos kernel and to the CI #1626
Conversation
|
@lucbv i cannot make this PR a draft or even assign it, so i am pinging you here. can you please add me as developer to kernels if I am not already? |
perf_test/CMakeLists.txt
Outdated
| @@ -50,5 +50,95 @@ if (KokkosKernels_ENABLE_PERFTESTS) | |||
| ADD_COMPONENT_SUBDIRECTORY(blas) | |||
| ADD_SUBDIRECTORY(performance) | |||
| #ADD_SUBDIRECTORY(common) | |||
|
|
|||
| IF (KOKKOS_HAS_TRILINOS) | |||
There was a problem hiding this comment.
Is that even defined? You should use KOKKOSKERNELS_HAS_TRILINOS instead
There was a problem hiding this comment.
Thanks, I changed it in the commit d953f8f. I saw that KOKKOS_HAS_TRILINOS is used in the file kokkos-kernels/perf_test/sparse/CMakeLists.txt too. Let me know if we should modify it too
|
@lucbv I realized that the PR was not in a draft mode. Sorry for that, in some other projects, adding draft as a prefix of the PR title was enough. I will follow the process now and for the future too |
There was a problem hiding this comment.
In general, this is fantastic. Thanks, @mperrinel ! Please see comments and questions below
| message(FATAL_ERROR "Benchmarks are not supported when building as part of Trilinos") | ||
| ENDIF() | ||
|
|
||
| find_package(benchmark QUIET) |
There was a problem hiding this comment.
Only search for google benchmark if it is enabled via KokkosKernels_ENABLE_BENCHMARK.
There was a problem hiding this comment.
Please add the KokkosKernels_ENABLE_BENCHMARK option around here: https://github.com/kokkos/kokkos-kernels/blob/develop/CMakeLists.txt#L53 and docs to BUILD.md.
There was a problem hiding this comment.
We didn't change the top level CMakeLists.txt because it was decided with @lucbv during one meeting with him. Our understanding it that eventually, benchmark and perfsuite need to be cleaned up or unified. We will bring this up again during the meeting with Luc in case we misunderstood.
| --benchmark_out=${BENCHMARK_NAME}_${BENCHMARK_TIME}.json | ||
| ) | ||
|
|
||
| ADD_TEST( |
There was a problem hiding this comment.
Will this be run with all other unit tests?
There was a problem hiding this comment.
No this would probably only be triggered in nightly build or even at a different frequency. The benchmark might be longer and could time out if included next to unit-tests
There was a problem hiding this comment.
That will be run with all other unit tests only if KokkosKernels_ENABLE_BENCHMARK is enabled, which is not the case by default
|
@e10harvey @lucbv @ndellingwood can you please take another look at this so that we can move forward? |
.jenkins/nightly.groovy
Outdated
| @@ -75,7 +75,7 @@ pipeline { | |||
| -DCMAKE_CXX_STANDARD=17 \ | |||
| -DCMAKE_CXX_EXTENSIONS=OFF \ | |||
| -DKokkosKernels_ENABLE_TESTS=ON \ | |||
| -DKokkosKernels_ENABLE_EXAMPLES=ON \ | |||
| -DKokkosKernels_ENABLE_BENCHMARKS=ON \ | |||
There was a problem hiding this comment.
| -DKokkosKernels_ENABLE_BENCHMARKS=ON \ | |
| -DKokkosKernels_ENABLE_EXAMPLES=ON \ |
| message(FATAL_ERROR "Benchmarks are not supported when building as part of Trilinos") | ||
| ENDIF() | ||
|
|
||
| find_package(benchmark QUIET) |
There was a problem hiding this comment.
Please add the KokkosKernels_ENABLE_BENCHMARK option around here: https://github.com/kokkos/kokkos-kernels/blob/develop/CMakeLists.txt#L53 and docs to BUILD.md.
There was a problem hiding this comment.
Please add the KokkosKernels_ENABLE_BENCHMARK option around here: https://github.com/kokkos/kokkos-kernels/blob/develop/CMakeLists.txt#L53 and documentation to BUILD.md.
|
Thanks @e10harvey for the comment. We answered it last week: |
|
I'd like to see the following environment information included in each benchmark output file that we plan to store. This is useful for post-facto analysis when something strange is observed to be happening. General:
For OpenMP
For CUDA:
For AMD / Intel GPUs:
|
Hi @cwpearson , this PR only adds googlebench as dependency. |
I see. Thank you. |
e10harvey
added
AT: PRE-TEST INSPECTED
kokkos-devops-admin
removed
the
AT: PRE-TEST INSPECTED
|
Status Flag 'Pre-Test Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED by label AT: PRE-TEST INSPECTED! Autotester is Removing Label; This inspection will remain valid until a new commit to source branch is performed. |
|
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: KokkosKernels_PullRequest_GCC930_Light_Tpls_GCC930
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_CUDA11_CUDA11_LayoutRight
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_CLANG13CUDA10
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_A64FX_Tpls_ARMPL2110
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_A64FX_GCC1020
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_VEGA908_ROCM520
Jenkins Parameters
Using Repos:
Pull Request Author: mperrinel |
|
Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run. Pull Request Auto Testing has FAILED (click to expand)Build InformationTest Name: KokkosKernels_PullRequest_GCC930_Light_Tpls_GCC930
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_CUDA11_CUDA11_LayoutRight
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_CLANG13CUDA10
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_A64FX_Tpls_ARMPL2110
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_A64FX_GCC1020
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_VEGA908_ROCM520
Jenkins Parameters
Console Output (last 100 lines) : KokkosKernels_PullRequest_GCC930_Light_Tpls_GCC930 # 212 (click to expand)Console Output (last 100 lines) : KokkosKernels_PullRequest_CUDA11_CUDA11_LayoutRight # 219 (click to expand)Console Output (last 100 lines) : KokkosKernels_PullRequest_CLANG13CUDA10 # 153 (click to expand)Console Output (last 100 lines) : KokkosKernels_PullRequest_A64FX_Tpls_ARMPL2110 # 48 (click to expand)Console Output (last 100 lines) : KokkosKernels_PullRequest_A64FX_GCC1020 # 46 (click to expand)Console Output (last 100 lines) : KokkosKernels_PullRequest_VEGA908_ROCM520 # 51 (click to expand) |
|
@fnrizzi thank you for the pointer, sorry to add noise here, I will comment over there |
|
The KokkosKernels_PullRequest_CLANG13CUDA10 failure is related to a CI configuration bug. It has been fixed now. |
e10harvey
added
AT: RETEST
|
Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing. |
|
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: KokkosKernels_PullRequest_GCC930_Light_Tpls_GCC930
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_CUDA11_CUDA11_LayoutRight
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_CLANG13CUDA10
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_A64FX_Tpls_ARMPL2110
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_A64FX_GCC1020
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_VEGA908_ROCM520
Jenkins Parameters
Using Repos:
Pull Request Author: mperrinel |
|
Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing. |
|
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: KokkosKernels_PullRequest_GCC930_Light_Tpls_GCC930
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_CUDA11_CUDA11_LayoutRight
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_CLANG13CUDA10
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_A64FX_Tpls_ARMPL2110
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_A64FX_GCC1020
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_VEGA908_ROCM520
Jenkins Parameters
Using Repos:
Pull Request Author: mperrinel |
|
Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing. |
|
Status Flag 'Pull Request AutoTester' - Failure: Timed out waiting for job KokkosKernels_PullRequest_VEGA908_ROCM520 to start: Total Wait = 3603
|
|
Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing. |
|
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: KokkosKernels_PullRequest_GCC930_Light_Tpls_GCC930
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_CUDA11_CUDA11_LayoutRight
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_CLANG13CUDA10
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_A64FX_Tpls_ARMPL2110
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_A64FX_GCC1020
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_VEGA908_ROCM520
Jenkins Parameters
Using Repos:
Pull Request Author: mperrinel |
|
Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED Pull Request Auto Testing has PASSED (click to expand)Build InformationTest Name: KokkosKernels_PullRequest_GCC930_Light_Tpls_GCC930
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_CUDA11_CUDA11_LayoutRight
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_CLANG13CUDA10
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_A64FX_Tpls_ARMPL2110
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_A64FX_GCC1020
Jenkins Parameters
Build InformationTest Name: KokkosKernels_PullRequest_VEGA908_ROCM520
Jenkins Parameters
|
|
Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ e10harvey lucbv ]! |
|
Status Flag 'Pull Request AutoTester' - Pull Request MUST BE MERGED MANUALLY BY Project Team - This Repo does not support Automerge |
This PR add google benchmark to Kokkos-kernel project.
This PR is work in progress because adapted from the one done in Kokkos core
To test the PR, please configure it with these options:
cmake <KOKKOS-KERNELS_SOURCE_DIR> -DKokkos_DIR=<KOKKOS_INSTALL_DIR>/lib/cmake/Kokkos -DKokkosKernels_ENABLE_BENCHMARK=ON -DKokkosKernels_ENABLE_TESTS=ON -DKokkosKernels_ENABLE_ALL_COMPONENTS=ON