From 8bc2454833aa1718c6d26d4147fa1ed6286d6a0e Mon Sep 17 00:00:00 2001 From: Rafael Sarmiento Date: Mon, 15 Apr 2019 15:13:03 +0200 Subject: [PATCH 1/4] add docs for the performance report --- docs/running.rst | 99 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) diff --git a/docs/running.rst b/docs/running.rst index 9bce3c437c..41ab9e9345 100644 --- a/docs/running.rst +++ b/docs/running.rst @@ -515,6 +515,105 @@ They are summarized below: * ``--max-retries NUM``: Specify the maximum number of times a failed regression test may be retried (default: 0). + + +Generating a Performance Report +------------------------------ + +A report of the performance values extracted during the regression tests can be generated by running ReFrame with the command line option `--performance-report`. + +The performance report is printed after the output of the regression and has the following format. + + .. code-block:: bash + PERFORMANCE REPORT + ------------------------------------------------------------------------------ + Check1 + - system:partition + - PrgEnv1 + * perf_variable1: + * perf_variable2: + * ... + - PrgEnv2 + : perf_variable1: + : perf_variable2: + * ... + ------------------------------------------------------------------------------ + Check2 + - system:partition + - PrgEnv1 + * perf_variable1: + * perf_variable2: + * ... + - PrgEnv2 + * perf_variable1: + * perf_variable2: + * ... + ------------------------------------------------------------------------------ + +where the performance variables are listed by programming environment and partition for each of the tests that are run. + +For instance + + .. code-block:: bash + reframe -C tutorial/config/settings.py -c tutorial/example7.py -r --performance-report + +prints something like this + + .. code-block:: bash + + Command line: ./reframe -r -c tutorial/example7.py --performance-report + Reframe version: 2.18-dev1 + Launched by user: USER + Launched on host: daint101 + Reframe paths + ============= + Check prefix : + Check search path : 'example7.py' + Stage dir prefix : /path/to/reframe/stage/ + Output dir prefix : /path/to/reframe/output/ + Perf. logging prefix : /path/to/reframe/perflogs + [==========] Running 1 check(s) + [==========] Started on Mon Apr 15 13:49:34 2019 + + [----------] started processing Example7Test (Matrix-vector multiplication (CUDA performance test)) + [ RUN ] Example7Test on daint:gpu using PrgEnv-cray + [ OK ] Example7Test on daint:gpu using PrgEnv-cray + [ RUN ] Example7Test on daint:gpu using PrgEnv-gnu + [ OK ] Example7Test on daint:gpu using PrgEnv-gnu + [ RUN ] Example7Test on daint:gpu using PrgEnv-pgi + [ OK ] Example7Test on daint:gpu using PrgEnv-pgi + [----------] finished processing Example7Test (Matrix-vector multiplication (CUDA performance test)) + + [ PASSED ] Ran 3 test case(s) from 1 check(s) (0 failure(s)) + [==========] Finished on Mon Apr 15 13:55:22 2019 + ============================================================================== + PERFORMANCE REPORT + ------------------------------------------------------------------------------ + Example7Test + - daint:gpu + - PrgEnv-cray + * perf: 49.994311 Gflop/s + - PrgEnv-gnu + * perf: 50.748701 Gflop/s + - PrgEnv-pgi + * perf: 49.844147 Gflop/s + ------------------------------------------------------------------------------ + +If a test needs to be run on a situation where there are no references, the `self.reference` dictionary may be written as + + .. code-block:: bash + + self.reference = { + '*': { + 'perf_variable1': (0, None, None), + 'perf_variable2': (0, None, None) + ... + } + } + +This will pass the performance checks and the performance report will be printed. + + Configuring ReFrame Directories ------------------------------- From 82f636c47d1a2e164e0c3974c9d63110092bb004 Mon Sep 17 00:00:00 2001 From: Rafael Sarmiento Date: Wed, 17 Apr 2019 09:18:35 +0200 Subject: [PATCH 2/4] fix comments --- docs/running.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/running.rst b/docs/running.rst index 41ab9e9345..bed1298950 100644 --- a/docs/running.rst +++ b/docs/running.rst @@ -518,11 +518,11 @@ They are summarized below: Generating a Performance Report ------------------------------- +------------------------------- A report of the performance values extracted during the regression tests can be generated by running ReFrame with the command line option `--performance-report`. -The performance report is printed after the output of the regression and has the following format. +The performance report is printed after the output of the regression tests and has the following format: .. code-block:: bash PERFORMANCE REPORT @@ -552,12 +552,12 @@ The performance report is printed after the output of the regression and has the where the performance variables are listed by programming environment and partition for each of the tests that are run. -For instance +For instance, .. code-block:: bash reframe -C tutorial/config/settings.py -c tutorial/example7.py -r --performance-report -prints something like this +produces the following output: .. code-block:: bash @@ -599,7 +599,7 @@ prints something like this * perf: 49.844147 Gflop/s ------------------------------------------------------------------------------ -If a test needs to be run on a situation where there are no references, the `self.reference` dictionary may be written as +If a test needs to be run on a situation where the reference values are unknown, the `self.reference` dictionary may be written as follows: .. code-block:: bash @@ -611,7 +611,7 @@ If a test needs to be run on a situation where there are no references, the `sel } } -This will pass the performance checks and the performance report will be printed. +Doing this, the performance checks pass and the performance report will be printed as expected. Configuring ReFrame Directories From 908be41d92b13a2dfbd4ef88bfcf28265570f3b6 Mon Sep 17 00:00:00 2001 From: Rafael Sarmiento Date: Wed, 17 Apr 2019 15:25:39 +0200 Subject: [PATCH 3/4] add units to non-reference case --- docs/running.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/running.rst b/docs/running.rst index bed1298950..1703ac6c56 100644 --- a/docs/running.rst +++ b/docs/running.rst @@ -605,8 +605,8 @@ If a test needs to be run on a situation where the reference values are unknown, self.reference = { '*': { - 'perf_variable1': (0, None, None), - 'perf_variable2': (0, None, None) + 'perf_variable1': (0, None, None, 'units'), + 'perf_variable2': (0, None, None, 'units') ... } } From 5d028140fc503fe456335376876272eeb1ac85e5 Mon Sep 17 00:00:00 2001 From: Vasileios Karakasis Date: Sat, 20 Apr 2019 00:08:47 +0200 Subject: [PATCH 4/4] Fix and fine tune docs on performance reports --- docs/running.rst | 114 ++++++++++++++++++++++++++++------------------- 1 file changed, 67 insertions(+), 47 deletions(-) diff --git a/docs/running.rst b/docs/running.rst index 1703ac6c56..6eb94863d9 100644 --- a/docs/running.rst +++ b/docs/running.rst @@ -520,61 +520,61 @@ They are summarized below: Generating a Performance Report ------------------------------- -A report of the performance values extracted during the regression tests can be generated by running ReFrame with the command line option `--performance-report`. - +If you are running performance tests, you may instruct ReFrame to produce a performance report at the end using the `--performance-report` command-line options. The performance report is printed after the output of the regression tests and has the following format: - .. code-block:: bash - PERFORMANCE REPORT - ------------------------------------------------------------------------------ - Check1 - - system:partition - - PrgEnv1 - * perf_variable1: - * perf_variable2: - * ... - - PrgEnv2 - : perf_variable1: - : perf_variable2: - * ... - ------------------------------------------------------------------------------ - Check2 - - system:partition - - PrgEnv1 - * perf_variable1: - * perf_variable2: - * ... - - PrgEnv2 - * perf_variable1: - * perf_variable2: - * ... - ------------------------------------------------------------------------------ - -where the performance variables are listed by programming environment and partition for each of the tests that are run. - -For instance, +.. code-block:: none - .. code-block:: bash - reframe -C tutorial/config/settings.py -c tutorial/example7.py -r --performance-report + PERFORMANCE REPORT + ------------------------------------------------------------------------------ + Check1 + - system:partition + - PrgEnv1 + * perf_variable1: + * perf_variable2: + * ... + - PrgEnv2 + : perf_variable1: + : perf_variable2: + * ... + ------------------------------------------------------------------------------ + Check2 + - system:partition + - PrgEnv1 + * perf_variable1: + * perf_variable2: + * ... + - PrgEnv2 + * perf_variable1: + * perf_variable2: + * ... + ------------------------------------------------------------------------------ -produces the following output: +Achieved performance values are listed by system partition and programming environment for each performance test that has run. +Performance variables are the variables collected through the :attr:`reframe.core.pipeline.RegressionTest.perf_patterns` attribute. - .. code-block:: bash +The following command will run the CUDA matrix-vector multiplication example from the `tutorial `__ and will produce a performance report: + +.. code-block:: bash + + ./bin/reframe -C tutorial/config/settings.py -c tutorial/example7.py -r --performance-report - Command line: ./reframe -r -c tutorial/example7.py --performance-report +.. code-block:: none + + Command line: ./bin/reframe -C tutorial/config/settings.py -c tutorial/example7.py -r --performance-report Reframe version: 2.18-dev1 Launched by user: USER Launched on host: daint101 Reframe paths ============= - Check prefix : + Check prefix : Check search path : 'example7.py' Stage dir prefix : /path/to/reframe/stage/ Output dir prefix : /path/to/reframe/output/ Perf. logging prefix : /path/to/reframe/perflogs [==========] Running 1 check(s) - [==========] Started on Mon Apr 15 13:49:34 2019 - + [==========] Started on Mon Apr 15 13:49:34 2019 + [----------] started processing Example7Test (Matrix-vector multiplication (CUDA performance test)) [ RUN ] Example7Test on daint:gpu using PrgEnv-cray [ OK ] Example7Test on daint:gpu using PrgEnv-cray @@ -583,9 +583,9 @@ produces the following output: [ RUN ] Example7Test on daint:gpu using PrgEnv-pgi [ OK ] Example7Test on daint:gpu using PrgEnv-pgi [----------] finished processing Example7Test (Matrix-vector multiplication (CUDA performance test)) - + [ PASSED ] Ran 3 test case(s) from 1 check(s) (0 failure(s)) - [==========] Finished on Mon Apr 15 13:55:22 2019 + [==========] Finished on Mon Apr 15 13:55:22 2019 ============================================================================== PERFORMANCE REPORT ------------------------------------------------------------------------------ @@ -599,19 +599,39 @@ produces the following output: * perf: 49.844147 Gflop/s ------------------------------------------------------------------------------ -If a test needs to be run on a situation where the reference values are unknown, the `self.reference` dictionary may be written as follows: - .. code-block:: bash - +For completeness, we show here the corresponding section from the ``Example7Test``, so that the connection between the test's code and the output becomes clear: + +.. literalinclude:: ../tutorial/example7.py + :lines: 20-28 + :dedent: 8 + + +If you are writing a benchmark, it is often the case that you will run it in an unknown system, where you don't have any reference value. +Normally, if ReFrame cannot find a reference for the system it is running on, it will complain and mark the test as a failure. +However, you may right your test in such a way, that it allows it to run successfully on any new system. +To achieve this, simply insert a "catch-all" ``*`` entry in the :attr:`reframe.core.pipeline.RegressionTest.reference` attribute: + + +.. code-block:: python + self.reference = { '*': { - 'perf_variable1': (0, None, None, 'units'), - 'perf_variable2': (0, None, None, 'units') + 'perf_var1': (0, None, None, 'units'), + 'perf_var2': (0, None, None, 'units') ... } } -Doing this, the performance checks pass and the performance report will be printed as expected. +The performance test will always pass on new systems and you may use the ``--performance-report`` option for getting the actual performance values. + + +.. note:: + + The performance report should not be confused with `performance logging <#performance-logging>`__. + It is simply a way of quickly visualizing the performance results and is useful for interactive testing. + Performance logging, if configured, occurs independently of the performance report and is meant for keeping performance data over time. + Its formatting facilitates parsing and it should be used for later analysis of the performance data obtained. Configuring ReFrame Directories