Skip to content

Commit

Permalink
Enable Charm++ Projections support
Browse files Browse the repository at this point in the history 
Removes documentation for using PAPI with projections. We need to revisit using
PAPI later, both in terms of how to best do it and its merits.
  • Loading branch information
@nilsdeppe
nilsdeppe committed Feb 15, 2019
1 parent c560fed commit b54848d
Show file tree
Hide file tree
Showing 3 changed files with 26 additions and 141 deletions.
33 changes: 16 additions & 17 deletions cmake/SetupCharmProjections.cmake
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,19 +35,19 @@ endfunction()

if (PROJECTIONS)
set(
CMAKE_CXX_LINK_FLAGS
"${CMAKE_CXX_LINK_FLAGS} -tracemode projections -tracemode summary"
)
# The values 1000 and 1001 are chosen arbitrary but assume that there are
# fewer than 1000 user-defined Actions (which should be a reasonable
# assumption).
set(
CMAKE_CXX_FLAGS
"${CMAKE_CXX_FLAGS} \
-D SPECTRE_CHARM_PROJECTIONS \
-D SPECTRE_CHARM_NON_ACTION_WALLTIME_EVENT_ID=1000 \
-D SPECTRE_CHARM_RECEIVE_MAP_DATA_EVENT_ID=1001"
)
CMAKE_CXX_LINK_FLAGS
"${CMAKE_CXX_LINK_FLAGS} -tracemode projections -tracemode summary"
)
# Sets the macro `SPECTRE_PAPI_COUNTERS` to a list of user-specfied PAPI
# counters (obtained from running `papi_avail`). The counters could be used
# in conjunction with Charm++'s user stats tracing facilities. However,
# care must be taken if/when this is implemented to not assume that an
# Action will start and end without any Actions being run in the middle.
# Because we sometimes elide calls to the Charm++ RTS when invoking
# Actions, an Action on a local component element could be executed as part
# of the current component element's execution. The PAPI counter stats
# must be kept separate for these two Actions in order for them to be
# meaningful.
if (PROJECTIONS_PAPI_COUNTERS)
check_charm_version_for_stat_tracing()

Expand Down Expand Up @@ -76,11 +76,10 @@ if (PROJECTIONS)
)
include(SetupPapi)
endif()
# Defines the macro SPECTRE_PROJECTIONS_USER_STATS if gathering user
# specified stats is supported by the Charm++ version.
if(PROJECTIONS_USER_STATS)
CHECK_CHARM_VERSION_FOR_STAT_TRACING()
set(
CMAKE_CXX_FLAGS
"${CMAKE_CXX_FLAGS} -D SPECTRE_PROJECTIONS_USER_STATS"
)
add_definitions(-DSPECTRE_PROJECTIONS_USER_STATS)
endif()
endif ()
106 changes: 10 additions & 96 deletions docs/DevGuide/ProfilingWithProjections.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,104 +21,17 @@ means compiling Charm++ with the `--with-production` flag. To enable trace
collecting you must build with the `--enable-tracing` flag as well. For
example, on a multicore 64-bit Linux machine the build command would be
``` shell
./build charm++ multicore-linux64 gcc -j8 --with-production --enable-tracing
./build charm++ multicore-linux-x86_64 gcc -j8 --with-production \
--enable-tracing
```

When building SpECTRE for profiling there are several options to control
what is profiled. First, you'll want to compile in Release mode by specifying
`-D CMAKE_BUILD_TYPE=Release`. To enable tracing with Projections you must
specify the CMake variable `-D PROJECTIONS=ON`. By default only Charm++ entry
methods will be profiled, which will not provide very much insight because
the AlgorithmChare infrastructure has only one entry method, which happens to
be a function template. You must specify a typelist of which Actions you would
like to profile before the first time `Evolution/EvolveSystem.hpp` is included
in the main executable. This typically means right after the
`the_ordered_actions_list` list is defined in the main executable. For example,

```cpp
using the_ordered_actions_list = tmpl::list<
Algorithms::CheckTriggers<the_system, the_volume_dim>,
Algorithms::SendDataForFluxes<the_system, the_volume_dim>,
Algorithms::ComputeVolumeDtU<the_system, the_volume_dim>,
Algorithms::ComputeBoundaryFlux<the_system, the_volume_dim>,
Algorithms::ImposeExternalBoundaryConditions<the_system, the_volume_dim>,
Algorithms::ComputeU<the_system, the_volume_dim>,
Algorithms::AdvanceSlab<the_system, the_volume_dim>,
Algorithms::UpdateInstance<the_system, the_volume_dim>>;

using the_trace_actions_list = tmpl::list<
Algorithms::CheckTriggers<the_system, the_volume_dim>,
Algorithms::ComputeVolumeDtU<the_system, the_volume_dim>,
Algorithms::ComputeBoundaryFlux<the_system, the_volume_dim>,
Algorithms::ImposeExternalBoundaryConditions<the_system, the_volume_dim>,
Algorithms::ComputeU<the_system, the_volume_dim>>;
```

Actions not in `the_trace_actions_list` will not be traced but will still be
executed. The code also records time in the AlgorithmChare's `receive_data`
method that is
not taken up by Actions (typically checking if all data needed for the next
Action has been received) and also time spent saving the received data into
local memory.

## Using PAPI With Projections

\warning Using PAPI with Projections's stat counters requires Charm++ v6.8.0
or newer.

It is possible to collect information from hardware counters using PAPI
for the Actions by specifying
`-D PROJECTIONS_PAPI_COUNTERS="PAPI_L1_DCM,PAPI_L2_DCM"`. That is, a comma
separated list of PAPI counters to record. To see the list of available counters
on your hardware run `papi_avail -a`. The recorded user statistics can then be
analyzed inside Charm++ Projections. It is also possible to record custom user
statistics using Charm++ by specifying `-D PROJECTIONS_USER_STATS=ON` and
defining the variable `user_stat_names` as

```cpp
static constexpr std::array<const char*, 2> user_stat_names{
{"name_1", "name_2"}};
```
The variable needs to be defined before `Evolution/EvolveSystem.hpp` is
included. To record statistics of PAPI counters it is recommended you disable
counters for Actions (you can still time profile them, though) and specify the
CMake variable `-D USE_PAPI=ON`. An example of how to record statistics from
PAPI counters is given below.
To provide a concrete example of tracing and analyzing hardware PMUs using PAPI
for only a subset of functions in an Action we will profile the function
`ScalarWaveEquations<Dim>::%compute_volume_dt_u`. There are several helper
functions provided in `Utilities/PAPI.hpp` that will come in useful. To start
the PAPI counters and record the time at the beginning of the function we call
```cpp
start_papi_counters(std::array<int, 2>{{PAPI_L1_DCM, PAPI_L2_DCM}});
const double start_time = get_time_from_papi();
```

and wherever we want to finish recording we run

```cpp
const double stop_time = get_time_from_papi();
auto counters = stop_papi_counters<2>();
```

The template parameter to `stop_papi_counters` must be the number of PAPI
counters being recorded, two in our example. The function `get_time_from_papi`
returns the time in seconds and `stop_time - start_time` gives the elapsed time
in seconds between the two calls. Using `get_time_from_papi` it is possible to
compute FLOP/s, or any other metric related to time. Finally, to store the
counter read outs into Projections's stat counter we use

```cpp
updateStat(projections_user_stat_offset, counters[0]);
updateStat(projections_user_stat_offset + 1, counters[1]);
```
The variable `projections_user_stat_offset` is used to ensure that the stat
numbers used by Charm++ internally do not collide with any used in the Actions,
unless you have over 9000 Actions.
methods will be profiled. Because we use an entry method template you'll need to
look at the template parameters to determine which action was called for the
specific entry method invocation.

## Running SpECTRE With Trace Output

Expand All @@ -133,11 +46,12 @@ tracing enabled use
./Evolve3DScalarWave +p4 +traceroot ./traces
```

For more information on runtime options to
control trace data see the
[Charm++ Projections manual](http://charm.cs.illinois.edu/manuals/html/projections/1.html).
For more information on runtime options to control trace data see the
[Charm++ Projections manual]
(http://charm.cs.illinois.edu/manuals/html/projections/1.html).

## Visualizing Trace %Data In Projections

See the [Charm++ Projections manual](http://charm.cs.illinois.edu/manuals/html/projections/2.html)
See the [Charm++ Projections manual]
(http://charm.cs.illinois.edu/manuals/html/projections/2.html)
for details.
28 changes: 0 additions & 28 deletions src/Parallel/Algorithm.hpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -313,11 +313,6 @@ class AlgorithmImpl<ParallelComponent, tmpl::list<ActionsPack...>> {
// @}

// Member variables

#ifdef SPECTRE_CHARM_PROJECTIONS
double non_action_time_start_;
#endif

Parallel::ConstGlobalCache<metavariables>* const_global_cache_{nullptr};
bool performing_action_ = false;
std::size_t algorithm_step_ = 0;
Expand Down Expand Up @@ -465,19 +460,12 @@ AlgorithmImpl<ParallelComponent,
if (performing_action_ or get_terminate()) {
return;
}
#ifdef SPECTRE_CHARM_PROJECTIONS
non_action_time_start_ = Parallel::wall_time();
#endif
lock(&node_lock_);
while (tmpl::size<actions_list>::value > 0 and not get_terminate() and
iterate_over_actions(
std::make_index_sequence<tmpl::size<actions_list>::value>{})) {
}
unlock(&node_lock_);
#ifdef SPECTRE_CHARM_PROJECTIONS
traceUserBracketEvent(SPECTRE_CHARM_NON_ACTION_WALLTIME_EVENT_ID,
non_action_time_start_, Parallel::wall_time());
#endif
}
/// \endcond

Expand Down Expand Up @@ -533,11 +521,6 @@ constexpr bool AlgorithmImpl<ParallelComponent, tmpl::list<ActionsPack...>>::
return;
}

#ifdef SPECTRE_CHARM_PROJECTIONS
traceUserBracketEvent(SPECTRE_CHARM_NON_ACTION_WALLTIME_EVENT_ID,
non_action_time_start_, Parallel::wall_time());
double start_time = detail::start_trace_action<this_action>();
#endif
performing_action_ = true;
algorithm_step_++;
make_overloader(
Expand Down Expand Up @@ -568,10 +551,6 @@ constexpr bool AlgorithmImpl<ParallelComponent, tmpl::list<ActionsPack...>>::
std::add_pointer_t<ParallelComponent>{}))>::type{});

performing_action_ = false;
#ifdef SPECTRE_CHARM_PROJECTIONS
detail::stop_trace_action<this_action>(start_time);
non_action_time_start_ = Parallel::wall_time();
#endif
// Wrap counter if necessary
if (algorithm_step_ >= tmpl::size<actions_list>::value) {
algorithm_step_ = 0;
Expand Down Expand Up @@ -601,9 +580,6 @@ void AlgorithmImpl<ParallelComponent, tmpl::list<ActionsPack...>>::
typename ReceiveTag::type::mapped_type::mapped_type>,
"The type of the data passed to receive_data for a tag that holds a map "
"must be a std::pair.");
#ifdef SPECTRE_CHARM_RECEIVE_MAP_DATA_EVENT_ID
double start_time = Parallel::wall_time();
#endif
auto& inbox = tuples::get<ReceiveTag>(inboxes_)[instance];
ASSERT(0 == inbox.count(t.first),
"Receiving data from the 'same' source twice. The message id is: "
Expand All @@ -613,10 +589,6 @@ void AlgorithmImpl<ParallelComponent, tmpl::list<ActionsPack...>>::
<< instance << "' with tag '" << pretty_type::get_name<ReceiveTag>()
<< "'.\n");
}
#ifdef SPECTRE_CHARM_RECEIVE_MAP_DATA_EVENT_ID
traceUserBracketEvent(SPECTRE_CHARM_RECEIVE_MAP_DATA_EVENT_ID, start_time,
Parallel::wall_time());
#endif
}

template <typename ParallelComponent, typename... ActionsPack>
Expand Down

0 comments on commit b54848d

Please sign in to comment.